Hello everyone, currently working on implementing QuestDB in our work organization, but ran in an issue with it, I am using telegraf to forward constantly a bunch of statsd metrics to influx working on 9009 port through udp, as from what I can see all metrics are kept on the storage and I encounter 100% disk io with machine load up to 6-7 for 1/5 & 15m, also 50G of disk storage was used for almost 24h timeframe, is there a way to configure quest to keep information in the memory?

RAM disk

I think it's the load displayed by top and similar tools . @Yordan Manov Could you share logs ?

It’s the load I am referring to, all data in this setup is kept on the disk, the machine is with 20G memory and it uses only 5% atm, with this setup we cannot afford a day worth of 50G disk, is there a way some of the information to be cached in the memory or to be moved in the memory, example innodb

The problem is the data footprint? What’s the table schema / row count?

I am forwarding 25 measurements with a couple of fields every 10 seconds (container metrics) and for now with a default retention policy (7days), the metrics are extracted from probably 300 containers

It is cached as disk / page cache

RAM

What are QuestDB table count, column types you write. Data types are important to get rights to reduce the footprint

That's the point that we don't want all of the information to be on the partitions, we want as much as possible to be moved on the RAM memory literally like innodb where you tell where mariadb should hold the information on the disk partitions or on the memory cards, the current setup we see 100% io operations mostly write where they cause the machine to increase the load and the performance is decreasing, if its possible we dont want the ram to stay low

We write to disk using mmap, if disk is not fast enough for a spike, ram takes the hit and then flushes to the disk

probably yes, thank you for your quick response

Sounds like you write about 500m data points per 24h, 50g means it's 100bytes each

hi @Yordan Manov kindly let us know your thoughts