https://questdb.io logo
Title
m

Matthew Halko

02/23/2023, 9:04 PM
Hello, is there a way to limit the maximum number of memory mapped files that quest db leaves in memory?
b

Bolek Ziobrowski

02/24/2023, 10:05 AM
Hi @Matthew Halko. Do you mean the number of file descriptors or the amount of memory used to mmap them ? The former is generally a function of number of workers x number of tables . The latter could use all available free memory but should release it if there's pressure . You could set memory limit with qdb running in docker though .
r

Ryan Hamilton

02/24/2023, 2:48 PM
ulimit on linux allows both limiting the amount of memory and number of open files on a per process basis: https://ss64.com/bash/ulimit.html I think it's -n and -m
On a typical dedicated server database setup I configure at least that, huge pages and the OOM killer.
See also numactl and nice.
m

Matthew Halko

02/24/2023, 3:07 PM
I am on windows server 2019. I have a very simple database setup, a single table with time and sales data. As I query the database, questdb leaves files in memory and eventually will consume all the memory on the machine
Questdb just leaves all the column files in memory after executing queries. I am running queries via postgres driver. Is this normal?
b

Bolek Ziobrowski

02/24/2023, 3:27 PM
Mmaped files are held by cached readers and writers . You can make questdb release them earlier by changing these server.conf properties :
# how often the writer maintenance job gets run,  in milliseconds
#cairo.idle.check.interval=300000

# defines frequency with which the reader pool checks for inactive readers. In milliseconds
#cairo.inactive.reader.ttl=120000

# defines frequency with which the writer pool checks for inactive readers. In milliseconds
#cairo.inactive.writer.ttl=600000
Are you getting memory errors from other applications running on the same box once questdb consumes all memory ?
m

Matthew Halko

02/24/2023, 3:37 PM
That worked!
Damn that was tricky
b

Bolek Ziobrowski

02/24/2023, 3:40 PM
Keep in mind that mmaping a file is slow in Windows so releasing readers more often is likely to slow down queries .
I think the difference between linux and windows here is that former evicts pages fine while latter clings to already touched ones and is likely to trigger paging .
m

Matthew Halko

02/24/2023, 4:02 PM
ahhh
Should be all good. Just need questdb to contain it's memory footprint