Title
b
bowen
05/25/2023, 9:09 PMHi, when I use psycopg2 to access questdb, an error occurs:
DatabaseError: could not open read-only [file=/opt/homebrew/var/questdb/db/BinanceUM-TRXUSDT-kline-1m/2023-03-20.1/unit.d]LINE 1: SELECT * FROM 'BinanceUM-TRXUSDT-kline-1m'This is my code:
import psycopg2 as pg
# 创建连接池
with pg.connect(
host='127.0.0.1',
port=8812,
user='admin',
password='quest',
database='questdb'
) as conn:
for symbol in tqdm(symbols_usdt):
table_name = f"BinanceUM-{symbol}-kline-1m"
QUERY = f"SELECT * FROM '{table_name}'"
cursor = conn.cursor()
cursor.execute(QUERY)
cursor.close()I need to use around 2 hundred cursors. It seems that the amount of cursors matters. When I request a little bit of data for each table, the number of cursors could be large without any error. However, if I request the whole data for each table, the number is very limited.
i
Imre
05/25/2023, 10:24 PMhi, the reason is very likely that you hit the max open files OS limit.
try to increase the limit if you have not done it yet.
https://questdb.io/docs/operations/capacity-planning/#maximum-open-files
b
bowen
05/25/2023, 11:17 PM@Imre Thanks! I noticed this document. However, it seems not suitable for MacOS
i
Imre
05/25/2023, 11:33 PMyes, these instructions are not for mac.
Mac seems to be more complicated.
unfortunately we do not have instructions.
if you happen to find a solution, please, share it. we could make it available to everyone.
b
bowen
05/25/2023, 11:38 PMOK. I found that the max-open file restriction is 245760 in my OS through the cmd:
sysctl kern.maxfilesperprockern.maxfilesperproc: 245760@Imre I decide to give up. I don't know why it opens so much files (20,000+) which exceeds the limit, and don't know how to modify the limit on MacOS. Maybe I have to try another more proven technique, like InfluxDB
i
Imre
05/26/2023, 1:06 AMafter some digging i found that we might need to pass
-XX:-MaxFDLimitre how many files opened by QuestDB, it is hard to guess from the logs.
when you run a select statement, typically the number of open files would be equals to (columns * partitions) for each table + index and meta data files.
by (columns * partitions) i mean all columns referenced in your select times the partitions contain the data involved in the query.
if you find the
could not open read-onlyerrno=xx24Too many open filesb
bowen
05/26/2023, 1:46 AMHi @Imre , thanks for letting me know the reasons!
Here is one line of the log, showing that the
errono242023-05-25T23:35:11.005398Z C i.q.c.p.PGConnectionContext error [msg=`could not open read-only [file=/opt/homebrew/var/questdb/db/BinanceUM-DOTUSDT-kline-1m/2023-02-20.0/unit.i]`, errno=24]i
Imre
05/26/2023, 9:02 AMhi @bowen, raised the following issue.
https://github.com/questdb/questdb/issues/3405
will test if
-XX:-MaxFDLimitb
bowen
05/27/2023, 12:52 AMHi @Imre, here is how I meet the bug