questdb #users-public

Docs

Join the conversation Join Slack

Channels

announcement

Franco Almonacid

03/13/2023, 2:17 PM

Hi everyone! We upgraded to version

7.0.1

and converted our tables to WAL tables. Before that we could query the inserted data almost immediately but after changing to WAL tables we have to wait several minutes to see the data, similar to having a high commit lag. The only way we found to fix this is going back to non-WAL tables, is there something else we can do?

Brandon E.

03/13/2023, 3:24 PM

Hi all, the questdb kubernetes is able to allocate more than one pod?

João Calapez

03/13/2023, 6:06 PM

Hello, questdb is giving me a JRE error, I was wondering if someone could shed some light on this log, maybe there is a config that I can change so that this wont happen? Any help would be appreciated

questdb_error_10_23.log

Izac

03/13/2023, 7:50 PM

Hey QuestDb, is it possible to set which database to ingest CSV data into via REST?

Nicolas Hourcard

03/14/2023, 10:25 AM

welcome @ajaxharg

Luiz Otavio

03/14/2023, 3:42 PM

Hello Team, quick question about the database 1 It already has implement replication? 2 It can guarantee dedup, in database? Thankss

Franco Almonacid

03/14/2023, 5:40 PM

Hi all! I'm trying to

DETACH

a partition with this query:

ALTER TABLE binance_dc DETACH PARTITION LIST '2023-03-13';

and I get this error:

could not detach partition [table=binance_dc, detachStatus=DETACH_ERR_ACTIVE, partitionTimestamp=2023-03-13T00:00:00.000Z, partitionBy=DAY]

also no error message in the questdb logs

Franco Almonacid

03/14/2023, 5:45 PM

sorry, there is an error in the logs

Franco Almonacid

03/14/2023, 5:46 PM

2023-03-14T17:45:29.125713Z C i.q.c.h.p.JsonQueryProcessorState [3009] error [q=`ALTER TABLE binance_dc DETACH PARTITION LIST '2023-03-13';`, msg=`could not detach partition [table=binance_dc, detachStatus=DETACH_ERR_ACTIVE, partitionTimestamp=2023-03-13T00:00:00.000Z, partitionBy=DAY]`, errno=-100]

Andy Beyer

03/14/2023, 5:48 PM

for SELECT queries, is sampling without aggregation allowed? e.g. just select 1 point per sample-window that matches the query (to downsample data) ? similar to the aggregateWindow(every) in flux?

Franco Almonacid

03/14/2023, 8:11 PM

Hi again guys - after detaching several partitions and inserted the data again we noticed that the

order by

doesn't work, here is an example

SELECT * FROM binance_dc WHERE symbol= 'ETHUSDT' AND timestamp IN '2023-02-10' ORDER BY timestamp DESC;

Franco Almonacid

03/14/2023, 8:11 PM

this is from the

EXPLAIN

query

alex.b

03/16/2023, 12:41 PM

Hi. Looks like ILP python client does not support a timestamp data with milliseconds precision, but dfs with only seconds or minutes were exported fine. Is there any workaround to save precision?

Steven Aldinger

03/17/2023, 5:11 AM

hello! we just ran into

max txn-inflight limit reached [txn=5556389, min=5540005, size=16384]

errors on

6.6.1

. we found a reference to it but from 2021 saying to delete the

_txn_scoreboard

file in the table directory. is that still the recommended solution?

Henri Asseily

03/17/2023, 7:04 AM

Hi. The docs aren't too clear about that, but can I have multiple database sessions open via the pgsql driver? One would be inserting, but multiple would be reading. Or should I instead use the REST api for reading?

Newskooler

03/17/2023, 8:28 AM

Does anyone here get this error? I am running

questdb

python library on a Mac M1.

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[7], line 21
     19 from research.pnl.order import Order
     20 from research.pnl.portfolio import Portfolio
---> 21 from questdb.ingress import Sender
     22 from questdb_mb.data_services import get_tables
     23 from <http://research.io|research.io> import load_time_series, remove_table

ImportError: dlopen(/Users/user/Software/python/virtual/mb/lib/python3.11/site-packages/questdb/ingress.cpython-311-darwin.so, 0x0002): tried: '/Users/user/Software/python/virtual/mb/lib/python3.11/site-packages/questdb/ingress.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/user/Software/python/virtual/mb/lib/python3.11/site-packages/questdb/ingress.cpython-311-darwin.so' (no such file), '/Users/user/Software/python/virtual/mb/lib/python3.11/site-packages/questdb/ingress.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

Ido Michael

03/17/2023, 7:01 PM

Hey not sure if that’s the right place to post this, I’m trying to integrate QDB with Jupysql so I can query from Jupyter notebooks. It seems the sql alchemy integration doesn’t exist? The only way to connect at the moment is with Psycopg2? Any forecast for this enhancement?

Sara Tavares

03/17/2023, 11:23 PM

[FOR HIRE] I'm a QA Engineer with background in mobile native apps, I have 9 years of experience in engineering, mainly in startups on a variety of products, including fintech, and others, including as an Lead Engineer and as an Engineering Manager • Experience in managing, growing and hiring a distributed and remote team • Experience setting up QA processes • Setting up testing frameworks, test scripts, and reviewing code changes as well as building technical documentation • Familiar with Typescript, Rust, AWS, Docker, Swift, Kotlin, Vue, and others • Proficiency in building and maintaining CI/CD pipelines • Experience with testing tools such as Cypress, XCUITest, Espresso, Selenium, Jest DM for LinkedIn/Github/rates

03/18/2023, 12:08 AM

I am using psycopg2 to read a questdb database. I am having trouble with specifying datetimes while filtering. psycopg2 automatically converts timeaware python datetimes to timestamptz types when I read from PG but with questdb, I get the following error

psycopg2.DatabaseError: invalid constant: timestamptz
LINE 1: ... timestamp <= '2023-03-17T19:55:46.342000+00:00'::timestampt...

Do I need to convert my Python datetimes into something else before using them in QuestDB queries? Edit: This is solved now. Seems like QuestDB needs datetimes to be naive and in utc, so what I did before using datetimes to filter rows in psycopg2 is the following (dt is a datetime object with some valid timezone):

dt = dt.astimezone(pytz.timezone("UTC")).replace(tzinfo=None)

03/19/2023, 6:44 AM

How do I restore a backup? I backed up a table in a qdb instance on one computer with BACKUP <table> command. I have copied the backup directory onto another computer which has also a qdb instance running. How do I import this table into this database. Tried copying the backup directory into the db subdirectory of where questdb data is stored but didn't work.

Andy Beyer

03/21/2023, 12:08 PM

I have a question about

ORDER BY

in queries: My tables have

timestamp

as the designated timestamp column If I run this query:

select * from '<my-table>' where _timestamp_ > '2023-01-01T00:00:00Z' order by _timestamp_ desc;

I get

results despite the fact that data is being regularly added to the table When I add a

LIMIT

clause to a

desc

sorted query:

select * from '<my-table>' where _timestamp_ > '2023-01-01T00:00:00Z' order by _timestamp_ desc limit 100000;

then I get 0 results If I do either of the above with

order by timestamp asc

I get the full results I am expected (~750000 results with no limit, or the full 100,000 results with the limit) Is there something special about sorting using time in descending order?

Newskooler

03/21/2023, 2:08 PM

hi 👋 Is there an easy and fast way to list all table partitions? 🤔

John M.

03/21/2023, 3:26 PM

I'm having a bit of a weird issue -- my QDB has been working fine, I have a WAL table "trade" that has many dates in it. Ever since a forced reboot last week though for a windows update when I insert data into this table it makes a "wal" folder (so "wal13" for example), which then makes subfolders 1,2,3,4... etc as I continue inserting into it. However, those wal folders never get turned into actual dated partitions... the data seems to just vanish/get deleted? So I'm actually unable to insert any more data into this table at the moment. Am I doing something obviously wrong? Do I need to flag the table or something after the forced reboot? I checked the "wal_tables()" command and the table is not suspended.

David Bellizzi

03/21/2023, 6:22 PM

Hey there questdb folks, I need to migrate our data to the new instance. Is there a documented process on how to do that?

Andy Beyer

03/22/2023, 1:49 PM

Wondering if you all any tuning suggestions for preventing periods of very high Disk I/O due to, as it was explained to me, out of order writes (details in thread)

Brandon E.

03/22/2023, 2:14 PM

Hi everyone, is there a way to always overwrite a table? so if the table already exists, you overwrite instead of appending?

Jetsukda Janthornthao

03/23/2023, 3:39 AM

How to

UPSERT

in QuestDB?

Nicolas Hourcard

03/23/2023, 9:52 AM

Hi @Yusuf, welcome to our community

John M.

03/24/2023, 6:04 PM

It might be helpful to write a blog post, or a little more documentation to the https://questdb.io/docs/concept/write-ahead-log/ page, about when WAL tables are advisable/not advisable for use. The comparison on that page is helpful, but calling out "using a WAL table in these types of scenarios is advisable: X, Y, Z, and is not advisable in these scenarios: A, B, C" explicitly could be helpful too. I have currently been using WAL tables because it seems like using them provides a more complete solution than not, however, seeing Andy's previous comment about disk thrashing has me realize that there probably are some costs to doing so (physical disk wear/tear, etc.), and perhaps I should switch to non-WAL for my usage case.

Michael

03/26/2023, 5:30 AM

@Michael