Title
#users-public
Daniel Schnetzer Fava

Daniel Schnetzer Fava

08/26/2022, 12:26 PM
Hi, I have an integration test with QuestDB running in a testcontainer. I see from the DB logs that data is being written (using ILP). But when I do a select * on the tables, I get no data back. Logs showing table writes:
2022-08-26T12:18:56.996189Z I i.q.c.l.t.LineTcpWriterJob assigned table to writer thread [tableName=event_errors, threadId=0]
    2022-08-26T12:18:56.996585Z I i.q.c.TableWriter switched partition [path='/var/lib/questdb/db/event_errors/2022-03-01']
    2022-08-26T12:18:56.997443Z I i.q.c.l.t.LineTcpWriterJob assigned table to writer thread [tableName=events, threadId=0]
    2022-08-26T12:18:57.006316Z I i.q.c.TableWriter switched partition [path='/var/lib/questdb/db/events/2022-03-01']
    2022-08-26T12:18:57.009357Z I i.q.c.TableWriter switched partition [path='/var/lib/questdb/db/event_errors/2022-03-03']
    2022-08-26T12:18:57.014857Z I i.q.c.TableWriter switched partition [path='/var/lib/questdb/db/events/2022-03-03']
Part of the logs showing the queries:
2022-08-26T12:18:57.853152Z E pg-server been unable to accept connections for 5000ms, closing listener [serverFd=85]
    2022-08-26T12:18:57.953599Z I i.q.c.p.PGConnectionContext parse [fd=90, q=select 1 from long_sequence(1)]
    2022-08-26T12:18:57.953645Z I i.q.c.p.PGConnectionContext query cache used [fd=90]
    2022-08-26T12:18:57.956994Z I i.q.c.p.PGConnectionContext parse [fd=90, q=SELECT * FROM events]
    2022-08-26T12:18:57.957820Z I i.q.g.SqlCompiler plan [q=`select-choose ts, id, provider, object_id, object_type, event_type, platform, invalid_count, valid_count from (select [ts, id, provider, object_id, object_type, event_type, platform, invalid_count, valid_count] from events timestamp (ts))`, fd=90]
    2022-08-26T12:18:57.964454Z I i.q.c.p.PGConnectionContext parse [fd=90, q=SELECT * FROM event_errors]
    2022-08-26T12:18:57.965008Z I i.q.g.SqlCompiler plan [q=`select-choose ts, event_id, path, message, quality_type, error_code, error_type from (select [ts, event_id, path, message, quality_type, error_code, error_type] from event_errors timestamp (ts))`, fd=90]
Any advice is welcome. 🙂 (It seems to me like similar code works okay in a dev environment. I see data when inspecting with the QuestDB console)
Jaromir Hamala

Jaromir Hamala

08/26/2022, 1:10 PM
Hello Daniel, this behaviour is very likely caused by “Commit Lag”. See: https://questdb.io/docs/troubleshooting/faq#why-is-ilp-data-not-immediately-available You can adjust commit lag. If you are using a TestContainers then the simplest thing is to use environment variables. Add to your container definition:
addEnv("QDB_CAIRO_COMMIT_LAG", "100");
<-- this set commit lag to 100ms.
Daniel Schnetzer Fava

Daniel Schnetzer Fava

08/26/2022, 1:16 PM
That did it @Jaromir Hamala! Thank you! 😃
Jaromir Hamala

Jaromir Hamala

08/26/2022, 1:17 PM
You are very welcome! Happy Questing! 🙂
1:28 PM
just for my curiosity: what are you using to feed data in? Java ILP client?
Daniel Schnetzer Fava

Daniel Schnetzer Fava

08/29/2022, 8:53 AM
Yes. The project is in Scala and uses the Java client.
Jaromir Hamala

Jaromir Hamala

08/29/2022, 8:54 AM
cool! is there anything you are missing in the client? anything you find annoying?
Daniel Schnetzer Fava

Daniel Schnetzer Fava

08/30/2022, 7:19 AM
So far so good with the client. Let me get a bit more experience with it and I’ll get back to you 🙂
9:34 AM
@Jaromir Hamala, ILP seems to be working well now. We’ve been doing insertions for a bit over a week now. My only comment is a small comment about documentation. It would be nice to know which methods can raise exceptions by reading the java docs. For example, the flush can raise an exception, but the doc above the method’s signature doesn’t mention it. 🙂
Jaromir Hamala

Jaromir Hamala

09/05/2022, 10:17 AM
good feedback, thank you!
10:17 AM
I’ll look at it