Title
#users-public
j

Jakub Kubík

09/14/2022, 7:45 PM
Hello there! What is the best practice when sending "a lot" of data (about 1k rows per second per service with 4 services, so 4k rows per second) to QuestDB using the ILP (more specifically the python-questdb-client)? The thing that may be specific to our environment - each of our services runs in 24 separate containers, so with 4 services, it is 4k rows per second using 96 connections. Each services writes to its own table. Flush automatically after every row, or only after some arbitrary buffer size is reached? If it is the latter, have you come across any "good" value empirically? Thanks in advance!
Adam Cimarosti

Adam Cimarosti

09/15/2022, 11:06 AM
Hi Jakub, For better throughput, flush infrequently. Your approach of flushing every row is slower and will have a significant overhead (as we disable Nagle's algorithm over TCP), but minimizes the delay of when the data will appear in the database for queries. To flush less frequently you can rely on the
auto_flush
feature (a flag to the Sender's constructor) that automatically flushes once a certain configurable buffer used size is reached: https://py-questdb-client.readthedocs.io/en/latest/api.html#questdb.ingress.Sender If you are concerned about the timelines of your flushes, you may also want to use the feature in conjunction with a timer. Here's an example for that:https://py-questdb-client.readthedocs.io/en/latest/examples.html#ticking-random-data-and-timer-based-flush Note: There's also a minor bug/limitation where auto_flush will not do a final flush on application shutdown. This will be fixed in the next patch release and in the meantime you can surround your code with a try/except for the
SystemExit
exception (which python raises during shutdown) and call
flush
explicitly in the except branch.
j

Jakub Kubík

09/15/2022, 11:16 AM
Hi Adam, thanks for your reply! Do you have any recommendations for the buffer size? We decided to switch to the default, 64 kB, after seeing an improvement with 16 kB.
Adam Cimarosti

Adam Cimarosti

09/15/2022, 12:00 PM
I would leave it high and then add a timer so you know that the timer flushes if there's a quiet period.
12:02 PM
The default 63K is a finger in the air. You can lower it too if you want.