Title
#users-public
Manoj DeSilva

Manoj DeSilva

09/27/2022, 8:56 AM
Hi Team, We have only one table in QuestDB. From the following can you please let me know, what is more efficient from QuestDB side.1. Sending data over one ILP connection to the single table 2. Sending data over multiple ILP connections to the single table. We tried both of above and learned that when using single connection QuestDB ingests more faster. Just wanted to confirm is it correct. Right now we are using a single ILP connection because of above observation but we still see that QuestDB cannot ingest data in the rate that we send. Is there any other parameters we can change to optimize ingestion performance.
Alex Pelagenko

Alex Pelagenko

09/27/2022, 10:01 AM
hi, yes there are few tricks
10:01 AM
and it depends on the hardware. What box do you use to write?
Manoj DeSilva

Manoj DeSilva

09/27/2022, 10:07 AM
We run QuestDB on kubernetes cluster. This Kubernetes runs on a Linode virtual machine with 64GB RAM and 16 CPUs. There are around 10 other kubernetes pods runs in this cluster running on the same virtual machine.
Alex Pelagenko

Alex Pelagenko

09/27/2022, 3:21 PM
to write to single table from multiple connections I'd suggest
line.tcp.writer.worker.count=1
line.tcp.io.worker.count=6
3:22 PM
6 is arbitrary, depends on how many connections in parallel you want to handle. I'd say 1 io thread can serve 2+ TCP connections
3:26 PM
Multiple ILP connections should speed up things anyway to certain limit, I'm surprised you see single connection is faster.
Manoj DeSilva

Manoj DeSilva

09/27/2022, 3:31 PM
Okay. Thank you. may be it is because we haven't set correct parameters like things that you mentioned above. Also we have lot of out of order data. When you say Multiple ILP connections should speedup things, is it because it will reduce the network latency or is it because QuestDB can ingest concurrently to improve the ingestion throughput to the same table.
Alex Pelagenko

Alex Pelagenko

09/27/2022, 4:01 PM
multiple connections with out of order within configured table lag will speed up things
4:02 PM
because it will be able to parse the data in parallel
4:02 PM
with Out of Order data the biggest slow down is to rewrite the partition, so smaller partitions help
Manoj DeSilva

Manoj DeSilva

09/27/2022, 7:09 PM
Thank you @Alex Pelagenko