Title
#users-public
p

Petr Postulka

10/05/2022, 12:46 PM
hi guys, one more issue ... we were writing to quest via ilp a suddenly the the apps got stuck on ILP writer.send method and it doesn't write any new data ... any idea what can cause this? we have possibly not ideally configued tcp.io and worker threads but would like to know if you any idea what can cause such behavior?
12:52 PM
version 6.5.3
12:56 PM
so it seems it is not completely stuck but it is writing something like few rows per sec
1:11 PM
ok, seems that there is possibility we're hitting disk limits
1:11 PM
will let you know
Jaromir Hamala

Jaromir Hamala

10/05/2022, 1:15 PM
hi Petr, it looks like you have a lot of out of order writes. is it the case?
p

Petr Postulka

10/05/2022, 1:19 PM
checking this now
1:30 PM
@Jaromir Hamala is it possible to see 03 per table now?
Jaromir Hamala

Jaromir Hamala

10/05/2022, 1:33 PM
I’m afraid you cannot. O3 metrics are not per table unfortunately. however you should be able to tell from logs. from the snippet you shared it looks like
l2_nanex_temp
could be the culprit?
p

Petr Postulka

10/05/2022, 1:34 PM
this is what we still see in the logs ... but at the same time we believe we've stopped all the writers already
1:34 PM
but this is still appearing in the logs all the time
1:35 PM
couldn't quest somehow get into some cycle?
Jaromir Hamala

Jaromir Hamala

10/05/2022, 1:36 PM
well, I cannot rule it out without knowing more. but I’ve never seen it before.
p

Petr Postulka

10/05/2022, 1:37 PM
any idea how to investigate further?
Alex Pelagenko

Alex Pelagenko

10/05/2022, 1:47 PM
you want to find out what writes to the table?
p

Petr Postulka

10/05/2022, 1:49 PM
I wanted to find out why we are seeing extreme O3 repeating all the time
1:50 PM
btw it suddenly stopped now and is gone ... we've stopped some writers to the tables we saw in logs but o3 was there for additional 15mins or so and then suddenly dropped to 0
1:50 PM
is it possible that it is somehow delayed?
1:50 PM
taking into account that disk was utilized to 100% all the time because of the O3
Alex Pelagenko

Alex Pelagenko

10/05/2022, 1:51 PM
there are some queues in ILP they can have few messages
1:51 PM
what’s table definitions and you config file?
p

Petr Postulka

10/05/2022, 1:52 PM
we saw that because of 100% disk utilization it was writing like 1 row per sec or terribly slow ... and doing O3 all the time ... possible that the queue could cause 15min delay in O3 after we stopped the writers? or not really?
Alex Pelagenko

Alex Pelagenko

10/05/2022, 1:53 PM
depends on table max uncommitted rows
p

Petr Postulka

10/05/2022, 1:53 PM
metrics.enabled=true
line.tcp.enabled=true
line.udp.enabled=true
cairo.sql.backup.root=/opt/questdb/backup
cairo.snapshot.instance.id=fffdd771-5840-5ff4-bde1-4233dd86e601
line.tcp.enabled=true
line.tcp.io.worker.count=6
line.tcp.writer.worker.count=2
line.udp.enabled=true
shared.worker.count=6
cairo.max.uncommitted.rows=1000
cairo.commit.lag=5000
pg.net.connection.limit=30
line.tcp.maintenance.job.interval=5000
cairo.sql.backup.root=/opt/questdb/backups
metrics.enabled=true
cairo.snapshot.instance.id=1720566A-DC57-4751-B3A1-E14E92FEE63F
1:53 PM
this is the config file
Alex Pelagenko

Alex Pelagenko

10/05/2022, 2:00 PM
what’s the tables DDL?
2:00 PM
actual max uncommitted rows in particular
p

Petr Postulka

10/05/2022, 2:00 PM
what you mean by DDL?
2:01 PM
schema?
Alex Pelagenko

Alex Pelagenko

10/05/2022, 2:01 PM
yes., in console right click on table and click “copy schema…”
p

Petr Postulka

10/05/2022, 2:02 PM
will send in private message