hi guys, one more issue ... we were writing to quest via ilp a suddenly the the apps got stuck on ILP writer.send method and it doesn't write any new data ... any idea what can cause this? we have possibly not ideally configued tcp.io and worker threads but would like to know if you any idea what can cause such behavior?
so it seems it is not completely stuck but it is writing something like few rows per sec
ok, seems that there is possibility we're hitting disk limits
will let you know
10/05/2022, 1:15 PM
hi Petr, it looks like you have a lot of out of order writes. is it the case?
10/05/2022, 1:19 PM
checking this now
@Jaromir Hamala is it possible to see 03 per table now?
10/05/2022, 1:33 PM
I’m afraid you cannot. O3 metrics are not per table unfortunately. however you should be able to tell from logs. from the snippet you shared it looks like
could be the culprit?
10/05/2022, 1:34 PM
this is what we still see in the logs ... but at the same time we believe we've stopped all the writers already
but this is still appearing in the logs all the time
couldn't quest somehow get into some cycle?
10/05/2022, 1:36 PM
well, I cannot rule it out without knowing more. but I’ve never seen it before.
10/05/2022, 1:37 PM
any idea how to investigate further?
10/05/2022, 1:47 PM
you want to find out what writes to the table?
10/05/2022, 1:49 PM
I wanted to find out why we are seeing extreme O3 repeating all the time
btw it suddenly stopped now and is gone ... we've stopped some writers to the tables we saw in logs but o3 was there for additional 15mins or so and then suddenly dropped to 0
is it possible that it is somehow delayed?
taking into account that disk was utilized to 100% all the time because of the O3
10/05/2022, 1:51 PM
there are some queues in ILP they can have few messages
what’s table definitions and you config file?
10/05/2022, 1:52 PM
we saw that because of 100% disk utilization it was writing like 1 row per sec or terribly slow ... and doing O3 all the time ... possible that the queue could cause 15min delay in O3 after we stopped the writers? or not really?