Hello everyone I am just writing an collector of data which

Question

Hello everyone I am just writing an collector of data which are persisted in QuestDB The average amount of data that are inserted as row is about 2kB My QuestDB instance is running as a standalone installation with fairly default configuration on a dedicated Azure instance I am having incoming traffic about 100msg s which all should be inserted and every 5minutes I am getting such error ```io questdb cutlass line LineSenderException 104 send error at io questdb cutlass line tcp PlainTcpLineChannel send PlainTcpLineChannel java 100 at io questdb cutlass line AbstractLineSender sendAll AbstractLineSender java 380 at io questdb cutlass line LineTcpSender send00 LineTcpSender java 86 at io questdb cutlass line AbstractLineSender put AbstractLineSender java 199 at io questdb cutlass line AbstractLineSender putUtf8Special AbstractLineSender java 319 at io questdb std str CharSink encodeUtf8 CharSink java 45 at io questdb std str CharSink encodeUtf8 CharSink java 37 at io questdb cutlass line AbstractLineSender field AbstractLineSender java 133 at xxxx core timeseries DbWriter write DbWriter kt 33 at xxxx core timeseries handlers AddDataPointRequestHandler handleRequestTyped AddDataPointRequestHandler kt 19 at xxxx core timeseries handlers AddDataPointRequestHandler handleRequestTyped AddDataPointRequestHandler kt 8 at xxxx core timeseries handlers TypedTimeSeriesRequestHandler handleRequest TimeSeriesRequestHandler kt 28 at xxxx core timeseries TimeSeriesService handleRequest TimeSeriesService kt 131 at xxxx core timeseries TimeSeriesService access$handleRequest TimeSeriesService kt 43 at xxxx core timeseries TimeSeriesService$serviceLoop$1$2$1$5 emit TimeSeriesService kt 107 at xxxx core timeseries TimeSeriesService$serviceLoop$1$2$1$5 emit TimeSeriesService kt 104 at kotlinx coroutines flow FlowKt ChannelsKt emitAllImpl$FlowKt ChannelsKt Channels kt 62 at kotlinx coroutines flow FlowKt ChannelsKt access$emitAllImpl$FlowKt ChannelsKt Channels kt 1 at kotlinx coroutines flow FlowKt ChannelsKt$emitAllImpl$1 invokeSuspend Channels kt at kotlin coroutines jvm internal BaseContinuationImpl resumeWith ContinuationImpl kt 33 at kotlinx coroutines DispatchedTask run DispatchedTask kt 106 at kotlinx coroutines EventLoopImplBase processNextEvent EventLoop common kt 284 at kotlinx coroutines BlockingCoroutine joinBlocking Builders kt 85 at kotlinx coroutines BuildersKt BuildersKt runBlocking Builders kt 59 at kotlinx coroutines BuildersKt runBlocking Unknown Source at xxxx common Service runBlocking Service kt 109 at xxxx core timeseries TimeSeriesService serviceLoop TimeSeriesService kt 71 at xxxx core timeseries TimeSeriesServiceMain main TimeSeriesServiceMain kt 14 ``` Interestingly there are not errors in log file of QuestDB

Jaromir Hamala · Answer

hi Milos the every 5 minutes aspect is suspicious I d check a firewall NAT if there is any

Miloš Kozák · Answer

Thanks for reply I have just checked and there is no firewall in our setup

Jaromir Hamala · Answer

~is there anything in questdb server log ~ edit I just saw your last sentence in the first message

Jaromir Hamala · Answer

well that s interesting as questdb the server should log disconnected clients even when they disconnect gracefully

Miloš Kozák · Answer

We have just setup another QuestDB instance in the same subnet as the problematic one and forwarded the traffic there Will keep you posted if the problem occurs even here

Jaromir Hamala · Answer

Hi Milos what s the outcome of the experiment

Miloš Kozák · Answer

Well the same setup but just a fresh azure instance and no occurrences of that exception Not even one in the course of 24hours

Miloš Kozák · Answer

Just thinking that could be related to the VM or state of data in that VM Planning to replicate the old database to the new server with rsync and carry on with the test to verify we have integrity error Any other suggestion

Jaromir Hamala · Answer

not really if you can reliably reproduce it every 5 mins then maybe I would try wireshark tcpdump running on both server and client boxes this could help to identify why is the connection broken it could help answering questions such as what party closed the connection first is there RST going on from what end

Miloš Kozák · Answer

We have to be connected but right now I got a few errors Having empty container in k8s and VM only lead to OK situation We have just swapped drive and inserted today s morning snapshot and here we go

Miloš Kozák · Answer

Looking into logs of questdb I noticed something strange The 3 line 104 where you san see series=pc at the end indicating I want to write to table named pc corresponds to log line ```Dec 16 12 10 37 questdb ana test 0 questdb 824061 2022 12 16T12 10 37 058467Z E i q c l t LineTcpConnectionContext 103 could not process line data table=anotherthable name msg=cast error for line protocol Dec 16 12 10 37 questdb ana test 0 questdb 824061 2022 12 16T12 10 37 058500Z E i q c l t LineTcpConnectionContext 103 could not parse measurement NONE at 987 line may be mangled due to partial parsing```

Miloš Kozák · Answer

The second line is very long but when I scroll right I can see the line is open ended Looks like some buffer has overflowed

Jaromir Hamala · Answer

`cast error for line protocol ` indicates a type problem

Jaromir Hamala · Answer

I am not sure if questdb always logs full lines even if they are super long maybe < Imre> knows by heart

Imre · Answer

hi yes it looks like a type cast error is this the log directly copied into the chat from the log file

Imre · Answer

seems like the end of the error message is missing

Imre · Answer

```could not process line data table=anotherthable name msg=cast error for line protocol ``` that should continue with more information about the error like which column it is and the types which we are trying to cast from and to

Imre · Answer

same for the second error message the input line should be printed there

Imre · Answer

```line may be mangled due to partial parsing this is the input here ```

Imre · Answer

we print the entire input no matter how long it is

Imre · Answer

second line is a result of the cast error

Miloš Kozák · Answer

I have just DM you that message but it looks trimmed to me

Imre · Answer

no that is the full message it shows the designated timestamp at the end

Imre · Answer

let me check

Imre · Answer

quick update we sorted this on private chat it was a symbol vs string column type mismatch