Title
#users-market-data
Jone Qiang

Jone Qiang

08/27/2022, 1:27 PM
what's reason for this issue: io.questdb.cutlass.line.LineSenderException: [111] could not connect to host [host=localhost] at io.questdb.cutlass.line.tcp.PlainTcpLineChannel.<init>(PlainTcpLineChannel.java:75) ~[questdb-6.4.3-jdk8.jar:6.4.3-jdk8]
1:28 PM
the server is running status. it was normal for many days. This error occurred today.
Andrey Pechkurov

Andrey Pechkurov

08/27/2022, 1:38 PM
Hi, The error suggests that the server is not listening on the port. Have you checked server logs?
Jone Qiang

Jone Qiang

08/27/2022, 1:43 PM
udp 0 0 0.0.0.0:9009 0.0.0.0😗 6506/java I can see this port.I check the log,just find the stdout log,and didnot see any error in log. @Andrey Pechkurov
1:43 PM
And postgresql to insert data is normal
Andrey Pechkurov

Andrey Pechkurov

08/27/2022, 1:46 PM
your above output is udp while
PlainTcpLineChannel
class suggests that you're using tcp
1:46 PM
how do you check ports?
Jone Qiang

Jone Qiang

08/27/2022, 1:49 PM
netstat -tunlp | grep 9009 @Andrey Pechkurov
Andrey Pechkurov

Andrey Pechkurov

08/27/2022, 1:50 PM
is there an entry for tcp?
1:51 PM
for instance, on my machine this command returns this line among other ones:``` tcp 0 0 0.0.0.0:9009 0.0.0.0😗 LISTEN - ``
Jone Qiang

Jone Qiang

08/27/2022, 1:52 PM
No just udp in my server.
1:54 PM
and in my test server ,run the same commad, I see udp and tcp entries. so is it cashed for tcp connection?
Andrey Pechkurov

Andrey Pechkurov

08/27/2022, 2:03 PM
for some reason TCP server is not started or failed on your machine, at least it looks so
Jone Qiang

Jone Qiang

08/27/2022, 2:11 PM
It had been normal for many days, but it just showed up today,should I restart the quest server to fix this issue or other way?
Andrey Pechkurov

Andrey Pechkurov

08/27/2022, 2:14 PM
you should check the server logs and, in any case, restart the server
2:15 PM
may it be firewall or any kind of anti-malware software blocking the port?
Jone Qiang

Jone Qiang

08/27/2022, 2:21 PM
just find data/log/stdout-2022-08-22T11-45-13.txt for example,didnot see any error.
2:22 PM
you said that to check the server log, is it quesdb log or linux server logs?
Andrey Pechkurov

Andrey Pechkurov

08/27/2022, 2:27 PM
QuestDB's one
Jone Qiang

Jone Qiang

08/30/2022, 11:48 AM
@Andrey Pechkurov what is the appropriate value of the tcp connection limit ? I set to 200, and when there's a lot of data being inserted ,there is an error about it
Andrey Pechkurov

Andrey Pechkurov

08/30/2022, 12:19 PM
What kind of error?
12:19 PM
Ideally, the number of TCP connections should be close to the number of ILP I/O workers
Jone Qiang

Jone Qiang

08/30/2022, 12:45 PM
I upgrade to 6.5.1(from 6.4.3) , when insert data frequently, the server was crashed, and I see the error: like tcp connection, I change the tcp connection limit,the server was crashed too. so I used 6.4.3.
Andrey Pechkurov

Andrey Pechkurov

08/30/2022, 12:54 PM
TCP connection limit shouldn't lead to crashes: new connections will be immediately closed when you reach the limit, nothing more than that
Jone Qiang

Jone Qiang

08/30/2022, 12:55 PM
6.5.1 ,it was crashed and I see this error.
Andrey Pechkurov

Andrey Pechkurov

08/30/2022, 12:57 PM
if you have a reproducer for the crash, could you create a GH issue for the issue?https://github.com/questdb/questdb/issues/new?assignees=&amp;labels=bug&amp;template=bug_report.yaml
Jone Qiang

Jone Qiang

08/30/2022, 12:58 PM
sure
1:11 PM
tcp 0 0 0.0.0.0:9009 0.0.0.0😗 LISTEN 1016/java udp 0 0 0.0.0.0:9009 0.0.0.0😗 1016/java tcp is normal, I can see this error when running for a while? could you give me some advice? io.questdb.cutlass.line.LineSenderException: [110] could not connect to host [host=localhost] at io.questdb.cutlass.line.tcp.PlainTcpLineChannel.<init>(PlainTcpLineChannel.java:75) ~[questdb-6.4.3-jdk8.jar:6.4.3-jdk8]
Andrey Pechkurov

Andrey Pechkurov

08/30/2022, 1:45 PM
Could you also share the logs? Hard to say what may go wrong without logs
12:05 PM
Hi @Jone Qiang I'm looking at the crash report right now and can't reproduce it, at least for now. Would you be able to share a minimal reproducer snippet for this crash or, at least, describe what's going on in your application? AFAIU you're writing to QuestDB from multiple threads in a Java application. Do you use a dedicated
LineTcpSender
instance per each thread? Is it long-lived or you close it after each batch write? What's your table schema?
12:46 PM
Duplicated this message on GH
Jone Qiang

Jone Qiang

08/31/2022, 1:40 PM
@Andrey Pechkurov
try (Sender sender = Sender.builder().address(host).build()) {
I used this to insert. when inserting data isnot frequently, it is normal.2022-08-31T03:15:41.661414Z I tcp-line-server below maximum connection limit, registered listener [serverFd=78] ,I can see this log,is it normal?
Andrey Pechkurov

Andrey Pechkurov

08/31/2022, 1:41 PM
yes, this log line means that the connection fits within the allowed max connection number and, thus, it was established successfully
1:42 PM
Does the crash happen all the time or only occasionally?
Jone Qiang

Jone Qiang

08/31/2022, 1:45 PM
Our application is consume data from kafka, I create more than one consumer,when one consumer poll data from kafka that is not frequently , it is normal, I start another consuemr,poll data frequently as same time, it will be crashed in some minutes.
1:46 PM
these consumer are different process. @Andrey Pechkurov
1:49 PM
for the version of 6.4.3 ,it can run longer than 6.5.1,the server will not crashed, but there is an error like ,cannot connect to 9009 tcp.
1:50 PM
io.questdb.cutlass.line.LineSenderException: [111] could not connect to host [host=localhost] at io.questdb.cutlass.line.tcp.PlainTcpLineChannel.<init>(PlainTcpLineChannel.java:75) ~[questdb-6.4.3-jdk8.jar:6.4.3-jdk8]
Andrey Pechkurov

Andrey Pechkurov

08/31/2022, 1:58 PM
So, this crash looks like a regression introduced after 6.4.3, I see
1:58 PM
Are there any errors in the server logs on 6.4.3 or 6.5.1?
1:59 PM
They should be something like the following:
2022-08-31T03:15:41.666325Z E module-name error message
or
2022-08-31T03:15:41.666325Z C module-name error message
Jone Qiang

Jone Qiang

08/31/2022, 2:00 PM
I attached the log file,it is 6.5.1, for 6.4.1, there are no error logs,just insert writer log,for example
Andrey Pechkurov

Andrey Pechkurov

08/31/2022, 2:04 PM
The attached 6.5.1 doesn't include logs from the server start timestamp. Do you have a full log?
2:08 PM
BTW how many tables do you have? Only a single one or many of them?
Jone Qiang

Jone Qiang

08/31/2022, 2:13 PM
6 tables,each process insert to 1 table
2:14 PM
hs_err_pid+18774.log is it this log that inlcude server start timestamp.?
Andrey Pechkurov

Andrey Pechkurov

08/31/2022, 2:15 PM
no, I mean the
quest_crash.log
one
2:16 PM
hs_err_pid* file is generated by JVM when something like a segfault happens
2:17 PM
BTW could you run
ulimit -c unlimited
before restarting 6.5.1 QuestDB? This command should enable core dump generation on JVM crash. The path to the core dump should be in the hs_err_pid* file
2:18 PM
Core dump helps us to diagnose the crash
Jone Qiang

Jone Qiang

08/31/2022, 2:28 PM
ok sure.
2:36 PM
questdb_crash.log is large,I will gave you complete file
2:41 PM
I will give you tommow, because it is on our production server, need to devops' help
Andrey Pechkurov

Andrey Pechkurov

08/31/2022, 3:20 PM
An update: no need to collect the core dump (we were able to locate the problematic place in the source code), but having the log file would be still helpful
Jone Qiang

Jone Qiang

08/31/2022, 3:25 PM
ok got it.
3:27 PM
Do you have any idea about why we cannot connect to tcp ? It has nothing to do with tcp connnection limit?
Andrey Pechkurov

Andrey Pechkurov

08/31/2022, 3:29 PM
No, it shouldn't be related with the connection limit
3:30 PM
Does it happen on 6.5.1 or only on 6.4.3?
Jone Qiang

Jone Qiang

08/31/2022, 3:33 PM
on 6.4.3, on 6.5.1, it is crashed directly.
Andrey Pechkurov

Andrey Pechkurov

08/31/2022, 3:35 PM
Could be another bug of a different manifestation of the same one
3:36 PM
We're going to fix the crash first and see how the database would behave then
6:22 PM
I'm able to reproduce a crash (not sure if this one is exactly yours), so for now, no need for additional logs
Jone Qiang

Jone Qiang

09/01/2022, 1:58 AM
Do we have timeline when to fix this issue and release a new version?
Andrey Pechkurov

Andrey Pechkurov

09/01/2022, 4:42 AM
The fix should be ready today or tomorrow. As for the release, I need to discuss it with the rest of the team. As temporary measure, I could share a snapshot build with you once the fix is merged. WDYT?
Jone Qiang

Jone Qiang

09/01/2022, 4:42 AM
ok sure,thx