https://questdb.io logo
Title
s

SM

03/27/2023, 12:46 AM
I am trying to setup a service file in Ubuntu (systemd) to start QuestDB automatically on boot. I looked at this message: https://questdb.slack.com/archives/C1NFJEER0/p1675616196079109?thread_ts=1675604388.418159&cid=C1NFJEER0 but unable to get it working. QuestDB starts but exits immediately. I am able to use the ExecStart statement on the comamnd line and it works perfectly fine. Here is my service file:
[Unit]
Description=QuestDB
Documentation=<https://www.questdb.io/docs/introduction>
After=network.target
[Service]
Type=simple
Restart=always
RestartSec=2
ExecStart=/home/myuser/questdb-7.0.1/bin/questdb.sh start -d /mnt/qdbdata/questdb
ExecStop=/mnt/myuser/questdb-7.0.1/bin/questdb.sh stop
ExecReload=/bin/kill -s HUP $MAINPID
ProtectSystem=full
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=questdb
[Install]
WantedBy=multi-user.target
What am I doing wrong?
a

Andrey Pechkurov

03/27/2023, 6:57 AM
Hello, The referenced example uses
java
to run the database.
questdb.sh
is different since it starts the process in the background. If you want to use
questdb.sh
, you should try
oneshot
service type like in the original message.
s

SM

03/28/2023, 5:21 AM
[Unit]
Description=QuestDB
Documentation=<https://www.questdb.io/docs/introduction>
After=network.target
[Service]
Type=oneshot
ExecStart=/home/user/questdb-7.0.1/bin/questdb.sh start -d /questdb
ExecStop=/home/user/questdb-7.0.1/bin/questdb.sh stop
ExecReload=/bin/kill -s HUP $MAINPID
ProtectSystem=full
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=questdb
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
I used oneshot. Now it starts but is still exits. Upon checking status of this service, I see Active (exited) while earlier it wasn;t even starting. Can you spot any mistakes I may be making?
a

Andrey Pechkurov

03/28/2023, 6:04 AM
Have you checked server logs? If the database can't start, they should contain something meaningful
I've just tried the following config and it seems to work fine:
[Unit]
Description=QuestDB
Documentation=<https://www.questdb.io/docs/introduction>
After=network.target
[Service]
Type=oneshot
ExecStart=/home/puzpuzpuz/Downloads/questdb-7.0.1-rt-linux-amd64/bin/questdb.sh start -d /tmp/questdb
ExecStop=/home/puzpuzpuz/Downloads/questdb-7.0.1-rt-linux-amd64/bin/questdb.sh stop
ProtectSystem=full
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=questdb
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
s

SM

03/28/2023, 11:54 PM
Strange. I used exactly this config (of course, changed the paths to questdb to match mine) and the service status shows as Active (exited). When I go to localhost:9000, I get a blank screen. Then I restart the service with systemctl restart questdb.service and everything works.
a

Andrey Pechkurov

03/29/2023, 7:38 AM
The status will be Active (exited), that's ok. This means that the original process exited, but the service is considered running
s

SM

03/29/2023, 7:45 AM
I thought so too, but the localhost UX doesn't load until I restart the service
a

Andrey Pechkurov

03/29/2023, 7:46 AM
Have you checked server logs?
s

SM

03/29/2023, 8:41 AM
Attached. Nothing jumped to me as a problem
a

Andrey Pechkurov

03/29/2023, 8:59 AM
2023-03-29T08:33:32.739229Z A http-server listening on 0.0.0.0:9500 [fd=127 backlog=64]
It listens on 9500, not on 9000, so you should be checking localhost:9500
s

SM

03/29/2023, 9:04 AM
That's right. I am on 9500 (9000 conflicts with another service). But same problem. I get a site cannot be reached message on the browser at localhost:9500
Here is the output to systemctl status questdb.service
● questdb.service - QuestDB
     Loaded: loaded (/etc/systemd/system/questdb.service; enabled; vendor preset: enabled)
     Active: active (exited) since Wed 2023-03-29 01:33:30 PDT; 37min ago
       Docs: <https://www.questdb.io/docs/introduction>
    Process: 856 ExecStart=/home/myuser/questdb-7.0.1/bin/questdb.sh start -d /questdb (code=exited, status=0/SUCCESS)
   Main PID: 856 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 18972)
     Memory: 2.9M
        CPU: 3.783s
     CGroup: /system.slice/questdb.service

Mar 29 01:33:32 desktop-nuc questdb[904]:         at io.questdb@7.0.1/io.questdb.std.ThreadLocal.get(ThreadLocal.java:46)
Mar 29 01:33:32 desktop-nuc questdb[904]:         at io.questdb@7.0.1/io.questdb.network.NetworkError.instance(NetworkError.java:47)
Mar 29 01:33:32 desktop-nuc questdb[904]:         at io.questdb@7.0.1/io.questdb.cutlass.line.udp.AbstractLineProtoUdpReceiver.bind(AbstractLineProtoUdpReceiver.java:141)
Mar 29 01:33:32 desktop-nuc questdb[904]:         at io.questdb@7.0.1/io.questdb.cutlass.line.udp.AbstractLineProtoUdpReceiver.<init>(AbstractLineProtoUdpReceiver.java:73)
Mar 29 01:33:32 desktop-nuc questdb[904]:         at io.questdb@7.0.1/io.questdb.cutlass.line.udp.LinuxMMLineUdpReceiver.<init>(LinuxMMLineUdpReceiver.java:40)
Mar 29 01:33:32 desktop-nuc questdb[904]:         at io.questdb@7.0.1/io.questdb.cutlass.Services.createLineUdpReceiver(Services.java:169)
Mar 29 01:33:32 desktop-nuc questdb[904]:         at io.questdb@7.0.1/io.questdb.ServerMain.<init>(ServerMain.java:202)
Mar 29 01:33:32 desktop-nuc questdb[904]:         at io.questdb@7.0.1/io.questdb.ServerMain.<init>(ServerMain.java:70)
Mar 29 01:33:32 desktop-nuc questdb[904]:         at io.questdb@7.0.1/io.questdb.ServerMain.<init>(ServerMain.java:66)
Mar 29 01:33:32 desktop-nuc questdb[904]:         at io.questdb@7.0.1/io.questdb.ServerMain.main(ServerMain.java:229)
You can see there is some sort of Java NetworkError at the bottom, so I don't think QuestDB is really starting (e.g. the re no ascii logo) . If I restart the service, everything starts to work fine and the error goes away
a

Andrey Pechkurov

03/29/2023, 9:20 AM
AbstractLineProtoUdpReceiver - I guess it can't bind UDP port. Do you have something on 9009?
Maybe you have another QDB running?
s

SM

03/29/2023, 9:52 AM
I only have one instance. ss -tunelp | grep 9009 shows this:
udp   UNCONN 0      0             0.0.0.0:9009       0.0.0.0:*    ino:66661 sk:200d cgroup:/system.slice/questdb.service <>                                                                                                                      
tcp   LISTEN 0      256           0.0.0.0:9009       0.0.0.0:*    ino:66660 sk:2014 cgroup:/system.slice/questdb.service <->
So it's questdb listening on 9009
a

Andrey Pechkurov

03/29/2023, 10:00 AM
Do you have full log with this error?
io.questdb@7.0.1/io.questdb.cutlass.line.udp.AbstractLineProtoUdpReceiver.bind(AbstractLineProtoUdpReceiver.java:141)
s

SM

03/29/2023, 10:59 AM
Where would this log be? I already pasted the log above from /questdb/log which doesn't have the line you mentioned
Also when I restart my PC, questdb is not listening on 9009 because of this network error. The output above I pasted where questdb seems to be listening is only after I restart the service and questdb works correctly
a

Andrey Pechkurov

03/29/2023, 12:00 PM
This log might be in the stdout of the exited process. Very likely that the server didn't get the chance to write it to the log file.
Could you check syslog?
s

SM

04/02/2023, 3:49 PM
Didn't find anything of consequence in syslog. systemctl status questdb.service outputs:
● questdb.service - QuestDB
     Loaded: loaded (/etc/systemd/system/questdb.service; enabled; vendor preset: enabled)
     Active: active (exited) since Sun 2023-04-02 08:09:11 PDT; 20min ago
       Docs: <https://www.questdb.io/docs/introduction>
   Main PID: 1349 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 57641)
     Memory: 38.6M
     CGroup: /system.slice/questdb.service

Apr 02 08:09:12 desktop-sm questdb[1396]:         at io.questdb@7.0.1/io.questdb.std.ThreadLocal.get(ThreadLocal.java:46)
Apr 02 08:09:12 desktop-sm questdb[1396]:         at io.questdb@7.0.1/io.questdb.network.NetworkError.instance(NetworkError.java:47)
Apr 02 08:09:12 desktop-sm questdb[1396]:         at io.questdb@7.0.1/io.questdb.cutlass.line.udp.AbstractLineProtoUdpReceiver.bind(AbstractLineProtoUdpReceiver.java:141)
Apr 02 08:09:12 desktop-sm questdb[1396]:         at io.questdb@7.0.1/io.questdb.cutlass.line.udp.AbstractLineProtoUdpReceiver.<init>(AbstractLineProtoUdpReceiver.java:73)
Apr 02 08:09:12 desktop-sm questdb[1396]:         at io.questdb@7.0.1/io.questdb.cutlass.line.udp.LinuxMMLineUdpReceiver.<init>(LinuxMMLineUdpReceiver.java:40)
Apr 02 08:09:12 desktop-sm questdb[1396]:         at io.questdb@7.0.1/io.questdb.cutlass.Services.createLineUdpReceiver(Services.java:169)
Apr 02 08:09:12 desktop-sm questdb[1396]:         at io.questdb@7.0.1/io.questdb.ServerMain.<init>(ServerMain.java:202)
Apr 02 08:09:12 desktop-sm questdb[1396]:         at io.questdb@7.0.1/io.questdb.ServerMain.<init>(ServerMain.java:70)
Apr 02 08:09:12 desktop-sm questdb[1396]:         at io.questdb@7.0.1/io.questdb.ServerMain.<init>(ServerMain.java:66)
Apr 02 08:09:12 desktop-sm questdb[1396]:         at io.questdb@7.0.1/io.questdb.ServerMain.main(ServerMain.java:229)
Now, when I do systemctl restart questdb.service, it starts up alright. I think the issue is in "io.questdb@7.0.1/io.questdb.network.NetworkError.instance(NetworkError.java:47)". Don't know what it is though that is causing this network error when the service start but goes away after the computer is fully booted up.
a

Andrey Pechkurov

04/03/2023, 6:34 AM
Unfortunately this part of the log doesn't contain the OS error code, so it's not very helpful. Could you try setting
line.udp.enabled
to
false
to disable UDP completely?
s

SM

04/04/2023, 1:43 AM
I solved this by adding this line in my service file. This delays the start for 60 seconds and then it works
ExecStartPre=/bin/sleep 60
a

Andrey Pechkurov

04/04/2023, 7:31 AM
Nice workaround, but I'm still puzzled on what was the original cause of the issue.