Title
#users-public
Newskooler

Newskooler

10/06/2022, 1:32 PM
Hi 👋 Where can I find documentation about the
# rolling file writer
in the
conf.log
? I am not sure how to use it exactly and I suspect it’s exactly what I am looking for. I could not find it here: https://questdb.io/docs/reference/configuration/#logging
1:53 PM
My goal is to: • Have the logs on a mounted volume • Keep only a few days worth of logs This is how I start QDB via Docker:
docker run --name questdb --netowkr="host" --user "$(id -u):$(id -g)"
-v /my/local/path/questdb:/questdb
-v /my/local/path/questdb/public:/questdb/public
-v /my/local/path/questdb/conf:/questdb/conf
-v /my/local/path/questdb/db:/questdb/db
-v /my/local/path/questdb/log:/questdb/log
questdb/quesstdb:6.5.2 /usd/bin/env QDB_PACKAGE=docker /app/bin/java 
-Dout=conf/log.conf
-m io.questdb/io.quesstdb.ServerMain
-d /questdb
-f
in my
/my/local/path/questdb/conf/log.conf
I have the following settings:
# list of configured writers
writers=file,stdout,http.min

# file writer
# w.file.class=io.questdb.log.LogFileWriter
# w.file.location=questdb-docker.log
# w.file.level=INFO,ERROR,DEBUG

# rolling file writer
w.file.class=io.questdb.log.LogRollingFileWriter
w.file.location=${log.dir}/questdb-rolling.log.${date:yyyyMMdd}
w.file.level=INFO,ERROR
w.file.rollEvery=day
w.file.rollSize=1g

# stdout
w.stdout.class=io.questdb.log.LogConsoleWriter
w.stdout.level=INFO

# min http server, used monitoring
w.http.min.class=io.questdb.log.LogConsoleWriter
w.http.min.level=ERROR
w.http.min.scope=http-min-server
12:55 PM
@Andrey Pechkurov can you please help with this?: )
2:10 PM
Can anyone please assist with this logging issue? 🙂 🤔
Pei

Pei

10/11/2022, 2:17 PM
Hey! I’ll get some help 🙂
Andrey Pechkurov

Andrey Pechkurov

10/11/2022, 2:39 PM
I'm not sure if we have docs for it. cc @Amy Wang
2:40 PM
Your config looks ok. Does it work?
Amy Wang

Amy Wang

10/11/2022, 2:41 PM
I am afraid not 😞 I was just looking through the docs to see if I could provide pointers. Watching this thread to add to the docs!
Andrey Pechkurov

Andrey Pechkurov

10/11/2022, 2:41 PM
BTW you may want to change
writers=file,stdout,http.min
to
writers=file,http.min
. I recall that we may be writing stdout into a file in
questdb.sh
(not sure if that's the entrypoint we use in Docker though)
2:42 PM
Keep only a few days worth of logs
This would have to be dome manually: https://github.com/questdb/questdb/issues/2477
Newskooler

Newskooler

10/11/2022, 3:49 PM
To be clear - my main issue is that - no data is saved to my volume mounted logs dir - where my qdb data is saved. The rolling logs is a secondary issue after the first one is fixed. I still can’t fix the first one actually 😞
3:52 PM
It was working well, but then I started running docker from a user account (not root) and then it broke.
Andrey Pechkurov

Andrey Pechkurov

10/11/2022, 4:09 PM
Hi @Marius Gheorghies Could you check the above Docker command and tell us if the mounted volumes look ok to you?
Newskooler

Newskooler

10/11/2022, 4:34 PM
Thanks!
4:34 PM
Then I guess @Andrey Pechkurov - please let me know if my config looks good? Ideally I’d like if the logs are retained only for a few days
Andrey Pechkurov

Andrey Pechkurov

10/11/2022, 5:20 PM
yes, the config is ok, but you'd have to run a separate cron job to remove older log files
Newskooler

Newskooler

10/11/2022, 5:25 PM
Okay - thanks. And the issue you have open in GitHub is about qdb supporting this natively?
Andrey Pechkurov

Andrey Pechkurov

10/11/2022, 5:50 PM
Exactly
Pei

Pei

10/11/2022, 5:55 PM
thanks a lot Andrei!
5:55 PM
also unfamiliar with this topic 😁
Newskooler

Newskooler

10/12/2022, 7:35 AM
@Andrey Pechkurov is it possible to split the rolling logs into INFO / ERROR / DEBUG / etc? Each in a separate file?
Andrey Pechkurov

Andrey Pechkurov

10/12/2022, 8:25 AM
You could try something like this:
writers=file-info,file-error

# file writer
w.file-info.class=io.questdb.log.LogFileWriter
w.file-info.location=${log.dir}/questdb-info.log
w.file-info.level=INFO
w.file-error.class=io.questdb.log.LogFileWriter
w.file-error.location=${log.dir}/questdb-error.log
w.file-error.level=DEBUG
8:26 AM
the only thing to keep in mind is that the defined level defines the lower bound level - all log levels above that value will be included into the file
8:27 AM
say, for ERROR level this means that all E (error), C (critical) and A (advisory) log entries will be written into the file
Newskooler

Newskooler

10/12/2022, 8:55 AM
I see… so then it does not make sense to have a separate one for INFO since INFO will include INFO, ERRO, Critical, etc. Is that correct?
Andrey Pechkurov

Andrey Pechkurov

10/12/2022, 9:16 AM
yes
Newskooler

Newskooler

10/12/2022, 9:23 AM
Does it make sense to submit an issue as a request to change or no? It will be nice if these were separate files
Andrey Pechkurov

Andrey Pechkurov

10/12/2022, 9:38 AM
Yes, why not
9:39 AM
BTW have you considered using Docker's own log drivers like the
journald
one?
9:41 AM
or
syslog
- then configuring retention policy would become simpler
Newskooler

Newskooler

10/12/2022, 9:50 AM
No. How would that go? Here is what I’d like to achieve: • have logs outside docker in a folder of my choice (already achieved that) • have them separate for each level (I will raise an issue for that) • have them kept for X number of time (I can do that via cronjob, but it will be nice if natively supported via QDB if you decide it’s a good idea, as the moving pieces become many…)
Andrey Pechkurov

Andrey Pechkurov

10/12/2022, 10:03 AM
It should use stdout
10:03 AM
And collect logs from there
Newskooler

Newskooler

10/12/2022, 10:21 AM
I will look into that - thanks!