Title
#users-public
j

John Warila

10/21/2022, 4:32 PM
Hey folks had an interesting issue last night: We deploy Quest via the k8s helm chart, and during a rebuild of our cluster we noticed that the qdb pod would not come up. Turns out helm took an update (we were pinned on qdb version but not helm chart), and the 0.22.0 version of the chart no longer comes up in our k8s. Reverting to 0.21.0 works again. It looks like the change is moving the server.conf file around, and the error is related:
chown: changing ownership of '/var/lib/questdb/conf/server.conf': Read-only file system
. Having server.conf not in the root is nice, but we will need to do some debugging on our cluster to get this change to work.
s

Steven Sklar

10/21/2022, 5:56 PM
The latest 0.22.0 change moved the default mount path of the config files from
/root/.questdb/db
to
/var/lib/questdb
. If you’re running as root, you should have access to
/var/lib
, but it seems like this isn’t the case. Could you possibly share your config so I can run some tests and try to reproduce the issue?
j

John Warila

10/21/2022, 5:57 PM
# Helm values for the QuestDB dev instance.
image:
  tag: 6.4.3

service:
  annotations: {}
  type: ClusterIP
  port: 9000
  expose:
    postgresql:
      enabled: true
      port: 8812
    influxdb:
      enabled: true
      port: 9009
    metrics:
      enabled: true
      port: 9003 

# Configuration options.
questdb:
  config:
    enabled: true
    options:
      cairo.commit.lag: 15000
      cairo.max.uncommitted.rows: 1000
      metrics.enabled: true

# Persistent disk setup. 512GB to hold lots of data.
persistence:
  enabled: true
  accessMode: ReadWriteOnce
  size: 512Gi

# Resource utilization.
resources:
  limits:
    memory: 8Gi
    cpu: 1
5:57 PM
That is the helm values we configure currently
s

Steven Sklar

10/21/2022, 5:59 PM
Thanks! I’ll have a look at those settings later today to see if I can repro the issue
j

John Warila

10/21/2022, 5:59 PM
The PVC is a non-readonly claim
5:59 PM
Thanks Steve
s

Steven Sklar

10/21/2022, 6:31 PM
Looking at your config, is there a specific reason why you are pinning the questdb version? I would recommend upgrading that first to potentially resolve the issue and also take advantage of new features and performance enhancements
j

John Warila

10/21/2022, 6:55 PM
Yeah we specifically are running that version in prod/dev since it is known to work with our schema/ingestion strategy - there were a couple regressions during the last update so we are pinning until we have time to fix them
6:55 PM
6.4.3 has been running fine, we plan to upgrade when we build out retention via detach/attach partition
s

Steven Sklar

10/21/2022, 9:46 PM
I believe the issue is that we are mounting the server and log configs to subdirectories of the
QUESTDB_DATA_DIR
and then trying to
chown
them as the
questdb
in the docker-entrypoint.sh. In previous chart versions, we were mounting the configs to
/root/.questdb
and thus skipped the
chown
step on line 38 of the entrypoint script.
9:47 PM
I can confirm that the helm chart v0.22.0 is working with questdb 6.5.3
9:49 PM
There were no relevant
Dockerfile
changes between questdb versions 6.4.3 and 6.5.3, and the only relevant
docker-entrypoint.sh
change is this bugfix by @Marius Gheorghies in 6.5: https://github.com/questdb/questdb/pull/2354
9:50 PM
I believe this is the fix that will allow us to mount configmaps in
/var/lib/questdb/
, but @Marius Gheorghies can confirm because he has more context
9:52 PM
I’m still playing with these various config settings to get qdb 6.4.3 working with helm chart 0.22.0, but we might need to make another helm chart release so we can mount the configs to
/root/.questdb
to avoid the
chown
in the first place…
j

John Warila

10/21/2022, 9:59 PM
Hmm copy, thanks for the insights on the background - what you posted matches up with what we are seeing in our dev cluster when using 6.4.3 and 0.22.0
s

Steven Sklar

10/21/2022, 10:14 PM
https://github.com/questdb/questdb-kubernetes/pull/89 I have a PR that makes the configmap mount paths configurable. It works locally for me, but I’ll wait for feedback from others.. please let me know if this works for you
10:15 PM
One other thing that I’d like to point out is that you need to update your
questdb.config
key to
questdb.serverConfig
per the changes in
0.21.0
in the Helm chart. We separated out the serverConfig and logConfig since they are two separate configmap mounts
j

John Warila

10/21/2022, 10:22 PM
Thanks for the check - I think we were several versions back so didn't get hit by that key change until today
Marius Gheorghies

Marius Gheorghies

10/21/2022, 11:29 PM
hi, sorry for late reply, you can also set
IGNORE_FIND_AND_OWN_DIR=true
as env variable and it will skip owning the files
11:29 PM
you will have to do it by hand if those files owned by root
s

Steven Sklar

10/24/2022, 1:18 PM
I’ll get this merged in today
5:13 PM
All set! You should be good to go with helm version 0.22.2. Please let me know if you continue to experience issues with the older questdb version
j

John Warila

10/24/2022, 5:13 PM
Cool, thanks Steve - I will give it a go in a few hours