Heyho everyone, did the db files get more sensitive between versions? I upgraded today (version 6.6.1 -> 7.0.0) and changed the docker mount from bind-mount to volume. I ran into a couple of problems and ended up not being able to keep my previous data. In this particular case no drama (else I would not have upgraded), but I want to share it so that I and hopefully others might learn something from it. First off some System Info: • Using the official qdb docker Image • Docker host running on CentOS 7 • File System XFS • Docker Storage Driver: Devicemapper Loop (I know this is not recommended and will change it soon) First thing I did: On version 6.6.1 changed the bind-mount to a docker volume at a different location. No problems. Started a new container, everything running fine and all the data available. Then I tried spinning up a new container (after taking down the 6.6.1 one) with version 7.0.0, yet it crashed right away. Tried the same with version 6.7 that I skipped, same thing. Both displayed the same error: CairoException [-1] name is reserved [table=sys.column_versions_purge_lock] Next I set it up in a separate empty bind-mount, all starting nice and tidy. However, stopping and restarting the container led to the same error yet again. Removing sys.column_versions_purge_lock manually from the db directory and... same thing but this time around for "telemetry". Rinse and repeat for telemetry and telemetry_config and it starts. Nice, so then I can set up the same changing from bind-mount to docker volume like I did with version 6.6.1, right? Well, unfortunately it would not let me do that. I copied over the questdb folder of the successful 7.0.0 container, removed the conflicting folders as above (2 actually since I turned off the telemetry in the settings in the meantime, so there was not table "telemetry" created). Successful startup of the container but, unfortunately, no data to read apart from the newly created sys.column_versions_purge_lock and telemetry_config tables. Any query to any other table (existing in the db directory) returns a table does not exist error. The browser interface also only showed the 2 mentioned tables. Lastly I deleted the whole db directory and, starting up the container again, newly generated data and tables can properly be queried. At the cost of the data from the past. Now I am curious if I that was unavoidable or if I was just inept to properly migrate. 😉

Hi, thanks for sharing the experience. Can you elaborate on what was involved in changing bind-mount to volume please?

Hi Alex, thank you for coming back at me :) Regarding the bind-mount to volume question: First I shut down the container, then created the volume using

sudo docker volume create questdb

, copied the files from the original location

/opt/questdb

to the volume location

/var/lib/docker/volumes/questdb/_data

. Then replaced the flag

-v "/opt/questdb:/var/lib/questdb"

with

--mount source=questdb,destination=/var/lib/questdb

in the docker run command. To my knowlegde the usage of -v or --mount should not change anything, I just used the opportunity to switch to the recommended syntax. As for the ownership, the data belongs to the user and group ID, 10001, which is not assigned to the host. Both before and after the copy, as well as the newly created data. I assume that is the ID of the questdb user within the container?

From all the symptoms I can only think that the data was copied without preserving permissions. I may be wrong but this would explain the weird error messages and the log you attached.

That crossed my mind too but how would it explain, that stopping and restarting the container leads to the same permission error? No files moved in that case. And even copying I checked that permissions were preserved (although human error can never be 100% excluded)

Do you think if I create empty bind-mounted questdb container with versin 6.7 or 7.0 it may fail to start?

It should not matter whether it is a bind-mount or volume. It will start, if the tables

sys.column_versions_purge_lock

,

telemetry

and

telemetry_config

do not exist in the db folder. Therefore it will also start if the mounted directory is empty (fresh install). If either of those exist, it fails with the given error (observed for qdb version 6.7 and 7.0.0).

I’ll try the steps, thanks