Title
#users-market-data
j

Jiri Pokorny

11/14/2022, 12:24 PM
Hi everyone, I have a (hopefully simple) question - what is the last part of partition directory name?
/tmp/cairo/writer/table/2022-11-12T10.40773157
                                      ^^^
Andrey Pechkurov

Andrey Pechkurov

11/14/2022, 12:27 PM
Hi Jiri, That's partition version. Each time when you insert rows somewhere at the beginning or middle of the partition, the database creates a new partition version and later deletes the old one.
12:28 PM
Since the version number is large, it seems to me that you have a lot of out-of-order writes
j

Jiri Pokorny

11/14/2022, 12:34 PM
Hi Andrei, thanks, that's very valuable info. So, if I understood it correctly, there is some kind of write buffer where reordering is relative cheap, in other words - until some timeout, late messages are ok. But after that QuestDB will rewrite whole partition end-to-end to insert single message into proper place?
Andrey Pechkurov

Andrey Pechkurov

11/14/2022, 12:36 PM
Yes, there is an in-memory buffer. See this doc section:https://questdb.io/docs/guides/out-of-order-commit-lag/
j

Jiri Pokorny

11/14/2022, 12:45 PM
Ok, I understand, thanks. 🙂
1:32 PM
Andrei, one more question - can you please elaborate in more detail how is the version incremented exactly? It seems to be always raising number in the scope of a single table. But the numbers seem be really large, can I deduce the exact number of o3 updates from that number or how does it work?
Andrey Pechkurov

Andrey Pechkurov

11/14/2022, 1:41 PM
the best way to get an understanding of how "bad" your O3 writes are is to use Prometheus metrics we have:https://questdb.io/docs/third-party-tools/prometheus/
1:41 PM
questdb_physically_written_rows_total
and
questdb_committed_rows_total
are the most valuable ones
1:42 PM
Write amplification is
questdb_physically_written_rows_total / questdb_committed_rows_total
1:43 PM
As for the partition version, it also gives some pointers on the volume of O3 commits. A large and/or fast growing version number means that your commits overwrite the partition frequently
j

Jiri Pokorny

11/14/2022, 1:49 PM
ok, so there's no direct relation there, thanks
Andrey Pechkurov

Andrey Pechkurov

11/14/2022, 2:11 PM
there is a relation, but it's better to use metrics when analyzing instance efficiency and performance