https://questdb.io logo
Title
n

Newskooler

02/09/2023, 10:26 AM
Does anyone know why I get this error: Query:
CREATE TABLE my_table (value float, symbol symbol index, timestamp TIMESTAMP) timestamp(timestamp)
PARTITION BY DAY 
WITH maxUncommittedRows=10000, commitLag=1000us;
Error:
unrecognized commitLag after WITH
I am using QDB 6.7 AFAI, this same query used to work on 6.5.x versions…
j

Jaromir Hamala

02/09/2023, 10:47 AM
hi, see the release notes: https://github.com/questdb/questdb/releases/tag/6.6 edit: Imre’s reply is nicer! 🙂
i

Imre

02/09/2023, 10:47 AM
hi @Newskooler, commit lag is automatically managed by QuestDB since 6.6.1. The DB will adjust commit lag dynamically by looking at past ingestion patterns. Since this is managed now automatically we removed this keyword.
n

Newskooler

02/09/2023, 11:03 AM
oh yes… that’s nice!
so is there an option to still set it up manually?
i

Imre

02/09/2023, 11:14 AM
you can set a max lag but generally not recommended
CREATE TABLE my_table (value float, symbol symbol index, timestamp TIMESTAMP) timestamp(timestamp)
PARTITION BY DAY 
WITH maxUncommittedRows=10000, o3MaxLag=1000us;
this will limit the database not to use bigger lag than that, even if that would result in better ingestion performance
on the other side this can help to make ingested data visible for queries faster
n

Newskooler

02/09/2023, 11:16 AM
okay, and when upgrading QDB from old instance to new instance - what happens to the already set commit lag on previous tables?
i

Imre

02/09/2023, 11:20 AM
it is used as o3MaxLag to stay backwards compatible. basically it makes sure that there is no additional delay introduced. the max lag set dynamically will not go above the original setting.
for new tables we do not recommend setting it, just let QuestDB manage it
n

Newskooler

02/09/2023, 11:22 AM
Thanks! In that case - how do we remove it limit such that old tables are set up as per QDB’s recommendation ?
i

Imre

02/09/2023, 11:22 AM
you could try without setting it and see if it works for you
if there is no heavy ingestion O3 QuestDB will set a small value on its own, and if there is a high rate ingestion lot of O3 it will increase it
previously the lag you set was constant
in practice there is always a max value:
long o3MaxLag = getLong(properties, env, PropertyKey.CAIRO_COMMIT_LAG, 10 * Dates.MINUTE_MILLIS);
just set high, 10 minutes
you could set the same
in practice the lag would never reach this value
n

Newskooler

02/09/2023, 11:43 AM
I understand. Thanks a lot for this detailed explainer @Imre I appreciate it
i

Imre

02/09/2023, 11:54 AM
no worries