Is there any way to enforce unique constraint in quest db? We are reading some log files and inserting time series data into Quest. But what if during batch insertion some of the records were written to table but ingestion job halted mid way. So some partial records might have been written to Quest. Now our application will mark this ingestion as FAILED and it will retry the same file for ingestion. But since partial data is already there in Quest how can I ensure that there are no duplicate rows?

Hello Ajay, QuestDB currently does not support native deduplication. You have multiple way to solve it: 1. Each row could include a unique identifier and then you would filter it out at a query time. think of UUID column: normally there would be no duplicated UUIDs. if there is a duplicated UUID then you know it’s caused by retrying and you should ignore the dup. 2. You could use Apache Kafka between your producer and QuestDB and then use QuestDB Kafka Connector to ingest data to QuestDB. This way you could rid off some duplicates. There are just examples, there are other strategies too.

@Jaromir Hamala I will be ingesting millions of rows and to check UUID I will need to fire a SELECT query before INSERT query and that seems inefficient

understood. we have a feature request open: https://github.com/questdb/roadmap/issues/3 please add your scenario, this helps us with prioritization.

Okay sure

yes, totally! it’s being cooked right now. we are implementing another strategy for table mutations (including inserts) and it’ll allow concurrent producers. it should be released …well, soon :)

Great… This seems like an exciting feature to have.

indeed! you can see this feature here: