Title
s
seralbdev
03/05/2023, 7:29 PMHi, I am starting an evaluation for QuestDB as the backend database for industrial device telemetry. It looks quite aligned with my needs but I have a question about ILP TCP. I guess this endpoint cannot guarantee that all sent data will be ingested, right? Client will be disconnected if the msg format is wrong but what about potential network issues? Is there any way for the client to detect that the last flushed data has arrived to the DB?
n
Nicolas Hourcard
03/05/2023, 10:42 PMHi Alberto, the team will be with you tomorrow to go through the specifics
@Imre could you pls help provide guidance on what we have now and what will be coming soon the ILP side? thanks
i
Imre
03/06/2023, 11:52 AMHi @seralbdev, if the problem is a malformed message, the client gets disconnected.
Everything before the problematic line is ingested into the database, and all lines after the broken one should be dropped.
You are right that currently, in case of a network issue, the client is unable to tell what made it into the database and what did not. That is because the client does not get feedback from the server.
Unfortunately you have to find out yourself where the ingestion stopped by querying the database.
Nic already hinted it above that we have plans to improve the situation.
The plan is that as ingestion goes the server will send ack messages back to the client, indicating what has been persisted to disk and available for queries.
If the client gets disconnected for any reason a handshake will take place with the server on reconnect.
In these quick handshake messages the client and the server will agree where they left things off exactly. The client will have an API to provide this information to you.
At the expense of keeping non-acked data in memory the client could even resend the missing data automatically.
s
seralbdev
03/06/2023, 2:35 PMHi,thanks a lot for reaching out so fast!
Well, I think that being ILP the main ingestion interface it makes sense to have some kind of ACK mechanism... It is great to know you are working on it 🙂 I think your product is a nice sweet spot between other options and it is going in a good direction!! Thanks a lot and best regards
n
Nicolas Hourcard
03/06/2023, 2:42 PMHi Alberto, would you be able to share a few more specifics about your use case?
s
seralbdev
03/06/2023, 2:45 PMWell, I work for a industrial automation company. I am doing a small research on gathering information from industrial controllers (PLCs) and deploying it in time series DB for further analysis. I would like to have a reliable system that can cope with bad network conditions. Sometimes this machines/devices do not have nice network connections or we go with 3G/4G
I am working on some edge device systems, so queues and memory are not a big deal, but I need to have a way of detecting dropped data to retry
I have done some small tests with Influx and TimeScale, but I think your product is a much nicer options (on the paper)
n
Nicolas Hourcard
03/06/2023, 2:48 PMthank you for the kind words! what do you like better than influx/timescale ?
we will let you know about the feedback for ILP
s
seralbdev
03/06/2023, 2:58 PMI think that ILP TCP is a much better idea than ILP HTTP. It is great that you can use SQL queries and the DB schema is auto-generated based on the ILP
You have the best of both world
Timescale requires a full blown Postgres there....the ingestion interface is the mail SQL interface...the attack surface exposed is big
n
Nicolas Hourcard
03/06/2023, 3:03 PMthank you
s
seralbdev
03/06/2023, 3:26 PMYou're welcome, I am looking forward for your news!!