Title
#users-public
Newskooler

Newskooler

08/19/2022, 9:42 AM
Does anyone have a python function for converting a pandas.DataFrame to ILP? 😃
Nicolas Hourcard

Nicolas Hourcard

08/19/2022, 12:04 PM
we’ve heard about users doing that already, plus we are going to build this within our python client eventually
12:04 PM
let me try to find useful information for you
Newskooler

Newskooler

08/19/2022, 12:49 PM
Thank you @Nicolas Hourcard that would be very helpful!
Nicolas Hourcard

Nicolas Hourcard

08/19/2022, 1:45 PM
@Adam Cimarosti can give some pointers to start with
Adam Cimarosti

Adam Cimarosti

08/19/2022, 2:14 PM
Hi @Newskooler, At the moment, there's no better way to send data from a Pandas dataframe to QuestDB via the ILP Python client other than iterating through the dataframe row by row (i.e. https://stackoverflow.com/questions/16476924/how-to-iterate-over-rows-in-a-dataframe-in-pandas). That said, we want to have a higher-level API specifically for this that can iterate through the data in native code whilst releasing the GIL. This should give a significant speed boost. I've sketched an API for this in comments: https://github.com/questdb/py-questdb-client/blob/main/src/questdb/ingress.pyx#L767 The way it would work is:
sender.pandas(   # or `buffer.pandas(`
    'table_name',
    dataframe,
    symbols=['col1', 'col2],
    at=['timestamp_col_name'])
At the moment there's no implementation for this yet. Is this in line with what you'd be looking? Do you have feedback on the sketched out API?
2:17 PM
.. I forgot to mention: The new
questdb
Python client
v1.0.1
now allows for
None
values in
symbols={...}
and
columns={...}
, so it should be a bit easier to construct the rows in the presence of missing values (no need to skip the keys now)
Newskooler

Newskooler

08/20/2022, 9:41 AM
Hi @Adam Cimarosti Yes this is what would help me. I can iterate row by row in pandas now too, but that’s slow too, so generally I’d like to get out of pandas ; but if you can speed it up by tinkering with how the GIL works, then that would be amazing!