Newskooler

    Newskooler

    1 month ago
    Does anyone have a python function for converting a pandas.DataFrame to ILP? 😃
    Nicolas Hourcard

    Nicolas Hourcard

    1 month ago
    we’ve heard about users doing that already, plus we are going to build this within our python client eventually
    let me try to find useful information for you
    Newskooler

    Newskooler

    1 month ago
    Thank you @Nicolas Hourcard that would be very helpful!
    Nicolas Hourcard

    Nicolas Hourcard

    1 month ago
    @Adam Cimarosti can give some pointers to start with
    Adam Cimarosti

    Adam Cimarosti

    1 month ago
    Hi @Newskooler, At the moment, there's no better way to send data from a Pandas dataframe to QuestDB via the ILP Python client other than iterating through the dataframe row by row (i.e. https://stackoverflow.com/questions/16476924/how-to-iterate-over-rows-in-a-dataframe-in-pandas). That said, we want to have a higher-level API specifically for this that can iterate through the data in native code whilst releasing the GIL. This should give a significant speed boost. I've sketched an API for this in comments: https://github.com/questdb/py-questdb-client/blob/main/src/questdb/ingress.pyx#L767 The way it would work is:
    sender.pandas(   # or `buffer.pandas(`
        'table_name',
        dataframe,
        symbols=['col1', 'col2],
        at=['timestamp_col_name'])
    At the moment there's no implementation for this yet. Is this in line with what you'd be looking? Do you have feedback on the sketched out API?
    .. I forgot to mention: The new
    questdb
    Python client
    v1.0.1
    now allows for
    None
    values in
    symbols={...}
    and
    columns={...}
    , so it should be a bit easier to construct the rows in the presence of missing values (no need to skip the keys now)
    Newskooler

    Newskooler

    1 month ago
    Hi @Adam Cimarosti Yes this is what would help me. I can iterate row by row in pandas now too, but that’s slow too, so generally I’d like to get out of pandas ; but if you can speed it up by tinkering with how the GIL works, then that would be amazing!