Hi guys, any thoughts about efficiently exporting table(2.5B records) to a csv file ?

For storage efficiency, I cannot think of anything other than paginating over the REST API. For time efficiency, maybe launching multiple queries in parallel for non-overlapping time segments that align with your partitions. You would end up with multiple files in this case that you would need to concat

We did the parallel paginated query to the rest api, but where looking for something at file level

Why would you export to csv in the first place?

As julia udf seems no to be in the near future will do heavy computation on bigquery and then back to questdb

Very interesting use case, Sebastián. I will point about this to the core team as we have recently talked about the possibility of both having some sort of in-database processing, and also exploring portable formats like apache arrow. No final decisions or dates at the moment, but your use case for sure can inspire us 🙂

Thanks @javier ramirez will have a look on spark