https://questdb.io logo
Title
i

Imre

01/12/2023, 3:12 PM
hi @cosmin, in your python code, which protocol do you use to send the data? do you use our python client? also, do you see any incoming connection on the ec2 instance (depending on the protocol you selected could be HTTP, PG Wire or ILP)?
and what about the network? can you ping the ec2 instance from the machine where you run your python client code?
c

cosmin

01/12/2023, 3:18 PM
Well firstly, the ec2 instance runs the python client i screenshot AND the questdb.
Not sure which client method cryptofeed uses to send data to questdb, but heres a link to the backend https://github.com/bmoscon/cryptofeed/blob/cbb4a6317c49bf4dcf419ea590a867c0138c5ced/cryptofeed/backends/quest.py#L51
Not sure exactly how to check if theres a incoming connection
The plan was to run cryptofeed in a docker container on a ec2 AMI built questdb
i

Imre

01/12/2023, 3:28 PM
ok, so it is using ILP, connects to localhost:9009
“Not sure exactly how to check if theres a incoming connection” -> you need to check QuestDB logs
you will see it on the log if there is any activity
and if there is any problem
c

cosmin

01/12/2023, 4:07 PM
Gotcha, i went on the console the log is empty
i

Imre

01/12/2023, 4:38 PM
cannot see ILP ingestion but likely this is only the end of the log… looks like you or the python client was querying tables and there was actually data in it.
2023-01-12 15:20:52 2023-01-12T14:20:52.413562Z I i.q.c.TableReader open partition /var/lib/questdb/db/book-BYBIT/2023-01-10 [rowCount=129125, partitionNameTxn=-1, transientRowCount=98, partitionIndex=0, partitionCount=2]
129125 rows in book-BYBIT, 2 partitions
you probably connected with the webconsole and run some queries?
c

cosmin

01/12/2023, 5:12 PM
There were also these logs at the end which I have missed 2023-01-12 17:07:58 2023-01-12T16:07:57.992164Z I pg-server connected [ip=172.17.0.1, fd=22] 2023-01-12 17:12:58 2023-01-12T16:12:58.514084Z I pg-server disconnected [ip=172.17.0.1, fd=22, src=idle] 2023-01-12 17:12:58 2023-01-12T16:12:58.563351Z I pg-server connected [ip=172.17.0.1, fd=22] 2023-01-12 17:17:58 2023-01-12T16:17:58.570479Z I pg-server disconnected [ip=172.17.0.1, fd=22, src=idle] 2023-01-12 17:17:58 2023-01-12T16:17:58.690186Z I pg-server connected [ip=172.17.0.1, fd=22] 2023-01-12 17:22:58 2023-01-12T16:22:58.709262Z I pg-server disconnected [ip=172.17.0.1, fd=22, src=idle]
Aside from this there is nothing else in terms of logs
You can ignore the pastebin that i sent you
I think the issue could be docker related, when i run the cryptofeed feed outside of a docker, it sends data to questdb no problem, but on a docker image, it doesnt send anything
i

Imre

01/12/2023, 5:29 PM
just realised that when running inside the docker container localhost is not the machine you are running docker on
you would need to use
host.docker.internal
instead of localhost. however, cryptofeed has this hardcoded:
class QuestCallback(SocketCallback):
    def __init__(self, host='127.0.0.1', port=9009, key=None, **kwargs):
        super().__init__(f"tcp://{host}", port=port, **kwargs)
you could use this instead:
docker run -d --network=host gunazzi/questdb:latest
this way you share your host’s network with the docker container, so
localhost
inside the container is the same as your machine
c

cosmin

01/12/2023, 6:00 PM
Ahh so that was the issue, tysm. It works now
i

Imre

01/12/2023, 6:03 PM
no worries