on C Npgsql I get `Exception while reading from stream` when

Question

on C Npgsql I get `Exception while reading from stream` when trying to execute many SELECTs simultaneously any ideas I have tried to change the timeout settings on some places confused

Bolek Ziobrowski · Answer

You should get an error message for query timeouts not disconnects

Super Richman · Answer

< Bolek Ziobrowski> how can I tell QDB to use all of the available resources it has CPU cores memory etc

Bolek Ziobrowski · Answer

Unless overriden in server conf by default QDB detects number of cores and uses all of them If you ran it in docker then you might want to have a look at container limits

Bolek Ziobrowski · Answer

If you use a big number of connections then the thing to look at is <http pg net|pg net> connection limit setting

Super Richman · Answer

and about memory

Bolek Ziobrowski · Answer

There are no limits on memory usage in default configuration

Bolek Ziobrowski · Answer

< Super Richman> Have you tried adjust the setting and re running SELECTs after instance restart

Super Richman · Answer

I am trying now

Super Richman · Answer

I have as well changed my code a bit so I am not sure anymore what is happening x for now the exceptions stopped

Super Richman · Answer

I am only on windows and I have noticed that one of my machines have the intel rapid storage thingy should I remove it

Bolek Ziobrowski · Answer

If you want better performance then I d recommend switching to Linux Window s page cache and OS calls are much slower

Bolek Ziobrowski · Answer

As for intell rapid storage if you ve optane ssd then I d recommend using it for qdb data directly

Super Richman · Answer

1 I have now ran a test with the connection limit 200 and got many exceptoins now I am running another test with the flag removed to ensure that it caused the exceptions now I got many exceptoins x shit 2 I have only one server and now its a problem to switch to linux 3 I have an SSD but from 2017 so I don t think its optane

Super Richman · Answer

I am closing the pg connection with `conn close ` but I think that qdb is leaving it open for a while in realtime I am only using 10 50 concurrent selects

Bolek Ziobrowski · Answer

Did you restart the instance after changing server conf

Bolek Ziobrowski · Answer

Qdb should close connections immediately

Super Richman · Answer

yes I am restarting every time

Super Richman · Answer

ok now I have set it to 9999 and no exceptoins

Bolek Ziobrowski · Answer

hmm

Bolek Ziobrowski · Answer

Do you pool connections or just open close them rapidly

Bolek Ziobrowski · Answer

If it s the latter then you might be hitting OS limits or firewall might be getting suspicious about the traffic

Bolek Ziobrowski · Answer

It s way better to pool connections because connection setup is quite costly

Super Richman · Answer

Its all on the same machine I have been doing pooling and it was good until I needed concurrent threads access to the db then it stopped working since I have been using the same connection for all SELECTs so I switched to openning and closing connections When I think about it actually I pass a NpgsqlFactory and a connection string to a 3rd party package which is doing the connections itself confused

Bolek Ziobrowski · Answer

Sorry but it doesn t make too much sense Connection pooling doesn t mean you ve just one connection Pooling tens or hundreds of connections shouldn t be a problem Instead of opening a connection and closing connection you should just fetch it from the pool and then put it back

Super Richman · Answer

yes you are right the issue is that I am using a 3rd party app which is connecting confused

Super Richman · Answer

if you have an example c code with npgsql it would be great