Just a quick question, how do you manage to insert multiple timestamps into QuestDB via the Golang ILP API ? For example, this does not work. The error is because there is no method for me to insert a "timeseries" column, the column types available are just StringColumn, BoolColumn, Int64Column, Float64Column. Is there such a thing as "TimeseriesColumn" ? I can't find it in the documentation. Should I make one ? The server closes the connection forcibly when this method is called:

// Push to db
err := sender.Table("urls").
   Symbol("ticker", ticker).
   StringColumn("start", dp.Start.Format(TimeLayout)).
   StringColumn("end", dp.End.Format(TimeLayout)).
   StringColumn("url", u.String()).
   BoolColumn("done", false).
   At(ctx, time.Now().UnixNano())

How would one correct this statement? Any examples would be of massive help.

I've also encountered another weird issue, I don't know if this is standard in SQL (I've been working with Postgres for the past 8 years and I've not encountered this). I'm open to all criticism. I've got a table full of aggregates called 'aggs', and I've got a table full of urls called 'urls' that are used to pull down the aggregates from a data provider. I use a small program to check which urls from 'urls' have not been used to populate 'aggs' yet, and then use them to fill up 'aggs'. But I need a query to determine which urls are "done", by checking 'aggs', as a sanity check. So I just update the table 'urls', based on what's in aggregates using the following query, or so I thought. 1. The below query does not work, it throws a

Invalid column: q.timestamp

error ONLY when I include both the conditions.

WITH q AS (
  SELECT ticker, timestamp
  FROM aggs 
  LATEST ON timestamp 
  PARTITION BY ticker
)
UPDATE urls u
SET done = true
FROM q
WHERE q.timestamp <= u.end
AND q.ticker = u.ticker;

2. This query below works however.

WITH q AS (
  SELECT ticker, timestamp
  FROM aggs 
  LATEST ON timestamp 
  PARTITION BY ticker
)
UPDATE urls u
SET done = true
FROM q
WHERE q.timestamp <= u.end;
--AND q.ticker = u.ticker;

3. And this query works.

WITH q AS (
  SELECT ticker, timestamp
  FROM aggs 
  LATEST ON timestamp 
  PARTITION BY ticker
)
UPDATE urls u
SET done = true
FROM q
WHERE q.ticker = u.ticker;

What's the issue with the first statement ? Obviously, I don't need just one condition, I need both of them. The schema for 'aggs' is as follows:

CREATE TABLE 'aggs' (
  ticker SYMBOL capacity 256 CACHE,
  timespan STRING,
  multiplier LONG,
  open DOUBLE,
  high DOUBLE,
  low DOUBLE,
  close DOUBLE,
  volume DOUBLE,
  vw DOUBLE,
  n DOUBLE,
  timestamp TIMESTAMP
) timestamp (timestamp) PARTITION BY DAY;

The schema for 'urls' is as follows:

CREATE TABLE 'urls' (ticker SYMBOL capacity 256 CACHE, start TIMESTAMP,end TIMESTAMP,
url STRING,
done BOOLEAN
) timestamp (end) PARTITION BY DAY;

I think it's related to this issue: https://github.com/questdb/questdb/issues/2329

Hi Shriram. It looks like an error indeed, and maybe you want to add your comments to the github issue you referenced (or start a new one). In the meantime, maybe you can do a workaround

Since it works with a single condition, you could do the join with your conditions before, then update using a single column. Something like this

WITH q AS (
  SELECT ticker, timestamp
  FROM aggs 
  LATEST ON timestamp 
  PARTITION BY ticker
), q_url AS (
  select q.* from q join urls on(ticker)
  WHERE `timestamp` <= end
)
UPDATE urls u
SET done = true
FROM q_url
where u.ticker = q_url.ticker;

Hi Javier, I've already commented on the specific issue, I don't want to open another (would be a duplicate). And thank you for your clever workaround!

Hi , I want to learn questDB storage and how to SIMD, where can I start reading the code

Hello - I have a specific functionality question to make sure its as intended. I am using the python questdb client and I am processing incoming TICK data from interactive brokers socket connection - when data is sent from the broker - I then place this incoming data into variables which feed into the Sender function (using TCP ILP) - use the variables to store in buffer row then flush. This TICK data comes in burst - through web socket connection its like the firehose is turned on. While ingesting data - on thelogging side I see these messages:

2022-10-03T16:00:33.665746Z I i.q.c.l.t.LineTcpMeasurementScheduler idle table going active [tableName=es_ticks]

2022-10-03T16:00:33.665747Z I i.q.c.l.t.TableUpdateDetails closing table writer [tableName=es_ticks]

2022-10-03T16:00:33.665748Z I i.q.c.l.t.TableUpdateDetails closing table parsers [tableName=es_ticks]

2022-10-03T16:00:33.665753Z I i.q.c.l.t.TableUpdateDetails closing table parsers [tableName=es_ticks]

`2022-10-03T16:00:33.665772Z I i.q.c.p.WriterPool open [table=

es_ticks

, thread=24]`

2022-10-03T16:00:33.665821Z I i.q.c.TableWriter open 'es_ticks'

2022-10-03T16:00:33.667156Z I i.q.c.TableWriter switched partition [path='/var/lib/questdb/db/es_ticks/2022-10-03.1842']

2022-10-03T16:00:33.667586Z I i.q.c.TableWriter purging non attached partitions [path=/var/lib/questdb/db/es_ticks]

`2022-10-03T16:00:33.667879Z I i.q.c.p.WriterPool >> [table=

es_ticks

, thread=24]`

2022-10-03T16:00:33.667888Z I i.q.c.l.t.LineTcpMeasurementScheduler assigned es_ticks to thread 0

2022-10-03T16:00:33.667892Z I i.q.c.l.t.TableUpdateDetails network IO thread using table [workerId=1, tableName=es_ticks, nNetworkIoWorkers=1]

2022-10-03T16:00:33.667897Z I i.q.c.p.ReaderPool open 'es_ticks' [at=0:0]

2022-10-03T16:00:33.668038Z I tcp-line-server scheduling disconnect [fd=39, reason=0]

2022-10-03T16:00:33.668041Z I tcp-line-server disconnected [ip=172.17.0.1, fd=39, src=queue]

2022-10-03T16:00:33.668093Z I tcp-line-server connected [ip=172.17.0.1, fd=39]

2022-10-03T16:00:33.668096Z I tcp-line-server connected [ip=172.17.0.1, fd=126]

2022-10-03T16:00:33.668098Z I tcp-line-server connected [ip=172.17.0.1, fd=127]

2022-10-03T16:00:33.668100Z I tcp-line-server connected [ip=172.17.0.1, fd=128]

2022-10-03T16:00:33.668102Z I tcp-line-server connected [ip=172.17.0.1, fd=129]

2022-10-03T16:00:33.668106Z I tcp-line-server connected [ip=172.17.0.1, fd=130]

2022-10-03T16:00:33.668124Z I tcp-line-server scheduling disconnect [fd=92, reason=0]

2022-10-03T16:00:33.668125Z I tcp-line-server disconnected [ip=172.17.0.1, fd=92, src=queue]

2022-10-03T16:00:33.668158Z I tcp-line-server scheduling disconnect [fd=93, reason=0]

2022-10-03T16:00:33.668159Z I tcp-line-server disconnected [ip=172.17.0.1, fd=93, src=queue]

2022-10-03T16:00:33.668180Z I tcp-line-server scheduling disconnect [fd=94, reason=0]

2022-10-03T16:00:33.668181Z I tcp-line-server disconnected [ip=172.17.0.1, fd=94, src=queue]

2022-10-03T16:00:33.668199Z I tcp-line-server scheduling disconnect [fd=95, reason=0]

2022-10-03T16:00:33.668200Z I tcp-line-server disconnected [ip=172.17.0.1, fd=95, src=queue]

2022-10-03T16:00:33.668218Z I tcp-line-server scheduling disconnect [fd=96, reason=0]

2022-10-03T16:00:33.668219Z I tcp-line-server disconnected [ip=172.17.0.1, fd=96, src=queue]

2022-10-03T16:00:33.668237Z I tcp-line-server scheduling disconnect [fd=97, reason=0]

2022-10-03T16:00:33.668238Z I tcp-line-server disconnected [ip=172.17.0.1, fd=97, src=queue]

2022-10-03T16:00:33.668261Z I tcp-line-server scheduling disconnect [fd=98, reason=0]

my question is - is this normal looking behavior with the repetitive oscillating of

2022-10-03T16:00:40.275261Z I tcp-line-server connected [ip=172.17.0.1, fd=39]

2022-10-03T16:00:40.275563Z I tcp-line-server scheduling disconnect [fd=39, reason=0]

of connecting and disconnect?

on stats - it says 0 bytes / 0 bytes disk read write BUT

<http://localhost:9000/#>

I can see the data is in there and I can run queries. I would say sometimes the data is not available for query until maybe 1 to 2 minutes later - i have tried to play with the buffer sizes in this regard. But really I am trying to gauge If my setup is correct.

the key thing im trying to achieve is keep up with data ingestion and be able to keep up with the real time time stamp of the tick data - every 1 minute - i query the database for data - so need fast inserts and data needs to be available - thats the goal at least... each tick data item is a new row - on incoming from interactive brokers socket connection i store them in variables - build out the rows for insert...

any insight appreciated !~

Ok i think its because I am calling

with Sender(HOST, PORT) as sender:
    try:

on each tick row.

I placed this in its own thread

def connectionSender():
    global sender
    sender = Sender(HOST, PORT)
    sender.connect()

Then I have a separate function which which creates the new rows from the incoming data from the broker:

def buffer_Send(bidPrice, askPrice, bidSize, askSize, year, month, day, hour, minute, second):
    try:
        buf = Buffer(init_capacity=1024, max_name_len=127)
        buf.row(
            'es_ticks',
            columns={
                'bidPrice': bidPrice,
                'askPrice': askPrice,
                'bidSize': int(bidSize),

etc... and sends the data to the database

appears to be working as intended.

In golang, does QuestDB have a connection Pool that I can call via multiple goroutines? The NewLineSender does not seem to be thread-safe.

The documentation confirms this, then how do I concurrently insert data into the database? Isn't that the main benefit of using Go? The code below is executed inside a goroutine, there are no senders that are shared between goroutines, each goroutine gets it's own sender... so this should technically work (as it's not mentioned within the documentation). And yet I get the following error:

runtime error: invalid memory address or nil pointer dereference

// Send the data to QDB, if response is not empty
	if res.Results != nil {
		// Connect to QDB and get sender
		sender, _ := qdb.NewLineSender(ctx)

		for _, v := range res.Results {
			err := sender.Table("aggs").
				Symbol("ticker", res.Ticker).
				StringColumn("timespan", "minute").
				Int64Column("multiplier", int64(1)).
				Float64Column("open", v.O).
				Float64Column("high", v.H).
				Float64Column("low", v.L).
				Float64Column("close", v.C).
				Float64Column("volume", v.V).
				Float64Column("vw", v.Vw).
				Float64Column("n", float64(v.N)).
				At(ctx, time.UnixMilli(int64(v.T)).UnixNano())
			db.CheckErr(err)
		}
		// Make sure the sender is flushed
		err = sender.Flush(ctx)
		db.CheckErr(err)

		// Close the sender.
		err = sender.Close()
		db.CheckErr(err)
	}

Why is the most upvoted answer here flat-files? https://quant.stackexchange.com/questions/29572/building-financial-data-time-series-database-from-scratch This makes no sense to me. Any thoughts by people using relevant data?

Hi everyone, hopefully someone will be able to help me here. 🙂 I am running a local docker instance of QuestDB and I am trying to send quite a large message via influx line protocol (few megabytes in a single field). Quest was dropping my connection so I started to tune some parameters and used something like:

-e QDB_LINE_TCP_MSG_BUFFER_SIZE=5000000 -e QDB_LINE_TCP_MAX_MEASUREMENT_SIZE=5000000

However, now the container just starts and ends without any explicit error, maybe it runs out of memory? Any advice regarding large messages and about how to tune these kind of parameters and how they sum up to total memory consumption of QuestDB process?

Hi guys, I am currently running QuestDB AMI and trying to loop through a dataframe to insert some test data; I am flushing the

sender

each iteration; intermittently I get errors like

Got error: Could not flush buffer: Connection reset by peer (os error 104)

can anyone point me in the right direction to understand this?

<!here> Any users working in the blockchain space? What do you think about this feature proposal? 🤔 Feel free to share your thoughts with us in this thread or comment on the GitHub issue directly. https://github.com/questdb/questdb/issues/2690

There's an issue I am facing with

sender

's not being released. I am on a Windows 11 system, and as mentioned there's a limit of 200 connections in Windows. I've modified that to 10000 from the config file as well, yet I'm getting the following error:

panic: runtime error: invalid memory address or nil pointer dereference
        panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x40 pc=0x104dfdc]

The error code points to this section of the code:

// Go Func to insert data into the database, reads from the channel
	go func() {
		// Makes sure wg closes
		defer wg.Done()

		// Make a goroutine that will accept data from a channel and push to questDB
		ctx, cancel := context.WithCancel(context.Background())

		// Get newline sender
		sender, err := qdb.NewLineSender(ctx)
		db.CheckErr(err)

		// Defer close the sender
		defer func() {
			if err = sender.Close(); err != nil {
				fmt.Println(err)
			}
		}()

		// Get the values from the channel
		for res := range c {
			for _, v := range res.Results {
				err := sender.Table("aggs").
					Symbol("ticker", res.Ticker).
					StringColumn("timespan", "minute").
					Int64Column("multiplier", int64(1)).
					Float64Column("open", v.O).
					Float64Column("high", v.H).
					Float64Column("low", v.L).
					Float64Column("close", v.C).
					Float64Column("volume", v.V).
					Float64Column("vw", v.Vw).
					Float64Column("n", float64(v.N)).
					At(ctx, time.UnixMilli(int64(v.T)).UnixNano())
				db.CheckErr(err)
			}

			// Make sure the sender is flushed
			err := sender.Flush(ctx)
			db.CheckErr(err)
		}

		// Cancel the context
		cancel()
	}()

It's definitely a case of connections to the database not being closed, as there was a delay in the error, AFTER increasing the number of connections to the db. I've even tied it to the context, and it simply does not release the connection, any suggestions are welcome.

Hi @Shriram Sunder The Go client releases the TCP connection on `Close()`: https://github.com/questdb/go-questdb-client/blob/fa4d7bd7b59ecc935e0e6867dc7c46f5c7e3f9c8/sender.go#L248 The question is whether you're calling

Close()

correctly. It looks like it will be closed only when the

c

channel is explicitly closed, so that the goroutine execution finishes. Are you sure you're closing the channel? One more question: how many of these goroutines do you start? If you're not trying to ingest more than 300-500 thousand rows per second, you should be fine with a single TCP connection - no need to open lots of them.

Hi team, I am using your c-client questdb repo. My OS is windows. I use CMake and use your repo through submodule. I build a custom static library inside my project. In that library, I link my library to your questdb_client library. at this stage, it's all good. but after i link the library to my executable, i got the following error: error LNK2005: "class basic_view<struct line_sender_column_name,&bool __cdecl line_sender_column_name_init(struct line_sender_column_name *,unsigned __int64,char const *,struct line_sender_error * *)>::questdb::basic_view<struct line_sender_column_name,&bool __cdecl line_sender_column_name_init(struct line_sender_column_name *,unsigned __int64,char const *,struct line_sender_error * *)> __cdecl questdb::ilp::literals::operator "" _cn(char const *,unsigned __int64)" (??__K_cn@literals@ilp@questdb@@YA?AV?$basic_view@Uline_sender_column_name@@$1?line_sender_column_name_init@@YA_NPEAU1@_KPEBDPEAPEAUline_sender_error@@@Z@23@PEBD_K@Z) already defined in TestQuestDB.cpp.obj

Hello everyone, we are having troubles while inserting data on QuestDB. A bit of context: we are inserting live crypto market-data into several tables at the same time, each table has it's own process writing data into it. Everything runs ok until there is a peak in the amount of data. Looking at the DB server there are lots of QuestDB Java processes in

D status

. A peak looks something like this:

Table 1: 6000 rows per second 
Table 2: 2000 rows per second
Table 3: 3000 rows per second
Table 4: 1000 rows per second

Table config:

maxUncommittedRows = 100000
commitLag = 4s

GCP Instance:

8 CPU
32GB RAM
SSD

Are we missing any config? or do you guys have any tip for tuning QuestDB?

also we are using

QuestDB 6.5.3

Are you using ILP or the PostgreSQL protocol for ingestion?

also.. is the data always in order (for each table)

@javier ramirez Hi Javier - we are using ILP, and the data is always in order we take it from the exchange websocket. We noticed some sorting messages in the logs though