https://questdb.io logo
Title
u

Utkarsh Chourasia

12/17/2022, 11:34 AM
Hey, can tell me the significance of
i
in the datatype? Is it an indicator that value is too small for to consume 64bits. Is it necessary to have i in long too??? https://questdb.io/docs/reference/api/ilp/columnset-types/#long256
i

Imre

12/17/2022, 11:45 AM
hi, the
i
ending simply means integer value. when there is no
i
at the end the ILP type is float. when the integer value starts with
0x
it is interpreted as long256.
u

Utkarsh Chourasia

12/17/2022, 11:46 AM
Okay, thanks 😊
i

Imre

12/17/2022, 11:46 AM
now depending on the ILP type these values can be inserted into different column types in QuestDB. see the cast tables.
if there is no column yet, and ILP creates the column on the fly, ILP float will result in a DOUBLE column, ILP integer will result in LONG type, ILP long256 will result in LONG256 column.
u

Utkarsh Chourasia

12/18/2022, 12:28 PM
Hey Imre, I am solving one of issues of questDB. I am kinda confused in which what kind of data does it accept in long256? Is it hexadecimal or pure integers or something else?
i

Imre

12/18/2022, 12:47 PM
hi, it is hexa, see the example in the ILP doc posted above:
temps,device=cpu,location=south value=0x123a4i 1638202821000000000\n
note that this datatype does not support arithmetic operations, only equality check. it was created specifically for storing crypto hashes where to have comparison is good enough. https://questdb.io/docs/reference/sql/datatypes/
it has come up that would be nice to bump it up to a full type, meaning to add support for arithmetic operations. i think it will happen but not sure when exactly.
u

Utkarsh Chourasia

12/18/2022, 12:50 PM
So I tried this, but it is still inserting my value as double, I can’t seem to understand why it that.
i

Imre

12/18/2022, 12:52 PM
do you have an example?
u

Utkarsh Chourasia

12/18/2022, 12:52 PM
f32 = "0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFi" // this is my string for long
// performing some operations to convert it into a number.
// converting the numbers into bytes.
// adding prefix '0x' and suffix i
// sending data over to using ipl
This is the output
I’ll share the code as well, just making final changes for a proper review.
If I convert the string to hexadecimal no like
FFFFFFFF
then into bytes. and the the prefix and suffix, then it is giving me an error.
2022-12-18T12:53:59.139708Z E i.q.c.l.t.LineTcpConnectionContext [22] could not parse measurement, INVALID_FIELD_VALUE at 44, line (may be mangled due to partial parsing): 'trades,name=test_ilp1 value=ffffffffffffffff'
i

Imre

12/18/2022, 1:01 PM
just tried it quickly and seems to be working for me
u

Utkarsh Chourasia

12/18/2022, 1:01 PM
can you share ur query?
i

Imre

12/18/2022, 1:01 PM
i did it like this:
tablename f32=0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFi 1465839830100400200\n"
this creates a tble with
f32
column with the type of LONG256
this is the content of the table created:
f32	timestamp
0xffffffffffffffffffffffffffffffff	2016-06-13T17:43:50.100400Z
how do you construct the message?
u

Utkarsh Chourasia

12/18/2022, 1:05 PM
err = sender.
		Table("trades").
		Symbol("name", "test_ilp1").
		Long256Column("value", bignum).
		At(ctx, time.Now().UnixNano())
i

Imre

12/18/2022, 1:05 PM
using a QuestDb client or creating the message yourself?
u

Utkarsh Chourasia

12/18/2022, 1:05 PM
This is my function.
client
i

Imre

12/18/2022, 1:07 PM
which client is it? Python?
C++?
u

Utkarsh Chourasia

12/18/2022, 1:08 PM
Its Go client, I am assigned with the issue to implement long256Column in go-questdb-client
i

Imre

12/18/2022, 1:09 PM
ok, got it… did not see support for long256 in the client
unfortunately i do not know much about go but essentially what you have to achieve is that the
0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFi
char sequence gets into the buffer (without quotes around it)
if you need help with the code, please, share and if i cannot help will pass you to the dev who worked on the go client
u

Utkarsh Chourasia

12/18/2022, 1:13 PM
Thanks, for all the help. I will push the code and share asap.
i

Imre

12/18/2022, 1:13 PM
the fact that you have a double column created tells me that probably the
i
is missing from the end?
but then it would be rejected because it cannot be parsed asa double…
ahhh, i see that there is another value in the table already, was the table already created by another message or manually?
with the column type being DOUBLE
u

Utkarsh Chourasia

12/18/2022, 1:16 PM
I thought that, so I cleared all the data and started off again. But like I shared above, its creating double column
i

Imre

12/18/2022, 1:18 PM
have you checked the message sent on the wire? either on the client side before sent or what the server received?
just looking at the go client codebase. would do something like this probably:
func (s *LineSender) Long256Column(name, val string) *LineSender {
	if !s.prepareForField(name) {
		return s
	}
	s.lastErr = s.writeColumnName(name)
	if s.lastErr != nil {
		return s
	}
	s.buf.WriteByte('=')
	s.lastErr = s.writeStrValue(val, true)
	if s.lastErr != nil {
		return s
	}
	s.buf.WriteByte('i')
	s.hasFields = true
	return s
}
and call it with
sender.Long256Column("value", "0xFFFFFFF")
u

Utkarsh Chourasia

12/18/2022, 1:28 PM
Man, now it’s working
i

Imre

12/18/2022, 1:28 PM
actually it should be
s.writeStrValue(val, false)
not
true
cool 🙂
u

Utkarsh Chourasia

12/18/2022, 1:29 PM
Thanks Imre, thanks a lot.
i

Imre

12/18/2022, 1:29 PM
the protocol is quite sensitive to the endings, quotes… and so on
hence we created the clients
no worries
u

Utkarsh Chourasia

12/18/2022, 1:29 PM
can you tell me, how did you figure that out?
i

Imre

12/18/2022, 1:29 PM
glad i could help
just looked at the code and modified the StringColumn not to send quotes but add the
i
at the end
this is the same field as a string with those modifications
u

Utkarsh Chourasia

12/18/2022, 1:33 PM
Okay, I guess, I missed it caused I was focused on how it was implemented in Int and doing the same for long256.
Again Thanks a lot!
Hey Imre, do you mind checking
0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffi
this as input in your query like you did here https://questdb.slack.com/archives/C1NFJEER0/p1671368510131959?thread_ts=1671276871.521659&cid=C1NFJEER0 The string method you wrote works great until the no. of character exceed 64bit. So I was wondering, that is it the implementation’s limit or something else?
i

Imre

12/18/2022, 3:05 PM
hi, when i run it i get this error on the server’s log:
2022-12-18T15:04:33.615451Z E i.q.c.l.t.LineTcpMeasurementEvent could not write line protocol measurement [tableName=addField, message=inconvertible value: `0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff` [STRING -> LONG256]
io.questdb.cairo.ImplicitCastException: inconvertible value: `0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff` [STRING -> LONG256]
	at io.questdb.cairo.ImplicitCastException.instance(ImplicitCastException.java:108)
	at io.questdb.cairo.ImplicitCastException.inconvertibleValue(ImplicitCastException.java:80)
	at io.questdb.std.Long256FromCharSequenceDecoder.decode(Long256FromCharSequenceDecoder.java:58)
i think it makes sense, this seems to be too big to fit into 256 bits
u

Utkarsh Chourasia

12/18/2022, 3:06 PM
You tested this in console??
i

Imre

12/18/2022, 3:07 PM
no, in a unit test
but an F represents 4 bits
256/4 = 64
so you can have 64 Fs max
just tried, 64 Fs passed, 65 Fs ended up with inconvertible value during STRING ->LONG256 conversion
u

Utkarsh Chourasia

12/18/2022, 3:12 PM
Got it, now it makes sense. Thanks for the data btw.
With 256bit, I was thinking that there must be 256F
i guess, we have successfully implemented long256 column in go client.
i

Imre

12/18/2022, 3:28 PM
sounds great! :)