https://questdb.io logo
Title
m

Michael IZKOOL

09/27/2022, 7:33 PM
I am looking for some guidance on something: We are trying to add a 'week' integer parameter to TimestampFormatUtils compute method. We tried to add it and also add the corresponding 'I' on this line of code https://github.com/questdb/questdb/blob/a7aaa464fba30e9ecc044ce63e62db517bfc5609/core/src/main/java/io/questdb/std/datetime/microtime/TimestampFormatCompiler.java#L1522 but we are getting an error:
java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.Unsafe$UnsafeClassDefiner.define(Unsafe.java:337)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.Unsafe.defineAnonymousClass(Unsafe.java:285)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.BytecodeAssembler.loadClass(BytecodeAssembler.java:418)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.BytecodeAssembler.newInstance(BytecodeAssembler.java:434)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.datetime.microtime.TimestampFormatCompiler.compile(TimestampFormatCompiler.java:1679)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.datetime.microtime.TimestampFormatCompiler.compile(TimestampFormatCompiler.java:245)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.datetime.microtime.TimestampFormatCompiler.compile(TimestampFormatCompiler.java:205)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.datetime.microtime.TimestampFormatCompiler.compile(TimestampFormatCompiler.java:201)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.datetime.microtime.TimestampFormatUtils.<clinit>(TimestampFormatUtils.java:407)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.cairo.PartitionByTest.assertFormatAndParse(PartitionByTest.java:374)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.cairo.PartitionByTest.testDirectoryFormattingWeek(PartitionByTest.java:194)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    ...
Caused by: java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    io/questdb/std/datetime/TimestampFormatAsm.parse(Ljava/lang/CharSequence;IILio/questdb/std/datetime/DateLocale;)J @420: invokestatic
  Reason:
    Type 'io/questdb/std/datetime/DateLocale' (current frame, stack[0]) is not assignable to integer
  Current Frame:
    bci: @420
    flags: { }
    locals: { 'io/questdb/std/datetime/TimestampFormatAsm', 'java/lang/CharSequence', integer, integer, 'io/questdb/std/datetime/DateLocale', integer, integer, integer, integer, integer, integer, integer, integer, long, long_2nd, integer, long, long_2nd, integer, integer, integer }
    stack: { 'io/questdb/std/datetime/DateLocale', integer, integer, integer, integer, integer, integer, integer, integer, integer, integer, long, long_2nd, integer }
  Bytecode:
    0000000: 0336 0b1c 360c 0236 0f14 0011 3710 0536
    0000010: 1204 3613 0336 1419 042b 150c 1db6 001a
    0000020: 370d 150c 160d b800 3f60 360c 102c 2b15
    0000030: 0c84 0c01 1db8 0057 1020 2b15 0c84 0c01
    0000040: 1db8 0057 2b15 0c1d b800 3837 0d16 0db8
    0000050: 0042 3605 150c 160d b800 3f60 360c 1020
    0000060: 2b15 0c84 0c01 1db8 0057 1904 2b15 0c1d
    0000070: b600 1d37 0d16 0db8 0042 0460 3606 150c
    0000080: 160d b800 3f60 360c 1020 2b15 0c84 0c01
    0000090: 1db8 0057 150c 1da2 002b 2b15 0cb9 00ca
    00000a0: 0200 102d a000 1e15 0c07 601d b800 4c2b
    00000b0: 150c 0460 840c 0515 0cb8 0046 7436 07a7
    00000c0: 0018 150c 0660 1db8 004c 2b15 0c84 0c04
    00000d0: 150c b800 4636 0710 202b 150c 840c 011d
    00000e0: b800 5715 0c04 601d b800 4c2b 150c 840c
    00000f0: 0215 0cb8 0046 3608 103a 2b15 0c84 0c01
    0000100: 1db8 0057 150c 0460 1db8 004c 2b15 0c84
    0000110: 0c02 150c b800 4636 0910 3a2b 150c 840c
    0000120: 011d b800 5715 0c04 601d b800 4c2b 150c
    0000130: 840c 0215 0cb8 0046 360a 1020 2b15 0c84
    0000140: 0c01 1db8 0057 2b15 0c1d b800 8d37 0d16
    0000150: 0d14 0011 949a 0018 1904 2b15 0c1d b600
    0000160: 2037 0d16 0db8 0042 360f a700 1016 0db8
    0000170: 0042 0085 1400 1369 3710 150c 160d b800
    0000180: 3f60 360c 150c 1db8 004f 1904 1513 1507
    0000190: 1506 1505 1508 1509 150a 150b 1514 150f
    00001a0: 1610 1512 b800 5bad                    
  Stackmap Table:
    full_frame(@194,{Object[#9],Object[#16],Integer,Integer,Object[#14],Integer,Integer,Top,Top,Top,Top,Integer,Integer,Long,Integer,Long,Integer,Integer,Integer},{})
    full_frame(@215,{Object[#9],Object[#16],Integer,Integer,Object[#14],Integer,Integer,Integer,Top,Top,Top,Integer,Integer,Long,Integer,Long,Integer,Integer,Integer},{})
    full_frame(@365,{Object[#9],Object[#16],Integer,Integer,Object[#14],Integer,Integer,Integer,Integer,Integer,Integer,Integer,Integer,Long,Integer,Long,Integer,Integer,Integer},{})
    same_frame(@378)
    at java.base/jdk.internal.misc.Unsafe.defineAnonymousClass0(Native Method)
    at java.base/jdk.internal.misc.Unsafe.defineAnonymousClass(Unsafe.java:1223)
    at jdk.unsupported/sun.misc.Unsafe.defineAnonymousClass(Unsafe.java:830)
    ... 43 more
java.lang.AssertionError
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.BytecodeAssembler.newInstance(BytecodeAssembler.java:435)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.datetime.microtime.TimestampFormatCompiler.compile(TimestampFormatCompiler.java:1679)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.datetime.microtime.TimestampFormatCompiler.compile(TimestampFormatCompiler.java:245)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.datetime.microtime.TimestampFormatCompiler.compile(TimestampFormatCompiler.java:205)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.datetime.microtime.TimestampFormatCompiler.compile(TimestampFormatCompiler.java:201)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.std.datetime.microtime.TimestampFormatUtils.<clinit>(TimestampFormatUtils.java:407)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.cairo.PartitionByTest.assertFormatAndParse(PartitionByTest.java:374)
    at io.questdb@6.5.3-SNAPSHOT/io.questdb.cairo.PartitionByTest.testDirectoryFormattingWeek(PartitionByTest.java:194)
...
Wondering if we missed any steps required to add a parameter to this method?
v

Vlad

09/28/2022, 10:05 AM
hi Michael, is there a fork i can checkout to have a look ?
i really deal with this by way of exclusion, you start with working code, then introduce changes one line at a time - see if it compiles
one problem i recently fixed, which may not be relevant here, but still. When you add items to pool, such as method. There is pool counter and each item gets an index. This index have to be used differently depending on its value. E.g. when it overflows byte, another bytecode that can take
short
has to be used
so all in all - it is a bit tricky
m

Michael IZKOOL

09/28/2022, 12:42 PM
Hey Vlad! Here is a branch of my fork with only the changes to add 1 new parameter to TimeFormatUtils.compute: https://github.com/Belair34/questdb/tree/addWeekToCompute We want to pass the new 'week' parameter from the GenericTimestampFormat.parse method into compute, but to test it I added a throw-away test at the top of PartitionByTests that you can use to reproduce the error.
v

Vlad

09/30/2022, 10:50 AM
hey Michael, just looking at the problem now. Got your fork and tried running test - how can i reproduce the issue ?
m

Michael IZKOOL

09/30/2022, 12:49 PM
Thanks for taking a look! You can reproduce the error by running a test I put at the top of PartitionByTest called testTimestampFormatUtilsCompute() line 51. This compute method will be used in the GenericTimestampFormat parse method, but it throws the exact same error from this test.
v

Vlad

09/30/2022, 3:11 PM
excellent, thank you!
checking
did you commit/push the test ?
m

Michael IZKOOL

09/30/2022, 3:39 PM
Yep, it should be in the addWeekToCompute branch in the file PartitionByTest. Here are the commit changes: https://github.com/Belair34/questdb/commit/85dff38439c89d0c16a5d59ccf0bf8a03cce724b#diff-dcaefb31928f42b904b30744f[…]2af29bc01137922b8a975a6ba322R53
v

Vlad

09/30/2022, 3:43 PM
alright, sorry!
got it now, can reproduce, give me 5
it looks like you went into a deep end. I will need to recollect how the method is assembled. What I would do is to add only one new flag in a non-impacting manner. What we currently have is that even for existing flags (http date format) bytecode is assembled incorrectly
it is type mismatch on method call essentially, but I will need to understand what caused type misalignment
will try to do it over the weekend
m

Michael IZKOOL

09/30/2022, 4:15 PM
I see, yeah it definitely felt like we went a bit deeper than we could handle, haha. Thanks a lot for taking the time to try and solve this!
Update: We ended up figuring it out. We needed to add a lot of stuff to the TimestampFormatCompiler's version of the parse method to get it to work. Previously we thought it was calling the generic version. It wasn't easy to figure out what all the assembly stuff meant, but we got it!