Title
f
Furkan Küçük
12/05/2022, 3:02 PMHello guys, I have a question that I'm stuck with, so here I am.
Here is the thing: I am collecting market data within a VPS for uninterrupted collection, which utilizes Kafka as a "message broker". I want to setup a Kafka Connect and QuestDB instance in my NAS at my home and configure the connect instance to fetch messages from the kafka instance at my VPS and save it in QuestDB. I am trying to use the "official" Connect plugin within this repo. My config looks like this:
{
"name": "questdb-sink",
"config": {
"connector.class": "io.questdb.kafka.QuestDBSinkConnector",
"host": "localhost:9009",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
}
}curl -X POST -H "
Accept:application/json" -H "Content-Type:application/json" --data @sink_conf.json http
://localhost:8083/connectors{
"error_code": 500,
"message": "Unexpected character ('}' (code 125)): was expecting double-quote to start field name\n at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 276] (through reference chain: org.apache.kafka.connect.runtime.rest.entities.CreateConnectorRequest[\"config\"])"
}j
Jaromir Hamala
12/05/2022, 3:13 PMHello Furkan,
a wild guess: can you try to remove the comma at the end of this line:
"value.converter": "org.apache.kafka.connect.json.JsonConverter",f
Furkan Küçük
12/05/2022, 6:13 PM@Jaromir Hamala Thanks mate, I missed that point. Nice catch!
Another question @Jaromir Hamala
I have a docker file like this:
FROM ubuntu:latest AS builder
WORKDIR /opt
RUN apt-get update && apt-get install -y curl wget unzip jq
RUN curl -s <https://api.github.com/repos/questdb/kafka-questdb-connector/releases/latest> | jq -r '.assets[]|select(.content_type == "application/zip")|.browser_download_url'|wget -qi -
RUN unzip kafka-questdb-connector-*-bin.zip
FROM confluentinc/cp-kafka-connect:7.2.2
COPY --from=builder /opt/kafka-questdb-connector/*.jar /kafka/connect/questdb-connector/
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:10.5.4
USER 1001{"error_code":500,"message":"Failed to find any class that implements Connector and which name matches io.questdb.kafka.QuestDBSinkConnector, available connectors are: PluginDesc{klass=class io.confluent.connect.jdbc.JdbcSinkConnector, name='io.confluent.connect.jdbc.JdbcSinkConnector', version='10.5.4', encodedVersion=10.5.4, type=sink, typeName='sink', location='file:/usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/'}, PluginDesc{klass=class io.confluent.connect.jdbc.JdbcSourceConnector, name='io.confluent.connect.jdbc.JdbcSourceConnector', version='10.5.4', encodedVersion=10.5.4, type=source, typeName='source', location='file:/usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='7.2.2-ccs', encodedVersion=7.2.2-ccs, type=source, typeName='source', location='file:/usr/share/java/kafka/'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='7.2.2-ccs', encodedVersion=7.2.2-ccs, type=source, typeName='source', location='file:/usr/share/java/kafka/'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='7.2.2-ccs', encodedVersion=7.2.2-ccs, type=source, typeName='source', location='file:/usr/share/java/kafka/'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='7.2.2-ccs', encodedVersion=7.2.2-ccs, type=sink, typeName='sink', location='file:/usr/share/java/confluent-control-center/'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='7.2.2-ccs', encodedVersion=7.2.2-ccs, type=source, typeName='source', location='file:/usr/share/java/confluent-control-center/'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='7.2.2-ccs', encodedVersion=7.2.2-ccs, type=source, typeName='source', location='file:/usr/share/java/confluent-control-center/'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='7.2.2-ccs', encodedVersion=7.2.2-ccs, type=sink, typeName='sink', location='file:/usr/share/java/confluent-control-center/'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='7.2.2-ccs', encodedVersion=7.2.2-ccs, type=source, typeName='soj
Jaromir Hamala
12/06/2022, 8:26 AMmerhaba Furkan, chances are the
cp-kafka-connect