Kafka

Publish captured changes to Apache Kafka topics as keyed JSON events, with TLS and SASL authentication.

A Kafka target publishes every captured change as a JSON message, keyed by the row's primary key, to one topic per table or to one shared topic. Use it when downstream consumers read a change stream rather than a replica table.

Requirements

Topic settings

SQL Flow works with any partition count and replication factor. When you create the topics:

• Partitions. Every event of a row carries the same key, so each row's events go to one partition and stay in order. Order across rows isn't kept. Adding partitions to a topic later moves keys to other partitions, so a row's newer events can land on a different partition than its older ones.
• Compaction. With cleanup.policy=compact, Kafka keeps the latest event per row. SQL Flow writes no tombstones: a deleted row's last event is its DELETE event, and compaction keeps it.
• Durability. SQL Flow waits for every in-sync replica to acknowledge each message, so a replication factor of 3 with min.insync.replicas=2 keeps an acknowledged event through the loss of one broker.

Permissions

With SASL, the user in SASL username needs these ACLs. SQL Flow only produces messages, so it needs no permissions on consumer groups or transactional IDs.

What SQL Flow creates on the target

SQL Flow creates no topics and no other objects. It produces messages to existing topics.

Topic names

The wizard shows the first topic names under the form, so you can check them before you continue.

Messages

Each message has:

• Key: a compact JSON object of the row's primary-key columns, for example {"order_id":42}. With One topic for every table, the key also carries the table, for example {"__table":"dbo.Orders","order_id":42}. Every event for one row, including its snapshot, has the same key, so log compaction and keyed upserts work.
• Value: a JSON envelope.
• Headers: cdc-table (the source table), cdc-operation (the operation) and content-type (application/json).

{
  "operation": "UPDATE",
  "table": "dbo.Orders",
  "timestamp": "2026-10-15T08:30:00+00:00",
  "data": { "order_id": 42, "status": "shipped" },
  "oldData": { "order_id": 42, "status": "new" }
}

Binary values are base64 strings, dates and times are ISO 8601 strings, and numbers that don't fit a JSON number exactly, such as very wide decimals, are strings. A column whose value the source didn't capture is left out of the event rather than sent as null.

Delete events

Where a DELETE event carries the deleted row depends on the source. Read oldData when it is present, and data otherwise.

Large values stored outside the row can be missing from a delete event: off-row large values from SQL Server DirectLog and Reconstruction, out-of-row LOBs that are no longer in the Oracle DirectLog redo window, and LOB columns under Oracle LogMiner.

Delivery

• Each row's events arrive in order, and a broker retry never reorders or duplicates messages within a partition.
• Delivery is at least once. After a crash or a lost acknowledgement, SQL Flow publishes the unacknowledged changes again, with the same key and value. Consumers should upsert by key or ignore repeats.

Options

• Initial snapshot. With Stream changes only, no existing row is published; consumers see changes only.
• Security. With Plaintext, the agent connects without authentication or encryption. With TLS, it verifies the brokers' certificates and connects without a SASL credential. GSSAPI (Kerberos), OAUTHBEARER and client certificates aren't supported.
• Host name check. Set sslEndpointIdentification with the sqlflow CLI, for example --opt sslEndpointIdentification=none on sqlflow connection add. Turning it off accepts any certificate the CA signed, whatever host it names.

Test connection proves that the brokers are reachable and that the credential and certificate work. It doesn't produce a message, so it passes before any topic exists or any topic ACL is granted.

Limitations

• Every captured table needs a primary key on the source. Otherwise the pipeline stops at startup with This target derives row identity from the source key; startup is refused rather than falling back to a table-name or conventional-id key.
• An UPDATE that changes a primary-key value stops the pipeline: one event can't move a row from one key to another.
• A TRUNCATE captured from the source stops the pipeline.
• Reload isn't available: Pipeline <id>: Reload isn't available for kafka targets, because published events and files can't be withdrawn. Create a new pipeline to publish a fresh snapshot.
• A message larger than 1,000,000 bytes stops the pipeline with Kafka event for '<table>' is larger than the 1,000,000-byte message limit. SQL Flow doesn't split or truncate events; reduce the row's size or leave the column out with output columns., or Serialized Kafka event is approximately <n> bytes, over the 1,000,000-byte message limit for topic '<topic>'. The limit can't be raised. To leave a large column out, see in-flight processing.
• Only JSON is produced. There is no Avro, Protobuf or schema registry support, and no dead-letter topic: an event that can't be published stops the pipeline.
• A pipeline stopped during the initial snapshot publishes the snapshot of every table again from the start when it restarts, so a topic that isn't compacted can hold duplicate snapshot rows.

See also

• Google Pub/Sub
• Multiple targets
• Initial load
• Database credentials and secret references