Google Pub/Sub

Publish captured changes to Google Cloud Pub/Sub topics as ordered JSON events, using the agent's Google credentials.

A Google Pub/Sub target publishes every captured change as a JSON message to a topic per table, with an ordering key built from the row's primary key. Use it when downstream services consume changes from Pub/Sub.

Requirements

The agent authenticates with the Google credentials available on its host, so the connection form has no credential field. Provide credentials on the agent host in any way Application Default Credentials supports, for example a service account attached to the virtual machine.

Permissions

SQL Flow publishes messages and, for Test connection, lists the project's topics. It doesn't create topics or subscriptions and doesn't read messages. The identity behind the agent's credentials needs:

Without pubsub.topics.list, Test connection fails with The Google identity authenticated, but it cannot list Pub/Sub topics in this project (pubsub.topics.list is required for the target probe). The pipeline itself needs only pubsub.topics.publish.

What SQL Flow creates on the target

SQL Flow creates no topics or subscriptions. It publishes messages to existing topics.

Topic names

Each table's topic is projects/<project>/topics/<prefix><table>, where <table> is the source table name with each . replaced by -. For example, with the prefix cdc-, table dbo.Orders publishes to topic cdc-dbo-Orders. Create a topic with that name for every table before you start the pipeline.

Messages

Each message has:

• Data: a UTF-8 JSON envelope with the same fields as a Kafka message: operation (INSERT, UPDATE, DELETE, or READ for an initial snapshot row), table, timestamp, data, and oldData when the source supplies a before-image. A DELETE carries the deleted row as described in Kafka: Delete events.
• Ordering key: a compact JSON object of the table and the row's primary-key values, for example {"__table":"dbo.Orders","order_id":42}. All events for one row share an ordering key.
• Attributes: cdc-table (the source table), cdc-operation (the operation) and cdc-dedup-id, which is the same every time the same change is published.

{
  "operation": "INSERT",
  "table": "dbo.Orders",
  "timestamp": "2026-10-15T08:30:00+00:00",
  "data": { "order_id": 42, "status": "new" }
}

Delivery

• Pub/Sub keeps each row's events in order. Enable message ordering on your subscriptions to receive them in that order.
• Delivery is at least once. After a failure, SQL Flow publishes the same changes again with the same cdc-dedup-id; use it to drop repeats.

Options

Set snapshotMode on a saved connection with the sqlflow CLI, for example --opt snapshotMode=never on sqlflow connection add.

Test connection proves the credentials and the project, but not that each table's topic exists or that the identity can publish to it.

Limitations

• Every captured table needs a primary key on the source, or the pipeline stops at startup.
• An UPDATE that changes a primary-key value stops the pipeline.
• A TRUNCATE captured from the source stops the pipeline.
• Reload isn't available: Pipeline <id>: Reload isn't available for pubsub targets, because published events and files can't be withdrawn. Create a new pipeline to publish a fresh snapshot.
• An event larger than 10,000,000 bytes stops the pipeline: Pub/Sub event for '<table>' (<operation>) exceeds the 10,000,000-byte service limit. SQL Flow doesn't split or truncate events; reduce the row's size or leave the column out with output columns.
• Credentials come only from the agent host. A connection can't carry a key file or a password.
• A pipeline stopped during the initial snapshot publishes the snapshot of every table again when it restarts.

See also

• Kafka
• Multiple targets
• Initial load
• Agents