Schema evolution

Schema changes by target

See how each kind of target receives a source schema change, from the DDL SQL Flow runs on databases to the message shape on Kafka and Pub/Sub.

Every target receives the schema changes of the tables it replicates, in order with the data. What that means depends on the kind of target: SQL Flow alters a database table, changes the shape of the messages on an event stream, or writes files with the new columns for a data platform.

Relational databases

On PostgreSQL, MySQL, Oracle, SQL Server, Azure SQL, SQL database in Fabric and Sybase ASE, SQL Flow alters the target table. It writes its own DDL for the target, rather than repeating the statement that ran on the source, and uses the target names you set.

The DDL SQL Flow runs

These are the statements SQL Flow runs for each kind of change, either automatically or as the proposed DDL of a held change that you approve. Names and types are examples.

-- Add a nullable column
ALTER TABLE [dbo].[Orders] ADD [Channel] nvarchar(20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL;

-- Add a NOT NULL column: existing rows get the source value through a
-- temporary default, which is dropped in the same transaction
ALTER TABLE [dbo].[Orders] ADD [Priority] int NOT NULL
    CONSTRAINT [__sqlflow_backfill_<hash>] DEFAULT (0) WITH VALUES;
ALTER TABLE [dbo].[Orders] DROP CONSTRAINT [__sqlflow_backfill_<hash>];

-- Drop a column
ALTER TABLE [dbo].[Orders] DROP COLUMN [Notes];

-- Rename a column
EXEC sys.sp_rename @objname=N'[dbo].[Orders].[Note]', @newname=N'Notes', @objtype=N'COLUMN';

-- Change a length, an integer or decimal type, or nullability
ALTER TABLE [dbo].[Orders] ALTER COLUMN [Quantity] bigint NOT NULL;

A character column keeps the source column's collation. If the target doesn't support that collation, the change is held for review.

A proposed DDL adds a NOT NULL column without the temporary default, with a comment that asks you to add a default first.

How a change is applied

  • In order. Every change made before the DDL is committed first, and every change made after it is applied under the new shape.
  • In one transaction, once. SQL Flow checks that the target table has exactly the expected shape, runs the DDL and records it together, so a failure rolls everything back and a restart never runs it twice.

A target table that doesn't have the expected shape stops the pipeline at the change. This happens when the table was altered by hand, for example when the new column was added in advance. See Changes that stop the pipeline.

SQL Flow records applied schema changes and reviews in its bookkeeping tables on the target. See what SQL Flow creates on SQL Server and PostgreSQL targets.

Permissions

The target login needs the right to alter the target tables. Without it, an automatic change stops the pipeline with the database's permission error, and an approved change moves to paused: last decision failed.

TargetWhat the login needs
SQL Server, Azure SQL, SQL database in FabricALTER on the schemas that hold the target tables, and the bookkeeping schema permissions listed on the SQL Server target page.
PostgreSQLOwnership of the target tables, and CREATE on the login's default schema, where SQL Flow records reviewed changes.
MySQLALTER on the target tables.
OracleOwnership of the target tables, or the ALTER ANY TABLE system privilege.
Sybase ASEOwnership of the target tables.

Event streaming & messaging

Kafka and Google Pub/Sub have no table to alter. A schema change changes the messages published after it, on the same topics, with the same keys and headers:

  • Added column: appears in data, and in oldData when the source supplies a before-image.
  • Dropped column: no longer appears.
  • Renamed column: appears under its new name.
  • Type change: its values are written in the JSON form of the new type.

No event is published for the rows that existed before the change. Their value of an added column appears in the next event that changes the row.

For example, after ALTER TABLE dbo.Orders ADD channel nvarchar(20) NULL on the source, the update events of a row change from the first shape to the second:

{ "operation": "UPDATE", "table": "dbo.Orders", "timestamp": "2026-10-15T08:30:00+00:00",
  "data": { "order_id": 42, "status": "shipped" } }
{ "operation": "UPDATE", "table": "dbo.Orders", "timestamp": "2026-10-15T09:10:00+00:00",
  "data": { "order_id": 42, "status": "delivered", "channel": "web" } }

No schema or version field

Messages carry no schema, schema version or DDL event. Consumers read the column set from each event, so write consumers that accept fields they don't know yet.

A column whose value the source didn't capture is also left out of an event, so a missing field doesn't mean the column was dropped. See the message format of Kafka and Google Pub/Sub.

Data platform integration

Fabric Open Mirroring

After a schema change, SQL Flow writes the table's change files with its new column set, in the same table folder. The table's _metadata.json, which holds only its key columns, doesn't change.

See Fabric Open Mirroring.

Source monitor

A source monitor writes nowhere, so nothing is altered. The change still appears among the pipeline's schema events, and a target you add later receives schema changes as this page describes for its group.

See also

If this was useful,
our newsletter will be too.

Get monthly insights on data engineering, AI, and building critical infrastructure - direct from the Spectral Core team and CEO Damir Bulic.