Reviewing held changes

Review a table held at a source schema change, apply or edit the proposed DDL or remove the table, and see how the table rejoins the stream.

When a target can't apply a source schema change by itself, SQL Flow holds that table at the change and asks you what to do. This page explains the Schema review card, each decision, and how the table rejoins the stream.

Held schema changes

A held change pauses only its table, only on the target that couldn't apply it, at the position of the change. The table's later changes are kept in order and none is applied, while every other table keeps streaming.

SQL Flow holds a table when:

• the target can't make the change exactly, for example a dropped or renamed key column, a type or nullability change the target can't reproduce, a column in a collation the target doesn't support, or a target column name that would collide with another;
• the table's row key changed on the source. See Primary key changes;
• an earlier change of the same table is still held. The later change is reviewed after it;
• a processing rule covers the table. See Tables with in-flight processing rules.

A hold survives agent restarts and takeovers. A held table raises no alert, so after DDL on the source, check the pipeline for a Schema review card.

The Schema review card

The Schema review card appears above the tabs of the pipeline page, on every tab, while any table is held. The pipeline's Schema tab also lists finished reviews.

Each held table has one entry per target. An entry shows:

What to do in each state

For a table with in-flight processing rules, reload the pipeline. See Tables with in-flight processing rules.

A reload of the pipeline clears every hold and closes every open review, because it copies every table again from a fresh snapshot. See Lifecycle.

Decide a held change

1. Open the pipeline and find the table in the Schema review card.
2. Read Why it was not applied automatically, the Proposed DDL and the Data consequence.
3. Choose one decision:
   • Apply proposed DDL runs the proposal as it stands.
   • Edit DDL opens the proposal in an editor. Change it, then select Apply edited DDL, or Resync with edited DDL.
   • Resync table runs the proposal and copies the table again from the source.
   • Remove table from pipeline stops applying the table on this target.
4. Watch the state change to applying, then to resumed. A resync shows paused: awaiting resync while the table is copied.

The pipeline's agent must be online: a decision for an offline agent is refused with The agent '<agent>' holding this review is not online.

Decide a table's reviews in order, earliest first. You can remove the table at any time.

Apply the proposed or edited DDL

SQL Flow runs the DDL once on the target, in one transaction, with a 10-minute timeout. A failing statement rolls the whole transaction back and changes nothing, and the state becomes paused: last decision failed with the error, such as apply DDL failed: <reason>. Fix the cause and apply again.

When the data consequence says "Requires table resync", the table doesn't resume after the DDL. The review moves to paused: awaiting resync: select Resync table to finish.

Edit the DDL when:

• the proposal says it needs an edit. Some proposals hold only comments, for example -- The source column is NOT NULL: add a DEFAULT, or add it NULL, before applying., and Apply proposed DDL is disabled with the hint "The proposal needs an edit first (see its comments).";
• you set your own column names in Configure. The proposal uses the automatic column names. See Name mapping;
• you want the target to differ, for example a wider type than the proposal.

Edited DDL can be up to 65,536 characters. DDL that holds only comments runs nothing on the target, and the table resumes.

Resync the table

Resync table runs the review's DDL on the target and then copies the table again from the source, while every other table keeps streaming. When the copy completes, the table rejoins the stream and the review reads resumed.

Use it for a table whose data consequence says "Requires table resync", or whose review reads paused: awaiting resync. It also works for any other held table, in place of replaying its kept changes.

• If the proposal needs an edit, Resync table is disabled with the hint "Edit the target DDL before resync." until you edit it.
• For a key change, restore a usable row key on the source before you resync.
• After the review's DDL has run, the resync must use that same DDL.

While the copy runs, the card shows its status and no decision buttons. If the copy fails, is refused or expires, the card says "This operation cannot be restarted from the review card; reload the pipeline to recover this table." Reload the pipeline.

While a review of a table is open, resync the table from the card, not from the Tables tab.

Remove the table

Remove table from pipeline closes the review and stops applying the table on this target. Its changes after the schema change are never applied, and the target table keeps the rows it had at the change. Other targets aren't affected.

The table stays in the pipeline's table list. To stop capturing it, remove it on the Tables tab. See Changing tables.

How the table resumes

After the DDL runs, SQL Flow applies the held table's kept changes, in order and under the new shape, until the table has caught up with the rest of the pipeline, then hands it back to the normal stream. The state then reads resumed. Nothing is copied again, and no change is applied twice.

A table whose data consequence says "Requires table resync" can't catch up this way, so it needs a fresh copy. See Resync the table.

Tables with in-flight processing rules

A table that an in-flight processing rule covers is never changed around its rules. Any schema change on a covered table holds it on that target, and with it every table the target's rules cover, until you reload the whole pipeline. The reload compiles the rules against the new schema.

The card's reason starts with processing_source_schema_reload_required. Apply proposed DDL and Resync table can't release the covered tables. Remove table from pipeline closes the review of the changed table, but the other covered tables stay held until the reload.

To change a covered table:

1. Make the source change.
2. If the change affects a rule, for example a dropped column that a filter or an output column reads, update the rules in Configure. See Validation and limits.
3. Reload the pipeline. Saving rules reloads it for you.

Name mapping

Schema changes follow the target names you set in Configure, with one exception:

• Changes applied automatically use your target table and column names. An added column gets the name you set for it, if any.
• Proposed DDL uses your target table name, but the automatic column names. If you renamed a column in Configure, edit the proposal to use your name before you apply it.
• A rename on the source renames the target column from the name you set to the new source column's automatic name, because your name belongs to the old source name. Check the names in Configure after a rename.

See Table and column names.

Errors on the card

A decision the control plane refuses appears in a red box under the card's heading:

A decision that fails on the agent appears on the review entry as Error, for example apply DDL failed: <reason> or resync failed: <reason>. A decision interrupted by an agent restart says so; decide again (DDL that already ran isn't run twice).

See also

• Overview
• Supported changes
• Schema changes by target
• Lifecycle
• Table and column names