Pipeline wizard

Tables and configure

Select the tables a pipeline replicates, add tables the source test can't see, and open Configure to set names, filters and columns.

The Tables step is the third step of the New pipeline wizard. It lists the tables the source test discovered, and you select the ones to replicate. From the same step you open Configure to set target names and in-flight rules for the selected tables.

Select tables

After a passing source test, the step lists the discovered tables grouped by schema, with every table you can replicate selected. A source that names tables without a schema, such as Sybase, shows one group called (no schema).

  • Select a table's or a schema's checkbox to include or exclude it.
  • Type in Search tables to narrow the list.
  • Choose All or Selected only to change what the list shows. The selection doesn't change.
  • Select Select all or Deselect all to change every discovered table at once.

Tables that can't be selected

SQL Flow applies changes by key, so a table needs a primary key or a unique index over non-null columns. Tables without one appear dimmed with "No primary key or non-null unique index", and can't be selected. Add a key on the source, then run the source test again.

When the source has too many tables for the test to check keys, the step says so. A table without a key is then refused when the pipeline starts.

Tables with text, ntext or image columns

For SQL Server DirectLog, the step shows an amber note when a selected table has text, ntext or image columns. Once DirectLog marks such a table for capture, SQL Server rejects UPDATETEXT (error 7137) and WRITETEXT (error 7138) from your application. Replace those statements with ordinary UPDATE statements before you start capture.

Add a table by hand

Select Add a table by hand (one discovery cannot see), type the name as schema.table, and select Add. Use this for a table the discovery can't see, or when you create a pipeline without an agent and so have no discovery. Tables you add appear under Added by hand (<n>). The agent still verifies that each exists and has a usable key.

If the list is empty, the step says why:

  • No agent assigned: add schema.table by hand, or assign an agent and run the source test.
  • Test not run: run the source test, or add schema.table by hand.
  • Nothing visible: "The source test connected but found no visible tables. Check the database or service and this user's table permissions, then test again."

Stale and unconfirmed selections

If a selected table no longer appears in the latest discovery, the step lists it in amber as not found in the source. Remove it, or run the source test again.

If you change the credentials or options of the same source database, the selection stays, and the step asks you to run the source test again to confirm it. If you change the host, port, database or capture mode, the selection is cleared, because it described a different database.

Configure names and rules

Select Configure at the top of the table list, or on a selected table, to open the Configure dialog for the target. In the dialog you can:

  • give a table or column a different name on the target;
  • keep only the rows that match a condition;
  • choose, compute or mask the output columns;
  • set a default for All tables, and override it for a single table.

Select Done to keep your changes. They are applied when you create the pipeline. A configured badge marks every table that has a name or a rule. The rules themselves are covered in other sections:

You can change names and rules after you create the pipeline, from Configure on the target in the pipeline's Targets card. Names change only while the pipeline is paused. Saving changed rules offers Save and reload…, because it reloads the whole pipeline. See Multiple targets.

Leave the step

Next is enabled when at least one table is selected and the names and rules in Configure are valid. Otherwise the red box says Add at least one table to replicate. or names the problem in Configure. A source monitor also needs a table list, so that capture knows what to watch.

When the target is Fabric Mirror, leaving this step tests the target again with your tables, because that test also checks who owns each table folder. A refusal keeps you here with the reason in the red box.

See also

Previous
Relay

If this was useful,
our newsletter will be too.

Get monthly insights on data engineering, AI, and building critical infrastructure - direct from the Spectral Core team and CEO Damir Bulic.