High availability
Automatic restart
Learn how the agent restarts a faulted pipeline, which faults it restarts, when it gives up, and how to resume a parked pipeline.
When a pipeline faults, its agent decides whether a restart can clear the fault. If it can, the agent restarts the pipeline on a growing delay, from its last committed position. If the fault keeps coming back, or can't be cleared by a restart, the pipeline waits for you.
There is nothing to configure: every pipeline uses the same schedule.
How a restart works
An automatic restart is the same as your own Pause and Start: the pipeline resumes from its last committed position, so nothing is reloaded.
One exception: if you started a Reload and it faulted before the pipeline reached streaming, the agent runs the same reload again, because there is no committed position to resume from yet.
Restart schedule
The first restart comes a few seconds after the fault, and each later one waits about twice as long, never more than 5 minutes. Together, five restarts ride out an outage of about two minutes, such as a target restarting, an Azure SQL database waking from auto-pause, or a failover.
A pipeline that runs for 5 minutes without a fault after a restart counts as recovered.
Fault classes
| Class | What the agent does | Fault codes |
|---|---|---|
| Transient | Restarts the pipeline on the schedule. | connection-lost, timeout, throttled, deadlock, broker-not-ready, broker-unavailable, target-unavailable, azure-paused, resource-exhausted |
| Terminal | Doesn't restart. The pipeline waits for you. | named-refusal, corruption, retention-gap, schema-refusal, auth-failure, poison, agent-memory, unrecognized-log-record |
| Unclassified | Restarts the pipeline on the same schedule, and names the fault as unclassified. | unknown |
Some examples:
- Transient: a dropped connection, a deadlock, Azure SQL throttling or resource governance, a database that is offline or unavailable, a full transaction log (cleared by a log backup or growth), a full spool volume.
- Terminal: a failed login or a missing permission, a missing table or column, a corrupted page or checksum, a source log that was truncated before SQL Flow read it, a start the agent's memory can't hold, a SQL Server log record that DirectLog doesn't recognize.
A failed login on a target that the pipeline has already written to counts as a transient target-unavailable fault, because SQL Server reports an offline database the same way.
Circuit breaker
The agent allows at most 5 automatic restarts in any 30 minutes. If the pipeline faults again after the fifth, the agent parks it: it stops restarting it and leaves it where it faulted, for an operator.
What you see
The Auto-restart card on the pipeline's Overview tab appears once the pipeline has faulted since you last started it. It shows the state, the next attempt, the restarts used, and the first and the last fault, each with its class, code, time and message.
| State | Label | Meaning |
|---|---|---|
backing-off | Restart scheduled | A restart is scheduled. The card counts down to it. |
restarting | Restarting | The agent is stopping and starting the pipeline. |
recovering | Recovering | The pipeline restarted and runs, but hasn't run 5 minutes without a fault yet. |
recovered | Recovered automatically | The pipeline ran 5 minutes without a fault after its restarts. |
parked | Failed: needs operator | The breaker tripped. Nothing restarts the pipeline. |
terminal | Failed: needs operator | The fault can't be cleared by a restart. Nothing restarts the pipeline. |
Above the tabs, a banner says the same thing while the pipeline runs:
- Pipeline failed, restarting automatically: with the fault and the attempt, in amber.
- Failed, needs operator: with the reason, in red. For a parked pipeline, it says how many restarts were used in 30 minutes and which fault came back.
When a later fault follows the first one, both banner and card show the First fault above the latest, because a later fault is often a consequence of the first.
Target outages
When the target stops accepting writes, the banner names the outage instead: Target unreachable, Target throttling writes or Target permission denied, with the time it started. Changes wait in the spool and the lag grows until the target accepts writes again; nothing is lost.
Resume a parked or failed pipeline
- Read the first and the last fault on the Auto-restart card.
- Fix the cause: bring the source, target or log broker back, restore the permission, or follow the refusal's own instructions.
- Click Pause, then Start.
A new start resets the restart count and the breaker.
A start refused because the agent doesn't have enough memory is never retried automatically. The banner reads Not started, not enough agent memory: and names what limits it. Change that, then Pause and Start.
An unrecognized log record
When SQL Server DirectLog meets a log record it doesn't recognize, the pipeline stops immediately before that record, with the fault unrecognized-log-record. No change is skipped or lost, but a restart would stop at the same record, so the agent doesn't restart it. While it's stopped, the pipeline keeps holding the source log; Pause releases the hold.
- Click Download diagnostics on the banner, or choose Actions > Download diagnostics, and send the file to SQL Flow support. See Diagnostics for support.
- To continue without waiting for a fix, choose Actions > Reload from source, which starts the pipeline over from a fresh copy of the source. See Lifecycle.
