Skip to content

Failover time ​

This page adds up Strait's share of the time until writes work again after a failover. It covers detection, holds, the workload's own move, and applying the new route. Each row gives the default and what to tune.

Strait never moves a primary. The workload's controller or the operator does that, and Strait follows (RFC0001). The time that the workload takes, such as a database promotion, is in the budget because Strait routes nowhere while it lasts.

Real timings come from a run on real clusters, not from CI or kind.

When the route became Active again ​

Read status.route.lastChangeTime together with status.route.reason:

sh
kubectl --context site-b -n app get siteservice pg \
  -o jsonpath='{.status.route.reason} {.status.route.members} {.status.route.lastChangeTime}{"\n"}'
FieldMeans
status.route.lastChangeTimeThe time of the decision whose status first showed the current members. It uses the controller's clock and is exact to the second. It does not change while the members stay the same, even while a member is held Unknown.
status.route.reasonActive once the one writable member is routed.
The Routed condition's lastTransitionTimeWhen the Service last went from no endpoint to an endpoint, or back.

After a failover during which the route was nowhere, a reason of Active, with a lastChangeTime after the failover began, gives the time when writes could reach the new primary again (RFC0002, requirement 24). Each site has its own SiteService, so read each site.

The parts ​

PartWhat decides itDefaultTune
A local member's change seenThe controller watches CloudNativePG's Clusters and decides again at once, and at least every 2s.Under 2sNothing.
A remote member's change seenEach site polls its peers every pollInterval. A change shows at the other sites within peerTimeout (RFC0004, requirement 13).pollInterval 2s, peerTimeout 6sSite.spec.exchange. peerTimeout must be at least twice pollInterval. Shorter values notice a change faster, and also turn a short flap into Unknown sooner.
A lost site seenIts peer's document goes stale after peerTimeout. Its members are then Unknown.6sSite.spec.exchange.peerTimeout.
A routed member that turned UnknownKept for unknownHold after it was last confirmed, then nowhere. It does not delay a promotion elsewhere: once another member is the one writable member, it is routed at once (RFC0002, rows 5 and 7).unknownHold 30sspec.policy.unknownHold, 0s to 10m. Shorter values stop routing to a dead member sooner. Longer ones ride out longer link flaps.
The workload's moveA CloudNativePG Cluster reads Transitioning while it switches over, promotes, demotes, or restarts its primary. Strait routes nowhere meanwhile (RFC0003).Depends on the workloadSee The promotion restart.
Declarations agreeingIn a switchover, the sites declare the new primary one after another. Until every known site does, the route is nowhere with DeclarationsDisagree (RFC0002).Up to peerTimeout after the last site changesChange every site's declaration at about the same time.
A returning site's controllerWith startupHold, a site that starts does not route to its own member until it has read every other site, or until the hold ends.Offspec.policy.startupHold.
A cut-off siteWith isolationTimeout, a site that has read no peer for that long stops routing to its own member. This limits lost writes at the old primary, and does not add to the new primary's time.Offspec.policy.isolationTimeout. Keep it below the time the other sites take to promote (RFC0002).
Applying the routeThe controller deletes old EndpointSlices, then writes the new ones, in one decision (RFC0002). The CNI then applies them.One API round trip per write, plus the CNINothing in Strait.
A remote member's pathThrough the gateway, the import must have a healthy link. A dead link fails over within 5s (RFC0005, requirement 4).Checks every 1s, 2 failuresNothing.
A gateway handoverA standby takes the gateway Lease within duration + 4.4 × retryPeriod of the holder's last renewal (RFC0005, requirement 50).6s + 4.4sSite.spec.gateway.lease.
Sessions to the old memberKept, unless policy.connections is EndGateway: then sessions through this site's gateway end after Envoy's drain time (RFC0002).Keptspec.policy.connections, and the chart value envoy.drainTimeSeconds.

The promotion restart ​

When CloudNativePG promotes a replica cluster, it restarts the new primary. The restart begins with a smart shutdown, which waits for open sessions to end, for up to smartShutdownTimeout. That is 180s by default. The Cluster reads Transitioning meanwhile, so Strait routes nowhere for as long as it lasts (RFC0003).

Set a short smartShutdownTimeout on every Cluster of the topology, for example 10s to 30s. A session still open then is cut when the timeout ends, and the restart goes on.

yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
spec:
  smartShutdownTimeout: 15

A planned switchover ​

From pg-a at site a to pg-b at site b:

StepRouteStrait's share
Site a is demoted, then declares pg-bNowhere at a at once. At b within peerTimeout.Up to peerTimeout until b sees it.
Site b is promotedNowhere while pg-b is Transitioning.None. This is the promotion and its restart.
pg-b is writablepg-b at b within one decision. At a within peerTimeout.Under 2s at b. Up to peerTimeout at a, plus the gateway import's link check.

A site loss ​

Site a, which held the primary, is lost:

StepRoute at site bStrait's share
Site a stops answeringpg-a is kept, HeldUnknown.Up to peerTimeout, then unknownHold after pg-a was last confirmed. Writes to pg-a fail anyway.
The operator promotes pg-bNowhere while pg-b is Transitioning.None. Detecting the loss and deciding to promote are the operator's or the workload's.
pg-b is writablepg-b, Active, within one decision. pg-a's hold does not delay it.Under 2s. With isolationTimeout and no other peer, the route stays nowhere until the operator sets it to 0s (Site loss).

Steps for the operator: Site loss.

Released under the Apache License 2.0.