Skip to content

Site return ​

A site comes back after a site loss, or a partition between sites heals. Its workload may still run a primary that the other sites have replaced. Until it is demoted, that primary takes the writes of the returning site's clients wherever Strait routes them to it.

Ahead of time: startupHold ​

Set spec.policy.startupHold on every Single mode SiteService, at every site, in production. A site's controller starts when the site comes back, so the hold must be in the spec before the loss.

ItemValue
Fieldspec.policy.startupHold, 0s to 10m, Single mode only. Off unless set.
EffectAfter the controller starts, it does not route to a local member until it has read every other site with a member, or until the hold ends. The route is nowhere meanwhile, with reason AwaitingPeers (RFC0002, Startup hold).
Without itThe returning site routes its clients to its old primary from its controller's first decision until it reads the other sites. In a two-site test, that lasted about 20 seconds (RFC0002, Motivation).
Suggested value30s, as in the end-to-end test. The hold ends once every other site with a member has been read, so it costs time only when a controller starts while cut off from those sites.
CostA controller that starts during a real partition, for example after an upgrade or a crash, keeps its local primary unrouted for up to startupHold.
Not coveredA controller that kept running through the loss, such as on a site that was only cut off, or whose nodes were frozen and resumed. It is not held (RFC0002, Limitations): A partition that heals.

Set it at each site, then check that the controller reads it:

sh
kubectl --context site-a -n app patch siteservice pg --type merge -p '{"spec":{"policy":{"startupHold":"30s"}}}' --dry-run=server
kubectl --context site-a -n app patch siteservice pg --type merge -p '{"spec":{"policy":{"startupHold":"30s"}}}'
strait --context site-a explain pg -n app

strait explain shows startupHold 30s on the Policy: line.

Steps ​

StepWhat to doCheck
1. Watch the returning siteAs soon as its API answers: strait --context site-b status -A.With startupHold, a name whose primary is local routes nowhere with AwaitingPeers until the site has read the other sites, or until the hold ends.
2. Demote the old primaryIf the returning site's workload still runs a primary that another site has replaced, demote it with the workload's own procedure, before the hold ends. For CloudNativePG in the distributed topology: set .spec.replica.primary of the returning site's Cluster to the new primary's name (CloudNativePG, replica clusters).strait --context site-b explain pg -n app: the returning site's member has ROLE replica.
3. Check every sitestrait --context SITE status -A at each site.Every peer REACHABLE. Every name routes to the new primary, reason Active.

What the sites show until step 2 is done:

CaseRouteReason, conditions
The returning site reads the other sites.Nowhere, at every site that reads both primaries.DeclarationsDisagree when the sites declare different primaries, else MultipleWritable. The Multiple condition is True, and a Warning event fires. Strait never picks one primary.
The returning site cannot read the other sites, and its hold has ended or is not set.Its own primary, at the returning site.Active. Its clients write to the old primary.

Step 2 for a returning site b, where pg-a at site a is the new primary:

sh
kubectl --context site-b -n app patch cluster.postgresql.cnpg.io pg-b --type merge \
  -p '{"spec":{"replica":{"primary":"pg-a"}}}' --dry-run=server
kubectl --context site-b -n app patch cluster.postgresql.cnpg.io pg-b --type merge \
  -p '{"spec":{"replica":{"primary":"pg-a"}}}'

The end-to-end step Two writable members runs step 1, and the two cases above, on two sites: test/e2e/README.md.

A partition that heals ​

A site that was only cut off kept its controller running, so no hold applies. During the partition, each side routes as in a site loss: a site with the primary keeps routing to it, and a site whose primary is on the other side holds it for policy.unknownHold, then routes nowhere (RFC0002, Why a partition does not stop the local primary). Once the sites read each other again:

CaseRoute at every siteReason, conditionsEnd-to-end step
No member was promoted during the partition.The one primary.ActivePartition: site a routes pg-b again within 15s of the heal. In 4 CI runs from v1.3.0 on, 1.6 to 2.9s.
A member was promoted at another site during the partition.Nowhere, until one primary is demoted.DeclarationsDisagree when the sites declare different primaries, else MultipleWritable. The Multiple condition is True, and a Warning event fires.Two writable members: both sites route nowhere, with Multiple True, within 30s of the heal. In 4 CI runs from v1.3.0 on, 2.1 to 3.7s.

In the second case, run steps 2 and 3 above: demote the primary that the other sites replaced, then check every site.

Released under the Apache License 2.0.