Skip to content

Troubleshooting ​

Each site decides its own routes and keeps its own view. Read each site with its own --context, and compare the sites.

A route to nowhere is a safe state, not a fault to work around. Find its cause and fix that. Strait follows. Do not edit Strait's resources to force a route.

Start here ​

StepCommandLook at
1strait --context site-a status -ANames whose ROUTE is nowhere, and their REASON. Peers whose REACHABLE is false.
2strait --context site-a explain pg -n appReason and Meaning, the members table, the declarations and the conditions.
3strait --context site-a peersWhen a member's STATE is Unknown.
4strait --context site-a linksWhen a peer is not reachable.

strait explain prints the meaning of the route's reason. The skill document, strait --skill, lists every reason with its meaning. The rules that pick a reason: RFC0002, Decision.

By route reason ​

ReasonLook atCause and next step
Active, PreferDeclared or Eligible, but connections failEndpoints:, and the Routed and Applied conditionsApplied is False with SyncFailed: a write of the route failed, and its message gives the error. A message that says the namespace is not one where the controller writes: add the namespace to the chart value controller.namespaces, then helm upgrade (RFC0007). Members that the route dropped are gone, and the next try comes within 2s. Routed is False with NoEndpoints: the routed member has no ready endpoint. With ServiceConflict: a Service with the SiteService's name exists, and Strait does not own it. Strait deletes its EndpointSlices for that name and writes none until that Service is gone. For a routed member whose PATH is Gateway, check strait links.
HeldUnknownThe REASON of the Unknown memberThe route is kept, since the member was confirmed less than policy.unknownHold ago. See Unknown members.
UnknownTimeout, MemberUnknownThe REASON of the Unknown memberSee Unknown members.
TransitioningThe REASON of the member whose STATE is TransitioningThe workload is moving. The route comes back once the move ends. The cnpg reasons: RFC0003, role and state. A routine configuration change and the restart after a promotion also count: RFC0003, Limitations.
DeclarationsDisagreeDeclarations: at each site, and the route's MessageThe sites name different primaries, or the Clusters of one site do: the message then names that site, and its Declarations: line is empty. For cnpg, compare .spec.replica.primary of every Cluster. For declared, compare spec.plugin.declared.active at each site. Make them agree.
NotDeclaredDeclarations: and the members tableOne member is writable, but a site names another member. Compare each site's declaration with where the primary runs, and fix the workload's configuration. A declaration that is no member's name sets the condition DeclarationUnmatched, whose message names the site and the name. With cnpg, a Cluster sets .spec.replica.self to a name other than its own metadata.name, or .spec.replica.primary names no member's Cluster (RFC0003, requirement 7).
MultipleWritableThe members whose ROLE is primary and ELIGIBLE is trueTwo members accept writes, for example after a partition heals. Strait never picks one. Demote one with the workload's own tools, and Strait follows. The Multiple condition is True.
NoEligibleMemberThe REASON of each memberNo member can take traffic. See Members that are not eligible.
SiteIsolatedstrait peers, and the route's Messagepolicy.isolationTimeout is set, and this site has read no peer for that long. The local member is not routed. It ends when a peer is read again. After a site loss with two sites, the survivor stays isolated: set isolationTimeout to 0s there once its member is the one to write to (Site loss).
AwaitingPeersstrait peers, and the route's MessageThe controller has started, and its first poll of every peer has not ended yet. The route is what it decided before reading any peer, and the message gives that decision's reason. It ends with that first poll, within about one pollInterval. With policy.startupHold, the controller also does not route to the local member until it has read every site with a member, or until the hold ends.

Connections after a route change ​

SymptomCause and next step
A client still reaches the previous member, while the SiteService's EndpointSlices no longer name itA route change never moves an established connection: the client must reconnect. A new connection that reaches the previous member comes from the CNI's connection tracking. For Cilium, check that cilium-dbg status --verbose in the agent shows Socket LB: Enabled, and that the agent's configuration does not set bpf-lb-sock-hostns-only to "true". RFC0002, Route changes and connections.
A session through the gateway stays open after its member left the route, with policy.connections: EndGatewayEnvoy closes it when its drain time has passed: 600s unless the chart value envoy.drainTimeSeconds sets it. Set it to a few seconds. Check that the member's Import shows spec.closed: true: kubectl get import <siteservice>-<member> -o jsonpath='{.spec.closed}'. A local client's session, or one over endpoint-direct, always stays (RFC0002, What EndGateway covers and costs).

Unknown members ​

The REASON column of the members table, for a member whose STATE is Unknown:

ReasonMeansNext step
SiteUnknownThe member's site is neither this site nor a reachable peer.strait peers. A SitePeer must exist whose name is the member's site. Then see Unreachable peers.
NotObservedThe member's site is reachable, but its document holds no observation of this member.Apply the same SiteService, with the same namespace, name and member names, at the member's site.
ObservationStaleThe member's site is reachable, but its controller last observed the member more than that site's peerTimeout ago, and at least 6s ago. Its reconciles fail or hang (RFC0004).The controller's logs at the member's site, and its API server.
ReadFailed: ...The plugin could not read the workload at this site. The rest of the reason is the error.Follow the error. For cnpg, check that CloudNativePG is installed at this site.
RoleMismatch: ...spec.plugin.cnpg.confirmRole is on, and the database at the -rw address named in the reason gives a pg_is_in_recovery() answer that contradicts the Cluster's role. CloudNativePG's status and the database disagree.Compare kubectl get cluster <name> -o yaml (.status.currentPrimary, .spec.replica) with the database. Fix the workload with its own tools. Strait routes nowhere rather than pick one (RFC0003).
RoleNotConfirmed: ...confirmRole is on, and the check got no answer. The rest of the reason is the error: no credentials, a refused login, a timeout, or no ready address.Set the chart value cnpg.confirmRole.secretName to a Secret with username and password of a role with LOGIN, at this site. Check that the role exists, may log in to postgres over TLS, and that the controller's pod reaches the database's pods on 5432.
PluginDisabledThe member's plugin is off at this site: the chart value cnpg.enabled is false.Set cnpg.enabled: true with helm upgrade at the member's site.

While a peer is unreachable, every member at that peer counts as Unknown (RFC0004). A routed member stays in the route for policy.unknownHold after it was last confirmed, then the name routes nowhere.

A peer is unreachable while its controller restarts, as in an upgrade. Upgrade one site at a time: Upgrade.

Members that are not eligible ​

ReasonMeansNext step
ClusterNotFoundNo CloudNativePG Cluster named members[].cnpg.cluster in the SiteService's namespace.The name, and the namespace: a member's Cluster and its SiteService share one.
ServiceNotFoundThe member's endpoint Service does not exist. For cnpg, the Cluster's Service for the role (RFC0003). For declared, members[].service.name.The workload's Services.
NoReadyEndpointThe endpoint Service has no ready address.The workload's pods.
PortNotFoundThe member's endpoint Service has no port of the number it needs. For cnpg, port 5432 on the Cluster's Service or the Pooler's. For declared, members[].service.port.The Service's ports.
PoolerNotFound, PoolerOfAnotherCluster, PoolerTypeMismatch, PoolerPausedThe Pooler named in members[].cnpg.pooler cannot be used.The Pooler must exist, belong to the member's Cluster, match the role, and not be paused.
ReplicaClusterThe Cluster is a replica cluster, so it is not writable.None. Expected for every member but the primary.
PrimaryChanging, Demoting, Promoting, NotYetWritable, NotYetReadOnly, Phase: ...CloudNativePG is moving the primary. The STATE is Transitioning.Wait for the move to end. RFC0003, role and state.

An eligible member shows PrimaryCluster, Ready or Declared.

Read both ends: a link needs the gateway at each site.

StepCommandLook at
1strait --context site-a peersREACHABLE false, and LAST ERROR.
2strait --context site-a linksWhether any link to that peer is HEALTHY.
3kubectl --context site-a get sitelink b-wan -o yamlThe reason and message of the Healthy condition.
4kubectl --context site-a get site a -o yamlThe GatewayActive, GatewayServices and GatewayConfig conditions.
SymptomCause and next step
LAST ERROR holds site mismatchThe peer's document names another site. Name the SitePeer after the peer's site.name.
A healthy link, but the peer is not reachableThe peer's controller does not serve its document. Check the peer: strait --context site-b status, and its controller's logs.
No healthy link to the peerSee the Healthy condition below.

The Healthy condition of a SiteLink:

ReasonMeansNext step
NotConfiguredThe link's WireGuard interface is not in place. The message says why.The site's private key: the Secret is missing, or holds no valid key. The SitePeer: missing, or no valid public key. A listen port taken by another link. Or the kernel's error when it creates the interface, for example on a node without the wireguard module. A message that starts with filter the WireGuard interfaces' traffic: the agent could not write its nftables table, for example on a node without nf_tables, and holds no interface until it can (RFC0007, requirement 12).
HealthCheckFailedThe peer's control port does not answer GET /healthz over this link.For wireguard: strait links -o yaml shows lastHandshakeTime. With no handshake, check that each SitePeer holds the other site's current public key, and that the waiting side's listen port is reachable over UDP. On the side with an endpoint, the gateway agent logs the WireGuard peer sent nothing when the peer stops answering, then drops the session each second until it answers, which clears lastHandshakeTime (RFC0005, requirement 37). For direct: the link's direct.address, and how the peer exposes directService. With a NodePort and externalTrafficPolicy: Local, the address must be the node of the peer's active gateway: RFC0005, Limitations.
NotInEnvoyEnvoy has no endpoint at this link's address yet.Brief after a change. If it stays, the controller's logs.
EndpointUnhealthyEnvoy marks the endpoint unhealthy.The controller's and the gateway agent's logs.
EnvoyUnreachableThe gateway agent cannot read Envoy's admin interface.The envoy container of the active gateway pod.

A direct SiteLink with endpointDirect also has the condition EndpointsLimited. When it is False, with NoEndpointCIDRs, endpoint-direct writes any member address from the peer's document, except unspecified, loopback, link-local and multicast addresses. Set endpointCIDRs to the peer's pod CIDRs (RFC0005, Endpoint-direct).

The conditions of the Site:

ConditionMeansNext step
GatewayActive False, NoActiveGatewayNo ready gateway pod holds the gateway lease.kubectl --context site-a -n strait-system get pods,events. The API server rejects the gateway pods unless the namespace's Pod Security level is privileged.
GatewayServices False, DirectServiceFailed or WireGuardServiceFailedThe controller could not write a gateway Service. The message holds the API server's error.For example, a WireGuard listen port outside the cluster's node port range, with a NodePort or LoadBalancer Service. Or a node port that another Service already holds: find it with kubectl --context site-a get services -A, and move that Service to another port (RFC0005, Limitations).
GatewayConfig False, RejectedA gateway pod's Envoy rejected its configuration, and keeps the one it accepted last. New exports, imports and links may be missing from it. The message names the pod, the type and Envoy's error.The controller's log, a gateway Envoy rejected its configuration. The condition clears once the pod accepts a later configuration. A rejection is a defect in Strait: report it with the error.
GatewayActive moves to another pod while nothing failedThe active gateway replica could not renew the gateway Lease within spec.gateway.lease.renewDeadline, 4s by default, for example while the API server stalled. Every connection across the gateway resets.The API server's availability. A longer renewDeadline and duration ride out longer stalls, and keep a failed replica active for longer (RFC0005, Timing of the gateway Lease).

Logs, events and metrics ​

SourceCommand or place
Versionsstrait version for the CLI. VERSION in strait status for the site's controller.
Eventskubectl --context site-a -n app get events --field-selector involvedObject.name=pg. Each route change is a RouteChanged event. MultipleWritable, DeclarationsDisagree, DeclarationUnmatched and UnknownTimeout are Warning events.
Controller logskubectl --context site-a -n strait-system logs deployment/strait-controller. An Import whose Service is not written logs Import Service not synced when its error is new. controller.logLevel: debug logs each retry, every 2s.
CloudNativePG watchThe controller logs cloudnative-pg at start. enabled follows cnpg.enabled. watched false with enabled true means CloudNativePG's CRDs were missing then: restart the controller. Getting started, Prerequisites.
Gateway agent logskubectl --context site-a -n strait-system logs POD -c agent, where POD is the active gateway pod that strait status shows. Each link that fails, comes back, or starts or stops carrying traffic logs SiteLink status. gateway.logLevel: debug logs every write of a link's status.
Gateway handoverskubectl --context site-a get events -A --field-selector reason=GatewayHandover. Each one resets every connection across the site's gateway.
MetricsPort metrics, 8080, of the controller pod, and 8091 of each gateway pod. Every metric, the scrape configuration and the alert rules: Metrics and alerts. A route to nowhere shows as strait_siteservice_endpoints 0.

Runbooks ​

CaseRunbook
A site is lost, and a member at another site must become primary.Site loss
A lost site comes back, or a partition heals, maybe with two primaries. startupHold for production.Site return
A site joins the others, or leaves them.Add or remove a site
A site goes into production: socket load balancing, clients that write, gateway node ports, endpointCIDRs.Before production
A site's WireGuard private key must change.WireGuard key rotation
The controller or a gateway pod does not become ready, or restarts.Not ready
The active gateway moves to another replica, or must move off a node.Gateway handover
An upgrade, or a node drain.Upgrade
A site must go back to an earlier 1.x release.Roll back within 1.x
The operator publishes a release, or the release workflow refuses one.Release
Someone checks that a release's CLI archive or image is what its tag builds.Rebuild a release

Released under the Apache License 2.0.