Limitations
What Strait doesn't do
These are choices, set as requirements in RFC0001:
| Strait doesn't | Who does |
|---|---|
| Fail over, switch over, promote, demote, pause or fence a workload. | The workload's own controller, or the operator. Strait follows the move. |
| Parse an application protocol. It carries TCP only. | The application. |
Route pod-to-pod traffic between clusters, beyond an application name or an Import. | A CNI. |
| Route through a third site. | Nobody: traffic between two sites uses a link between those two sites. |
Known limitations
Each RFC lists the limitations of its subject: what to do about each one, and when to revisit it. They are shown below as the RFCs stood when this site was built:
- RFC0002, Limitations
- RFC0003, Limitations
- RFC0004, Limitations
- RFC0005, Limitations
- RFC0007, Limitations
RFC0002: SiteService, Modes and Policies
| Limitation | What to do | Revisit when |
|---|---|---|
A second primary hidden behind a partition is not detected until the partition heals. isolationTimeout stops only a site that sees no peer: two sites that each still see a third one both keep routing their own primary. |
With three or more sites, set isolationTimeout. Before promoting elsewhere, stop the cut-off site's database and clients (Site loss). |
None. Detecting it needs a quorum, which is the workload's job. |
| The startup hold and the status at startup begin at the controller's first decision. Until then, the EndpointSlices and the status that the previous controller wrote stay in place. A site that comes back serves its old route, and shows its old status, until the new controller decides. That takes the time the controller pod needs to start. When the previous controller did not release the leader election's lease, as after a crash, it also takes the lease's 15s ([RFC#0004]). | Keep a returning site's clients or database stopped until its controller has decided (Site return). | A site that must not serve at all between its return and the controller's first decision, or a status reader that must tell an old status from a new one. |
The startup hold covers a controller that starts. A controller that was frozen with its node and resumes keeps its memory of the peers it read, so it is not held. It reads them again within one pollInterval. |
After a long freeze of the controller's node, delete the controller pod, so that the new one holds. | A platform that freezes nodes for long, such as live migration. |
A route change does not stop new connections to the previous member while the CNI's connection tracking still names it and it is a backend of any Service. Strait owns no CNI state ([RFC#0001]). EndGateway stops this only for a member reached through the gateway. |
Use socket load balancing that covers pods, and clients that refuse a read-only server (Before production). | A CNI that offers a supported way to drop the tracking of one Service backend. |
A remote site learns of a demotion through the exchange, so for up to peerTimeout ([RFC#0004]) it may still route to a member that its own site has just demoted. Writes in that window reach a read-only database and fail. |
Clients retry a failed write. A shorter peerTimeout shortens the window. |
A workload whose clients cannot retry a failed write. |
After a controller start, a routed member with no EndpointSlice, as when it had no endpoint, is not in the previous route. It is not held while Unknown, and routes nowhere until it is confirmed again. |
Nothing. The route is nowhere, never a guess, until the member is confirmed again. | A hold that must survive a restart for a member with no endpoint. |
The status refreshes member timestamps at most every 10s while nothing else changes. After a controller restart, a hold starts from the last written time, so it can end up to 10s early. The previous controller wrote that time by its own node's clock. If that clock is behind the new controller's, the hold ends early by the difference. A time ahead of the new controller's clock counts as the restart's time (requirement 17), so the hold then lasts at most unknownHold after the restart. The early end routes nowhere, never to a guess. |
Keep unknownHold well above 10s, and the nodes' clocks in sync. |
Holds shorter than 10s that must survive a restart exactly. |
RFC0003: Plugin Contract and v1 Plugins
| Limitation | What to do | Revisit when |
|---|---|---|
A routine configuration change also shows Applying configuration, so it routes nowhere for as long as CloudNativePG applies it. |
Change a Cluster's configuration when refused connections are acceptable. |
A user whose configuration changes must not refuse connections. |
A site whose local Clusters name different primaries declares none to its peers, since the document holds one declaration per site ([RFC#0004]). Only its own decision routes nowhere for it. |
Make the Clusters agree. strait explain at that site names them. |
A document version that carries more than one declaration per site. |
A promotion restarts the new primary with a smart shutdown that waits up to smartShutdownTimeout, 180s by default, while sessions stay open. The Cluster reads Transitioning meanwhile, so Strait routes nowhere, even while a Pooler still takes writes. A short smartShutdownTimeout keeps the window short. |
Set a short smartShutdownTimeout (Failover time). |
A user who needs writes during that window. |
RFC0004: Observation Exchange
| Limitation | What to do | Revisit when |
|---|---|---|
| No signature on the document. Integrity rests on the link. | Use wireguard links, or direct links over a network that is trusted and encrypted below Strait. |
A transport without its own authentication. |
RFC0005: Data Plane
| Limitation | What to do | Revisit when |
|---|---|---|
| One gateway replica carries traffic at a time. | Size the gateway pod for the site's cross-site traffic (Benchmark). | A site whose cross-site traffic exceeds one replica. |
| A WireGuard interface needs the kernel module and nftables on the node. | Check the nodes that can run a gateway pod. A link whose node lacks them shows NotConfigured. |
A cluster whose nodes lack them. A userspace driver would then be added. |
A controller that starts counts the gateway Lease as valid for its full leaseDurationSeconds from when it first reads it (requirement 32), as client-go's leader election does. A holder whose agent stopped renewing just before then stays the active gateway for up to 6s more, while its pod is ready. Reading renewTime instead would compare the holder's clock with the controller's, which requirement 32 forbids. |
Nothing. A handover right after a controller start can take up to 6s longer. | A way to tell, at first sight, a Lease that is still renewed from one that is not, without the holder's clock. |
The default gateway ports that peers reach, the control port 30900, the export ports 30901-31899 and the WireGuard listen port 31820, are in the default node port range, from which Kubernetes gives a port to each Service that names none. Another Service can get a gateway port before the gateway Service names it. The API server then refuses the whole gateway Service, so it takes no new port, and the GatewayServices condition holds the API server's error. No API serves the cluster's node port range, so Strait cannot check the ports against it. The operator's checks: Before production. |
Before production, Gateway node ports. | A cluster where another Service takes a gateway port. |
With a NodePort direct Service and externalTrafficPolicy: Local, only the node of the active gateway serves the Service. A direct.address names one node, so the link is unhealthy while the active gateway runs on another node, and traffic takes the next link. |
Keep the gateway on the node that direct.address names, or rely on the next link. |
A direct link with more than one address. |
| A request that hangs ends at its timeout (requirement 49). A connection to the API server that stops answering altogether stays open until client-go's HTTP/2 health check closes it: after 30s with no frame from the server, and 15s more with no answer to a ping. Until then, each request of that replica on that connection fails at its timeout. Inferred from client-go's defaults, not tested. | Set HTTP2_READ_IDLE_TIMEOUT_SECONDS and HTTP2_PING_TIMEOUT_SECONDS in the gateway agent to find a dead connection sooner. |
A site whose connections to the API server stop answering while the API server still serves. The environment variables HTTP2_READ_IDLE_TIMEOUT_SECONDS and HTTP2_PING_TIMEOUT_SECONDS set the health check's timings. |
RFC0007: Security
| Limitation | What to do | Revisit when |
|---|---|---|
The metrics of the controller and of the gateway agent are plain HTTP. networkPolicy.metricsFrom limits who reaches them. They hold names of objects and their states, no address and no secret. |
Set networkPolicy.metricsFrom to the scrapers only. |
A cluster that requires metrics over TLS. The controller and the gateway agent would then need the right to create TokenReviews and SubjectAccessReviews. |
A CNI that does not support endPort admits only the first port of each range of the gateway's policy, so the pods of the cluster reach only the first import port and the first export port. |
Use a CNI that supports endPort, or leave networkPolicy.gateway off. |
A site whose CNI does not support endPort. |
| The xDS stream between the controller and Envoy is not encrypted. It stays inside the Strait namespace, and a NetworkPolicy admits only gateway pods. | Run a CNI that enforces NetworkPolicy. | A cluster without NetworkPolicy enforcement. |
Rotating the WireGuard key needs the peers' SitePeer updated at the same time. |
Follow WireGuard key rotation. | Sites that must rotate keys often. |
Without controller.namespacedReads, the controller reads every Service and EndpointSlice of the cluster, also with controller.namespaces. With it, an Export of another namespace has no ready endpoint. In both cases, the controller reads and writes Exports, Imports, their status and events in every namespace. |
Set controller.namespaces and controller.namespacedReads where the controller must read less (Sizing). |
A site whose Strait objects must stay in the namespaces of its SiteServices. |
RBAC cannot limit patch on pods to the gateway pods, or to one annotation. On envtest's API server, a patch changed a pod's image and labels. A compromised controller could use this to run code in a gateway pod, with the gateway agent's NET_ADMIN and WireGuard private key, or to change the labels that select the pods of its namespace. |
Limit who can act as the controller's service account, and watch the gateway pods' changes. | A site that needs a narrower right. A ValidatingAdmissionPolicy could admit from the controller a change of the deletion cost only. |
Without direct.endpointCIDRs, endpoint-direct accepts any address from the peer that is not special. A compromised peer, or anyone on the path of a direct link, can point a local name at another address that the site's pods reach. strait validate warns ([RFC#0006] requirement 18), and the link's condition EndpointsLimited is False ([RFC#0005] requirement 48). v1 accepts such a link, so refusing it would break objects that v1 stored. |
Set endpointCIDRs on each direct link with endpointDirect (Before production). |
The next API version: it requires endpointCIDRs with endpointDirect. |