Skip to content

Sizing the controller ​

The controller's memory grows with the Services and EndpointSlices that its cache holds. By default, that is every one in the cluster.

Measured ​

TestCacheMemory in cmd/strait-controller/cache_test.go starts the controller's cache on envtest's API server, and measures its live heap. Each Service has four labels, two annotations and two ports. Each has one EndpointSlice with three ready endpoints, each with a pod reference, a node and a zone. Each object has one managedFields entry. The 10,000 Services are spread over 20 namespaces.

CacheServices cachedLive heapPer Service and its EndpointSlice
1.2.x10,00067 MiB7,056 bytes
1.3.0, which leaves out managedFields10,00053 MiB5,621 bytes
1.3.0 with controller.namespacedReads, one namespace of the 205003 MiB346 bytes per Service of the cluster

With no Service, the controller's live heap was 2 MiB more than the test's own. An object with more labels, endpoints or managedFields entries takes more.

sh
STRAIT_CACHE_MEMORY=10000 make test

Setting the limit ​

ValueEffect
controller.resources.limits.memoryThe container's limit, 256Mi by default. The chart passes it to the controller, which sets the Go runtime's soft memory limit to 90% of it. Near that limit, the garbage collector runs more often, instead of letting the heap grow to twice what it holds.
controller.resources.requests.memory64Mi by default. Raise it together with the limit, so that the scheduler finds the room.

Leave twice the measured heap, and a margin of 128Mi for the binary, the Go runtime and the controller's other objects:

limits.memory >= 128Mi + 2 x 5.6 KiB x (Services of the cluster)
Services in the clusterlimits.memory
up to 10,000256Mi, the default
30,000512Mi
80,0001Gi

A controller that runs out of memory restarts. Until the next one serves, the other sites count this site's members Unknown (RFC0004). The alert StraitNoController fires at this site, and StraitPeerUnreachable at the others (Metrics and alerts).

Fewer objects in the cache ​

With controller.namespaces set, controller.namespacedReads: true limits the controller's reads of Services, EndpointSlices and CloudNativePG's objects to those namespaces and the release namespace (RFC0007 requirement 26). The cache then holds only theirs, and the chart grants those reads in those namespaces only.

Object in another namespaceWith namespacedReads
SiteServiceIts local members are Unknown with the read's error, it routes nowhere, and its Applied condition is False and names the namespace. With controller.namespaces alone, the controller cannot write its Service there either.
ExportNo ready endpoint: its Ready condition is False, reason ReadFailed, and peers see it not ready.
Import with spec.portIts Service condition is False, reason SyncFailed, and names the namespace.
Import without spec.portWorks as before.

Released under the Apache License 2.0.