Skip to content

Metrics and alerts ​

Every metric that Strait exports, how to scrape it, and the alert rules that the chart ships. Each metric is read when it is scraped. No label holds an address or a port.

Scrape ​

PodPortPathServes
Controllermetrics, 8080/metricsThe controller's metrics. Only the elected controller decides, polls and serves xDS.
Each gateway podmetrics, 8091/metricsThe gateway agent's metrics.
Each gateway podmetrics, 8091/stats/prometheusEnvoy's stats that the agent lets through (Envoy).

Scrape every gateway pod, including the standbys. Their Envoy checks the links just as the active pod's Envoy does.

With the Prometheus Operator, the chart renders the scrape configuration and the rules. Both are off by default, since their CRDs may be absent:

ValueRenders
metrics.podMonitor.enabledThe PodMonitors strait-controller and strait-gateway. metrics.podMonitor.labels holds the labels that the Prometheus's podMonitorSelector requires.
metrics.prometheusRule.enabledThe PrometheusRule strait, with the rules of Alerts. metrics.prometheusRule.labels holds the labels that the Prometheus's ruleSelector requires.

Without the operator, a Prometheus scrape configuration:

yaml
scrape_configs:
  - job_name: strait
    kubernetes_sd_configs:
      - role: pod
        namespaces: {names: [strait-system]}
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name, __meta_kubernetes_pod_container_port_name]
        regex: strait;metrics
        action: keep
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
  - job_name: strait-envoy
    metrics_path: /stats/prometheus
    kubernetes_sd_configs:
      - role: pod
        namespaces: {names: [strait-system]}
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component, __meta_kubernetes_pod_container_port_name]
        regex: gateway;metrics
        action: keep
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

The metrics are plain HTTP. networkPolicy.metricsFrom limits the sources that reach the controller's port 8080 and, with networkPolicy.gateway.enabled, the gateway's port 8091 (RFC0007).

Controller ​

MetricTypeLabelsValue
strait_siteservice_endpointsgaugenamespace, siteserviceEndpoints written for the application name. 0 is a route to nowhere.
strait_siteservice_route_reasongaugenamespace, siteservice, reason1 for the reason of the current route.
strait_siteservice_conditiongaugenamespace, siteservice, condition1 while the condition is True.
strait_siteservice_last_decision_timestamp_secondsgaugenamespace, siteserviceWhen the elected controller last decided the route, in Unix seconds. It decides each SiteService at least every 2s.
strait_peer_reachablegaugepeer1 while the peer's document is fresh (RFC0004).
strait_peer_last_seen_timestamp_secondsgaugepeerWhen the peer's document was last read, in Unix seconds, from SitePeer.status.lastSeenTime.
strait_link_healthygaugelink, peer, transport1 while the link is healthy, from SiteLink.status.
strait_link_activegaugelink, peer, transport1 while the link carries the pair's traffic, from SiteLink.status.
strait_site_conditiongaugecondition1 while the condition of the local Site is True: GatewayActive, GatewayServices, GatewayConfig.
strait_gateway_lease_transitions_totalcounterHow many times a gateway replica took the gateway Lease, from another holder or after a release: the Lease's leaseTransitions. Each move resets every connection across the site's gateway.
strait_xds_rejected_totalcountertypeConfigurations that a gateway Envoy rejected, by resource type, such as Cluster (RFC0005).
strait_status_read_failedgaugekind1 when the scrape could not read the objects of the kind. Their metrics are then missing, not 0.
leader_election_master_statusgaugename1 while this controller holds the Lease name="strait-controller". From controller-runtime.

controller-runtime also exports its own metrics, such as controller_runtime_reconcile_total, workqueue_depth and rest_client_requests_total, and those of Go and the process.

Gateway agent ​

MetricTypeLabelsValue
strait_gateway_leadinggauge1 while this replica holds the gateway Lease and carries the site's traffic.
strait_gateway_link_connectionsgaugelink, peerOpen connections of the imports to the peer over the link, from this replica's Envoy.
strait_gateway_wireguard_receive_bytes_totalcounterlink, peerBytes received on the link's WireGuard interface since it was created. Active replica only.
strait_gateway_wireguard_transmit_bytes_totalcounterlink, peerBytes sent on the link's WireGuard interface since it was created. Active replica only.

The agent also exports the metrics of Go and the process.

Envoy ​

At /stats/prometheus, the agent serves these of Envoy's stats, and no other path of Envoy's admin interface. Envoy names each one envoy_ followed by the stat, with each . replaced by _.

StatsLabelsHolds
cluster.*.upstream_cx_{total,active,connect_fail,connect_timeout,none_healthy,rx_bytes_total,tx_bytes_total}, membership_{healthy,total}, health_check.{success,failure,network_failure}, update_rejectedenvoy_cluster_namePer cluster: each export and import, each peer, the controller. Connections, bytes, connect failures, healthy endpoints.
tcp.*.downstream_cx_{total,rx_bytes_total,tx_bytes_total,no_route}envoy_tcp_prefixPer export and import: connections and bytes from clients.
http.{control,peers}.downstream_rq_{total,1xx,..,5xx}envoy_http_conn_manager_prefix, envoy_response_code_classRequests to the control port and to the peer listener.
server.{live,state,uptime,memory_allocated,memory_heap_size,total_connections}Envoy's state.
control_plane.connected_state1 while Envoy holds its xDS stream to the controller.
cluster_manager.{active_clusters,cds.update_success,cds.update_rejected}, listener_manager.{total_listeners_active,lds.update_success,lds.update_rejected}Configuration updates.
Name in the labelsMeans
export/<namespace>/<name>, import/<namespace>/<name>An Export or an Import. A . in the name shows as _: Envoy's labels end a name at its first ..
peer/<peer>The control port of a peer, over each link.
controllerThe controller's observation port.

Per-link bytes exist for wireguard links only, from the interface. A direct link has no interface, and Envoy's stats per endpoint are labelled by address.

Alerts ​

The rules of deploy/helm/strait/files/alerts.yaml. They suit one Prometheus per site, since absent() and max() cover every series in that Prometheus.

AlertFires whenNext step
StraitNoControllerNo controller has led for 2m. Routes are frozen, and the peers count this site's members Unknown.kubectl -n strait-system get pods,events. Troubleshooting.
StraitPeerUnreachableA peer's document has been stale for 2m.strait peers. Troubleshooting.
StraitNoActiveLinkNo link to a peer has been healthy for 2m.strait links.
StraitRouteNowhereAn application name has routed nowhere for 2m.strait explain.
StraitNotAppliedThe controller has not applied a route for 2m.The Applied condition's message.
StraitNoActiveGatewayNo ready gateway pod has held the gateway Lease for 1m.Troubleshooting, the Site's conditions.
StraitGatewayConfigRejectedA gateway Envoy has rejected its configuration for 5m.The GatewayConfig condition.
StraitDecisionsStalledThe controller has decided no route of an application name for over a minute.The controller's log, and the API server's latency.
StraitStatusUnreadableThe controller's metrics have not read a kind for 5m.The controller's log.
StraitGatewayHandoversThe gateway Lease moved more than 3 times in an hour.The GatewayHandover events of the Site, and the gateway pods' restarts.
StraitGatewayNotConfiguredA gateway pod has had no xDS stream to the controller for 5m.The controller's Service, and the network policy on it.

The for of each rule is longer than what Strait rides out on its own: a link failover, a gateway handover, a hold of an Unknown member.

Probes ​

Liveness checks only the work that waits for neither the API server nor a link. So a slow API server, or a link that is down, restarts nothing. Work that waits for them shows in the metrics above instead.

ContainerLiveness fails when
ControllerThe elected controller's poll round, or its xDS rebuild loop, has not begun for twice its period plus 30s: 34s with the default pollInterval of 2s.
Gateway agentThe active replica's WireGuard session loop has not turned for 30s. It reads the interfaces only.
EnvoyNo probe: its admin interface is on 127.0.0.1 only. Envoy kills itself when a thread has not answered its watchdog for 60s, and the kubelet restarts it.

Released under the Apache License 2.0.