Skip to content

Gateway benchmark ​

This benchmark measures what the gateway costs an application at site b that talks to a server at site a. It compares Strait's Envoy gateway with a plain HAProxy pair on the same direct link, and with endpoint-direct, which has no proxy. RFC0005 cites its results.

Paths ​

PathHops from the client pod at b to the server pod at a
Strait gateway (Envoy), direct linkbench.bench.svc:7000 (ClusterIP) -> gateway b, import listener -> node a's direct address, export NodePort -> gateway a, export listener -> server pod
HAProxy pair, direct linkhaproxy-b.bench.svc:7000 (ClusterIP) -> HAProxy b -> node a's direct address, NodePort 32100 -> HAProxy a -> server pod
Strait endpoint-direct, no proxybench.bench.svc:7000 (ClusterIP) -> server pod, routed over the direct link
Strait gateway (Envoy), WireGuard linkAs the first row, with the gateways joined by the wireguard link

The direct link has no tunnel, so the first three rows compare the proxies, not encryption. The fourth row shows what the tunnel adds.

Fixture ​

ItemValue
Siteskind clusters strait-bench-a and strait-bench-b, one node each, on one host.
NetworksDocker network strait-net carries each node's primary address and the wireguard link. Docker network strait-direct is the private link: node a at 172.31.0.12, node b at 172.31.0.22.
StraitThe chart in deploy/helm/strait, one gateway replica per site. Links per site: direct, cost 10, and wireguard, cost 20.
Application nameSiteService bench, plugin declared, active member srv-a at a. Member srv-b at b has no pod.
ServerOne pod at a. Its Service maps port 7000 to target port 7001, so endpoint-direct must carry the target port.
ClientOne pod at b.
Node routesEach node routes the other site's pod subnet over strait-direct, without masquerade. Endpoint-direct needs this. It stands in for a network that makes pod addresses routable between sites. Strait does not set it up.

Files: bench/. The runner is bench/run.sh. The client and server are one Go program, bench/tcpbench.

Measurements ​

Every message is 64 bytes. The server echoes each request.

MetricHow
Request/response latencyOne open connection, 1000 requests per second for 10 s after 1 s of warmup. p50, p90 and p99 of the round trip.
Connection setup500 new connections, one after another, after 20 unmeasured. Time from dial to the first echoed response. A bare connect would end at the local proxy, so the time includes one round trip through the whole path.
ThroughputThe server sends as fast as it can. Bytes the client receives over 10 s, after 1 s of warmup, on 1 and on 4 connections.
CPU per connection50 connections, each 100 requests per second, 5000 per second in all. Proxy container CPU from its cgroup's cpu.stat, over 16 s from second 4 of the load. Minus the same container's CPU over 16 s with no traffic, in the same round. Divided by 50. Envoy container only: the gateway agent is not on the data path. Endpoint-direct has no proxy and counts 0.
Latency under loadp50 and p99 of the round trips of those 50 connections.

Fair comparison ​

SettingEnvoyHAProxy
Imageenvoyproxy/envoy:distroless-v1.40.2, the chart'shaproxy:3.2.26-alpine
Worker threads2, the chart's --concurrency2, nbthread 2
CPU request, CPU limit1, none. The chart's default request is 50m.1, none
Access logsnonenone
Idle timeoutdisabled1 h
Upstream TCP keepalive30 s, every 5 s, 3 probes, at both sitesthe same, at b only. No run is long enough for a probe.
Buffersdefaults, 1 MiB per connectiondefaults, tune.bufsize 16 KiB, no splice
Health checksevery second, as Strait runs themnone
Reached at b througha ClusterIP Servicea ClusterIP Service
Reached at a througha NodePort on node a's direct addressa NodePort on node a's direct address
Upstream at athe server pod's address, from EDSthe server pod's address, from a headless Service

The client and the server also request 1 CPU with no limit. No pod is pinned to a core: the host is shared, and the benchmark cannot reserve cores on it.

Checks ​

Before and after each path, the runner checks that the traffic took that path. A round whose check fails is left out of the results and listed.

CheckPaths
The server sees the expected source: gateway a's pod, HAProxy a's pod, or the client pod.all
The EndpointSlice of bench at b holds gateway b and the import's port, or the server pod and port 7001.Strait paths
b's SiteLink for the path is healthy and active.Strait paths
A one-second transfer crosses gateway b's WireGuard interface, or does not.Strait paths
WireGuard bytes received during the path's measurements cover every streamed byte, or stay under 50 MB.all
The load held at least 98% of its target rate.all

Run order ​

ItemValue
Rounds5
Order in each roundgateway with direct link, HAProxy, endpoint-direct, gateway with WireGuard link
Between pathsThe runner patches b's links: spec.direct.endpointDirect on a-direct, and the cost of a-wireguard, 5 or 20. Then it waits for the checks.
Recorded before each roundnproc, load average, host CPU busy, docker stats, idle proxy CPU
Recorded before each load windowhost CPU busy, docker stats

Each cell of the results is the median over rounds, with the lowest and highest round in brackets. Differences against endpoint-direct, and between Envoy and HAProxy, are taken within each round before the median. That cancels most of the drift between rounds.

Environment ​

ItemValue
HostOne Linux host with Docker, 8 vCPU, shared with other workloads. Both sites, the client and the server run on it.
Linux7.0
Docker29.9.2
kindv0.33.0, node image kindest/node:v1.37.0
Kubernetesv1.37.0, kube-proxy in iptables mode, kindnet
Envoyv1.40.2
HAProxy3.2.26
Go1.27.3, for tcpbench

Results ​

make bench, five rounds, every check passed. Each cell is the median, with the lowest and highest round in brackets.

Latency, µs:

PathRequest/response p50p90p99Connection setup p50p90p99
Strait gateway (Envoy), direct link339 (299 to 341)404 (343 to 408)483 (407 to 489)841 (807 to 879)909 (857 to 1001)1040 (986 to 1127)
HAProxy pair, direct link270 (249 to 293)329 (287 to 350)387 (344 to 428)629 (609 to 663)689 (673 to 736)784 (736 to 863)
Strait endpoint-direct, no proxy112 (92 to 114)141 (113 to 143)174 (140 to 177)299 (274 to 317)351 (316 to 373)422 (358 to 478)
Strait gateway (Envoy), WireGuard link481 (458 to 489)616 (576 to 630)2298 (2187 to 2488)1141 (1100 to 1227)1467 (1352 to 1791)3401 (3173 to 3976)

Added over endpoint-direct in the same round, µs:

PathRequest/response p50Request/response p99Connection setup p50
Strait gateway (Envoy), direct link225 (206 to 229)306 (268 to 317)546 (529 to 571)
HAProxy pair, direct link156 (138 to 183)210 (175 to 256)344 (292 to 357)
Strait gateway (Envoy), WireGuard link366 (345 to 397)2124 (2047 to 2316)835 (783 to 928)

Throughput, CPU and latency under load. CPU is in millicores per connection, at 100 requests per second on each connection:

Path1 stream, Gbit/s4 streams, Gbit/sCPU at aCPU at bCPU, bothUnder load p50, µsUnder load p99, µs
Strait gateway (Envoy), direct link8.71 (7.21 to 10.55)11.62 (11.13 to 12.50)10.6 (9.9 to 10.9)10.7 (10.3 to 11.1)21.3 (20.2 to 22.0)332 (309 to 354)566 (467 to 640)
HAProxy pair, direct link3.34 (3.07 to 3.52)6.67 (6.52 to 7.08)9.9 (9.3 to 10.1)10.1 (9.4 to 10.2)20.0 (18.7 to 20.3)282 (244 to 285)443 (376 to 451)
Strait endpoint-direct, no proxy15.59 (12.46 to 16.26)48.10 (47.46 to 50.58)000152 (139 to 155)296 (266 to 300)
Strait gateway (Envoy), WireGuard link0.75 (0.72 to 0.79)0.75 (0.60 to 0.79)8.8 (8.6 to 9.4)8.8 (8.7 to 9.5)17.6 (17.3 to 18.8)528 (504 to 563)1550 (1485 to 1607)

Envoy minus HAProxy, on the direct link, in the same round. Positive latency and CPU favour HAProxy. Positive throughput favours Envoy.

MetricEnvoy minus HAProxy
Request/response p50, µs50 (36 to 90)
Request/response p99, µs63 (36 to 137)
Connection setup p50, µs212 (172 to 270)
Connection setup p99, µs273 (159 to 343)
Under load p50, µs66 (45 to 76)
Under load p99, µs137 (91 to 198)
1 stream, Gbit/s5.38 (3.69 to 7.17)
4 streams, Gbit/s4.80 (4.58 to 5.43)
CPU, both sites, millicores per connection1.5 (0.7 to 2.0)

Host during the run:

ItemValue
Load average, 1 min, before each round3.4 to 4.5
Host CPU busy over 2 s before each load window, benchmark included0.4 to 0.7 cores
Host CPU busy during each load window, benchmark included1.1 to 2.6 cores
Idle CPU, both Envoys10 to 13 millicores
Idle CPU, both HAProxysunder 1 millicore

Under CPU contention ​

bench/contention.sh, on the clusters of a kept run. Both nodes are held to the same 4 host CPUs. The load is the one above: 50 connections, 100 requests per second each. Envoy and HAProxy alternate A B B A, 8 runs each, so neither always goes first. Then a neighbour pod per site keeps 2 threads busy, with the same CPU request as each proxy, and the runs repeat. Every run held its rate. The load average was 1.5 to 2.1 without the neighbour, and 7 to 25 with it, other workloads included.

ConditionProxyUnder load p50, µsUnder load p99, µs
No neighbourEnvoy331 (329 to 335)1363 (1087 to 1880)
No neighbourHAProxy260 (259 to 296)1025 (853 to 1301)
NeighbourEnvoy11482 (9300 to 12260)25877 (20584 to 40850)
NeighbourHAProxy10559 (10091 to 11508)19593 (12955 to 36046)

Reading ​

A pooled PostgreSQL or Valkey connection pays the request/response cost once per query. A new connection pays the setup cost once.

QuestionAnswer
What does the gateway add per round trip?225 µs with Envoy, 156 µs with HAProxy, over the two hops. Endpoint-direct avoids both.
What does Envoy add over HAProxy per round trip?50 µs at p50 and 63 µs at p99 on one connection. 66 µs at p50 and 137 µs at p99 under load.
What does a real link add?Light in fibre covers about 200 km per millisecond. A 100 km fibre path adds at least 1 ms per round trip, 1000 km at least 10 ms. In this fixture the whole endpoint-direct round trip is 112 µs, so a share of it overstates the cost.
Is 50 µs meaningful against that?No. It is 5% of a 1 ms round trip and 0.5% of 10 ms, before any server time.
What does Envoy add per new connection?212 µs. A new PostgreSQL connection already needs at least one more round trip across the link for its startup, and more for password authentication or TLS. A pool pays it once per connection.
Which moves more data?Envoy, at defaults: 2.6 times HAProxy on one stream, 1.7 times on four.
Which uses less CPU?HAProxy, by 1.5 millicores per connection, about 7%.
What does WireGuard add?142 µs at p50 and 1.8 ms at p99 over the direct link, and one stream tops out near 0.75 Gbit/s on this host.
What happens when CPU runs short?Both proxies slow down 35 to 40 times at p50, from about 0.3 ms to about 11 ms. Within each A B B A block, Envoy's p99 is 1.1 to 1.5 times HAProxy's, with and without the neighbour, and its p50 is higher too. That gap is small next to the slowdown.

Conclusion: the difference between Envoy and HAProxy is not meaningful for an application that talks to PostgreSQL or Valkey across sites, while the gateway has CPU to spare. Envoy costs about 50 µs more per round trip over two hops and about 7% more CPU per request, and moves 1.7 to 2.6 times as much data at defaults. Against a cross-site round trip of 1 ms or more, the latency difference is 5% or less. Between sites so close that the round trip is shorter, the gateway itself, 156 to 225 µs here, matters more than the choice of proxy. Endpoint-direct removes it. When the gateway's node runs short of CPU, both proxies slow down far more than they differ: the gateway needs CPU headroom whichever proxy runs.

Limits ​

LimitEffect
Both sites, the client and the server share one host and one kernel. The link is a Linux bridge.Absolute numbers compare only within one run. The fixture's round trip is far shorter than a real link's.
The host is shared, and no core is reserved.Scheduling delays add to every proxy hop. In make bench, Envoy always runs just before HAProxy, so a host whose load falls during a round favours HAProxy. The contention check alternates them.
Kernel work is charged to no container: iptables, conntrack, softirq, WireGuard encryption in kernel workers.The CPU columns cover the proxy processes only. The WireGuard row understates the tunnel's CPU.
One client, one server, 64-byte messages, an echo with no server time.Real replies are larger and servers take time, which shrinks the proxy's share.
Both proxies run at their defaults.HAProxy offers splice and larger buffers, which this run does not use. The throughput gap is at defaults only.
One node and one gateway replica per site.Scaling is not measured.

Repeat ​

sh
make bench

It needs Docker, kind, kubectl, helm, Go and the wireguard kernel module. It creates the Docker networks strait-net and strait-direct when they are missing, and the clusters strait-bench-a and strait-bench-b. It writes _output/bench/results.md, results.json, the raw measurements in raw/, and the strait CLI's view of site b. It deletes the clusters at the end.

VariableEffect
STRAIT_BENCH_KEEP=1Keeps the clusters. A later run reuses them.
STRAIT_BENCH_SKIP_BUILD=1Uses the images of an earlier run.
STRAIT_BENCH_ROUNDS=NRounds. 5 by default.
STRAIT_BENCH_PATHS="..."Paths, in measuring order. pod-ip adds the client dialing the server pod's address, with no Service.

The contention check runs on kept clusters and writes _output/bench/contention.md:

sh
STRAIT_BENCH_KEEP=1 make bench
bench/contention.sh
VariableEffect
STRAIT_BENCH_CPUS=4-7Host CPUs that both nodes are held to. Given back at the end.
STRAIT_BENCH_BLOCKS=4A B B A blocks per condition.

Released under the Apache License 2.0.