Perf/watt benchmarks in a k8s home cluster

From dtype.org

Perf/watt of a heterogeneous Kubernetes cluster: Apple silicon vs x86 vs Raspberry Pi CM5

Measured performance-per-watt for eight nodes of a real, running Kubernetes cluster — Apple M1/M4 Mac minis, three x86 boxes, and Raspberry Pi CM5s — with power taken at the wall rather than from spec sheets. The lens is deliberately "what does a container on this node get, and what does the node cost to own", not bare-silicon SPEC. Written for anyone sizing a home or small-office cluster for energy efficiency. Claims are limited to what we directly measured or can cite; inference and open questions are marked as such. Every number below is from a single afternoon's run on 2026-07-14 and the raw method is given so it can be repeated.

Summary

  • Apple M4 wins performance-per-watt by roughly 2× over every x86 part tested, and it does so from inside a virtual machine while the x86 boxes run on bare metal.
  • x86 wins absolute throughput by a wide margin. The i9-13900 produced more than twice the multi-core work of any other node — it is simply expensive to run.
  • The CM5 and low-power x86 are tied on perf/watt once the measurement boundary is corrected for. The CM5's apparent lead is an artifact of how it is metered.
  • Single-threaded perf/watt is where x86 collapses — up to 6× worse than Apple silicon, because the whole package, DRAM and PSU stay awake to run one core.
  • Two benchmarks disagreed by ~3× on cross-architecture ranking. Anyone comparing ARM and x86 with a single CPU benchmark is likely publishing a wrong number. Details below.
  • The idle floor dominates the result. A node that idles at 25 W has to win hard on throughput to break even against one that idles at 2.6 W.

The cluster under test

All nodes are workers in one k3s cluster (v1.36.2+k3s1), all running the same containerised benchmark. RAM differs across nodes but is irrelevant to a CPU benchmark and was not normalised.

Node CPU Arch Threads Host OS Notes
cx1 Intel i9-13900 x86_64 32 Ubuntu 26.04 Desktop board (ASRock Z790M), 96 GB, ATX PSU
cx3 Intel i7-14700T x86_64 28 Ubuntu 26.04 "T" 35 W-class part; carries Postgres/Plex/Thanos
cx4 AMD Ryzen 5 5600U x86_64 12 Ubuntu 26.04 Mini-PC, mobile APU
mx3, mx4 Apple M4 arm64 10 Ubuntu 26.04 in a Lima VM on macOS VM gets the whole box (10 cpu / 12 GiB)
mx1 Apple M1 arm64 8 Fedora Asahi Remix 44 Bare metal — no VM, no macOS
x1, x2 Broadcom BCM2712 (Cortex-A76) arm64 4 Ubuntu 26.04 Raspberry Pi CM5, PoE-powered

Method

Benchmarks

Two benchmarks, deliberately, so they could be cross-checked against each other. Both run from one multi-arch debian:trixie-slim container (glibc everywhere, so the C library is a constant), as a Kubernetes Job pinned to each node with nodeName.

  • sysbench cpu 1.0.20 — --threads=1 and --threads=$(nproc), --time=300. Score = events/s.
  • 7z b (7-Zip 25.01) — -mmt1 and -mmt$(nproc), looped to fill 300 s. Score = the "Tot" rating in MIPS, averaged over passes.

Each of the four tests ran for a full 5 minutes per node. That duration is not arbitrary — see Measuring the power below.

Measuring the power

Every node is individually metered, and the meters are polled into Prometheus/Thanos, so a benchmark's power is a time-range query over the exact epochs the benchmark logged.

  • x86 and Mac nodes — a UniFi smart PDU, one outlet each: true AC at the wall, including each machine's own PSU losses.
  • CM5 nodes — PoE ports on a UniFi switch: DC delivered at the port, which excludes the switch PSU's AC→DC conversion loss.

These are not the same boundary, and that matters — see Caveats.

Three practical findings about the instrument, each of which would have silently corrupted the results:

  1. The meter's real cadence is 60 s, not the ~2 min we had assumed.
  2. The PDU lags a load change by ~60–120 s. Every multi-thread window opened with one or two samples still reporting the previous test's power. A 5-minute window survives this (take the last 3 samples); a 2–3 minute window would have consisted almost entirely of stale pre-load readings and produced plausible-looking, entirely wrong numbers.
  3. Do not use a Prometheus query_range with a step finer than the scrape interval. It carries the last sample forward to fill empty steps, so the head of every window replays the previous test's power as if it were real data. Query raw samples with a range selector instead.

The plateau rule adopted: take the last 3 samples of each 5-minute window; if they disagree by more than ~15%, treat the run as contaminated rather than believing it.

Baseline (idle) power, and why it is included

The headline metric is score ÷ total wall watts, with the idle floor included, not subtracted. An always-on cluster node burns its floor 24/7 whether or not it is computing, so that draw is a real cost of owning it. We deliberately did not normalise it away.

Idle was taken as the 10th percentile over a verified-quiet 26-minute window with the cluster's batch work stopped. This matters more than it sounds: a naive "10th percentile over 24 h" gave 48 W for cx3, because a transcoding job had been running for most of the day. A percentile is only an idle estimate if the machine was actually idle for most of the window.

Node Idle (W) Boundary
cx1 (i9-13900) 25.2 PDU (AC)
cx3 (i7-14700T) 17.2 PDU (AC)
cx4 (Ryzen 5600U) 9.9 PDU (AC)
mx1 (M1) 5.8 PDU (AC)
mx3 (M4) 2.9 PDU (AC)
mx4 (M4) 2.6 PDU (AC)
x1, x2 (CM5) 2.7, 2.6 PoE (DC)

The spread here is the story of the whole article: an M4 Mac mini idles at about the same power as a Raspberry Pi, despite roughly ten times the peak performance, while the desktop i9 idles at nearly ten times either.

Results

7z, all cores — the headline

This is the number we trust for cross-architecture comparison (see the next section for why).

Node CPU Threads Score (MIPS) Idle W Load W MIPS/W Boundary
mx3 Apple M4 10 74,698 2.9 24.7 3,024 PDU
mx4 Apple M4 10 74,402 2.6 28.2 2,641 PDU
mx1 Apple M1 8 42,413 5.8 18.7 2,269 PDU
x2 CM5 (A76) 4 14,194 2.6 8.4 1,682* PoE
cx4 Ryzen 5 5600U 12 48,355 9.9 32.7 1,479 PDU
cx1 i9-13900 32 144,572 25.2 100.9 1,432 PDU
cx3 i7-14700T 28 77,010 17.2 54.2 1,421 PDU

* PoE figure — not directly comparable to the PDU rows. Corrected, it lands ~1,450–1,630; see Caveats.

Read it as three tiers:

  • Apple M4 — ~2× everything else. And it earns that through a virtualization layer.
  • CM5 and low-power x86 — tied, once the boundary is corrected.
  • The i9 is the throughput king and the efficiency loser: 144,572 MIPS is nearly 2× the next best node, at 100.9 W to get it.

Single-thread — where x86 falls apart

Node CPU 7z single (MIPS) Load W MIPS/W
mx3 Apple M4 9,681 9.5 1,021
mx4 Apple M4 9,529 9.8 969
x2 CM5 4,269 4.7 912*
mx1 Apple M1 7,033 10.8 652
cx4 Ryzen 5 5600U 7,096 27.8 255
cx3 i7-14700T 6,972 37.6 186
cx1 i9-13900 9,902 62.8 158

Note what this table is not saying. On raw single-thread speed the i9 wins (9,902, ahead of the M4's 9,681). But it burns 62.8 W to run one core, against the M4's 9.8 W — the package, 96 GB of DRAM and an ATX PSU all stay awake to feed a single thread. The result is a 6× perf/watt deficit on exactly the kind of latency-sensitive, lightly-threaded work a home cluster spends most of its life doing.

We also caught the i9's turbo behaviour: an all-core run transiently pulled 157.6 W at the wall before settling to a 99 W plateau — the PL2 boost window, visible as a single 60 s sample.

The two benchmarks disagree, and one of them is lying

This is the most portable lesson here. The same eight machines, ranked by the same metric, two respected benchmarks:

Node sysbench single (events/s) 7z single (MIPS) Verdict
mx4 (M4) 13,519 9,529 sysbench says M4 wins by 2.9×
cx1 (i9-13900) 4,690 9,902 7z says the i9 wins outright

Both cannot be right. [https://blog.cloud-mercato.com/why-you-should-use-sysbench/ sysbench's CPU test is a simplistic prime-number benchmark built on 64-bit integer arithmetic, and is heavily dependent on 64-bit division] — it was written with database-like integer workloads in mind, not as a general CPU metric. Integer-division throughput is one of the most microarchitecture-variable operations there is, so a division-bound loop is close to a worst case for cross-architecture comparison.

The public Geekbench 6 spread between these parts is roughly 1.3×, which corroborates 7z and not sysbench. We therefore treat 7z as the cross-architecture number and sysbench as unreliable for that purpose. Note the rankings still agree (M4 > M1 > CM5 > x86) — it is the magnitudes that are untrustworthy.

Open question: we cannot explain the 2.9× from first principles, and we are flagging that rather than inventing a story. The obvious guess — that ARM divides faster — is not obviously right: [https://forums.raspberrypi.com/viewtopic.php?t=243567&start=25 x86's DIV returns quotient and remainder in a single instruction, while ARM64's UDIV returns only the quotient and needs an extra MSUB for the remainder] — and the prime loop needs the remainder, which argues ARM should be disadvantaged here. The most likely explanation is simply that Apple's divider vastly outperforms Raptor Lake's, but we have not verified that, and it is not required for the conclusion: whatever the cause, a division-bound benchmark is the wrong tool.

Two things we learned about measuring, not about CPUs

Kubernetes makes a small node's power visibly lumpy

The CM5 nodes' power wandered by ±1 W at idle — enough to matter when the entire all-core load delta is only ~3 W. Our first instinct was to blame the meter. That was wrong, and the control test is worth copying:

Device Mean W Stdev W What it is
usl-gw 1.74 0.05 Dumb constant PoE load
pikvm 2.37 0.11 Dumb constant PoE load
u1, u2 2.75, 2.51 0.17, 0.21 CM5s, but not running k3s
x1, x2, x3 5.14, 3.81, 3.30 1.69, 0.97, 0.76 CM5s running k3s + etcd

The meter resolves 0.05 W. Two identical CM5 boards that are not cluster members sit at 0.17–0.21 W of variation. The cluster members show 10–30× that. The wobble is real power: k3s and etcd make a small ARM node's draw genuinely bursty at idle. There is no stable "idle floor" to measure on such a node, only a distribution — which is why a percentile over a verified-quiet window, rather than a spot reading, is the only honest baseline.

Never benchmark the machine you are working from

x1 hosts the operator tooling that was orchestrating this very benchmark. It looked fine on most tests, so we re-ran the identical benchmark on x2 — hardware-identical, same cooling, same clock, but idle and uninvolved:

Test x1 (busy) x2 (clean) x2 advantage
sysbench single 2,645.3 2,644.8 0.0%
7z single 4,158 4,269 2.7%
sysbench multi 9,918 10,331 4.2%
7z multi 12,428 14,194 14.2%

Single-thread is identical to four significant figures — one core of four, no contention. But 7z multi was off by 14.2%, far more than the ~4% the other tests suggested, because 7z is cache-hungry and the co-resident work was evicting its working set. A contaminated node does not degrade uniformly: it degrades where the benchmark is sensitive. If we had only run sysbench, we would have concluded the contamination was ~4% and shipped a number that was wrong by 14%.

Caveats

  • Mixed measurement boundaries. CM5s are metered at the PoE port (DC delivered, excluding the switch PSU's conversion loss); everything else at the PDU (AC at the wall). The switch draws 105.1 W AC to deliver 77.2 W of PoE across all ports; allowing 15–25 W for the switch's own electronics puts its PSU at ~86–96% efficient, i.e. a ×1.04–×1.17 correction to the CM5 figures. That moves x2 from 1,682 to ~1,450–1,630 MIPS/W — statistically indistinguishable from cx4's 1,479. So "the CM5 beats low-power x86" is not a supportable claim; "they are tied" is. The M4's ~2× lead is far outside this correction and is unaffected.
  • The Lima VM tax is unquantified. The M4 nodes ran inside a VM and still won. We did not measure the native-vs-virtualised delta (doing so requires stopping the VM, which would have taken the node out of the cluster), so we can only say the tax is ≥ 0 and the M4's true native figure can only be better than reported. The ~2× is a floor.
  • CM5 precision is ~±6%, not better: with ~5 samples against ~0.8 W of real burstiness, the standard error on a ~6–8 W reading is ~0.35 W. Adequate for tier placement, not for splitting a tie.
  • "Idle" is as-deployed, not bare. cx3 permanently hosts Postgres, Plex and Thanos and never goes below 12.4 W in 30 days; cx1 hosts Prometheus. Their idle figures include that resident work. This is deliberate — it is what those nodes actually cost to own — but it is not a board-level idle spec.
  • Single run, single afternoon. No repeat trials, so no confidence intervals beyond the within-window spread. The cluster was verified quiet first.
  • musl vs glibc was eliminated (one glibc container everywhere) but compiler/tuning differences between the Debian arm64 and amd64 builds of these tools were not.

Reproducing this

The container is stock debian:trixie-slim plus apt-get install sysbench 7zip (note: on current Debian the binary is 7z, from the 7zip package — p7zip is a removed transitional package). The benchmark logs BENCH_START/BENCH_END epochs around each test; power is then a range query over exactly those epochs, taking the last 3 samples. Run it when the cluster is genuinely idle, one node per metered domain, and re-run anything whose plateau samples disagree.

Retrieved from "https://dtype.org/index.php?title=Perf/watt_benchmarks_in_a_k8s_home_cluster&oldid=2364"