View source for Perf/watt benchmarks in a k8s home cluster

= Perf/watt of a heterogeneous Kubernetes cluster: Apple silicon vs x86 vs Raspberry Pi CM5 =

''Measured performance-per-watt for eight nodes of a real, running Kubernetes cluster — Apple M1/M4
Mac minis, three x86 boxes, and Raspberry Pi CM5s — with power taken at the wall rather than from
spec sheets. The lens is deliberately "what does a container on this node get, and what does the
node cost to own", not bare-silicon SPEC. Written for anyone sizing a home or small-office cluster
for energy efficiency. Claims are limited to what we directly measured or can cite; inference and
open questions are marked as such. Every number below is from a single afternoon's run on 2026-07-14
and the raw method is given so it can be repeated.''

== Summary ==

* '''Apple M4 wins performance-per-watt by roughly 2×''' over every x86 part tested, and it does so '''from inside a virtual machine''' while the x86 boxes run on bare metal.
* '''x86 wins absolute throughput by a wide margin.''' The i9-13900 produced more than twice the multi-core work of any other node — it is simply expensive to run.
* '''The CM5 and low-power x86 are tied''' on perf/watt once the measurement boundary is corrected for. The CM5's apparent lead is an artifact of how it is metered.
* '''Single-threaded perf/watt is where x86 collapses''' — up to 6× worse than Apple silicon, because the whole package, DRAM and PSU stay awake to run one core.
* '''Two benchmarks disagreed by ~3× on cross-architecture ranking.''' Anyone comparing ARM and x86 with a single CPU benchmark is likely publishing a wrong number. Details below.
* '''The idle floor dominates the result.''' A node that idles at 25 W has to win hard on throughput to break even against one that idles at 2.6 W.

== The cluster under test ==

All nodes are workers in one k3s cluster (v1.36.2+k3s1), all running the same containerised
benchmark. RAM differs across nodes but is irrelevant to a CPU benchmark and was not normalised.

{| class="wikitable"
! Node !! CPU !! Arch !! Threads !! Host OS !! Notes
|-
| <code>cx1</code> || Intel i9-13900 || x86_64 || 32 || Ubuntu 26.04 || Desktop board (ASRock Z790M), 96 GB, ATX PSU
|-
| <code>cx3</code> || Intel i7-14700T || x86_64 || 28 || Ubuntu 26.04 || "T" 35 W-class part; carries Postgres/Plex/Thanos
|-
| <code>cx4</code> || AMD Ryzen 5 5600U || x86_64 || 12 || Ubuntu 26.04 || Mini-PC, mobile APU
|-
| <code>mx3</code>, <code>mx4</code> || Apple M4 || arm64 || 10 || Ubuntu 26.04 '''in a Lima VM on macOS''' || VM gets the whole box (10 cpu / 12 GiB)
|-
| <code>mx1</code> || Apple M1 || arm64 || 8 || Fedora Asahi Remix 44 || '''Bare metal''' — no VM, no macOS
|-
| <code>x1</code>, <code>x2</code> || Broadcom BCM2712 (Cortex-A76) || arm64 || 4 || Ubuntu 26.04 || Raspberry Pi CM5, PoE-powered
|}

== Method ==

=== Benchmarks ===

Two benchmarks, deliberately, so they could be cross-checked against each other. Both run from one
multi-arch <code>debian:trixie-slim</code> container (glibc everywhere, so the C library is a
constant), as a Kubernetes <code>Job</code> pinned to each node with <code>nodeName</code>.

* '''<code>sysbench cpu</code>''' 1.0.20 — <code>--threads=1</code> and <code>--threads=$(nproc)</code>, <code>--time=300</code>. Score = events/s.
* '''<code>7z b</code>''' (7-Zip 25.01) — <code>-mmt1</code> and <code>-mmt$(nproc)</code>, looped to fill 300 s. Score = the "Tot" rating in MIPS, averaged over passes.

Each of the four tests ran for a '''full 5 minutes''' per node. That duration is not arbitrary — see
''Measuring the power'' below.

=== Measuring the power ===

Every node is individually metered, and the meters are polled into Prometheus/Thanos, so a
benchmark's power is a time-range query over the exact epochs the benchmark logged.

* '''x86 and Mac nodes''' — a UniFi smart PDU, one outlet each: '''true AC at the wall''', including each machine's own PSU losses.
* '''CM5 nodes''' — PoE ports on a UniFi switch: '''DC delivered at the port''', which '''excludes''' the switch PSU's AC→DC conversion loss.

'''These are not the same boundary, and that matters — see Caveats.'''

Three practical findings about the instrument, each of which would have silently corrupted the
results:

# '''The meter's real cadence is 60 s''', not the ~2 min we had assumed.
# '''The PDU lags a load change by ~60–120 s.''' Every multi-thread window opened with one or two samples still reporting the ''previous'' test's power. A 5-minute window survives this (take the last 3 samples); '''a 2–3 minute window would have consisted almost entirely of stale pre-load readings''' and produced plausible-looking, entirely wrong numbers.
# '''Do not use a Prometheus <code>query_range</code> with a step finer than the scrape interval.''' It carries the last sample forward to fill empty steps, so the head of every window replays the previous test's power as if it were real data. Query raw samples with a range selector instead.

The plateau rule adopted: take the '''last 3 samples''' of each 5-minute window; if they disagree by
more than ~15%, treat the run as contaminated rather than believing it.

=== Baseline (idle) power, and why it is included ===

'''The headline metric is score ÷ total wall watts, with the idle floor included, not subtracted.'''
An always-on cluster node burns its floor 24/7 whether or not it is computing, so that draw is a real
cost of owning it. We deliberately did not normalise it away.

Idle was taken as the 10th percentile over a verified-quiet 26-minute window with the cluster's batch
work stopped. This matters more than it sounds: a naive "10th percentile over 24 h" gave '''48 W''' for
<code>cx3</code>, because a transcoding job had been running for most of the day. A percentile is only
an idle estimate if the machine was actually idle for most of the window.

{| class="wikitable"
! Node !! Idle (W) !! Boundary
|-
| <code>cx1</code> (i9-13900) || 25.2 || PDU (AC)
|-
| <code>cx3</code> (i7-14700T) || 17.2 || PDU (AC)
|-
| <code>cx4</code> (Ryzen 5600U) || 9.9 || PDU (AC)
|-
| <code>mx1</code> (M1) || 5.8 || PDU (AC)
|-
| <code>mx3</code> (M4) || 2.9 || PDU (AC)
|-
| <code>mx4</code> (M4) || 2.6 || PDU (AC)
|-
| <code>x1</code>, <code>x2</code> (CM5) || 2.7, 2.6 || PoE (DC)
|}

The spread here is the story of the whole article: '''an M4 Mac mini idles at about the same power as a
Raspberry Pi''', despite roughly ten times the peak performance, while the desktop i9 idles at nearly
ten times either.

== Results ==

=== 7z, all cores — the headline ===

This is the number we trust for cross-architecture comparison (see the next section for why).

{| class="wikitable sortable"
! Node !! CPU !! Threads !! Score (MIPS) !! Idle W !! Load W !! data-sort-type="number" | '''MIPS/W''' !! Boundary
|-
| <code>mx3</code> || Apple M4 || 10 || 74,698 || 2.9 || 24.7 || '''3,024''' || PDU
|-
| <code>mx4</code> || Apple M4 || 10 || 74,402 || 2.6 || 28.2 || '''2,641''' || PDU
|-
| <code>mx1</code> || Apple M1 || 8 || 42,413 || 5.8 || 18.7 || 2,269 || PDU
|-
| <code>x2</code> || CM5 (A76) || 4 || 14,194 || 2.6 || 8.4 || 1,682* || PoE
|-
| <code>cx4</code> || Ryzen 5 5600U || 12 || 48,355 || 9.9 || 32.7 || 1,479 || PDU
|-
| <code>cx1</code> || i9-13900 || 32 || '''144,572''' || 25.2 || 100.9 || 1,432 || PDU
|-
| <code>cx3</code> || i7-14700T || 28 || 77,010 || 17.2 || 54.2 || 1,421 || PDU
|}

<nowiki>*</nowiki> ''PoE figure — not directly comparable to the PDU rows. Corrected, it lands ~1,450–1,630; see Caveats.''

Read it as three tiers:

* '''Apple M4 — ~2× everything else.''' And it earns that ''through'' a virtualization layer.
* '''CM5 and low-power x86 — tied''', once the boundary is corrected.
* '''The i9 is the throughput king and the efficiency loser''': 144,572 MIPS is nearly 2× the next best node, at 100.9 W to get it.

=== Single-thread — where x86 falls apart ===

{| class="wikitable sortable"
! Node !! CPU !! 7z single (MIPS) !! Load W !! data-sort-type="number" | MIPS/W
|-
| <code>mx3</code> || Apple M4 || 9,681 || 9.5 || '''1,021'''
|-
| <code>mx4</code> || Apple M4 || 9,529 || 9.8 || 969
|-
| <code>x2</code> || CM5 || 4,269 || 4.7 || 912*
|-
| <code>mx1</code> || Apple M1 || 7,033 || 10.8 || 652
|-
| <code>cx4</code> || Ryzen 5 5600U || 7,096 || 27.8 || 255
|-
| <code>cx3</code> || i7-14700T || 6,972 || 37.6 || 186
|-
| <code>cx1</code> || i9-13900 || 9,902 || 62.8 || '''158'''
|}

Note what this table is ''not'' saying. On raw single-thread speed the i9 '''wins''' (9,902, ahead of
the M4's 9,681). But it burns '''62.8 W''' to run one core, against the M4's 9.8 W — the package,
96 GB of DRAM and an ATX PSU all stay awake to feed a single thread. The result is a '''6× perf/watt
deficit''' on exactly the kind of latency-sensitive, lightly-threaded work a home cluster spends most
of its life doing.

We also caught the i9's turbo behaviour: an all-core run '''transiently pulled 157.6 W''' at the wall
before settling to a 99 W plateau — the PL2 boost window, visible as a single 60 s sample.

=== The two benchmarks disagree, and one of them is lying ===

'''This is the most portable lesson here.''' The same eight machines, ranked by the same metric, two
respected benchmarks:

{| class="wikitable"
! Node !! sysbench single (events/s) !! 7z single (MIPS) !! Verdict
|-
| <code>mx4</code> (M4) || '''13,519''' || 9,529 || sysbench says M4 wins by 2.9×
|-
| <code>cx1</code> (i9-13900) || 4,690 || '''9,902''' || 7z says the i9 wins outright
|}

Both cannot be right. [https://blog.cloud-mercato.com/why-you-should-use-sysbench/ sysbench's CPU
test is a simplistic prime-number benchmark built on 64-bit integer arithmetic, and is heavily
dependent on 64-bit division] — it was written with database-like integer workloads in mind, not as a
general CPU metric. Integer-division throughput is one of the most microarchitecture-variable
operations there is, so a division-bound loop is close to a worst case for cross-architecture
comparison.

The public Geekbench 6 spread between these parts is roughly 1.3×, which corroborates 7z and not
sysbench. '''We therefore treat 7z as the cross-architecture number and sysbench as unreliable for
that purpose.''' Note the rankings still agree (M4 > M1 > CM5 > x86) — it is the ''magnitudes'' that
are untrustworthy.

'''Open question:''' we cannot ''explain'' the 2.9× from first principles, and we are flagging that
rather than inventing a story. The obvious guess — that ARM divides faster — is not obviously right:
[https://forums.raspberrypi.com/viewtopic.php?t=243567&start=25 x86's <code>DIV</code> returns
quotient and remainder in a single instruction, while ARM64's <code>UDIV</code> returns only the
quotient and needs an extra <code>MSUB</code> for the remainder] — and the prime loop needs the
remainder, which argues ARM should be ''disadvantaged'' here. The most likely explanation is simply
that Apple's divider vastly outperforms Raptor Lake's, but we have not verified that, and it is not
required for the conclusion: whatever the cause, a division-bound benchmark is the wrong tool.

== Two things we learned about measuring, not about CPUs ==

=== Kubernetes makes a small node's power visibly lumpy ===

The CM5 nodes' power wandered by ±1 W at idle — enough to matter when the entire all-core load delta
is only ~3 W. Our first instinct was to blame the meter. That was '''wrong''', and the control test is
worth copying:

{| class="wikitable"
! Device !! Mean W !! Stdev W !! What it is
|-
| <code>usl-gw</code> || 1.74 || '''0.05''' || Dumb constant PoE load
|-
| <code>pikvm</code> || 2.37 || 0.11 || Dumb constant PoE load
|-
| <code>u1</code>, <code>u2</code> || 2.75, 2.51 || 0.17, 0.21 || '''CM5s, but not running k3s'''
|-
| <code>x1</code>, <code>x2</code>, <code>x3</code> || 5.14, 3.81, 3.30 || '''1.69, 0.97, 0.76''' || '''CM5s running k3s + etcd'''
|}

The meter resolves 0.05 W. Two ''identical CM5 boards'' that are not cluster members sit at 0.17–0.21 W
of variation. The cluster members show 10–30× that. '''The wobble is real power: k3s and etcd make a
small ARM node's draw genuinely bursty at idle.''' There is no stable "idle floor" to measure on such
a node, only a distribution — which is why a percentile over a verified-quiet window, rather than a
spot reading, is the only honest baseline.

=== Never benchmark the machine you are working from ===

<code>x1</code> hosts the operator tooling that was orchestrating this very benchmark. It looked fine
on most tests, so we re-ran the identical benchmark on <code>x2</code> — hardware-identical, same
cooling, same clock, but idle and uninvolved:

{| class="wikitable"
! Test !! <code>x1</code> (busy) !! <code>x2</code> (clean) !! <code>x2</code> advantage
|-
| sysbench single || 2,645.3 || 2,644.8 || '''0.0%'''
|-
| 7z single || 4,158 || 4,269 || 2.7%
|-
| sysbench multi || 9,918 || 10,331 || 4.2%
|-
| 7z multi || 12,428 || 14,194 || '''14.2%'''
|}

Single-thread is '''identical to four significant figures''' — one core of four, no contention. But
'''7z multi was off by 14.2%''', far more than the ~4% the other tests suggested, because 7z is
cache-hungry and the co-resident work was evicting its working set. A contaminated node does not
degrade uniformly: it degrades ''where the benchmark is sensitive''. If we had only run sysbench, we
would have concluded the contamination was ~4% and shipped a number that was wrong by 14%.

== Caveats ==

* '''Mixed measurement boundaries.''' CM5s are metered at the PoE port (DC delivered, excluding the switch PSU's conversion loss); everything else at the PDU (AC at the wall). The switch draws 105.1 W AC to deliver 77.2 W of PoE across all ports; allowing 15–25 W for the switch's own electronics puts its PSU at ~86–96% efficient, i.e. a '''×1.04–×1.17 correction''' to the CM5 figures. That moves <code>x2</code> from 1,682 to '''~1,450–1,630 MIPS/W''' — statistically indistinguishable from <code>cx4</code>'s 1,479. '''So "the CM5 beats low-power x86" is not a supportable claim; "they are tied" is.''' The M4's ~2× lead is far outside this correction and is unaffected.
* '''The Lima VM tax is unquantified.''' The M4 nodes ran inside a VM and still won. We did not measure the native-vs-virtualised delta (doing so requires stopping the VM, which would have taken the node out of the cluster), so we can only say the tax is ≥ 0 and the M4's true native figure '''can only be better than reported'''. The ~2× is a floor.
* '''CM5 precision is ~±6%''', not better: with ~5 samples against ~0.8 W of real burstiness, the standard error on a ~6–8 W reading is ~0.35 W. Adequate for tier placement, not for splitting a tie.
* '''"Idle" is as-deployed, not bare.''' <code>cx3</code> permanently hosts Postgres, Plex and Thanos and never goes below 12.4 W in 30 days; <code>cx1</code> hosts Prometheus. Their idle figures include that resident work. This is deliberate — it is what those nodes actually cost to own — but it is not a board-level idle spec.
* '''Single run, single afternoon.''' No repeat trials, so no confidence intervals beyond the within-window spread. The cluster was verified quiet first.
* '''musl vs glibc was eliminated''' (one glibc container everywhere) but compiler/tuning differences between the Debian arm64 and amd64 builds of these tools were not.

== Reproducing this ==

The container is stock <code>debian:trixie-slim</code> plus <code>apt-get install sysbench 7zip</code>
(note: on current Debian the binary is <code>7z</code>, from the <code>7zip</code> package —
<code>p7zip</code> is a removed transitional package). The benchmark logs
<code>BENCH_START/BENCH_END</code> epochs around each test; power is then a range query over exactly
those epochs, taking the last 3 samples. Run it when the cluster is genuinely idle, one node per
metered domain, and re-run anything whose plateau samples disagree.

[[Category:Infrastructure]]
[[Category:Benchmarks]]