Mik Mak Muk

A 160 GB bare-metal CachyOS homelab cluster built on three HP Mini PCs. No heavy hypervisors, no cloud lock-in—just Ansible, custom kernel tuning, zero-touch enterprise PKI, and physical hardware hacks.

Repositoryelastic-direct branchhanskruse.eu

Instead of buying a loud, power-hungry server rack, I built a quiet, compact 3-node cluster using HP EliteDesk and ProDesk Mini G4 PCs mounted under my DIY desk shelf.

Rather than wrapping everything inside heavy virtual machine hypervisors like Proxmox, Mik Mak Muk runs directly on bare-metal CachyOS. Using Ansible, the project builds a fast, rock-solid baseline OS layer before deploying workload such as multi-node Elasticsearch 9 & Kibana cluster or running distributed llama.cpp AI cluster for embedding and inference as initial experiments.

In future experiments I will explore systemd containers, incus and K0s with Cillium

Cluster Topology (160 GB RAM Total)

Host Model CPU RAM Primary Role
mik HP EliteDesk 800 Mini G4 Intel i5-8500 (65W) 32 GB Primary Master / Heavy Compute
mak HP ProDesk 600 Mini G4 Intel i5-8500T (35W) 64 GB Worker Node / Kibana UI
muk HP ProDesk 400 Mini G4 Intel i5-8500T (35W) 64 GB Worker / Ingress Node

Benchmark Results

Measured with ./benchmark.yml on bare-metal CachyOS. Last update: 2026-11-11T21:36:17Z.

Hardware & Power Audit

Collected 2026-11-11T21:21:29Z

Metric mik mak muk
CPU model Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz Intel(R) Core(TM) i5-8500T CPU @ 2.10GHz Intel(R) Core(TM) i5-8500T CPU @ 2.10GHz
Kernel 7.3.3-1-cachyos 7.3.3-1-cachyos 7.3.3-1-cachyos
Microcode 0xfa 0xfa 0xfa
Scaling driver intel_pstate intel_pstate intel_pstate
Governor performance performance performance
EPP performance performance performance
Turbo boost yes yes yes
CPU max clock 4100 MHz 3500 MHz 3500 MHz
Installed memory 31.1 GiB 62.6 GiB 62.6 GiB
Populated SO-DIMMs 2 2 2
Memory speed 2400 MT/s 2667 MT/s 2667 MT/s
iGPU driver i915 i915 i915
iGPU max clock n/a 1100 MHz 1100 MHz
VT-x yes yes yes
IOMMU (VT-d) no no no

CPU & Memory

Collected 2026-11-11T21:26:20Z

Metric mik mak muk
Sysbench single-thread 486.2 events/s 398.8 events/s 398.9 events/s
Sysbench multi-thread 2917.3 events/s 2393.6 events/s 2393.5 events/s
stress-ng matrixprod (60 s) 9431 bogo ops/s 7636 bogo ops/s 7855 bogo ops/s
Memory sequential read 36057 MiB/s 30147 MiB/s 29477 MiB/s
Memory sequential write 13537 MiB/s 10688 MiB/s 10750 MiB/s
CPU package idle 79.0 °C 64.0 °C 67.0 °C
CPU package after test 84.0 °C 70.0 °C 71.0 °C
Throttle events 0 0 0

Integrated GPU (UHD 630)

Collected 2026-11-11T21:34:40Z

Metric mik mak muk
GPU Intel(R) UHD Graphics 630 (CFL GT2) Intel(R) UHD Graphics 630 (CFL GT2) Intel(R) UHD Graphics 630 (CFL GT2)
OpenCL compute units 3 3 3
Max clock 1100 MHz 1100 MHz 1100 MHz
OpenCL FP32 peak 405.2 GFLOPS 334.6 GFLOPS 316.0 GFLOPS
Vulkan FP32 peak 412.1 GFLOPS 340.4 GFLOPS 321.6 GFLOPS
OpenCL memory bandwidth 30.2 GB/s 32.7 GB/s 32.0 GB/s
Vulkan memory bandwidth 30.0 GB/s 33.0 GB/s 31.9 GB/s
VA-API profiles 32 32 32
vkmark score 7617 5976 5908
glmark2-es2 score 5049 3771 3617
Package idle 83.0 °C 72.0 °C 71.0 °C
Package after test 79.0 °C 65.0 °C 66.0 °C
Throttle events 0 0 0

Combined CPU + iGPU Load

Collected 2026-11-11T21:36:17Z

Metric mik mak muk
Load duration 60 s 60 s 60 s
CPU matrixprod under load 8124 bogo ops/s 7327 bogo ops/s 7414 bogo ops/s
iGPU clpeak runs during load 5 5 5
iGPU FP32 at end of load window 403.5 GFLOPS 311.1 GFLOPS 313.6 GFLOPS
Package idle 80.0 °C 66.0 °C 69.0 °C
Package at end of load window 95.0 °C 74.0 °C 74.0 °C
Package after test 90.0 °C 71.0 °C 71.0 °C
Throttle events 0 0 0

NVMe Storage

Collected 2026-11-11T21:24:34Z

Metric mik mak muk
Drive WDC WDS500G2B0C-00PXH0 Samsung SSD 980 PRO 1TB KINGSTON SNVS1000G
Firmware 211210WD 5B2QGXA7 S8B40103
Wear (percentage used) 0 % 0 % 1 %
Data written 3.59 TB 6.92 TB 1.23 TB
Power-on hours 254 h 129 h 6520 h
Direct sequential write (128k) 662.7 MiB/s 2630.0 MiB/s 129.6 MiB/s
Direct sequential write IOPS 5302 21040 1037
fsync random write IOPS (4k) 960 162 276
fsync latency avg 17.4 µs 21.6 µs 19.7 µs
fsync latency p99 20.4 µs 24.4 µs 21.6 µs
NVMe idle 55.9 °C 52.9 °C 44.9 °C
NVMe after test 62.9 °C 60.9 °C 60.9 °C

Network (iperf3 full mesh)

Collected 2026-11-11T21:23:47Z

Path 1 GbE 2.5 GbE
mik → mak 0.93 Gbps 2.47 Gbps
mik → muk 0.93 Gbps 2.47 Gbps
mak → mik 0.93 Gbps 2.47 Gbps
mak → muk 0.93 Gbps 2.47 Gbps
muk → mik 0.94 Gbps 2.47 Gbps
muk → mak 0.94 Gbps 2.47 Gbps

Design Choices

Building on compact 8th-Gen mini-PC chassis comes with real hardware limits.:

SATA Drive Caddy issues

All nodes run OS and data strictly on high-speed NVMe drives. However, removing the unused 2.5" SATA caddy to save space breaks cooling: on 65W models like mik, the CPU fan mounts directly to the SATA bracket! Removing it causes the CPU fan to ramp up to 100% full speed permanently. We keep the empty caddys installed solely to maintain stock fan mounting and quiet airflow. On 35 watt TDP models I might consider removing the caddy later to install networking via issue.The 65 Watt Elite Desk model does have two Nvme slots. The second one is unusable for normal NVME SSDs when the caddy is kept. A 30mm SSD does fit. I originally had old 120Gb SSD drives in two machines. given 2026 storage prices I decided to postpone adding big Sata SSDs or adding an extra Nvme SSD to my elitedesk machine. the 120 Gb SSDs were moved to another project.

2.5 GbE USB-C Option Port Networking

  • HP never made official 2.5 GbE option modules for 4th gen chassis
  • Thunderbolt is exclusive to EliteDesk (missing on ProDesk) and the required option port modules are rare
  • M.2 Wi-Fi/NVMe network adapters require custom 3D-printed rear covers to slot in the option port
  • 10 GbE USB adapters throttle under heat.
  • Adding USB-C 2.5 GbE dongles via Type-C option slots provides a clean, cheap 2.5x speed boost over the 1 GbE onboard port without MacGyver hacks.

Automated Network Benchmarking & Routing

As the old Dutch proverb goes: "Meten is weten" ("To measure is to know"). We built an automated full-mesh iperf3 benchmark playbook to verify line-rate throughput across both network tiers (1 GbE management and 2.5 GbE cluster interconnect). With jumbo frames we get 2.48 Gbps. Without jumbo frames we get 2.36 Gbps. On the 1 gb link we get 0.93 Gbps.

  • Jumbo Frames (MTU 9000): Configured exclusively on the isolated 2.5 GbE backend to eliminate L2-L4 protocol overhead and maximize TCP payload transfer speed (~2.36 Gbps payload saturation).
  • Zero Default Gateway on 2.5 GbE: The 2.5 GbE network runs strictly flat-L2 without a default route. Internet and management traffic route solely through the 1 GbE interface (MTU 1500), avoiding routing asymmetric bugs or PPPoE related MTU drops at the gateway.

Tales from the Lab: Lockouts & Vogon PKI

The Great pam_faillock Incident: Shortly after setting up the bare-metal OS, a few mistyped passwords during background Ansible tests triggered CachyOS brute-force protection (pam_faillock). The system assumed I was hacking myself and completely locked my personal login account out of sudo! The clean fix was creating a dedicated myansibleuser account with passwordless sudo and dedicated ED25519 SSH keys, keeping personal human accounts separate and password-protected.

Digital Trust & Border Controls

"I previously worked on ReadID at Inverid(now part of Signicat). During a project for a Norwegian bank, I traveled through Harwich border control so often that I became a familiar face. One day, a new Border Force officer asked what I was doing. My answer: 'Checking passports. Just like you.'

Behind him, his supervisor smiled which gave me the confidence that I would catch my train. He didn't know me personally, he trusted the established process. Trust begins with verified identity. In our cluster, certificates do the exact same thing."

To manage TLS certificates across Elasticsearch nodes without taking ourselves too seriously, the automated PKI pays explicit tribute to Douglas Adams’ Hitchhiker’s Guide to the Galaxy. Leaf certificates are issued by the B.I.R.D. (BIRD Interstellar Recursive Defenestration) Root CA under the authority of the Vogon Deep-Space Avian & DTN Transit Board (RFC 1149 / RFC 9171). Behind the jokes lies a zero-touch cryptographic foundation issuing mTLS node certificates and local browser trust configuration.

Repository Branch Breakdown

main Base OS & Kernel Tuning

  • Mitigations Off: Disables Spectre/Meltdown kernel mitigations (mitigations=off) to recover up to 25% CPU performance in an isolated homelab.
  • NVMe Sleep Fix: Sets nvme_core.default_ps_max_latency_us=0 to stop deep power-saving states from dropping drives into read-only mode.
  • Intel Plundervolt Handling: Safely removes software undervolting to prevent systemd service crashes caused by HP BIOS microcode locks.
  • USB Network Routing: Configures 192.168.107.0/24 for inter-node transport with zero default gateway conflicts.

elastic-direct Native Systemd Stack

  • Native Arch PKGBUILDs: Compiles Elasticsearch and Kibana directly into .pkg.tar.zst packages using custom Jinja2 templates.
  • Native Systemd Services: Runs directly on host, without Docker/Kubernetes containers.
  • Automated CA & Token Handling: Generates Root CA certificates, exports browser trust files to ~/Desktop, and fetches kibana-ansible-token dynamically over HTTPS.
  • Dual-Subnet Binding: Restricts node transport sync (port 9300) strictly to the fast 2.5 GbE USB network.
  • UFW managed firewall settings I did not forget setup and teardown of firewall rules.

Declarative Lifecycle Execution

Management is clean, simple, and 100% idempotent:

What's Next? Jina Ebook Vectorization & Cluster Inference

Now that the bare-metal CachyOS cluster is tuned and the isolated 2.5 GbE transport network is active, we are tackling two major workloads across the 3 nodes:

  • Jina AI Ebook Vector Pipeline: Harnessing Jina AI embedding models across both Intel UHD 630 iGPUs (via Vulkan) and AVX2 CPU cores simultaneously. We vectorize entire Calibre EPUB ebooks into Elasticsearch 9.
  • Distributed LLM Inference (160 GB RAM Pool): Clustering mik, mak, and muk over 2.5 GbE RPC to run quantized 70B+ parameter models across pooled host RAM via llama.cpp.

Can integrated mini-PC iGPUs and 2.5 GbE USB networks handle high-throughput RAG indexing and local AI inference? Stay tuned for the next experiment.