Home · Getting Started · Cluster Guide · Troubleshooting · About
Goal: buy everything the cluster needs before you start.
This guide supports two boards: the Raspberry Pi 5, the newest and fastest model, and the Raspberry Pi 4 Model B, which the original cluster ran Hadoop and Spark on (see Builds). Every step works on both; where they differ, the guide shows both.
The table shows which boards suit a Hadoop/Spark cluster.
| Board | RAM options | Power supply | Fit for this cluster |
|---|---|---|---|
| Raspberry Pi 5 | 1, 2, 4, 8 or 16 GB | 5 V 5 A USB-C (27 W) | Supported, fastest. About two to three times as fast as a Pi 4 at training. 4 GB minimum, 8 GB recommended. Needs the Active Cooler under sustained load |
| Raspberry Pi 4 Model B | 1 to 8 GB | 5.1 V 3 A USB-C (15.3 W) | Supported. The original cluster ran on 2 GB boards. 2 GB minimum, 4 GB recommended. Needs heatsinks and a fan under sustained load |
| Raspberry Pi 3 Model B+ | 1 GB | 5.1 V 2.5 A micro-USB | No: 1 GB is too little for Hadoop and Spark together |
| Raspberry Pi Zero 2 W | 512 MB | 5.1 V 2.5 A micro-USB | No: too little memory and no Ethernet port |
Pick one model and use the same amount of RAM on all five nodes if you can. Mixing works, but the memory settings in Spark have to suit the smallest board, and slower workers finish fewer models.
Use the 64-bit edition of Raspberry Pi OS on every board. Java, Hadoop and Spark all need it to use more than a few gigabytes of memory efficiently, and Hadoop publishes ARM 64-bit builds.
This list is for one master and four workers, as in the rest of the guide. Quantities scale with the number of nodes. Where the two boards need different parts, the table has a column for each.
| Component | Raspberry Pi 5 | Raspberry Pi 4 | Qty | Priority |
|---|---|---|---|---|
| Board | Raspberry Pi 5, 4 GB minimum, 8 GB recommended | Raspberry Pi 4 Model B, 2 GB minimum, 4 GB recommended | 5 | Required |
| Power supply | Official 27 W USB-C (5 V 5 A) | Official 15.3 W USB-C (5.1 V 3 A) | 5 | Required |
| Cooling | Raspberry Pi Active Cooler | Heatsinks and a fan, or a case with fans | 5 | Required |
| Storage | 32 GB A2 microSD card, or an NVMe SSD on the M.2 HAT+ | 32 GB A2 microSD card, or a USB 3 SSD | 5 | Required |
| microSD card reader | same | same | 1 | Required |
| 8-port gigabit network switch | same | same | 1 | Required |
| Ethernet cable, Cat 5e or better | same | same | 6 | Required |
| Cluster case (stacking plates with standoffs) | Room above each board for the Active Cooler, and the M.2 HAT+ if used | Room for heatsinks and fans | 1 | Recommended |
| PoE+ HAT, instead of power supplies | A HAT made for the Pi 5 | A HAT made for the Pi 4 | 5 | Optional |
| Keyboard, mouse, monitor | same | same | 1 each | Optional |
| Micro-HDMI to HDMI cable | same | same | 1 | Optional |
Notes:
master runs the NameNode, the Spark master and your driver program. More RAM gives Spark more room; Spark has memory settings for every size. The original cluster worked on 2 GB Pi 4 boards.Prices change often, so check current ones on the Raspberry Pi 5 and Raspberry Pi 4 product pages and your local reseller.
Use the official supply for your board on every Pi:
| Board | Official supply | Five boards can draw up to |
|---|---|---|
| Raspberry Pi 5 | 27 W, 5 V at up to 5 A (product page) | 25 A |
| Raspberry Pi 4 | 15.3 W, 5.1 V at up to 3 A (specifications) | 15 A |
With too little power, the Pis see a voltage drop under load, slow themselves down, and can corrupt their cards. That defeats the point of the cluster, which is to train faster. A multi-port USB charger rarely delivers full current on every port at once, so avoid one unless its spec sheet says it does.
Good options:
Check: after the cluster is running, every node should print throttled=0x0:
cluster-cmd vcgencmd get_throttled
Anything else means a power or heat problem. 0x50005, for example, means under-voltage and throttling are happening now and have happened since boot. Fix power before tuning anything else.
Both boards start slowing their CPU down at 80 °C, and slow further at 85 °C (a Pi 5 drops to about 1.5 GHz). Without cooling, they reach that within minutes of training a model on all four cores.
Check the temperature of every node while a job runs:
cluster-cmd vcgencmd measure_temp
Under about 70 °C at full load is comfortable.
microSD cards are slow at random writes and wear out with heavy writing. HDFS writes data blocks and logs constantly, so:
Flash an SSD with Raspberry Pi Imager the same way you flash a card, using a USB adapter for an NVMe drive.
Skip this unless your cluster already runs cool and stable. It’s not part of building the cluster: come back to it once the cluster runs. Overclocking adds heat and power draw, and a cluster under hours of full load is the worst case for both. Get vcgencmd get_throttled to report throttled=0x0 first (Power).
Overclocking needs good cooling: heatsinks and a fan.
Raspberry Pi 5: don’t overclock a Pi 5 cluster. It already runs at 2.4 GHz, and under hours of full load the gain is small compared with the extra heat and the risk of an unstable node. If you want to try anyway, its settings go in a [pi5] section; see the official overclocking options.
Raspberry Pi 4: overclocking gives a Pi 4 a more noticeable boost. The values and steps below are for the Pi 4, as used in the original build.
| Setting | Default | Reported stable maximum |
|---|---|---|
CPU (arm_freq) |
1500 MHz on early boards, 1800 MHz on newer ones | 2147 MHz |
GPU (gpu_freq) |
500 MHz | 750 MHz |
over_voltage |
0 | 6 |
Check what your board runs at today:
vcgencmd get_config arm_freq
The maximum values come from Tom’s Hardware’s overclocking tests; not every board reaches them.
over_voltage limitsEach step of over_voltage adds 0.025 V. For example, over_voltage=3 adds 0.075 V (75 mV).
Values above 6 are only allowed together with force_turbo=1, and that combination permanently sets a bit in the chip that voids your warranty. Stay at 6 or below. Details are in the official overclocking options.
Open the boot configuration file on the Pi:
sudo nano /boot/firmware/config.txt
(On your computer, it’s config.txt on the card’s bootfs partition.)
Add these lines under the [pi4] section, or add the section if it’s missing:
[pi4]
over_voltage=2
arm_freq=2000
gpu_freq=600
Reboot with sudo reboot.
Check that the new speed applied and that the board isn’t throttling under load:
vcgencmd measure_clock arm
vcgencmd measure_temp
vcgencmd get_throttled
If the Pi doesn’t boot or get_throttled is not 0x0, raise over_voltage by 1 (up to 6) or lower arm_freq. If a Pi won’t boot at all, put its card in your computer and remove the lines from config.txt on the bootfs partition.
Step 1 of 9 · Step 2: Install OS →