pi-cluster

Home · Getting Started · Cluster Guide · Troubleshooting

About

Why this cluster exists, what Hadoop and Spark are, and the history of the build.

Purpose

The main reason for building and configuring this cluster is to leverage Hadoop and Spark’s distributed storage and processing capabilities to build, train, test and assess Machine Learning models in parallel for projects at work and also to demonstrate you may not need to invest in full-powered computers or servers to build a cluster.

I found myself training 311 XGBoost models on time-series financial data that would take about 7 hours to train total. As a result, the goal here is to at least reduce the amount of time it takes to train all models. By having a separate cluster, I allow myself to continue working on other tasks while the cluster takes the load. That is when the idea of a Raspberry Pi cluster kicked in.

Design

The 311 models are independent: each learns from its own series. The cluster therefore splits the work by series and trains one model at a time on each worker, with all four of the worker’s cores working on that one model. Four workers train four models at once.

Hadoop’s file system (HDFS) stores the data and the trained models, and Spark hands out the series to the workers. Learning Hadoop and Spark on real hardware is part of the point of this project, alongside the speed-up. Measuring success explains how to tell whether the cluster is actually faster.

Background: Hadoop and Spark

We live in a digital world full of data. Whether that is un-, semi- or structured data, there are many things we can do with it if we make use of Machine Learning (ML). More specifically, supervised, unsupervised or reinforcement learning algorithms are used to build models that learn (train) and discover patterns from historical data to predict an outcome. There are numerous applications of ML such as weather forecast (prediction), storm tracking models, among others.

It all sounds similar, so here is how the pieces fit together on this cluster:

How Hadoop and Spark fit together: your work runs on Spark, which gets CPU and memory from its own standalone scheduler and reads data from HDFS. MapReduce with YARN is optional. The master runs the HDFS NameNode and the Spark master; each worker runs a DataNode and a Spark worker.

The cluster lets you extract, transform and load (ETL) data and train models with Spark, while HDFS stores the data. Analysis and visualization of the results are up to you.

Builds

The cluster was first built in 2020 on Raspberry Pi 4 boards and ran Hadoop and Spark successfully. The guide has since been updated for current software and for the Raspberry Pi 5, the newest model, while keeping full support for the Raspberry Pi 4. Where the two boards need different parts or settings, the guide shows both.

Build When Boards Operating system Result
Original build 2020 5 × Raspberry Pi 4 Model B, 2 GB Raspbian (2020 release) Worked, including HDFS and Spark. The Hadoop and Spark pages were written later, in the 2026 update
2026 update October 2026 Written for 5 × Raspberry Pi 5 or 5 × Raspberry Pi 4 Raspberry Pi OS Lite 64-bit (Debian 13) Not built on hardware. Steps updated from the original build; the training job was tested on a Spark 4.2 cluster on a regular computer

The original build

The 2020 cluster: five Raspberry Pi 4 boards in a stacked acrylic case with a fan on top, master at the top and four workers below. Each board has its own USB power cable on the left and an Ethernet cable on the right, running down to the network switch the stack sits on.

Front view of the original cluster: five Raspberry Pi 4 boards stacked in a black acrylic case on top of a network switch, with red USB power cables on the left and colour-coded Ethernet cables on the right. Side view of the original cluster powered on, showing the fan on the top plate, Ethernet cables from each board into the switch at the bottom, and the switch's green port lights. The original cluster running next to the monitor it was set up with, with power cables bundled by Velcro straps and red power lights on each board.

Measuring success

The baseline. On my computer, each model took about 81 seconds to train, including cross-validation and a randomized grid search. For all 311 models:

311 models × 81 s ≈ 25,200 s ≈ 7 hours

Success means the cluster trains all 311 models in less than that, measured with the same script and data on both sides.

Why a slower Pi can still win. A Raspberry Pi core is slower than a laptop core, especially on a Pi 4, so one model will usually take longer than 81 seconds on a Pi. That alone doesn’t mean the cluster loses, because four workers train four models at the same time:

  1. Rounds. 311 models ÷ 4 workers = 77¾, so the cluster needs 78 rounds of “one model on each worker”.
  2. Cluster time. Each round takes about as long as one Pi takes to train one model, so the whole run takes about 78 × the time per model on one Pi.
  3. Break-even. To finish under 7 hours, each round must take less than 25,200 s ÷ 78 ≈ 323 s, about 5.4 minutes.

So the line between a win and a loss is about 5.4 minutes per model on one Pi, not 81 seconds:

Time per model on one Pi Cluster time for 311 models Result
81 s or less 78 × 81 s ≈ 1¾ hours or less Clear win, about 4 times faster than one computer
81 s to about 5.4 min between 1¾ and 7 hours Still a win. Each model is slower, but training four at once makes up for it
More than about 5.4 min more than 7 hours Slower than one computer

On the original Pi 4 build, a model took longer than 81 seconds, but the exact time wasn’t recorded, so it’s unknown whether that build beat 7 hours. A Pi 5 is faster, so it has a better chance; the only way to know is to measure it. Workload shows how.

Even when a fast computer is quicker, the cluster still frees that computer for other work while the models train, and each extra worker adds another model in parallel.

Built it with these steps? Please open an issue with your boards, RAM, OS image date and benchmark time, or if a step didn’t work for you.


Back to the Cluster Guide