Home · Getting Started · Cluster Guide · Troubleshooting · About
Goal: Java on every node, and Hadoop’s file system (HDFS) running on master alone, so you can confirm Hadoop works before adding the workers.
Run on: master. Java goes to every node with cluster-cmd.
Before you start: The Pi Network, Cluster Commands and Cluster Security are done, and nodes-up shows every node up. Versions are listed in Software.
Hadoop 3.5 needs Java 17 exactly. Raspberry Pi OS offers Java 21 and 25 but not 17, so install Eclipse Temurin 17 from the Adoptium repository.
cluster-cmd 'sudo apt install -y wget gpg'
cluster-cmd 'wget -qO- https://packages.adoptium.net/artifactory/api/gpg/key/public | sudo gpg --dearmor --yes -o /usr/share/keyrings/adoptium.gpg'
cluster-cmd 'echo "deb [signed-by=/usr/share/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb $(. /etc/os-release && echo $VERSION_CODENAME) main" | sudo tee /etc/apt/sources.list.d/adoptium.list'
cluster-cmd 'sudo apt update && sudo apt install -y temurin-17-jdk'
Check:
cluster-cmd 'ls -d /usr/lib/jvm/temurin-17-jdk-*'
Every node should print /usr/lib/jvm/temurin-17-jdk-arm64. That folder is JAVA_HOME below.
If apt update says the repository has no Release file for your OS codename, Adoptium hasn’t published that codename yet: replace it with the previous Debian codename (bookworm) in /etc/apt/sources.list.d/adoptium.list and run the last command again.
Download the ARM 64-bit build and its checksum, then check that the download is complete and unmodified:
cd /tmp
wget https://dlcdn.apache.org/hadoop/common/hadoop-3.5.0/hadoop-3.5.0-aarch64.tar.gz
wget https://downloads.apache.org/hadoop/common/hadoop-3.5.0/hadoop-3.5.0-aarch64.tar.gz.sha512
sha512sum -c hadoop-3.5.0-aarch64.tar.gz.sha512
The last command must print hadoop-3.5.0-aarch64.tar.gz: OK.
Unpack it into /opt/hadoop, owned by your user so Hadoop can write its logs there:
sudo mkdir -p /opt/hadoop /opt/hadoop_data
sudo chown "$USER": /opt/hadoop /opt/hadoop_data
tar -xzf hadoop-3.5.0-aarch64.tar.gz -C /opt/hadoop --strip-components=1
Keep the .tar.gz in /tmp for now: Cluster Setup copies it to the workers.
Put the cluster’s settings in their own file, ~/.cluster_env:
cat > ~/.cluster_env <<'EOF'
# pi-cluster environment
export JAVA_HOME=/usr/lib/jvm/temurin-17-jdk-arm64
export HADOOP_HOME=/opt/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/opt/spark
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SPARK_HOME/bin
EOF
Load it from the first line of ~/.bashrc:
sed -i '1i [ -f ~/.cluster_env ] && . ~/.cluster_env' ~/.bashrc
source ~/.bashrc
It must go at the top, not the bottom. ~/.bashrc on Raspberry Pi OS stops early for commands sent over SSH (like cluster-cmd hdfs ...), so anything below that point wouldn’t apply to them.
Check: hadoop version prints Hadoop 3.5.0.
HDFS keeps two kinds of data:
master, records which file is made of which blocks and where each block is stored.master is its own DataNode; in Cluster Setup the workers take over.All Hadoop settings live in $HADOOP_CONF_DIR (/opt/hadoop/etc/hadoop).
Java and memory. Append to hadoop-env.sh:
cat >> $HADOOP_CONF_DIR/hadoop-env.sh <<'EOF'
# pi-cluster
export JAVA_HOME=/usr/lib/jvm/temurin-17-jdk-arm64
export HDFS_NAMENODE_OPTS="-Xmx512m"
export HDFS_SECONDARYNAMENODE_OPTS="-Xmx256m"
export HDFS_DATANODE_OPTS="-Xmx256m"
EOF
Hadoop gives each service up to a quarter of the machine’s memory by default. These limits keep the services small so Spark and your models get the rest.
Where HDFS lives. Replace core-site.xml:
cat > $HADOOP_CONF_DIR/core-site.xml <<'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop_data/tmp</value>
</property>
</configuration>
EOF
Where data is stored, and how many copies. Replace hdfs-site.xml:
cat > $HADOOP_CONF_DIR/hdfs-site.xml <<'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///opt/hadoop_data/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///opt/hadoop_data/datanode</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
EOF
dfs.replication is how many copies of each block HDFS keeps. With a single DataNode, only 1 is possible.
Which nodes are DataNodes. For now, only master:
echo master > $HADOOP_CONF_DIR/workers
Format the NameNode. This creates an empty file system; do it once, since formatting again erases everything HDFS stores:
hdfs namenode -format
Look for has been successfully formatted near the end.
Start HDFS:
start-dfs.sh
Check the running services with jps. It should list NameNode, DataNode and SecondaryNameNode. If one is missing, its log in /opt/hadoop/logs/ says why.
Store a file and read it back:
hdfs dfs -mkdir -p /user/$USER
echo "hello from the pi cluster" | hdfs dfs -put - /user/$USER/hello.txt
hdfs dfs -cat /user/$USER/hello.txt
Check the report shows one live DataNode:
hdfs dfsadmin -report | grep "Live datanodes"
You can also open the NameNode’s web page from your computer at http://master.local:9870.
A warning such as Unable to load native-hadoop library is harmless: Hadoop falls back to Java code.
Stop HDFS before moving on:
stop-dfs.sh
Step 6 of 9 · ← Step 5: Cluster Security · Step 7: Cluster Setup →