Home · Getting Started · Cluster Guide · About
Find the symptom, then try the fix. Most problems show up in a log: Hadoop’s are in /opt/hadoop/logs/ and Spark’s in /opt/spark/logs/, on the node where the service runs.
| Symptom | Likely cause | Fix |
|---|---|---|
WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! |
The card was reflashed, so the node has new SSH host keys | On the computer that warns: ssh-keygen -R worker1, plus ssh-keygen -R worker1.local and the IP if you used those, then connect again |
ssh: Could not resolve hostname master.local |
Your network or computer doesn’t support .local names |
Use the node’s IP address from your router’s device list |
Permission denied (publickey) |
Your key isn’t on that node | Redo SSH keys from master, or reflash with your key in Imager |
nodes-up shows a node DOWN |
Off, still booting, cable or switch port problem, or a changed IP address | Check the Pi’s green light and the switch port light; find its address in the router; Give each node a fixed address |
sudo: unable to resolve host worker2 |
The node’s name is missing from the hosts block | Check hostname, then run setup_hosts.sh again on that node |
| The hosts block disappears after a reboot | Only /etc/hosts was edited, and cloud-init rebuilt it from its template |
Run setup_hosts.sh, which also updates the template |
cluster-cmd prints !! worker3: command failed |
The command failed there, or SSH couldn’t connect within 5 seconds | Run the command on that worker directly (ssh worker3 '<command>') to see its error |
A sudo command fails over cluster-cmd with a password is required |
Your user needs a password for sudo |
Run it per node with ssh -t, or see Cluster Security |
| Lost SSH access after turning on the firewall | SSH wasn’t allowed before ufw enable |
Connect a keyboard and monitor and run sudo ufw disable, or reflash the card; then follow Firewall in order |
| Symptom | Likely cause | Fix |
|---|---|---|
vcgencmd get_throttled isn’t throttled=0x0 |
Under-voltage or overheating | See Power and Cooling. If the last hex digit is odd (1, 3, 5…), the Pi has too little power right now; if the fifth digit from the right is odd, it has had too little since boot |
| Random crashes, corrupted cards, read-only file system | Weak power supply, or a worn card | The official supply for each Pi (Power); replace the card; consider an SSD |
| One worker trains far fewer models than the others | That worker is throttling | cluster-cmd vcgencmd measure_temp during a job; improve its cooling |
| Symptom | Likely cause | Fix |
|---|---|---|
hdfs dfsadmin -report shows fewer than 4 live DataNodes |
DataNode didn’t start, or can’t reach master |
Read the newest hadoop-*-datanode-*.log on that worker |
DataNode log says Incompatible clusterIDs |
The NameNode was formatted again after the DataNode first joined | On master: stop-dfs.sh; then master-cmd 'rm -rf /opt/hadoop_data/datanode' and start-dfs.sh. This deletes all HDFS data |
DataNode log shows Connection refused to master:9000 |
The NameNode listens on the wrong address, or the firewall blocks it | On master, ss -tln \| grep 9000 must show 192.168.0.100:9000, not 127.0.1.1. If not, run setup_hosts.sh on master and restart HDFS. Check that ufw allows your subnet |
Name node is in safe mode |
Normal for a minute after start, while DataNodes report in | Wait, or check with hdfs dfsadmin -safemode get. If it stays on, some DataNodes are missing |
JAVA_HOME is not set or UnsupportedClassVersionError |
Wrong or missing Java | cluster-cmd 'ls -d /usr/lib/jvm/temurin-17-jdk-*'; JAVA_HOME in hadoop-env.sh must point to it (Single Node Setup) |
hdfs: command not found over SSH |
~/.cluster_env isn’t loaded at the top of ~/.bashrc |
Check head -1 ~/.bashrc on that node; see Set environment variables |
| Symptom | Likely cause | Fix |
|---|---|---|
| Spark master page shows 0 or too few workers | Worker can’t reach master:7077, or didn’t start |
Read /opt/spark/logs/*Worker*.out on that worker; check the firewall and SPARK_MASTER_HOST in spark-env.sh |
Initial job has not accepted any resources |
The job asks for more memory or cores than a worker offers | spark.executor.memory must be at most SPARK_WORKER_MEMORY, and spark.executor.cores at most SPARK_WORKER_CORES (Configure Spark) |
ModuleNotFoundError: No module named 'xgboost' (or pandas) |
Workers use a Python without the packages | Check PYSPARK_PYTHON=/opt/pyenv/bin/python in spark-env.sh on every node, and the import check in Python on every node |
Python in worker has different version than that in driver |
Nodes run different Python versions | Use the same Raspberry Pi OS release on every node, and the same /opt/pyenv |
ExecutorLostFailure or a worker drops out mid-job |
Out of memory: Linux stopped a process | ssh worker1 'sudo dmesg \| grep -i killed'; use the smaller row in the memory table, or 4 GB boards |
max concurrent is more than 1 on a worker |
spark.executor.cores and the script’s cores per model differ |
Keep spark.executor.cores 4 and run train_models.py without --threads (it defaults to 4) |
| Training is much slower than the benchmark | Throttling, or more threads than cores | Check temperature and power; don’t raise --threads above the worker’s core count |
Back to the Cluster Guide