DIY Cluster “Super” Computer
DIY Cluster Super Computer
Let me start off by saying this is a work in progress. Largely because it is a solution without a specific problem at this point, but in general I wanted to see if I could supplement the performance of my main computer (2015 MacBook Pro) with the use of Raspberry Pi’s in a cluster configuration to take advantage of parallel computing. Ideally this would be used with MATLAB, but as of now MATLAB does not support installation on PI 4’s ARM architecture even though it is 64 bit. Trust me I tried haha. MATLAB does support being compiled elsewhere and run on the PI, but you can’t install their parallel server toolkit. So it will only be used with Python’s ipyparallel for now.
What peaked my interest in this project was the surprising specs the new Pi 4 has for only ~$60/ each:
- Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz
- 1GB, 2GB or 4GB LPDDR4-3200 SDRAM (depending on model)
- 2.4 GHz and 5.0 GHz IEEE 802.11ac wireless, Bluetooth 5.0, BLE
- Gigabit Ethernet
- 2 USB 3.0 ports; 2 USB 2.0 ports.
- Raspberry Pi standard 40 pin GPIO header (fully backwards compatible with previous boards)
- 2 × micro-HDMI ports (up to 4kp60 supported)
- 2-lane MIPI DSI display port
- 2-lane MIPI CSI camera port
- 4-pole stereo audio and composite video port
- H.265 (4kp60 decode), H264 (1080p60 decode, 1080p30 encode)
- OpenGL ES 3.0 graphics
- Micro-SD card slot for loading operating system and data storage
- 5V DC via USB-C connector (minimum 3A*)
- 5V DC via GPIO header (minimum 3A*)
- Power over Ethernet (PoE) enabled (requires separate PoE HAT)
- Operating temperature: 0 – 50 degrees C ambient
- Design Software : Solidworks
- 3D Printer : PRUSA i3 - 15% zigzag fill
- Coding Language : Python --> Jupyter Lab
- Cluster Controller : iPyParallel
I tried to find a good Raspberry Pi 4 cluster case with Power Over Ethernat (POE) hat design on Thingverse or GrabCAD, but was not happy with what I saw. There were a ton of great PI 4 only cases, but I wanted to make sure the POE hat fit. I chose to go the POE hat route vs powering the nodes with a USB hub to make cabling easier and to allow me to power on the nodes via my managed switched. Note this is more expensive though because each POE hat is like $25. Information on my design can be found here: https://www.thingiverse.com/thing:4320050
While I don't intend this to be a complete guide, the basics of getting this setup are below. This assumes you have a main PC that wil run the cluster controller and the PIs will just run the individual engine nodes. I also assume you have an SMB share to allow all nodes access to the same data. 1) Buy all the components & print case (see Thingverse link above) 2) Configure each node: -Attach to router/dns server and assign static routes. I picked names of pi# where # is different for each node -Generic Ubuntu setup found here: https://docs.google.com/document/d/1NSzC07X6Ad6qsCdwTwUVzlNKLFFyvW77p6wUjyOTcPM/edit?usp=sharing -Install and configure iPyParallel on your main machine: https://ipyparallel.readthedocs.io/en/latest/ -Then you should be able to type "ipcluster start" and it will automatically get all of your engines running -See my picture above for code to test to see if all engines are running and accessible
At this point I just have it running. I am now working on benchmarking the performance for various tasks I do commonly (e.g. FFTs, Filtering, etc.. on large amounts of time domain data). I might get into more machine learning, but that would ideally be more for GPUs and I would want a cluster of something like the NVIDIA Jetson and not Pi. Let me know if you have other suggestions of us!