40-Core arm cluster using the nanopc-t3 climbers.net anoxie

Projects » 40-core ARM cluster using the nanopc-T3 building a high performance 5-node cluster for under £100 (£375 including five t3s)

After the interest in my cluster of raspberry pi 3s in april, I was keen to try building clusters with some of the other excellent sbcs now on the market. FriendlyARM in china generously sent me 5 of their latest nanopc-T3 64-bit ARM boards, each with an EIGHT core ARM A53 soc running at 1.4GHz, gigabit ethernet and 8GB of emmc storage onboard.

The completed cluster measures 146.4 (w) x 151 (h) x 216mm (d) and weighs 1.6kg (5.6 x 5.9 x 8.3, 56oz), including the AC-DC power supply.


NanoPC-T3 vs raspberry pi 3

The nanopc-T3 board is a considerable advance over the raspberry pi 3 both in terms of performance and features, but unfortunately this comes at nearly twice the price (at least for those of us in the UK):

anoxie

Model

NanoPC-T3

Raspberry pi 3

SoC

8-core ARM A53

1.4GHz

4-core ARM A53

1.2GHz

Memory

1GB DDR3

1GB DDR2

Network

1000Mbps

100Mbps

WiFi

802.11bgn

802.11bgn

Bluetooth

4.0 dual

4.1 + BLE

Storage

8GB emmc onboard

SD card

MicroSD card

USB spare

2 fitted + 2 headers

1 microusb

4 fitted

Video

HDMI 1.4, LVDS, RGB-LCD, MIPI-DSI

HDMI, DSI

Camera ports

DVP, CSI

CSI

Audio

3.5mm

3.5mm

Microphone

Onboard

No

Size

100 x 60mm

85 x 56mm

Power

1.6 → 4.3W

2A max, DC 4/1.7mm

1.2 → 2.1W

2.5A max, microusb

Launched

Q2 2016

Q1 2016

Price (UK)

£59.02 1

£33.59

1US$60 T3 + US$1.50 heatsink + $8 shipping + 20% VAT + 0% import duty = £59.02

3D design in sketchup

I modified my original raspberry pi cluster design using the free version of sketchup and built rough 3D templates of the nanopc-T3’s, network switch, sockets, etc.Anoxie I didn’t bother to include ventilation slots/grids in the 3D model.

• download SKP file for sketchup 2013

2D design in inkscape

I used the free inkscape application for 2D design, ready for exporting to the laser cutter. Each colour is a different pass of the laser, at different power/speed levels, so the green lines are cut first to make holes for ports/screws/ventilation, pink are extra cuts to help extract delicate parts, orange is text/lines that are etched and finally blue cuts the outside of each panel.

Download files for laser cutting on one 600x400x3mm sheet:

• SVG format sheet or

• DXF format sheet

Read more about the laser cutting and the screwless case-clipping system in my original article.Anoxie I probably went a little overboard with so many ventilation holes, which meant the laser-cutting took 45 minutes to complete.

Design changes from the raspberry pi 3 cluster

Although very similar to my cluster of raspberry pi 3s, I made a number of changes and improvements:

• I kept the horizontal mounting rail design but the T3 has M3 holes which are easier to find parts for, rather than M2.5 on the pi. And the holes are further apart because the overall board size is slightly bigger than the pi. Screwing the plastic nuts onto horizontal rails is a bit tedious (and a tight squeeze to get past the ethernet port on each T3), and I’d like to 3D print a C-shaped ‘clip’ that holds the boards in place along each rail, or perhaps use tight elastic washers.Anoxie

• external PSU vs internal USB hub – the t3s are powered via 4mm/1.7mm DC plugs rather than the pi’s microusb, so it made sense to swap out the internal USB hub power supply for an AC-DC power ‘brick’ that sits outside the case. This gives more space inside the case (for the larger T3 boards, and a fan), and should help with dissipating the heat from the power supply. Each T3 can draw up to a maximum of 2A, but will actually be much less in this cluster, without extra USB and GPIO accessories.

• DC cable splitter. I was pleased to find a ready-made 5-way splitter cable (by myvolts for korg volcas) with the less common 4mm/1.7mm DC plugs that the t3s use. However at 1.75m it is far too long to fit inside the case, and my utter bodge-job of shortening the cable was…Anoxie well, best forgotten . Splitter cable v2.0 was far more successful – a cheap 45cm 6-way splitter cable sold for CCTV cameras is around £2, and cutting off the (wrong size) DC plugs and soldering new right-angled 4mm/1.7mm DC plugs gives a neat result for less than half the price of the myvolts cable. The 6-way cable also allows for a 5V connection to power the ethernet switch, saving on the cost of a separate cable.

• internal case fan vs external – I was sure that the high performance T3 boards would need active cooling, and so I stuck with an ultraquiet 92mm fan, but moved it inside the case, and to the back rather than on top.

• case USB ports – while perfectly functional, I never really liked the look of the twin USB ports on my pi cluster, so for the T3 cluster I sourced a more compact 2-port USB mount.Anoxie

• right-angle USB extenders – I didn’t realise until everything else was finished that the USB cables would stick up too far vertically to shut the case lid (!) so I had a last-minute purchase of two right-angle extenders so I could connect up the USB. A neater solution would be making/buying a single cable.

• no internal shelf vs shelf – not needing a shelf to attach the USB hub to simplifies the design, and also means it can be cut from a single sheet of 600x400mm acrylic. Removing the shelf would have reduced the rigidity of the case, but screwing the horizontal mounting rails to the side panels keep it secure.

• flat 25cm LAN cables vs round 15cm – I loved the rainbow LAN cables in my rpi3 build, but it was a tight fit bending them in the case.Anoxie these flat cables bend far more easily, and 25cm is enough extra length to reach the LAN ports on the top of

The T3 board.

• green LAN cables vs boring grey – the green really brings out the blue of their eyespcbs … Plus the friendlyarm logo is blue+green.

• gigabit vs 10gigabit-switch – the T3 network ports are each 1000mbps (10x faster than the pi), so using at least a 1000mbps switch is a no-brainer really. A 10gbps switch would eliminate any bottleneck (e.G., if the T3’s were simultaneously saturating their links talking to the outside network), however these are still very expensive at £600+, and too large to fit inside this case. The new asus XG-D2008 switch looks promising.

Read about some of the design choices on the original pi cluster.Anoxie

Power, temperature cooling

At idle, the entire system of five t3s, network switch 5V fan sips a mere 13W, and at 100% load it still only uses 31W in total. There is the possibility of further reducing the power requirements by disabling wifi, bluetooth and HDMI?

Is a heatsink worth it? With double the number of cores, the T3 soc can generate far more heat than the pi3, so I’d suggest a heatsink is pretty much an essential purchase. Luckily friendlyarm sell a substantial heatsink with thermal paste for only US$1.50 that securely clips onto the T3 board. It is much bigger than other aftermarket SBC heatsinks I’ve seen, and in my tests it reduced the soc temperatures by a remarkable 10-35°C !

The power adaptor can supply up to 50W (2A per T3), which is more than enough for a couple of power-hungry external devices to be plugged into the USB ports.Anoxie using:

Cat /sys/class/hwmon/hwmon0/device/temp_label

To measure the soc core temperature, the cluster idles at 36°C (97°F) with cooling from the 5V fan.

At 100% load and the 5V fan, using:

Sysbench –test=cpu –cpu-max-prime=200000 –num-threads=8 run

The soc core temperatures reached a stable 68°C (154°F). If you don’t provide at least some active cooling, the socs will get as high as 85°C and automatically throttle down their clock speed, to avoid overheating. They can safely run long-term at that temperature, but you don’t get maximum performance.

Exactly the same case design should work with nanopc t2s and t1s which run cooler than the T3, and so might not need a fan to cool them. For cooling a single T3 you could use a much smaller fan, perhaps 40-60mm.Anoxie

Silent cooling:

To cool down the cluster I fitted a single 92mm fan inside the rear of the case. I used a (effectively) silent fan recommended by quietpc.Com, the nanoxia deep silence (£10.15). At 5V I have to get my ear within 50-75mm (2-3) to hear even the slightest whisper from the fan, and the supplied rubber gromets definitely do a good job of isolating the case from any small vibrations.

Description

Heatsinks?

Idle

100% load

Performance

Case, 12V fan, 1400 rpm

No

39°C

85°C

Throttles

Case, 12V fan, 1400 rpm

Yes

30°C

50°C

OK

Case, 7V fan, 800 rpm

Yes

33°C

59°C

OK

Case, 5V fan, ? Rpm

Yes

36°C

68°C

OK

Unlike PC motherboards, a T3 doesn’t have a standard 3 or 4-pin fan header, but it is quite easy to power the fan using the GPIO 5V power rail instead.Anoxie WARNING you can’t run anything that needs a lot of current this way – however a low-power fan should be fine. You do need to make sure you have a quality power supply, and not too many power-hungry USB devices plugged into your T3, otherwise you’ll get crashes rebooting. The main GPIO pins are for data, but using the VDD_5V power rail on pin 29 should work in theory.

This should be enough to run the fan with 5V, or if you need more cooling, splice a step-up/boost converter in the middle to adjust the speed of the fan, by controlling its voltage between 5V and 12V.

Building the T3 cluster

• remove network switch case (2 small screws)

Attach the network switch to case base, using 4x 6mm brass spacers + 4x 14mm bolts + nuts.Anoxie this only fits one way around

• cut off only 5 (of the 6) plugs on the DC power splitter cable, and solder on 5 new 4.0/1.7mm right angled plugs

Bolt DC socket port to case back

Cut off the combined end of the splitter cable, and solder onto the DC screw mount socket.

• screw 2 external LAN ports to inside of the case back (has “DC5V 10A” etched on the outside)

Fit the 92mm case fan to inside of the case back using the supplied rubber grommets. The fan logo should face inwards if you want the fan to blow cold air into the case. The rubber grommets might need to be pushed into place with a small screwdriver/pen – don’t pull too hard!

Clip case back to case base (marked C+D)

Plug external LAN into network switch ports 1+3 (no room to use 1+2)

anoxie

• place the pads of heatsink compound to the soc chip of each T3, and firmly clip on each heatsink

Slide 5 T3’s onto 4 threaded rods using 48 nuts to secure. This might be faster with the rod in an electric screwdriver? Leave 30mm space at left end, space each T3 25mm apart. The LAN+USB side of the T3’s point upwards and towards the front of the case

Attach case sides to T3 rods ( EF+AB), secure with 8 metal nuts.

• fit the 3-pin fan plug into the GPIO pins of one of the T3’s, so that black (ground) connects to pin 30, red (power 5-12V) to pin 29, and yellow (fan speed) hangs unconnected over the PCB.

• plug T3 LAN cables into network switch ports 4-8 (be very careful if removing these – it is easy to break tiny plastic clips on the switch ports)

anoxie

Clip case sides to case bottom ( E+F, A+B)

• route HDMI cable through case

Screw external twin USB to front case

Bolt external HDMI to front case, 2 nuts

Plug 2 external usbs + right-angle adaptors into any T3 (or 2 separate T3’s)

Plug T3 LAN cables into T3’s

Plug T3 DC power cables into T3’s

Plug HDMI cable + right-angle adaptor into any T3

Clip case front to case bottom ( G+H)

Clip case lid to case sides, front back ( M+N+O+P+Q+R+S+T)

Attach self-adhesive rubber feet to underside of base

Bill of materials

Most of these parts were sourced from individual sellers on amazon or ebay, which of course racks up the postage charges. If there were enough demand, it would be cheaper to bulk buy the parts and have a kit with everything you need to build the cluster.Anoxie

Edimax ES-5800G V3 gigabit ethernet switch

£18.45

Flat 25cm cat6 LAN cables (5 pack)

£6.80

5V 10A AC-DC external power supply, 5.5/2.5mm plug

£12.99

5.5/2.5mm chassis mount DC socket (1 from a 2 pack)

£1.95

6-way DC power splitter cable

£2.05

DC plug right angled 4/1.7mm (5 pack)

£3.49

M3 steel screws 8mm (4 from a 10 pack)

£1.10

RJ45 male to female screw mount (2 pack)

£1.98

M3 steel hex nuts (4 from a 5 pack)

£1.10

M3 steel bolts 14mm (4 from a 5 pack)

£1.35

M3 brass female standoff 6mm (4 from a 10 pack)

£0.99

M3 steel threaded bar 150mm inc. Nuts (4 from a 5 pack)

£5.50

0.5m HDMI male to female panel mount (inc. Bolts)

£2.39

Twin USB female socket to male cable (inc. Screws)

£2.19

USB right-angle male to female socket extender (2 pack)

anoxie

£1.98

M3 nylon hex nuts (48 from a 50 pack)

£3.45

3mm extruded clear perspex 600x400mm

£5.32

Laser cutting charge

N/a

HDMI 270 degree adaptor

£0.99

Nanoxia 92mm deep silence 1400rpm fan

£10.15

Polyurethane rubber feet (4 from a 12 pack)

£2.99

Subtotal inc PP

£87.21

NanoPC-T3 at US$60/each (5 pack) 1

£272.78

NanoPC-T3 heat sink 5mm (5 pack)

£6.24

Total inc PP

£366.23

1 the nanopc-T3 is duty free to import into the UK, and only costs US$21 to ship 5 boards from china, but then there is UK VAT at 20%, bringing the total to £279.02.

The cluster of t3s (excluding the boards themselves) is slightly cheaper to build than the pi3s: you don’t need a USB hub, separate microusb cables, SD cards for each board, etc. However it is a shame they used 4/1.7mm DC power connectors, rather than the far more common 5.5/2.5mm or 5.5/2.1mm sizes.Anoxie

A cheaper high-performance option? The nanopi M3 was announced by friendlyarm after this T3 cluster was built, and looks like a great alternative for a low-cost, compact cluster. It has the same 8-core 1.4GHz soc and gigabit ethernet, but on a smaller board with fewer extra interfaces and nearly half the price – a 5-node cluster should give the same performance, but for a total of £260 instead of £366.

Show me the benchmarks!!

I’m writing a separate article benchmarking this nanopc-T3 cluster against my original raspberry pi 3 cluster, as well as detailed benchmarks comparing the individual boards. Watch this space…

Clusters of other single board computers

So far I’ve also built:

• DIY 5 node cluster of raspberry pi 3s

anoxie

• 40-core ARM cluster using the nanopc-T3

• 5 node cluster of orange pi plus 2e

• bargain 5 node cluster of PINE A64+

• clusterhat with 4x raspberry pi zero

The nanopc-T3, orange pi plus 2e and PINE A64+ clusters mostly share the same components, including the acrylic case panels – only the 2 side panels are unique because the boards are different sizes. I’m planning to revise my original raspberry pi cluster design so it will share most of these same case/components.

I’d like to build a small cluster of all the current crop of sub-$100 ARM sbcs, comparing the different features, and with detailed benchmarks. E.G., odroid C2/XU4 and the banana pi M3. Please email me if you’d like to send boards for review.Anoxie

Software to run on a cluster?

Or… What is it for??

Education, training, blah, blah… Well personally I’m just running a debian linux provided by friendlyarm on each T3 for now, and I’m going to experiment with things like load-balanced web/database servers.

Running docker on ARM on each node looks like an excellent way of controlling the cluster.