-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy pathindex.html
1662 lines (1500 loc) · 69.4 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Building a Little Cluster</title>
<meta name="description" content="a guide to setting up a raspberry pi cluster">
<link rel="stylesheet" href="css/main.css">
</head>
<body>
<img src="images/walogo.png"/>
<h1>Raspberry Pi Cluster Setup</h1>
<ul>
<li><a href="#intro">Setting up a Raspberry Pi Cluster</a></li>
<li><a href="#command">Command Terminals</a></li>
<li><a href="#materials">Materials</a></li>
<li><a href="#connections">Connecting the Pis</a></li>
<li><a href="#mainnode">Setting Up the Main Node</a></li>
<li><a href="#workers">Setting up the worker nodes</a></li>
<li><a href="#bench">Benchmarking</a></li>
<li><a href="https://www.epcc.ed.ac.uk/discover-and-learn/resources-and-activities/what-is-a-supercomputer/wee-archie">Wee Archie</a></li>
<!--TODO Add link to command glossary which I am going to make-->
<!--<li>Glossary of commands</li>-->
</ul>
<h2>Version and Authorship</h2>
<table>
<tr><td><b>Version</b></td><td>0.96</td></tr>
<tr><td><b>Author</b></td><td>A Grant (EPCC)</td></tr>
<tr><td><b>Contributors</b></td><td>M Antonioletti (EPCC), O T Brown (EPCC), J Kennedy (EPCC), G Gibb (EPCC), R MacLeod (EPCC)</td></tr>
<tr><td><b>Notes</b></td><td>Linux Network Connection to be completed. </td></tr>
<tr><td><b>Notes</b></td><td>Raspbian SSH Access Note added. </td></tr>
</table>
<a name="intro"><h2>Before You Start</h2></a>
<p>
These instructions will help you set up a cluster of Raspberry Pi computers as a small parallel system. They will take you through the process of connecting the Pis together, setting up the environment and checking that it works correctly.
</p>
<p>
These instructions are not for complete beginners with computers - they assume knowledge of concepts such as networks, command line and file systems. There are some explanations in this document, but it still require some experience and a willingness to try things out.
</p>
<p>
<b>Note:</b> these instructions deal with a cluster of 5 Raspberry Pis but the cluster can be of any size you want (minimum of 2, as otherwise it is not a parallel computer!)
</p>
<p>
<b>Note:</b> Raspbian refers to the version of Linux commonly distributed with the Raspberry Pi computers. New Out Of Box Software (NOOBS) is an easy-to-use installation manager for Raspian which you can use if you're setting up your Pis for the first time.
Please ensure you have Rasbian installed on each SD card before you begin and the SD card is inserted into a Raspberry Pi.
</p>
<p>
<b>Note:</b>If you try these instructions and find any issues or have suggestions, please submit an issue to <a href="https://github.com/EPCCed/wee_archlet/issues">our GitHub repo</a>.
</p>
<a name="command"><h3>Command Terminals</h3></a>
<p>
This document will refer to command terminals throughout, when accessing, configuring, and controlling the Raspberry Pis from your computer. A command terminal provides a textual interface to a user, rather than a graphical interface that you might be more used to. Command line interfaces often look like:
<figure>
<img src="images/terminal.png" alt="Terminal"/>
<figcaption>Example Terminal</figcaption>
</figure>
Sometimes they will have light text on a dark background - you can configure them to your own liking. Regardless, a Raspberry Pi uses the Linux operating system and will have a terminal that functions and behaves in the same way this document describes.
</p>
<p>
On a Linux system, you will need to open a command terminal on your computer - you may be able to get this using a right mouse click and choosing "Terminal", or from a list of programs in the program menu.
</p>
<p>
On Windows 10 you can use PowerShell, which you can find through the search feature in the taskbar. If you prefer, you can install a Secure Shell (SSH) client to connect to the Raspberry Pis. Two free options are: - <a href="http://www.chiark.greenend.org.uk/~sgtatham/putty/">PuTTY</a>, and <a href="https://mobaxterm.mobatek.net/">MobaXterm</a>. These instructions assume that you are working from the command line, so if you are using an SSH client please check its documentation so you know how it works.
</p>
<p>
On a Mac OS machine, Terminal is under Utilities in the Applications folder.
</p>
<a name="materials"><h2>Materials</h2></a>
<h3>You will need:</h3>
<ul>
<li>5 Raspberry Pis (Note the examples used in these instructions are Raspberry Pi 3s - the instructions should work for most Raspberry Pi models. If problems are encountered please submit an issue to <a href="https://github.com/EPCCed/wee_archlet/issues">Cluster Instructions</a>).</li>
<li>5 Power Supplies/1 Multisupply Unit.</li>
<li>1 Switch (with at least 8 Ports).</li>
<li>6 Ethernet Cables.</li>
<li>5 Power cables. </li>
<li>Laptop or desktop PC with a shareable ethernet connection.</li>
</ul>
<h3>Optionally:</h3>
<ul>
<li>5 Raspberry Pi cases - we've used <a href="http://cpc.farnell.com/multicomp/cbpiblox-red/pi-blox-enclosure-red-rev3/dp/SC14019">Lego-Style cases</a>, which are stackable.</li>
</ul>
<h3>Suitable and Unsuitable Raspberry Pi Models</h3>
<p>
<b>NOTE:</b>There are many products with the Raspberry Pi name so this is a list of the ones that are suitable and unsuitable for these instructions.
</p>
<p>
Suitable Raspberry Pi Models (these instructions work for these types of Raspberry Pi):
</p>
<ul>
<li>Raspberry Pi 3</li>
<li>Raspberry Pi 2</li>
</ul>
<p>
Unsuitable Raspberry Pi Models (these instruction will not work or require substantial changes and/or additional hardware):
</p>
<ul>
<li>Raspberry Pi Zero</li>
<li>Raspberry Pi Zero W</li>
<li>Raspberry Pi Compute Module</li>
</ul>
<p>
At the time of writing there are three types of Raspberry Pi in the market - to distinguish between them we provide the following images:
</p>
<figure>
<img src="images/11546-04.jpg" alt="Raspberry Pi 1"/>
<figcaption>Raspberry Pi 1 Model B</figcaption>
</figure>
<figure>
<img src="images/Raspberry-Pi-2-Bare-BR.jpg" alt="Raspberry Pi 2"/>
<figcaption>Raspberry Pi 2 Model B</figcaption>
</figure>
<figure>
<img src="images/Raspberry_Pi_3_Model_B.png" alt="Raspberry Pi 3"/>
<figcaption>Raspberry Pi 3 Model B</figcaption>
</figure>
<p>
If you are using cases - this is the point where you should put the Raspberry Pi into its case. An example of a cased Raspberry Pi would look like this:
</p>
<figure>
<img src="images/IMG_20160906_105359475.jpg" alt="Cased Raspberry Pi Open"/>
<figcaption>Lego Style Case - Open</figcaption>
</figure>
<figure>
<img src="images/IMG_20160906_105327393.jpg" alt="Cased Raspberry Pi Closed"/>
<figcaption>Lego Style Case - Closed</figcaption>
</figure>
<h3>Materials used in these instructions:</h3>
<p>
We've used Raspberry Pi 3s for our cluster. In place of 5 mains power supplies, we've chosen <a href="http://www.farnell.com/datasheets/1948264.pdf">1 Multi-USB Power Socket</a>. Note - This particular model appears to have been discontinued. Each pi is connected to this via a USB to B-type Micro USB cable. Our switch is a Netgear GS108E.
</p>
<h3>Important Note for New Raspbian Installations</h3>
<p>
Due to changes in the recent versions of Raspbian and the NOOBS distributions, to follow these instructions you have to make a change to your Raspberry Pi SD Card before you boot it up for the first time.
</p>
<p>
<!--We may need to expand more on this step, as it does not go into as much detail as others in this document-->
Newer versions of Raspbian have SSH access disabled by default. To reenable it, you will need to insert the microSD cards containing Raspbian into your computer and create a blank text file called 'SSH' in the boot partition. This will be the disk that appears in your file system when you insert the microSD card.
</p>
<p>
Once you have created the file, you will be able to boot Raspbian with SSH remote access enabled.
</p>
<h2><a name="connections">Connecting the Raspberry Pi Cluster</a></h2>
<ol>
<li>Connect the ethernet cables to the ethernet ports:</li>
<figure>
<img src="images/IMG_20170210_132818620.jpg" alt="Ether Pi"/>
<figcaption>Connect Ethernet to Pi</figcaption>
</figure>
<li>Connect the ethernet cables to the switch:</li>
<figure>
<img src="images/IMG_20170210_132909899.jpg" alt="Ether Switch"/>
<figcaption>Connect Ethernet to Switch</figcaption>
</figure>
Now it should look something like:
<figure>
<img src="images/IMG_20170210_132914225.jpg" alt="Ethernet"/>
<figcaption>Connected</figcaption>
</figure>
<li>Connect your laptop or desktop to the switch via an ethernet cable. (Note that while Raspberry Pi 3 has a built-in wireless network - we are not using this and at this point it probably would not have been set up.)</li>
<li>Now all the network cables are connected - we need to put the power cables in. First connect the power cable for the switch and turn it on:</li>
<figure>
<img src="images/IMG_20170210_132948565.jpg" alt="Ethernet"/>
<figcaption>Switch Power</figcaption>
</figure>
<li>Now connect each of the Raspberry Pis to your power supply and turn them on. In this case we are using an externally powered USB-hub:</li>
<figure>
<img src="images/IMG_20170210_133056149.jpg" alt="Ethernet"/>
<figcaption>Raspberry Pi Power</figcaption>
</figure>
<figure>
<img src="images/IMG_20170210_133126552_HDR.jpg" alt="Ethernet"/>
<figcaption>Plug in</figcaption>
</figure>
And that's it! All the Pi units should have LED lights on or flashing, and look something like this:
<figure>
<img src="images/IMG_20170210_133145864.jpg" alt="Ethernet"/>
<figcaption>Our Cluster</figcaption>
</figure>
</ol>
<h2><a name="sharing">Network Software Setup</a></h2>
<p>
<b>Note: </b>These instructions assume you are using a computer connected to the internet via Wi-Fi.
</p>
<p>
First on your laptop/desktop you will have to configure the internet sharing, so that the Raspberry Pi's can connect to the Wifi via an ethernet connection to your laptop or desktop. How you do this depends on your operating system:
</p>
<h3>OS X (Mac OS)</h3>
<ol>
<li>Go to System Perferences and choose 'Sharing'.</li>
<figure>
<img src="images/mac-preferences.png" alt="Mac preferences"/>
<figcaption>Mac preferences</figcaption>
</figure>
<li>Click on Internet Sharing and choose share from Wifi to Ethernet.</li>
</ol>
<h3>Windows 10</h3>
<ol>
<li>Go into 'Network and Sharing Centre' in the Control Panel</li>
<li>Choose 'Change adapter options' from the options on the left, and right-click on the Wi-Fi connection (identifiable by a signal strength indicator in the icon, wired networks will not have this). Select 'Properties' from the menu.</li>
<li>Select the Sharing Tab, and from there check the box at "Allow other network users to connect through this computer's Internet Connection".</li>
<li>Ensure that the Home networking connection is your Ethernet network in the drop-down menu below.</li>
<li>Close the Wi-Fi Properties window to return to the Network Connections. Right-click on the Ethernet connections and select 'Properties' again you have just done for Wi-Fi.</li>
<li>In the list headed "This connection uses the following items", click on the one called "Internet Protocal Version 4". This will enable a 'Properties' button just below, click on this.</li>
<li>Take a note of the IP Address and Subnet Mask listed. You can now close down all the windows you've opened.</li>
</ol>
<h3>Linux</h3>
<h4>Ubuntu</h4>
<ol>
<li>Open a terminal and run the command <pre>nm-connection-editor</pre> to open the network connection editor.</li>
<li>Select your internet connection, then click the cog icon to edit the connection settings.
<figure>
<img src="images/ubuntu_nm-connection-editor.png" alt="nm-connection-editor"/>
<figcaption>Edit network connection (note that this screenshot was taken on a virtual machine and so shows a Wired Connection, not Wi-Fi, which is what you should be using).</figcaption>
</figure>
<li>Go to the "IPv4 Settings" tab and select "Shared to other computers" from the "Method" drop-down list</li>
<li>Click "Save", and close the connection editor</li>
</ol>
<h2>Getting into the Pi</h2>
<p>
The next step is to obtain the IP addresses for each Raspberry Pi in your cluster. Again, how you do this will depend on what operating system you use.
</p>
<p>
IP addresses are what we can use to identify a Pi on the network and use it as a target to log in - its the same as a street address when going somewhere - though IP addresses can change.
A brief explanation of IP Addresses and DHCP are available at the following links:
</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/IP_address">IP Address</a></li>
<li><a href="https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol">DHCP</a></li>
</ul>
<h3>MacOS and Linux</h3>
<p>
For the setup used here, the shared network connection showed up as an adapter called <code>bridge100</code> after running <code>ifconfig</code>. This is what allows the WiFi connection to be shared to the Ethernet connection.
</p>
<p>
<b>Note:</b> Bridge100 may be using some other name on your computer, especially if you are using Linux. To check which interface is created - type <code>ifconfig</code> before turning on your sharing or plugging in your adapter then run it again afterwards to find out what difference the changes made.
</p>
<p>
<b>Note:</b> On more recent versions of Linux (Ubuntu 18.04+) ifconfig is deprecated. Instead use:
<pre>ip -c addr show</pre>
</p>
<ol>
<li>Open a command terminal:</li>
<figure>
<img src="images/terminal.png" alt="Terminal"/>
<figcaption>Example Terminal</figcaption>
</figure>
<li>Type: <code>ifconfig</code>. This should output something like:</li>
<pre>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
nd6 options=1<PERFORMNUD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ether a4:5e:60:e7:21:0f
inet6 fe80::a65e:60ff:fee7:210f%en0 prefixlen 64 scopeid 0x4
inet 172.20.152.14 netmask 0xfffff000 broadcast 172.20.159.255
nd6 options=1<PERFORMNUD>
media: autoselect
status: active
en1: flags=963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX> mtu 1500
options=60<TSO4,TSO6>
ether 6a:00:00:41:c6:f0
media: autoselect <full-duplex>
status: inactive
en2: flags=963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX> mtu 1500
options=60<TSO4,TSO6>
ether 6a:00:00:41:c6:f1
media: autoselect <full-duplex>
status: inactive
bridge0: flags=8822<BROADCAST,SMART,SIMPLEX,MULTICAST> mtu 1500
options=63<RXCSUM,TXCSUM,TSO4,TSO6>
ether a6:5e:60:7e:1d:00
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
ipfilter disabled flags 0x2
member: en1 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 5 priority 0 path cost 0
member: en2 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 6 priority 0 path cost 0
media: <unknown type>
status: inactive
p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304
ether 06:5e:60:e7:21:0f
media: autoselect
status: inactive
awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1484
ether ca:90:43:20:19:ba
inet6 fe80::c890:43ff:fe20:19ba%awdl0 prefixlen 64 scopeid 0x9
nd6 options=1<PERFORMNUD>
media: autoselect
status: active
vboxnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
ether 0a:00:27:00:00:00
inet 192.168.59.3 netmask 0xffffff00 broadcast 192.168.59.255
vboxnet1: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ether 0a:00:27:00:00:01
</pre>
<p>
You are most likely to be looking for the bridge100 interface:
</p>
<pre>
bridge100: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=3<RXCSUM,TXCSUM>
ether a6:5e:60:7e:1d:64
inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
inet6 fe80::a45e:60ff:fe7e:1d64%bridge100 prefixlen 64 scopeid 0xe
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
ipfilter disabled flags 0x2
member: en4 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 10 priority 0 path cost 0
Address cache:
nd6 options=1<PERFORMNUD>
media: <unknown type >
status: inactive
</pre>
<p>
In this example, the IP address is 192.168.2.1 (line beginning with <code>inet</code> above.)
<p>
<li>There are two ways to find out what is connected. the first is to run the command <code>arp</code>:</li>
<pre>
arp -i bridge100 -a
</pre>
<p>
This runs a program which returns information about what is connected currently to the <code>bridge100</code> interface. When you run this command you will get output which contains lines like:
</p>
<pre>
? (239.255.255.250) at 1:0:5e:7f:ff:fa on bridge100 ifscope permanent [ethernet]
</pre>
<p>
Running the <code>arp</code> command will give you a list of connected IP addresses - there should be the same number of addresses as you have Raspberry Pis connected.
</p>
<li>
The other way is to install a program like nmap, which is available from <a href="https://nmap.org/">https://nmap.org/</a>. You can run it in two ways (replacing the IP addresses with the one you noted when running <code>ifconfig</code>):</li>
<ul>
<li>Scan a range of IP addresses (use this option if you know what range your connection sharing will use): <pre> nmap 192.168.2.1-255</pre></li>
<li>Scan a subnet on the network (use this option if you only know the start of the IP address range, e.g 192.168.2): <pre>nmap 192.168.2.0/24</pre></li>
</ul>
</ol>
<h3> <a name="winnmap">Windows</a></h3>
<p>
The easiest way to determine the IP addresses on Windows is with nmap, which you can download from <a href="https://nmap.org/">here</a>.
</p>
<ol>
<li>Using the Ethernet IP address you noted before when sharing the Internet Connection, run the following command, but change the last number to 0:</li>
<pre>
c:\nmap 192.168.137.0/24
</pre>
<p>
This should give output like:
</p>
<pre>
Starting Nmap 7.40 ( https://nmap.org ) at 2017-04-26 15:53 GMT Daylight Time
Nmap scan report for 192.168.137.225
Host is up (0.00013s latency).
All 1000 scanned ports on 192.168.137.225 are closed
MAC Address: 98:5A:EB:C7:15:8B (Apple)
Nmap scan report for 192.168.137.1
Host is up (0.0013s latency).
Not shown: 997 closed ports
PORT STATE SERVICE
135/tcp open msrpc
139/tcp open netbios-ssn
445/tcp open microsoft-ds
Nmap done: 256 IP addresses (2 hosts up) scanned in 23.48 seconds
</pre>
<p>
All the IP addresses which show up starting with "Nmap scan report for <IP address>" will be your Raspberry Pis - except for the one with your written down IP address.
</p>
</ol>
<a name="mainnode"><h2>Setting Up the Main Node</h2></a>
<p>
Now we have our Pis started, we know their current IP addresses, we can start doing some configuration and setup. In this guide, the Pi with address 192.168.2.18 the master node in the cluster, and the other Pis will be worker nodes.
In our cluster, the master node will run the network assigning IP addresses via the DHCP server, share its Wifi connection (if used), share a networked drive for files, and start the programs we will run on the whole cluster.
</p>
<p>
For example, my five Pi addresses are:
</p>
<pre>
192.168.2.18
192.168.2.19
192.168.2.20
192.168.2.21
192.168.2.22
</pre>
<p>
<b>Note:</b> These IP addresses are only valid with the current connection to your computer. After the setup is complete, the Raspberry Pis will be assigned addresses by the cluster server.
</p>
<h3>Logging in to the Master Node</h3>
<p>
In your command terminal type the following command, using the username (if not the default <code>pi</code>) and IP address for the Pi that will be your master node:
</p>
<pre>
</pre>
<p>
The first time you do this, the output will look something like this:
</p>
<pre>
The authenticity of host '192.168.2.18 (192.168.2.18)' can't be established.
RSA key fingerprint is a0:97:1f:d6:df:8c:a5:45:db:fe:b4:94:46:1d:0f:48.
Are you sure you want to continue connecting (yes/no)?
</pre>
<p>
Type <code>yes</code> to allow this and future connections:
</p>
<pre>
Warning: Permanently added '192.168.2.18' (RSA) to the list of known hosts.
[email protected]'s password:
</pre>
<p>
Enter the password for that user (the default will be <code>raspberry</code>).
</p>
<p>
If the password is correct, you will get the following output:
</p>
<pre>
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Feb 26 03:25:51 2016
pi@raspberrypi:~ $
</pre>
<p>
You are now remotely logged in to that account on your Raspberry Pi, via your laptop/desktop. Any commands you enter while logged in will be executed as if you were doing so on the Pi directly. This is a standard way of logging into a remote system - like a supercomputer such as ARCHER.
</p>
<a name="configrpi"><h3>System Configuration</h3></a>
<p>
To open the Raspberry Pi confiuration program, enter the command <code>sudo raspi-config</code>:
</p>
<pre>
pi@raspberrypi:~ $ sudo raspi-config
</pre>
<p>
The <code>sudo</code> prefix means that the following commands will be run as the super-user which has permissions to make system-wide changes rather than just changes to user files.
</p>
<p>
This will bring up a menu. Select the following options:
</p>
<ol>
<li>The first step is to configure the master node to boot to a command line interface (CLI) like you are using now, instead of a graphical user interface (GUI) like most modern operating systems have. Choose the following options in turn, navigating with the keyboard arrow keys and selecting each option with the Enter key:
<ul>
<li><code>3 Boot Options</code></li>
<li><code>B1 Desktop/CLI</code></li>
<li><code>B1</code> again for <code>Console</code>.</li>
</ul>
<li>Next, change the Pi's hostname so you can easily identify it from the others in future. Follow the following options in the same ways as the previous step:</li>
<ul>
<li><code>2 Network Options</code></li>
<li><code>N2 Hostname</code></li>
</ul>
<li>An information screen will appear, detailing what characters are allowed in the hostname - read it and select <code><Ok></code> to continue.</li>
<li>On this screen, you will see the Raspberry Pi's current hostname. To make it more identifiable and customise your cluster, enter a new name of your choice. In my example I will change it to <i>beira</i> and hit enter/return to confirm it.</li>
<li>Finally need to set the time zone so that the date and time on the Pi are correct. Once you are back at the main menu, select:</li>
<ul>
<li><code>4 Localisation Options</code></li>
<li><code>I2 Change Timezone</code></li>
</ul>
<li>You will be presented with a list of geographic regions. Select where you are and hit Enter, then pick a city from the next list which is in your timezone. For example, if you were here in EPCC, you would pick 'Europe', and then 'London', since Edinburgh is not an option.</li>
<li>Once back in the main menu, use the right arrow keys to select <code><Finish></code> and hit Enter/return.</li>
<li>You will be asked if you want to reboot - choose yes. Your ssh session will end, and you will have to wait for a couple of minutes while the Raspberry Pi restarts</li>
</ol>
<p>
If you log back in to your master node at this stage, you will see your new hostname now appears on the command line:
</p>
<pre>
pi@beira:~ $
</pre>
<h3>Software Update and Passwordless Login</h3>
<p>
We will now install some updates on the Raspberry Pi main software:
</p>
<ol>
<li>The first command to run is <code>rpi-update</code>:</li>
<pre>
pi@beira:~ $ sudo rpi-update
</pre>
<p>
This will update the Raspberry Pi with the latest version of the core software and firmware for the system - it will take a few minutes and start with output like:
</p>
<pre>
*** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
*** Performing self-update
*** Relaunching after update
*** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
#############################################################
WARNING: This update bumps to rpi-4.9.y linux tree
Be aware there could be compatibility issues with some drivers
Discussion here:
https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=167934
##############################################################
Would you like to proceed? (y/N)
</pre>
<p>
Type <code>y</code>, then hit Enter. You will need to reboot the Raspberry Pi for this to take effect:
</p>
<pre>
pi@beira:~ $ sudo reboot
</pre>
<li>Wait a few minutes while your master node reboots, then log back in via ssh.</li>
<li>Before updating or installing any more software, begin setting up the passwordless login system that the cluster will need to run programs in parallel.
<p>
At the command line run the following:
</p>
<pre>
pi@beira:~ $ ssh-keygen -t rsa
</pre>
<p>
This will generate what is called a public/private key pair - this is a system used for encryption and security to allow information to be shared in private. This will generated public key is stored in a file, which can be given to any other machine or person you want to communicate with and a private file which has to be kept secret so that other people know that only you sent information to them or the information they send to you using your public key can only be read by you.
</p>
<li>Use the default file path for the first prompt, and then don’t put in a passphrase for this cluster (just press return/enter when you are asked for a pass phrase). Your interaction should look a little like:</li>
<pre>
pi@beira:~ $ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/pi/.ssh/id_rsa):
Created directory '/home/pi/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/pi/.ssh/id_rsa.
Your public key has been saved in /home/pi/.ssh/id_rsa.pub.
The key fingerprint is:
9b:98:c7:86:17:0a:1e:32:95:65:ee:1c:0f:48:48:ef pi@beira
The key's randomart image is:
+---[RSA 2048]----+
| .... o |
| .o * |
| = + |
| o o + |
| o E o S |
| + o * + |
| . = B |
| + |
| |
+-----------------+
pi@beira:~ $
</pre>
<li>Finally, we will update some of the operating system and installed software, run the command:</li>
<pre>
sudo apt-get update
</pre>
<p>
When the prompt asks enter ‘<code>Y</code>’ and it will install the upgrades.
</p>
<li>Then upgrade the installed software to its latest versions - which include bug fixes and improvements - with this command:</li>
</p>
<pre>
sudo apt-get upgrade
</pre>
<p>
Depending on the number of upgrades, this can take a long time to complete. Once it's done, you are ready to move onto the next stage of setting up the master node.
</p>
</ol>
<h3>Setting up the Shared File System</h3>
<p>
The next step is to set up the Network File System (NFS) server component.
</p>
<ol>
<li>First thing we want to do is to add a network module, so run the following command:</li>
<pre>
sudo nano /etc/modules
</pre>
<p>
This will open up a file like this one in the nano text editor:
</p>
<figure>
<img src="images/nanomodules.png" alt="Text Editor"/>
<figcaption>Adding ipv6</figcaption>
</figure>
<li>At the end of the file, using the cursor keys to move, if it is not present (as it is above), add the following:</li>
<pre>
ipv6
</pre>
<li>To save the file and exit nano, enter the following keyboard shortcuts:</li>
<ul>
<li>CRTL + O to save.</li>
<li>CTRL + X to close.</li>
</ul>
<li>Run the command:</li>
<pre>
sudo service rpcbind start
</pre>
<li>Next, install the NFS Kernel server. This is needed to run the server for the NFS drive.</li>
<pre>
sudo apt-get install nfs-kernel-server
</pre>
<li>Now create the actual shared location. Start by creating a directory in /home for sharing:</li>
<pre>
sudo mkdir -p /home/shared_dir
</pre>
<p>
<code>mkdir</code> is a command used to create a directory (which can contain other directories and files).
</p>
<li>Change the new directory access permissions so that everyone can read or write to this directory:</li>
<pre>
sudo chmod 777 /home/shared_dir
</pre>
<p>
<code>chmod</code> is a command to change the access permissions on a file or directory. Here 777 means that all users can read, write or execute commands in that directory. If you only want the owner to be able to write and others to only read you would use 644. This link will take you to the <a href="https://en.wikipedia.org/wiki/Chmod">chmod Wikipedia entry</a>.
</p>
<li>Now we need to mount the shared_dir and bind it in order to make it available to all nodes.</li>
<pre>
sudo mount --bind /home/shared_dir/ /home/shared_dir/
</pre>
<li>Now we need to make sure the directory gets mounted and exported each time the system boots up. Run the following command:</li>
<pre>
sudo nano /etc/fstab
</pre>
<p>
At the end of the now open file add the following line, and save and close the file as before.
</p>
<pre>
/home/shared_dir /home/shared_dir none bind 0 0
</pre>
<li>Next, open the following file:</li>
<pre>
sudo nano /etc/default/nfs-kernel-server
</pre>
<p>
Make sure that any one of the following options is present in the opened file:
</p>
<ul>
<li>
<pre>
NEED_SVCGSSD=no
</pre>
</li>
<li>
<pre>
NEED_SVCGSSD=“no”
</pre>
</li>
<li>
<pre>
NEED_SVCGSSD=“”
</pre>
</li>
<li>
<pre>
NEED_SVCGSSD=
</pre>
</li>
</ul>
<p>
Save and close the file.
</p>
<li>The configuration process continues with a mapping file:</li>
<pre>
sudo nano /etc/idmapd.conf
</pre>
<p>
Ensure that under the [Mapping] section the following text is present:
</p>
<pre>
[Mapping]
Nobody-User = nobody
Nobody-Group = nogroup
</pre>
<li>Now run the following to expose the new directory to the network:</li>
<pre>
sudo nano /etc/exports
</pre>
<p>
At the end of the file add the following to expose your shared_dir to the network, and allow read and write access (replacing with the first part of your IP addresses):
</p>
<pre>
/home/shared_dir 192.168.2.0/24(rw,nohide,insecure,no_subtree_check,async)
</pre>
<li>Use the following command to reload the file now it's been modified:</li>
<pre>
exportfs -a
</pre>
<li>As a final check, examine the following files in nano:</li>
<pre>
/etc/init.d/nfs-kernel-server
/etc/init.d/nfs-common
/etc/init.d/rpcbind
</pre>
<p>
Near the top of the file, there should be a line like this one:
</p>
<pre>
# Default-Start: 2 3 4 5
</pre>
<p>
<b>However</b> in some, it may say:
</p>
<pre>
# Default-Start: S
</pre>
<p>
You need to change all the files that have an <code>S</code> to <code>2 3 4 5</code>.
</p>
<li>Run the following six commands:</li>
<pre>
sudo update-rc.d -f rpcbind remove
sudo update-rc.d rpcbind defaults
sudo update-rc.d -f nfs-common remove
sudo update-rc.d nfs-common defaults
sudo update-rc.d -f nfs-kernel-server remove
sudo update-rc.d nfs-kernel-server defaults
</pre>
<p>
(If any of these commands fail with error messages, try the following, then retry):
</p>
<pre>
sudo apt-get purge rpcbind
sudo apt-get install nfs-kernel-server
</ol>
<h3>Setting Up MPI</h3>
<p>
<a href="https://en.wikipedia.org/wiki/Message_Passing_Interface">MPI (Message Passing Interface)</a> is a core technology used in high performance computing that allows processes to communicate in a standard way, and so solve a problems together in parallel.
We will install and test it with a simple program on a single node.
</p>
<ol>
<li>Run the following to install necessary libraries:</li>
<pre>
sudo apt-get install libxml2-dev
sudo apt-get install zlib1g zlib1g-dev
sudo apt-get install mpich
</pre>
<li>Now try running the command:</li>
<pre>
mpiexec —version
</pre>
<p>
The output should look something like this:
</p>
<pre>
pi@beira:~ $ mpiexec --version
HYDRA build details:
Version: 3.2
Release Date: Wed Nov 11 22:06:48 CST 2015
CC: gcc -Wl,-z,relro
CXX: g++ -Wl,-z,relro
F77: gfortran -Wl,-z,relro
F90: gfortran -Wl,-z,relro
Configure options: ...
Process Manager: pmi
Launchers available: ssh rsh fork slurm ll lsf sge manual persist
Topology libraries available: hwloc
Resource management kernels available: user slurm ll lsf sge pbs cobalt
Checkpointing libraries available: blcr
Demux engines available: poll select
</pre>
<p><b>Note:</b> The output above has been shortened. Yours should have more detail.</p>
<li>Now we will write a small program in C to test that MPI works on just one Raspberry Pi.</li>
<p>
Navigate to the shared_dir directory, create a new directory called <code>testprogram</code>, then go into that directory and create an empty file called <code>hello.c</code> with the following set of commands:
</p>
<pre>
cd /home/shared_dir
mkdir testprogram
cd testprogram
nano hello.c
</pre>
<li>Executing <code>nano hello.c</code> will open the empty file as well as create it. Type the following code, save, and exit:</li>
<pre>
#include <mpi.h>
#include <stdio.h>
int main(int argc, char** argv) {
// Initialize the MPI environment
MPI_Init(NULL, NULL);
// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
// Print off a hello world message
printf("Hello world from processor %s, rank %d"
" out of %d processors\n",
processor_name, world_rank, world_size);
// Finalize the MPI environment.
MPI_Finalize();
}
</pre>
<li>To compile the code into an executable, do the following command:</li>
<pre>
mpicc -o hello hello.c
</pre>
<li>Type the command:</li>
<pre>
ls -al
</pre>
<p>
This should show the directory contents - which will include an executable file called <code>hello</code>.
</p>
<li>Now we will run this as an MPI program. First we need to create a hostfile:</li>
<pre>
nano hostfile
</pre>
<p>
and populate it with the address of your Raspberry Pi master node, followed by a <code>:4</code>. For example I need to type in:
</p>
<pre>
192.168.2.18:4
</pre>
<p>
Now save the file and exit nano.
</p>
<li>To run the <code>hello</code> program, type in the command:</li>
<pre>
mpiexec -n 4 -f hostfile ./hello
</pre>
<p>
<code>mpiexec</code> command will run the program <code>hello</code> as an MPI program across 4 cores on the systems identified in your hostfile. This means that the program will run as four processes which can communicate with each other - in this case each process will only write out which core on the Raspberry Pi it is working on.
</p>
<p>
You should get output like this:
</p>
<pre>
pi@beira:/home/shared_dir/testprogram $ mpiexec -n 4 -f hostfile ./hello
Hello world from processor beira, rank 0 out of 4 processors
Hello world from processor beira, rank 1 out of 4 processors
Hello world from processor beira, rank 2 out of 4 processors
Hello world from processor beira, rank 3 out of 4 processors
</pre>
<p>
If you do see this output like this then congratulations - you have setup MPI on your system and run a basic MPI program on it.
</p>
<h3>Setting Up the DHCP Server</h3>
<ol>
<li>The first step is to enable Wifi on your master node. To do this, you need to add a network to the wpa_supplicant.conf file.</li>
<p>
Open wpa_supplicant.conf in nano:
</p>
<pre>
sudo nano /etc/wpa_supplicant/wpa_supplicant.conf
</pre>
<p>
and the end of the file add:
</p>
<pre>
network={
ssid=“<your ssid>“
psk=“<your wifi code>“
}
</pre>
<p>
<b>Note:</b> The SSID is the WiFi network identifier - this is an identifier broadcast to allow you to select the right WiFi network. If you are running a home network - this will likely be on your router. Otherwise you can check to see what WiFi network your main computer is connected to from the WiFi network properties. As an example:
</p>
<pre>
network={
ssid="SKY64DBE"
psk="UDAXEESQ"
}
</pre>
<p>
If you have a system like <i>eduroam</i> please have a look <a href="eduroam.html">here</a>.
</p>
<li>You may need to restart the wifi connection on your Raspberry Pi. Run the command:</li>
<pre>
sudo ifconfig wlan0 down
</pre>
<p>
Wait for a few seconds then run:
</p>
<pre>
sudo ifconfig wlan0 up
</pre>
<p>
When you run the command, you should get the output for the wifi with an IP address. :
</p>
<pre>
ifconfig wlan0
</pre>
<p>