Introducing the correlation functi
The code here shows the evolution of \( \kappa_d \) as a function of \( d \) for a series of random numbers. We see that the function \( \kappa_d \) approaches \( 0 \) as \( d\rightarrow \infty \).
+ +Note: code will be inserted here later.
Introducing the correlation functi
The code here shows the evolution of \( \kappa_d \) as a function of \( d \) for a series of random numbers. We see that the function \( \kappa_d \) approaches \( 0 \) as \( d\rightarrow \infty \).
+Note: code will be inserted here later.
+Resampling methods: Blocking
diff --git a/doc/pub/week9/html/week9.html b/doc/pub/week9/html/week9.html index 98929dfe..06ac4661 100644 --- a/doc/pub/week9/html/week9.html +++ b/doc/pub/week9/html/week9.html @@ -804,6 +804,8 @@Introducing the correlation functi
The code here shows the evolution of \( \kappa_d \) as a function of \( d \) for a series of random numbers. We see that the function \( \kappa_d \) approaches \( 0 \) as \( d\rightarrow \infty \).
+Note: code will be inserted here later.
+Resampling methods: Blocking
diff --git a/doc/pub/week9/ipynb/ipynb-week9-src.tar.gz b/doc/pub/week9/ipynb/ipynb-week9-src.tar.gz index e1f7c1fa..622f1df5 100644 Binary files a/doc/pub/week9/ipynb/ipynb-week9-src.tar.gz and b/doc/pub/week9/ipynb/ipynb-week9-src.tar.gz differ diff --git a/doc/pub/week9/ipynb/week9.ipynb b/doc/pub/week9/ipynb/week9.ipynb index e6cc2d25..f1a04377 100644 --- a/doc/pub/week9/ipynb/week9.ipynb +++ b/doc/pub/week9/ipynb/week9.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "7f934c76", + "id": "9343f8bf", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "5d8c5800", + "id": "66464e58", "metadata": { "editable": true }, @@ -27,7 +27,7 @@ }, { "cell_type": "markdown", - "id": "7bd2e26c", + "id": "44214dd1", "metadata": { "editable": true }, @@ -50,7 +50,7 @@ }, { "cell_type": "markdown", - "id": "181ed6f2", + "id": "fdb87fa7", "metadata": { "editable": true }, @@ -67,7 +67,7 @@ }, { "cell_type": "markdown", - "id": "e47adac5", + "id": "21549353", "metadata": { "editable": true }, @@ -86,7 +86,7 @@ }, { "cell_type": "markdown", - "id": "e143647e", + "id": "df4d2efc", "metadata": { "editable": true }, @@ -104,7 +104,7 @@ }, { "cell_type": "markdown", - "id": "b8c9399e", + "id": "08b48c0e", "metadata": { "editable": true }, @@ -121,7 +121,7 @@ }, { "cell_type": "markdown", - "id": "72de4651", + "id": "cc7b9945", "metadata": { "editable": true }, @@ -133,7 +133,7 @@ }, { "cell_type": "markdown", - "id": "8e9371fe", + "id": "a1933b45", "metadata": { "editable": true }, @@ -143,7 +143,7 @@ }, { "cell_type": "markdown", - "id": "cd3e9e74", + "id": "d415d278", "metadata": { "editable": true }, @@ -155,7 +155,7 @@ }, { "cell_type": "markdown", - "id": "9cc43d3d", + "id": "e73e9f65", "metadata": { "editable": true }, @@ -170,7 +170,7 @@ }, { "cell_type": "markdown", - "id": "e4d44112", + "id": "ede0d2ac", "metadata": { "editable": true }, @@ -185,7 +185,7 @@ }, { "cell_type": "markdown", - "id": "cd1734a3", + "id": "7b1560b6", "metadata": { "editable": true }, @@ -197,7 +197,7 @@ }, { "cell_type": "markdown", - "id": "0edda6fb", + "id": "eff82084", "metadata": { "editable": true }, @@ -207,7 +207,7 @@ }, { "cell_type": "markdown", - "id": "485ab088", + "id": "62654667", "metadata": { "editable": true }, @@ -219,7 +219,7 @@ }, { "cell_type": "markdown", - "id": "428e6c44", + "id": "68cb0f0f", "metadata": { "editable": true }, @@ -229,7 +229,7 @@ }, { "cell_type": "markdown", - "id": "703c3170", + "id": "f42bf77a", "metadata": { "editable": true }, @@ -241,7 +241,7 @@ }, { "cell_type": "markdown", - "id": "3404f05c", + "id": "5b148aba", "metadata": { "editable": true }, @@ -253,7 +253,7 @@ }, { "cell_type": "markdown", - "id": "56c64606", + "id": "ad425337", "metadata": { "editable": true }, @@ -263,7 +263,7 @@ }, { "cell_type": "markdown", - "id": "5c1f6ee5", + "id": "4b7c80ed", "metadata": { "editable": true }, @@ -275,7 +275,7 @@ }, { "cell_type": "markdown", - "id": "a192cbd4", + "id": "b9a0bf94", "metadata": { "editable": true }, @@ -285,7 +285,7 @@ }, { "cell_type": "markdown", - "id": "a4458392", + "id": "4a903b42", "metadata": { "editable": true }, @@ -297,7 +297,7 @@ }, { "cell_type": "markdown", - "id": "085cafc0", + "id": "859bdf66", "metadata": { "editable": true }, @@ -309,7 +309,7 @@ }, { "cell_type": "markdown", - "id": "f8b1bd83", + "id": "a85255c9", "metadata": { "editable": true }, @@ -324,7 +324,7 @@ }, { "cell_type": "markdown", - "id": "085dc147", + "id": "18960ace", "metadata": { "editable": true }, @@ -334,7 +334,7 @@ }, { "cell_type": "markdown", - "id": "b07f69d6", + "id": "46d3602b", "metadata": { "editable": true }, @@ -346,7 +346,7 @@ }, { "cell_type": "markdown", - "id": "524a3980", + "id": "063f2b21", "metadata": { "editable": true }, @@ -360,7 +360,7 @@ }, { "cell_type": "markdown", - "id": "aa229836", + "id": "41bbfc7e", "metadata": { "editable": true }, @@ -379,7 +379,7 @@ }, { "cell_type": "markdown", - "id": "a74e265c", + "id": "c96a94ab", "metadata": { "editable": true }, @@ -391,7 +391,7 @@ }, { "cell_type": "markdown", - "id": "48a0b0b5", + "id": "0d871074", "metadata": { "editable": true }, @@ -403,7 +403,7 @@ }, { "cell_type": "markdown", - "id": "4fa10f66", + "id": "1b11719d", "metadata": { "editable": true }, @@ -413,7 +413,7 @@ }, { "cell_type": "markdown", - "id": "10aaa6c6", + "id": "e194cad4", "metadata": { "editable": true }, @@ -425,7 +425,7 @@ }, { "cell_type": "markdown", - "id": "6dc77832", + "id": "624a935e", "metadata": { "editable": true }, @@ -435,7 +435,7 @@ }, { "cell_type": "markdown", - "id": "78b3c31c", + "id": "a0403e99", "metadata": { "editable": true }, @@ -447,7 +447,7 @@ }, { "cell_type": "markdown", - "id": "4c584946", + "id": "f656557a", "metadata": { "editable": true }, @@ -459,7 +459,7 @@ }, { "cell_type": "markdown", - "id": "bb141426", + "id": "884e97e8", "metadata": { "editable": true }, @@ -473,17 +473,19 @@ }, { "cell_type": "markdown", - "id": "6f36565b", + "id": "2d2ed4e1", "metadata": { "editable": true }, "source": [ - "The code here shows the evolution of $\\kappa_d$ as a function of $d$ for a series of random numbers. We see that the function $\\kappa_d$ approaches $0$ as $d\\rightarrow \\infty$." + "The code here shows the evolution of $\\kappa_d$ as a function of $d$ for a series of random numbers. We see that the function $\\kappa_d$ approaches $0$ as $d\\rightarrow \\infty$.\n", + "\n", + "**Note**: code will be inserted here later." ] }, { "cell_type": "markdown", - "id": "250b1b3e", + "id": "37201f23", "metadata": { "editable": true }, @@ -501,7 +503,7 @@ }, { "cell_type": "markdown", - "id": "233f70b8", + "id": "55e9d3e1", "metadata": { "editable": true }, @@ -515,7 +517,7 @@ }, { "cell_type": "markdown", - "id": "b17d9e13", + "id": "d3492b16", "metadata": { "editable": true }, @@ -530,7 +532,7 @@ }, { "cell_type": "markdown", - "id": "179d551e", + "id": "90283800", "metadata": { "editable": true }, @@ -546,7 +548,7 @@ }, { "cell_type": "markdown", - "id": "f89ed58e", + "id": "8791c09b", "metadata": { "editable": true }, @@ -558,7 +560,7 @@ }, { "cell_type": "markdown", - "id": "44308237", + "id": "881dde94", "metadata": { "editable": true }, @@ -577,7 +579,7 @@ }, { "cell_type": "markdown", - "id": "1a9ef202", + "id": "a5648050", "metadata": { "editable": true }, @@ -601,7 +603,7 @@ }, { "cell_type": "markdown", - "id": "0ca89ff0", + "id": "979611cc", "metadata": { "editable": true }, @@ -616,7 +618,7 @@ }, { "cell_type": "markdown", - "id": "6bc37505", + "id": "c17c5ab4", "metadata": { "editable": true }, @@ -628,7 +630,7 @@ }, { "cell_type": "markdown", - "id": "87775adf", + "id": "6df608d1", "metadata": { "editable": true }, @@ -640,7 +642,7 @@ }, { "cell_type": "markdown", - "id": "14149079", + "id": "004092a6", "metadata": { "editable": true }, @@ -658,7 +660,7 @@ }, { "cell_type": "markdown", - "id": "73987934", + "id": "d0be2c66", "metadata": { "editable": true }, @@ -676,7 +678,7 @@ }, { "cell_type": "markdown", - "id": "69fbf5d1", + "id": "69ca5a78", "metadata": { "editable": true }, @@ -687,7 +689,7 @@ }, { "cell_type": "markdown", - "id": "81fe5990", + "id": "f8328d81", "metadata": { "editable": true }, @@ -698,7 +700,7 @@ }, { "cell_type": "markdown", - "id": "d4706212", + "id": "4c798f11", "metadata": { "editable": true }, @@ -716,7 +718,7 @@ }, { "cell_type": "markdown", - "id": "eb661f47", + "id": "a4ecc77b", "metadata": { "editable": true }, @@ -726,7 +728,7 @@ }, { "cell_type": "markdown", - "id": "2f1d607f", + "id": "f9f794e9", "metadata": { "editable": true }, @@ -744,7 +746,7 @@ }, { "cell_type": "markdown", - "id": "eecababc", + "id": "3fface33", "metadata": { "editable": true }, @@ -754,7 +756,7 @@ }, { "cell_type": "markdown", - "id": "49a6dfb0", + "id": "064ba2a5", "metadata": { "editable": true }, @@ -766,7 +768,7 @@ }, { "cell_type": "markdown", - "id": "2c62cc63", + "id": "b112cab0", "metadata": { "editable": true }, @@ -778,7 +780,7 @@ }, { "cell_type": "markdown", - "id": "06fde16f", + "id": "343ef06b", "metadata": { "editable": true }, @@ -796,7 +798,7 @@ }, { "cell_type": "markdown", - "id": "61e106ab", + "id": "b9fc08c8", "metadata": { "editable": true }, @@ -806,7 +808,7 @@ }, { "cell_type": "markdown", - "id": "a9f5001a", + "id": "9e41a50c", "metadata": { "editable": true }, @@ -823,7 +825,7 @@ }, { "cell_type": "markdown", - "id": "aed624ba", + "id": "4dad2b57", "metadata": { "editable": true }, @@ -843,7 +845,7 @@ }, { "cell_type": "markdown", - "id": "421f9f8b", + "id": "804f4e29", "metadata": { "editable": true }, @@ -854,7 +856,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "8d23bb07", + "id": "e659813f", "metadata": { "collapsed": false, "editable": true @@ -1083,7 +1085,7 @@ }, { "cell_type": "markdown", - "id": "7d5001dc", + "id": "265f2fe3", "metadata": { "editable": true }, @@ -1098,7 +1100,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "6da7142f", + "id": "3397cf44", "metadata": { "collapsed": false, "editable": true @@ -1164,7 +1166,7 @@ }, { "cell_type": "markdown", - "id": "6d0b7e85", + "id": "1de1b08e", "metadata": { "editable": true }, @@ -1189,7 +1191,7 @@ }, { "cell_type": "markdown", - "id": "17407842", + "id": "4013f9b4", "metadata": { "editable": true }, @@ -1202,7 +1204,7 @@ }, { "cell_type": "markdown", - "id": "ad210718", + "id": "36c01fcc", "metadata": { "editable": true }, @@ -1213,7 +1215,7 @@ }, { "cell_type": "markdown", - "id": "69401651", + "id": "405ad9e9", "metadata": { "editable": true }, @@ -1229,7 +1231,7 @@ }, { "cell_type": "markdown", - "id": "68b36411", + "id": "5377c5da", "metadata": { "editable": true }, @@ -1239,7 +1241,7 @@ }, { "cell_type": "markdown", - "id": "6c321ece", + "id": "cb6cd8bb", "metadata": { "editable": true }, @@ -1253,7 +1255,7 @@ }, { "cell_type": "markdown", - "id": "e8c5dca5", + "id": "1e3fb35c", "metadata": { "editable": true }, @@ -1264,7 +1266,7 @@ }, { "cell_type": "markdown", - "id": "5dbfe538", + "id": "84e6c1d2", "metadata": { "editable": true }, @@ -1274,7 +1276,7 @@ }, { "cell_type": "markdown", - "id": "f67659a4", + "id": "72e6c3cd", "metadata": { "editable": true }, @@ -1286,7 +1288,7 @@ }, { "cell_type": "markdown", - "id": "77187b55", + "id": "ea5d6f90", "metadata": { "editable": true }, @@ -1297,7 +1299,7 @@ }, { "cell_type": "markdown", - "id": "92aa4055", + "id": "3887b522", "metadata": { "editable": true }, @@ -1307,7 +1309,7 @@ }, { "cell_type": "markdown", - "id": "1e09c863", + "id": "3ca2b19a", "metadata": { "editable": true }, @@ -1317,7 +1319,7 @@ }, { "cell_type": "markdown", - "id": "ec018a86", + "id": "439672d1", "metadata": { "editable": true }, @@ -1329,7 +1331,7 @@ }, { "cell_type": "markdown", - "id": "0d745d5f", + "id": "1eb68081", "metadata": { "editable": true }, @@ -1341,7 +1343,7 @@ }, { "cell_type": "markdown", - "id": "9d0406a1", + "id": "de86ccd8", "metadata": { "editable": true }, @@ -1352,7 +1354,7 @@ }, { "cell_type": "markdown", - "id": "7c0f74b9", + "id": "4842593c", "metadata": { "editable": true }, @@ -1367,7 +1369,7 @@ }, { "cell_type": "markdown", - "id": "018d7811", + "id": "6d207ed0", "metadata": { "editable": true }, @@ -1383,7 +1385,7 @@ }, { "cell_type": "markdown", - "id": "897c72c9", + "id": "a96a8b01", "metadata": { "editable": true }, @@ -1397,7 +1399,7 @@ }, { "cell_type": "markdown", - "id": "adec4792", + "id": "f5e3e574", "metadata": { "editable": true }, @@ -1407,7 +1409,7 @@ }, { "cell_type": "markdown", - "id": "cb465899", + "id": "239e6239", "metadata": { "editable": true }, @@ -1421,7 +1423,7 @@ }, { "cell_type": "markdown", - "id": "3d14ad95", + "id": "a7a833fa", "metadata": { "editable": true }, @@ -1432,7 +1434,7 @@ }, { "cell_type": "markdown", - "id": "002fc562", + "id": "29636e71", "metadata": { "editable": true }, @@ -1450,7 +1452,7 @@ }, { "cell_type": "markdown", - "id": "d03ee544", + "id": "bf382314", "metadata": { "editable": true }, @@ -1466,7 +1468,7 @@ }, { "cell_type": "markdown", - "id": "9a764260", + "id": "f500592c", "metadata": { "editable": true }, @@ -1482,7 +1484,7 @@ }, { "cell_type": "markdown", - "id": "0de76fb9", + "id": "90e3f414", "metadata": { "editable": true }, @@ -1496,7 +1498,7 @@ }, { "cell_type": "markdown", - "id": "7fd90368", + "id": "2ea4473a", "metadata": { "editable": true }, @@ -1510,7 +1512,7 @@ }, { "cell_type": "markdown", - "id": "18ac99c8", + "id": "5fb521d1", "metadata": { "editable": true }, @@ -1529,7 +1531,7 @@ }, { "cell_type": "markdown", - "id": "10f0f1c5", + "id": "dfb47bdc", "metadata": { "editable": true }, @@ -1541,7 +1543,7 @@ }, { "cell_type": "markdown", - "id": "76cbe575", + "id": "1d7fe6e0", "metadata": { "editable": true }, @@ -1552,7 +1554,7 @@ }, { "cell_type": "markdown", - "id": "ac611c90", + "id": "1d29258b", "metadata": { "editable": true }, @@ -1581,7 +1583,7 @@ }, { "cell_type": "markdown", - "id": "ce267e9b", + "id": "f0f561a9", "metadata": { "editable": true }, @@ -1592,7 +1594,7 @@ }, { "cell_type": "markdown", - "id": "b5e50c74", + "id": "6212347a", "metadata": { "editable": true }, @@ -1604,7 +1606,7 @@ }, { "cell_type": "markdown", - "id": "4834c522", + "id": "2ce79f3b", "metadata": { "editable": true }, @@ -1617,7 +1619,7 @@ }, { "cell_type": "markdown", - "id": "cd5abf08", + "id": "1a5dab49", "metadata": { "editable": true }, @@ -1628,7 +1630,7 @@ }, { "cell_type": "markdown", - "id": "c0cfbeb1", + "id": "db9adf75", "metadata": { "editable": true }, @@ -1640,7 +1642,7 @@ }, { "cell_type": "markdown", - "id": "1c2b9caf", + "id": "cb2d35a4", "metadata": { "editable": true }, @@ -1657,7 +1659,7 @@ }, { "cell_type": "markdown", - "id": "b3627ae3", + "id": "335c5f06", "metadata": { "editable": true }, @@ -1669,7 +1671,7 @@ }, { "cell_type": "markdown", - "id": "c85f3d13", + "id": "0817b562", "metadata": { "editable": true }, @@ -1684,7 +1686,7 @@ }, { "cell_type": "markdown", - "id": "b3d4e458", + "id": "21b11a74", "metadata": { "editable": true }, @@ -1694,7 +1696,7 @@ }, { "cell_type": "markdown", - "id": "c45ed72e", + "id": "63cb8bda", "metadata": { "editable": true }, @@ -1712,7 +1714,7 @@ }, { "cell_type": "markdown", - "id": "6b9f160e", + "id": "c7767939", "metadata": { "editable": true }, @@ -1723,7 +1725,7 @@ }, { "cell_type": "markdown", - "id": "9251ed34", + "id": "89e5e83d", "metadata": { "editable": true }, @@ -1782,7 +1784,7 @@ }, { "cell_type": "markdown", - "id": "d4cb34c8", + "id": "622ef350", "metadata": { "editable": true }, @@ -1793,7 +1795,7 @@ }, { "cell_type": "markdown", - "id": "aba109f4", + "id": "c5e37166", "metadata": { "editable": true }, @@ -1803,7 +1805,7 @@ }, { "cell_type": "markdown", - "id": "b29b64ce", + "id": "6c6dec26", "metadata": { "editable": true }, @@ -1813,7 +1815,7 @@ }, { "cell_type": "markdown", - "id": "ecb51ee3", + "id": "cdf688b8", "metadata": { "editable": true }, @@ -1823,7 +1825,7 @@ }, { "cell_type": "markdown", - "id": "ef5054ab", + "id": "a3acd8de", "metadata": { "editable": true }, @@ -1834,7 +1836,7 @@ }, { "cell_type": "markdown", - "id": "f09ca19c", + "id": "ecea71ed", "metadata": { "editable": true }, @@ -1847,7 +1849,7 @@ }, { "cell_type": "markdown", - "id": "0efe59e0", + "id": "7f90f854", "metadata": { "editable": true }, @@ -1858,7 +1860,7 @@ }, { "cell_type": "markdown", - "id": "bc05c8e0", + "id": "b406ced8", "metadata": { "editable": true }, @@ -1871,7 +1873,7 @@ }, { "cell_type": "markdown", - "id": "57fe77fa", + "id": "d689a529", "metadata": { "editable": true }, @@ -1881,7 +1883,7 @@ }, { "cell_type": "markdown", - "id": "ee0eb549", + "id": "f12ac466", "metadata": { "editable": true }, @@ -1892,7 +1894,7 @@ }, { "cell_type": "markdown", - "id": "a9b8a0cb", + "id": "d7b74229", "metadata": { "editable": true }, @@ -1902,7 +1904,7 @@ }, { "cell_type": "markdown", - "id": "05e175db", + "id": "4d6354ad", "metadata": { "editable": true }, @@ -1912,7 +1914,7 @@ }, { "cell_type": "markdown", - "id": "f6ecf4e7", + "id": "eeffbcd7", "metadata": { "editable": true }, @@ -1922,7 +1924,7 @@ }, { "cell_type": "markdown", - "id": "c0ee7b75", + "id": "a2977107", "metadata": { "editable": true }, @@ -1932,7 +1934,7 @@ }, { "cell_type": "markdown", - "id": "74dcf2f8", + "id": "145e25ac", "metadata": { "editable": true }, @@ -1942,7 +1944,7 @@ }, { "cell_type": "markdown", - "id": "b72d5a50", + "id": "676d7ed0", "metadata": { "editable": true }, @@ -1952,7 +1954,7 @@ }, { "cell_type": "markdown", - "id": "6c936d22", + "id": "146bc27a", "metadata": { "editable": true }, @@ -1962,7 +1964,7 @@ }, { "cell_type": "markdown", - "id": "e57b5c9b", + "id": "0dc2e1d7", "metadata": { "editable": true }, @@ -1976,7 +1978,7 @@ }, { "cell_type": "markdown", - "id": "591ea5f8", + "id": "98e50fec", "metadata": { "editable": true }, @@ -1988,7 +1990,7 @@ }, { "cell_type": "markdown", - "id": "86e62af7", + "id": "878bb19c", "metadata": { "editable": true }, @@ -1998,7 +2000,7 @@ }, { "cell_type": "markdown", - "id": "d4eadc28", + "id": "0b80829f", "metadata": { "editable": true }, @@ -2012,7 +2014,7 @@ }, { "cell_type": "markdown", - "id": "9fb55622", + "id": "4b7841dd", "metadata": { "editable": true }, @@ -2025,7 +2027,7 @@ }, { "cell_type": "markdown", - "id": "b4c7cce6", + "id": "5bc29126", "metadata": { "editable": true }, @@ -2035,7 +2037,7 @@ }, { "cell_type": "markdown", - "id": "63f1d357", + "id": "f5bac65f", "metadata": { "editable": true }, @@ -2049,7 +2051,7 @@ }, { "cell_type": "markdown", - "id": "a485face", + "id": "69a7c7c4", "metadata": { "editable": true }, @@ -2067,7 +2069,7 @@ }, { "cell_type": "markdown", - "id": "ad55a449", + "id": "ec082f22", "metadata": { "editable": true }, @@ -2077,7 +2079,7 @@ }, { "cell_type": "markdown", - "id": "776ddba9", + "id": "22ab5df8", "metadata": { "editable": true }, @@ -2089,7 +2091,7 @@ }, { "cell_type": "markdown", - "id": "ca642f90", + "id": "48e3d47a", "metadata": { "editable": true }, @@ -2103,7 +2105,7 @@ }, { "cell_type": "markdown", - "id": "14123a0d", + "id": "fae476ae", "metadata": { "editable": true }, @@ -2113,7 +2115,7 @@ }, { "cell_type": "markdown", - "id": "3a9e1c79", + "id": "314371af", "metadata": { "editable": true }, @@ -2127,7 +2129,7 @@ }, { "cell_type": "markdown", - "id": "0224f0d3", + "id": "42d8c853", "metadata": { "editable": true }, @@ -2139,7 +2141,7 @@ }, { "cell_type": "markdown", - "id": "4643b973", + "id": "970bc318", "metadata": { "editable": true }, @@ -2149,7 +2151,7 @@ }, { "cell_type": "markdown", - "id": "bd6152a6", + "id": "90869c50", "metadata": { "editable": true }, @@ -2162,7 +2164,7 @@ }, { "cell_type": "markdown", - "id": "3d86ddd8", + "id": "890532a2", "metadata": { "editable": true }, @@ -2175,7 +2177,7 @@ }, { "cell_type": "markdown", - "id": "adb31489", + "id": "992df998", "metadata": { "editable": true }, @@ -2185,7 +2187,7 @@ }, { "cell_type": "markdown", - "id": "77d0b5d3", + "id": "fb008853", "metadata": { "editable": true }, @@ -2198,7 +2200,7 @@ }, { "cell_type": "markdown", - "id": "c31215ca", + "id": "a4470689", "metadata": { "editable": true }, @@ -2208,7 +2210,7 @@ }, { "cell_type": "markdown", - "id": "46ed8de3", + "id": "b862d1e3", "metadata": { "editable": true }, @@ -2221,7 +2223,7 @@ }, { "cell_type": "markdown", - "id": "85a36227", + "id": "fe4a7658", "metadata": { "editable": true }, @@ -2234,7 +2236,7 @@ }, { "cell_type": "markdown", - "id": "8e5714f0", + "id": "1ff661d2", "metadata": { "editable": true }, @@ -2245,7 +2247,7 @@ }, { "cell_type": "markdown", - "id": "efcc358e", + "id": "2b3f0ad3", "metadata": { "editable": true }, @@ -2259,7 +2261,7 @@ }, { "cell_type": "markdown", - "id": "925846f6", + "id": "4e0c1946", "metadata": { "editable": true }, @@ -2273,7 +2275,7 @@ }, { "cell_type": "markdown", - "id": "30c20cb6", + "id": "d882193b", "metadata": { "editable": true }, @@ -2301,7 +2303,7 @@ }, { "cell_type": "markdown", - "id": "40b4b92c", + "id": "6fd5fa43", "metadata": { "editable": true }, @@ -2321,7 +2323,7 @@ }, { "cell_type": "markdown", - "id": "da5377a0", + "id": "68de093f", "metadata": { "editable": true }, @@ -2333,7 +2335,7 @@ }, { "cell_type": "markdown", - "id": "36b63048", + "id": "16bb3303", "metadata": { "editable": true }, @@ -2349,7 +2351,7 @@ }, { "cell_type": "markdown", - "id": "b73861c2", + "id": "ee8650ac", "metadata": { "editable": true }, @@ -2368,7 +2370,7 @@ }, { "cell_type": "markdown", - "id": "9aec5620", + "id": "9edf7838", "metadata": { "editable": true }, @@ -2388,7 +2390,7 @@ }, { "cell_type": "markdown", - "id": "f9313bb0", + "id": "b9cc9f8c", "metadata": { "editable": true }, @@ -2406,7 +2408,7 @@ }, { "cell_type": "markdown", - "id": "7a2037ca", + "id": "5ecfabb8", "metadata": { "editable": true }, @@ -2427,7 +2429,7 @@ }, { "cell_type": "markdown", - "id": "2e1caede", + "id": "d413e36d", "metadata": { "editable": true }, @@ -2444,7 +2446,7 @@ }, { "cell_type": "markdown", - "id": "08ecf53f", + "id": "a8affa8f", "metadata": { "editable": true }, @@ -2454,7 +2456,7 @@ }, { "cell_type": "markdown", - "id": "1d4b1934", + "id": "29e4a4c8", "metadata": { "editable": true }, @@ -2485,7 +2487,7 @@ }, { "cell_type": "markdown", - "id": "ea777020", + "id": "5152bc9c", "metadata": { "editable": true }, @@ -2496,7 +2498,7 @@ }, { "cell_type": "markdown", - "id": "41c4860a", + "id": "391ad490", "metadata": { "editable": true }, @@ -2508,7 +2510,7 @@ }, { "cell_type": "markdown", - "id": "d3d46adc", + "id": "42196c57", "metadata": { "editable": true }, @@ -2520,7 +2522,7 @@ }, { "cell_type": "markdown", - "id": "75b8a579", + "id": "997f90e8", "metadata": { "editable": true }, @@ -2530,7 +2532,7 @@ }, { "cell_type": "markdown", - "id": "d554b123", + "id": "09c0fd0e", "metadata": { "editable": true }, @@ -2542,7 +2544,7 @@ }, { "cell_type": "markdown", - "id": "936fd39c", + "id": "f5b6ca8d", "metadata": { "editable": true }, @@ -2552,7 +2554,7 @@ }, { "cell_type": "markdown", - "id": "a862d627", + "id": "c98b964a", "metadata": { "editable": true }, @@ -2582,7 +2584,7 @@ }, { "cell_type": "markdown", - "id": "5684ece5", + "id": "e758d05e", "metadata": { "editable": true }, @@ -2593,7 +2595,7 @@ }, { "cell_type": "markdown", - "id": "ef75d9c4", + "id": "727d974a", "metadata": { "editable": true }, @@ -2605,7 +2607,7 @@ }, { "cell_type": "markdown", - "id": "9cce31b6", + "id": "1d6271fe", "metadata": { "editable": true }, @@ -2615,7 +2617,7 @@ }, { "cell_type": "markdown", - "id": "f9769935", + "id": "1a876a5a", "metadata": { "editable": true }, @@ -2627,7 +2629,7 @@ }, { "cell_type": "markdown", - "id": "5f461a3e", + "id": "8534c118", "metadata": { "editable": true }, @@ -2637,7 +2639,7 @@ }, { "cell_type": "markdown", - "id": "3cdbaea1", + "id": "332d31d5", "metadata": { "editable": true }, @@ -2649,7 +2651,7 @@ }, { "cell_type": "markdown", - "id": "23e1510f", + "id": "b78e43f4", "metadata": { "editable": true }, @@ -2671,7 +2673,7 @@ }, { "cell_type": "markdown", - "id": "0856ac03", + "id": "521f357b", "metadata": { "editable": true }, @@ -2692,7 +2694,7 @@ }, { "cell_type": "markdown", - "id": "454ee61c", + "id": "9f65dc0f", "metadata": { "editable": true }, @@ -2702,7 +2704,7 @@ }, { "cell_type": "markdown", - "id": "e44cdc6b", + "id": "9eda2cc0", "metadata": { "editable": true }, @@ -2759,7 +2761,7 @@ }, { "cell_type": "markdown", - "id": "a9959a86", + "id": "d24069ca", "metadata": { "editable": true }, @@ -2770,7 +2772,7 @@ }, { "cell_type": "markdown", - "id": "da017900", + "id": "66e0fbdf", "metadata": { "editable": true }, @@ -2844,7 +2846,7 @@ }, { "cell_type": "markdown", - "id": "8cc611e7", + "id": "25a8ada7", "metadata": { "editable": true }, @@ -2863,7 +2865,7 @@ }, { "cell_type": "markdown", - "id": "6f8e7fda", + "id": "96334923", "metadata": { "editable": true }, @@ -2879,7 +2881,7 @@ }, { "cell_type": "markdown", - "id": "fc160e54", + "id": "1086e7f6", "metadata": { "editable": true }, @@ -2894,7 +2896,7 @@ }, { "cell_type": "markdown", - "id": "35d4c6fc", + "id": "9a29659c", "metadata": { "editable": true }, @@ -2912,7 +2914,7 @@ }, { "cell_type": "markdown", - "id": "42361424", + "id": "1c92c7b7", "metadata": { "editable": true }, @@ -2929,7 +2931,7 @@ }, { "cell_type": "markdown", - "id": "1aaeb684", + "id": "7d8a1f2c", "metadata": { "editable": true }, @@ -2940,7 +2942,7 @@ }, { "cell_type": "markdown", - "id": "b81bb408", + "id": "d69c7bc3", "metadata": { "editable": true }, @@ -2952,7 +2954,7 @@ }, { "cell_type": "markdown", - "id": "b0311044", + "id": "4b683a55", "metadata": { "editable": true }, @@ -2962,7 +2964,7 @@ }, { "cell_type": "markdown", - "id": "5804b8d3", + "id": "8accb3c3", "metadata": { "editable": true }, @@ -2974,7 +2976,7 @@ }, { "cell_type": "markdown", - "id": "ee810179", + "id": "5efec843", "metadata": { "editable": true }, @@ -2984,7 +2986,7 @@ }, { "cell_type": "markdown", - "id": "ec2840b5", + "id": "b5e15977", "metadata": { "editable": true }, @@ -2996,7 +2998,7 @@ }, { "cell_type": "markdown", - "id": "ae2e230a", + "id": "032ee329", "metadata": { "editable": true }, @@ -3006,7 +3008,7 @@ }, { "cell_type": "markdown", - "id": "d67d97e1", + "id": "6d0abbee", "metadata": { "editable": true }, @@ -3018,7 +3020,7 @@ }, { "cell_type": "markdown", - "id": "1525f768", + "id": "b4b3b622", "metadata": { "editable": true }, @@ -3030,7 +3032,7 @@ }, { "cell_type": "markdown", - "id": "3e5a0366", + "id": "8d6b48de", "metadata": { "editable": true }, @@ -3046,7 +3048,7 @@ }, { "cell_type": "markdown", - "id": "0a3d70dd", + "id": "882c8bb8", "metadata": { "editable": true }, @@ -3064,7 +3066,7 @@ }, { "cell_type": "markdown", - "id": "3693a713", + "id": "188afc27", "metadata": { "editable": true }, @@ -3085,7 +3087,7 @@ }, { "cell_type": "markdown", - "id": "5a5ccdc1", + "id": "be80bfdb", "metadata": { "editable": true }, @@ -3099,7 +3101,7 @@ }, { "cell_type": "markdown", - "id": "c59cac43", + "id": "3061ce2d", "metadata": { "editable": true }, @@ -3112,7 +3114,7 @@ }, { "cell_type": "markdown", - "id": "4f099832", + "id": "18e5ec20", "metadata": { "editable": true }, @@ -3125,7 +3127,7 @@ }, { "cell_type": "markdown", - "id": "f34cc9cb", + "id": "05ee10db", "metadata": { "editable": true }, @@ -3138,7 +3140,7 @@ }, { "cell_type": "markdown", - "id": "991094cb", + "id": "97403a45", "metadata": { "editable": true }, @@ -3154,7 +3156,7 @@ }, { "cell_type": "markdown", - "id": "68e14aec", + "id": "3036ce8a", "metadata": { "editable": true }, @@ -3164,7 +3166,7 @@ }, { "cell_type": "markdown", - "id": "947df6e7", + "id": "372c7f77", "metadata": { "editable": true }, @@ -3174,7 +3176,7 @@ }, { "cell_type": "markdown", - "id": "3b580a58", + "id": "ad4ce4ba", "metadata": { "editable": true }, @@ -3184,7 +3186,7 @@ }, { "cell_type": "markdown", - "id": "cf57172d", + "id": "7aa3ee46", "metadata": { "editable": true }, @@ -3195,7 +3197,7 @@ }, { "cell_type": "markdown", - "id": "8954eb04", + "id": "e4145636", "metadata": { "editable": true }, @@ -3206,7 +3208,7 @@ }, { "cell_type": "markdown", - "id": "4671e326", + "id": "c7dfa255", "metadata": { "editable": true }, @@ -3217,7 +3219,7 @@ }, { "cell_type": "markdown", - "id": "b08a52e8", + "id": "821936b4", "metadata": { "editable": true }, @@ -3227,7 +3229,7 @@ }, { "cell_type": "markdown", - "id": "a861537e", + "id": "f50ba0b0", "metadata": { "editable": true }, @@ -3237,7 +3239,7 @@ }, { "cell_type": "markdown", - "id": "d4185b13", + "id": "78c6d578", "metadata": { "editable": true }, @@ -3247,7 +3249,7 @@ }, { "cell_type": "markdown", - "id": "fdbf1c9f", + "id": "9dca4d6b", "metadata": { "editable": true }, @@ -3257,7 +3259,7 @@ }, { "cell_type": "markdown", - "id": "4fa81e38", + "id": "fa2ae697", "metadata": { "editable": true }, @@ -3270,7 +3272,7 @@ }, { "cell_type": "markdown", - "id": "5fda1f95", + "id": "e3fc56ca", "metadata": { "editable": true }, @@ -3290,7 +3292,7 @@ }, { "cell_type": "markdown", - "id": "5a56b5b5", + "id": "c3b38944", "metadata": { "editable": true }, @@ -3306,7 +3308,7 @@ }, { "cell_type": "markdown", - "id": "e9bda343", + "id": "45b95d3e", "metadata": { "editable": true }, @@ -3316,7 +3318,7 @@ }, { "cell_type": "markdown", - "id": "db0d73df", + "id": "5e70388f", "metadata": { "editable": true }, @@ -3326,7 +3328,7 @@ }, { "cell_type": "markdown", - "id": "f3a3c4b5", + "id": "b0ed2ea2", "metadata": { "editable": true }, @@ -3336,7 +3338,7 @@ }, { "cell_type": "markdown", - "id": "373dc478", + "id": "d446a947", "metadata": { "editable": true }, @@ -3359,7 +3361,7 @@ }, { "cell_type": "markdown", - "id": "6958b4e5", + "id": "8b31c635", "metadata": { "editable": true }, @@ -3372,7 +3374,7 @@ }, { "cell_type": "markdown", - "id": "b33344da", + "id": "7bfd2f1e", "metadata": { "editable": true }, @@ -3382,7 +3384,7 @@ }, { "cell_type": "markdown", - "id": "fdeedc35", + "id": "b076a236", "metadata": { "editable": true }, @@ -3392,7 +3394,7 @@ }, { "cell_type": "markdown", - "id": "eee795e9", + "id": "281e69e0", "metadata": { "editable": true }, @@ -3402,7 +3404,7 @@ }, { "cell_type": "markdown", - "id": "ad0bbfc5", + "id": "7369452c", "metadata": { "editable": true }, @@ -3412,7 +3414,7 @@ }, { "cell_type": "markdown", - "id": "6a84fe02", + "id": "44fb9048", "metadata": { "editable": true }, @@ -3427,7 +3429,7 @@ }, { "cell_type": "markdown", - "id": "71b44913", + "id": "e3519ab9", "metadata": { "editable": true }, @@ -3437,7 +3439,7 @@ }, { "cell_type": "markdown", - "id": "042775ec", + "id": "20e5dd91", "metadata": { "editable": true }, @@ -3449,7 +3451,7 @@ }, { "cell_type": "markdown", - "id": "03b93bde", + "id": "9ac13a49", "metadata": { "editable": true }, @@ -3473,7 +3475,7 @@ }, { "cell_type": "markdown", - "id": "8f65b101", + "id": "3621cf41", "metadata": { "editable": true }, @@ -3485,7 +3487,7 @@ }, { "cell_type": "markdown", - "id": "4d44f32f", + "id": "72c6466d", "metadata": { "editable": true }, @@ -3508,7 +3510,7 @@ }, { "cell_type": "markdown", - "id": "f5fca32b", + "id": "50326108", "metadata": { "editable": true }, @@ -3518,7 +3520,7 @@ }, { "cell_type": "markdown", - "id": "6e7dbd2c", + "id": "f407c5a4", "metadata": { "editable": true }, @@ -3538,7 +3540,7 @@ }, { "cell_type": "markdown", - "id": "5c0f2635", + "id": "737c6309", "metadata": { "editable": true }, @@ -3554,7 +3556,7 @@ }, { "cell_type": "markdown", - "id": "61669cf2", + "id": "ea6e04d2", "metadata": { "editable": true }, @@ -3564,7 +3566,7 @@ }, { "cell_type": "markdown", - "id": "b067e83a", + "id": "e179d073", "metadata": { "editable": true }, @@ -3585,7 +3587,7 @@ }, { "cell_type": "markdown", - "id": "76158876", + "id": "88698fd3", "metadata": { "editable": true }, @@ -3604,7 +3606,7 @@ }, { "cell_type": "markdown", - "id": "f50ed3d1", + "id": "499296fd", "metadata": { "editable": true }, @@ -3614,7 +3616,7 @@ }, { "cell_type": "markdown", - "id": "b548728b", + "id": "80f61521", "metadata": { "editable": true }, @@ -3638,7 +3640,7 @@ }, { "cell_type": "markdown", - "id": "8085bb44", + "id": "ef7b7655", "metadata": { "editable": true }, @@ -3651,7 +3653,7 @@ }, { "cell_type": "markdown", - "id": "20b2a3b9", + "id": "0ee6b027", "metadata": { "editable": true }, @@ -3663,7 +3665,7 @@ }, { "cell_type": "markdown", - "id": "04cce86a", + "id": "b196f1c4", "metadata": { "editable": true }, @@ -3679,7 +3681,7 @@ }, { "cell_type": "markdown", - "id": "158cd5e2", + "id": "b06e842e", "metadata": { "editable": true }, @@ -3692,7 +3694,7 @@ }, { "cell_type": "markdown", - "id": "10926207", + "id": "710086f2", "metadata": { "editable": true }, @@ -3704,7 +3706,7 @@ }, { "cell_type": "markdown", - "id": "d0dece69", + "id": "4d3a1771", "metadata": { "editable": true }, @@ -3723,7 +3725,7 @@ }, { "cell_type": "markdown", - "id": "3d053e2f", + "id": "07e57403", "metadata": { "editable": true }, @@ -3738,7 +3740,7 @@ }, { "cell_type": "markdown", - "id": "9668b407", + "id": "6b931e78", "metadata": { "editable": true }, @@ -3750,7 +3752,7 @@ }, { "cell_type": "markdown", - "id": "12eb35b9", + "id": "84ebd8a9", "metadata": { "editable": true }, @@ -3760,7 +3762,7 @@ }, { "cell_type": "markdown", - "id": "02f90f49", + "id": "1d5c5663", "metadata": { "editable": true }, @@ -3770,7 +3772,7 @@ }, { "cell_type": "markdown", - "id": "24fb6f69", + "id": "8f7a6314", "metadata": { "editable": true }, @@ -3795,7 +3797,7 @@ }, { "cell_type": "markdown", - "id": "1eda1f15", + "id": "e9c01031", "metadata": { "editable": true }, @@ -3805,7 +3807,7 @@ }, { "cell_type": "markdown", - "id": "62c86791", + "id": "f19a6651", "metadata": { "editable": true }, @@ -3828,7 +3830,7 @@ }, { "cell_type": "markdown", - "id": "d203a19a", + "id": "d611825a", "metadata": { "editable": true }, @@ -3838,7 +3840,7 @@ }, { "cell_type": "markdown", - "id": "be8560b4", + "id": "a70a0737", "metadata": { "editable": true }, @@ -3863,7 +3865,7 @@ }, { "cell_type": "markdown", - "id": "683a1a3a", + "id": "fafdaa43", "metadata": { "editable": true }, @@ -3875,7 +3877,7 @@ }, { "cell_type": "markdown", - "id": "4a3d1ec8", + "id": "fd41d43b", "metadata": { "editable": true }, @@ -3886,7 +3888,7 @@ }, { "cell_type": "markdown", - "id": "b96a8840", + "id": "6f6f7ccc", "metadata": { "editable": true }, @@ -3903,7 +3905,7 @@ }, { "cell_type": "markdown", - "id": "bcc235b8", + "id": "24581a53", "metadata": { "editable": true }, @@ -3920,7 +3922,7 @@ }, { "cell_type": "markdown", - "id": "6f521ed1", + "id": "0e6fdf1f", "metadata": { "editable": true }, @@ -3931,7 +3933,7 @@ }, { "cell_type": "markdown", - "id": "307ef4f1", + "id": "c02c5d53", "metadata": { "editable": true }, @@ -3944,7 +3946,7 @@ }, { "cell_type": "markdown", - "id": "1a2f72b4", + "id": "215cfb7c", "metadata": { "editable": true }, @@ -3960,7 +3962,7 @@ }, { "cell_type": "markdown", - "id": "214ebedd", + "id": "f4267db8", "metadata": { "editable": true }, @@ -3970,7 +3972,7 @@ }, { "cell_type": "markdown", - "id": "96619240", + "id": "584fed2f", "metadata": { "editable": true }, @@ -3997,7 +3999,7 @@ }, { "cell_type": "markdown", - "id": "740865e4", + "id": "fff0c1fa", "metadata": { "editable": true }, @@ -4007,7 +4009,7 @@ }, { "cell_type": "markdown", - "id": "484916ee", + "id": "16ad2582", "metadata": { "editable": true }, @@ -4473,7 +4475,7 @@ }, { "cell_type": "markdown", - "id": "a2672ee2", + "id": "8001465e", "metadata": { "editable": true }, @@ -4503,7 +4505,7 @@ }, { "cell_type": "markdown", - "id": "ab7b0b53", + "id": "c58e3959", "metadata": { "editable": true }, @@ -4514,7 +4516,7 @@ }, { "cell_type": "markdown", - "id": "50cd6880", + "id": "7e30d9dc", "metadata": { "editable": true }, @@ -4524,7 +4526,7 @@ }, { "cell_type": "markdown", - "id": "4b877c85", + "id": "d523b4fa", "metadata": { "editable": true }, @@ -4534,7 +4536,7 @@ }, { "cell_type": "markdown", - "id": "edc15526", + "id": "1cde3797", "metadata": { "editable": true }, @@ -4544,7 +4546,7 @@ }, { "cell_type": "markdown", - "id": "4f80c44e", + "id": "477f580a", "metadata": { "editable": true }, @@ -4560,7 +4562,7 @@ }, { "cell_type": "markdown", - "id": "ae290278", + "id": "e9b3615c", "metadata": { "editable": true }, @@ -4571,7 +4573,7 @@ }, { "cell_type": "markdown", - "id": "0b47582b", + "id": "b14e5170", "metadata": { "editable": true }, @@ -4581,7 +4583,7 @@ }, { "cell_type": "markdown", - "id": "29680064", + "id": "004c2ab7", "metadata": { "editable": true }, @@ -4591,7 +4593,7 @@ }, { "cell_type": "markdown", - "id": "add80e8f", + "id": "ab806620", "metadata": { "editable": true }, @@ -4601,7 +4603,7 @@ }, { "cell_type": "markdown", - "id": "badeef32", + "id": "c0dd8be6", "metadata": { "editable": true }, @@ -4615,7 +4617,7 @@ }, { "cell_type": "markdown", - "id": "1655cea6", + "id": "a28c16f1", "metadata": { "editable": true }, @@ -4636,7 +4638,7 @@ }, { "cell_type": "markdown", - "id": "542d46d6", + "id": "02c67899", "metadata": { "editable": true }, @@ -4646,7 +4648,7 @@ }, { "cell_type": "markdown", - "id": "5f3a057b", + "id": "9f22aa20", "metadata": { "editable": true }, @@ -4674,7 +4676,7 @@ }, { "cell_type": "markdown", - "id": "e600a25b", + "id": "adfc98a6", "metadata": { "editable": true }, @@ -4687,7 +4689,7 @@ }, { "cell_type": "markdown", - "id": "7cd27256", + "id": "d9d71aaa", "metadata": { "editable": true }, @@ -4697,7 +4699,7 @@ }, { "cell_type": "markdown", - "id": "4af8979b", + "id": "eeace845", "metadata": { "editable": true }, @@ -4715,7 +4717,7 @@ }, { "cell_type": "markdown", - "id": "d314f63a", + "id": "bd6b6109", "metadata": { "editable": true }, @@ -4725,7 +4727,7 @@ }, { "cell_type": "markdown", - "id": "476b27d6", + "id": "fe258a73", "metadata": { "editable": true }, @@ -4751,7 +4753,7 @@ }, { "cell_type": "markdown", - "id": "b3150e25", + "id": "2d3800ea", "metadata": { "editable": true }, @@ -4761,7 +4763,7 @@ }, { "cell_type": "markdown", - "id": "d7994f3d", + "id": "b74cb4ab", "metadata": { "editable": true }, @@ -4783,7 +4785,7 @@ }, { "cell_type": "markdown", - "id": "a1b0ca46", + "id": "0757e417", "metadata": { "editable": true }, @@ -4794,7 +4796,7 @@ }, { "cell_type": "markdown", - "id": "af8f316a", + "id": "e03e3257", "metadata": { "editable": true }, @@ -4804,7 +4806,7 @@ }, { "cell_type": "markdown", - "id": "f2def2ed", + "id": "9d6da977", "metadata": { "editable": true }, @@ -4815,7 +4817,7 @@ }, { "cell_type": "markdown", - "id": "5a2e2450", + "id": "7b972a17", "metadata": { "editable": true }, @@ -4833,7 +4835,7 @@ }, { "cell_type": "markdown", - "id": "66bbd83e", + "id": "6faa8d57", "metadata": { "editable": true }, @@ -4844,7 +4846,7 @@ }, { "cell_type": "markdown", - "id": "6d307151", + "id": "a72c6377", "metadata": { "editable": true }, @@ -4858,7 +4860,7 @@ }, { "cell_type": "markdown", - "id": "2a35060f", + "id": "1b3261d6", "metadata": { "editable": true }, @@ -4872,7 +4874,7 @@ }, { "cell_type": "markdown", - "id": "9721e8f2", + "id": "8c038a25", "metadata": { "editable": true }, @@ -4883,7 +4885,7 @@ }, { "cell_type": "markdown", - "id": "a99bffb1", + "id": "357bace0", "metadata": { "editable": true }, @@ -4900,7 +4902,7 @@ }, { "cell_type": "markdown", - "id": "bf58be92", + "id": "2d6839dc", "metadata": { "editable": true }, @@ -4911,7 +4913,7 @@ }, { "cell_type": "markdown", - "id": "b9ad7fc0", + "id": "c623ee25", "metadata": { "editable": true }, @@ -4921,7 +4923,7 @@ }, { "cell_type": "markdown", - "id": "dfd937c8", + "id": "3b608171", "metadata": { "editable": true }, @@ -4945,7 +4947,7 @@ }, { "cell_type": "markdown", - "id": "39645b2c", + "id": "6f92aec8", "metadata": { "editable": true }, @@ -4957,7 +4959,7 @@ }, { "cell_type": "markdown", - "id": "31bf0645", + "id": "993af8db", "metadata": { "editable": true }, @@ -4968,7 +4970,7 @@ }, { "cell_type": "markdown", - "id": "867c7391", + "id": "fc10904a", "metadata": { "editable": true }, @@ -4980,7 +4982,7 @@ }, { "cell_type": "markdown", - "id": "bf4d473a", + "id": "99b8781e", "metadata": { "editable": true }, @@ -4998,7 +5000,7 @@ }, { "cell_type": "markdown", - "id": "a1e4c6e9", + "id": "70611f2f", "metadata": { "editable": true }, @@ -5008,7 +5010,7 @@ }, { "cell_type": "markdown", - "id": "288b6627", + "id": "dff4b4c4", "metadata": { "editable": true }, @@ -5032,7 +5034,7 @@ }, { "cell_type": "markdown", - "id": "bcbc9208", + "id": "95ed13d4", "metadata": { "editable": true }, @@ -5042,7 +5044,7 @@ }, { "cell_type": "markdown", - "id": "a9bcffe4", + "id": "c80f95e7", "metadata": { "editable": true }, @@ -5066,7 +5068,7 @@ }, { "cell_type": "markdown", - "id": "2e3fc84f", + "id": "c5505d2e", "metadata": { "editable": true }, @@ -5087,7 +5089,7 @@ }, { "cell_type": "markdown", - "id": "f8bc1f7a", + "id": "e46cbeb1", "metadata": { "editable": true }, @@ -5097,7 +5099,7 @@ }, { "cell_type": "markdown", - "id": "b573c665", + "id": "d215692e", "metadata": { "editable": true }, @@ -5107,7 +5109,7 @@ }, { "cell_type": "markdown", - "id": "1549ba33", + "id": "7c2689f4", "metadata": { "editable": true }, @@ -5117,7 +5119,7 @@ }, { "cell_type": "markdown", - "id": "2c132037", + "id": "aea1cada", "metadata": { "editable": true }, @@ -5129,7 +5131,7 @@ }, { "cell_type": "markdown", - "id": "823a30b6", + "id": "35d873d7", "metadata": { "editable": true }, @@ -5140,7 +5142,7 @@ }, { "cell_type": "markdown", - "id": "e5ba8e90", + "id": "02a58894", "metadata": { "editable": true }, @@ -5150,7 +5152,7 @@ }, { "cell_type": "markdown", - "id": "7e924da6", + "id": "01e8e8c7", "metadata": { "editable": true }, @@ -5162,7 +5164,7 @@ }, { "cell_type": "markdown", - "id": "1c5b0ce4", + "id": "efa4a661", "metadata": { "editable": true }, @@ -5172,7 +5174,7 @@ }, { "cell_type": "markdown", - "id": "a94aa426", + "id": "4e341558", "metadata": { "editable": true }, @@ -5184,7 +5186,7 @@ }, { "cell_type": "markdown", - "id": "adbde99b", + "id": "d1993e22", "metadata": { "editable": true }, @@ -5200,7 +5202,7 @@ }, { "cell_type": "markdown", - "id": "1ae20e61", + "id": "20999aa1", "metadata": { "editable": true }, @@ -5212,7 +5214,7 @@ }, { "cell_type": "markdown", - "id": "d8886ebb", + "id": "6df214bc", "metadata": { "editable": true }, @@ -5233,7 +5235,7 @@ }, { "cell_type": "markdown", - "id": "ac0b7955", + "id": "34973e22", "metadata": { "editable": true }, @@ -5243,7 +5245,7 @@ }, { "cell_type": "markdown", - "id": "d43c39dd", + "id": "657c216b", "metadata": { "editable": true }, @@ -5253,7 +5255,7 @@ }, { "cell_type": "markdown", - "id": "4130a2e9", + "id": "59069634", "metadata": { "editable": true }, @@ -5265,7 +5267,7 @@ }, { "cell_type": "markdown", - "id": "6b6329b8", + "id": "28a4e062", "metadata": { "editable": true }, @@ -5275,7 +5277,7 @@ }, { "cell_type": "markdown", - "id": "79b51c95", + "id": "ca59b4b5", "metadata": { "editable": true }, @@ -5285,7 +5287,7 @@ }, { "cell_type": "markdown", - "id": "2d5289b2", + "id": "49695cd6", "metadata": { "editable": true }, @@ -5295,7 +5297,7 @@ }, { "cell_type": "markdown", - "id": "be2cac4e", + "id": "d10f80c8", "metadata": { "editable": true }, @@ -5305,7 +5307,7 @@ }, { "cell_type": "markdown", - "id": "8ce871cd", + "id": "53dd202f", "metadata": { "editable": true }, @@ -5315,7 +5317,7 @@ }, { "cell_type": "markdown", - "id": "3ac0c15c", + "id": "79834124", "metadata": { "editable": true }, @@ -5325,7 +5327,7 @@ }, { "cell_type": "markdown", - "id": "b1ee283d", + "id": "835f487e", "metadata": { "editable": true }, @@ -5336,7 +5338,7 @@ }, { "cell_type": "markdown", - "id": "ecf9f5f3", + "id": "3741a498", "metadata": { "editable": true }, @@ -5346,7 +5348,7 @@ }, { "cell_type": "markdown", - "id": "abdd42de", + "id": "1a7b247c", "metadata": { "editable": true }, @@ -5356,7 +5358,7 @@ }, { "cell_type": "markdown", - "id": "954b0f67", + "id": "aced1efc", "metadata": { "editable": true }, @@ -5366,7 +5368,7 @@ }, { "cell_type": "markdown", - "id": "c4f2c788", + "id": "11ab2f2b", "metadata": { "editable": true }, @@ -5376,7 +5378,7 @@ }, { "cell_type": "markdown", - "id": "f5d8f023", + "id": "22be04b9", "metadata": { "editable": true }, @@ -5386,7 +5388,7 @@ }, { "cell_type": "markdown", - "id": "c3275b66", + "id": "ac9751f9", "metadata": { "editable": true }, @@ -5412,7 +5414,7 @@ }, { "cell_type": "markdown", - "id": "99156479", + "id": "96e15531", "metadata": { "editable": true }, @@ -5430,7 +5432,7 @@ }, { "cell_type": "markdown", - "id": "17b40842", + "id": "a4287251", "metadata": { "editable": true }, @@ -5442,7 +5444,7 @@ }, { "cell_type": "markdown", - "id": "2a8760df", + "id": "5c7c551c", "metadata": { "editable": true }, @@ -5456,7 +5458,7 @@ }, { "cell_type": "markdown", - "id": "3a7bc98f", + "id": "4adc3d4a", "metadata": { "editable": true }, @@ -5466,7 +5468,7 @@ }, { "cell_type": "markdown", - "id": "7516a6a4", + "id": "38e013c4", "metadata": { "editable": true }, @@ -5481,7 +5483,7 @@ }, { "cell_type": "markdown", - "id": "b05312ed", + "id": "f7ece92d", "metadata": { "editable": true }, @@ -5493,7 +5495,7 @@ }, { "cell_type": "markdown", - "id": "5e955e7f", + "id": "0604579c", "metadata": { "editable": true }, @@ -5506,7 +5508,7 @@ }, { "cell_type": "markdown", - "id": "75d8fc33", + "id": "268e7c49", "metadata": { "editable": true }, @@ -5523,7 +5525,7 @@ }, { "cell_type": "markdown", - "id": "d071cea0", + "id": "47463a46", "metadata": { "editable": true }, @@ -5533,7 +5535,7 @@ }, { "cell_type": "markdown", - "id": "682d789a", + "id": "f692aedb", "metadata": { "editable": true }, @@ -5552,7 +5554,7 @@ }, { "cell_type": "markdown", - "id": "790aa2d3", + "id": "1034c179", "metadata": { "editable": true }, @@ -5563,7 +5565,7 @@ }, { "cell_type": "markdown", - "id": "30b561e2", + "id": "bdd76a05", "metadata": { "editable": true }, @@ -5577,7 +5579,7 @@ }, { "cell_type": "markdown", - "id": "4d2a0797", + "id": "4653e65f", "metadata": { "editable": true }, @@ -5587,7 +5589,7 @@ }, { "cell_type": "markdown", - "id": "3cd95899", + "id": "7eff17bd", "metadata": { "editable": true }, @@ -5605,7 +5607,7 @@ }, { "cell_type": "markdown", - "id": "68c2a4fe", + "id": "301a7620", "metadata": { "editable": true }, @@ -5619,7 +5621,7 @@ }, { "cell_type": "markdown", - "id": "047f7c41", + "id": "8fa3ad4d", "metadata": { "editable": true }, @@ -5634,7 +5636,7 @@ }, { "cell_type": "markdown", - "id": "e60ece38", + "id": "571816aa", "metadata": { "editable": true }, @@ -5646,7 +5648,7 @@ }, { "cell_type": "markdown", - "id": "f56129e7", + "id": "a9f819a5", "metadata": { "editable": true }, @@ -5656,7 +5658,7 @@ }, { "cell_type": "markdown", - "id": "7b02497f", + "id": "986b2d1c", "metadata": { "editable": true }, @@ -5666,7 +5668,7 @@ }, { "cell_type": "markdown", - "id": "466f17e6", + "id": "fbf8852e", "metadata": { "editable": true }, @@ -5676,7 +5678,7 @@ }, { "cell_type": "markdown", - "id": "35fb343a", + "id": "d92cb234", "metadata": { "editable": true }, @@ -5688,7 +5690,7 @@ }, { "cell_type": "markdown", - "id": "50838011", + "id": "3a9e198b", "metadata": { "editable": true }, @@ -5700,7 +5702,7 @@ }, { "cell_type": "markdown", - "id": "1ac70c07", + "id": "ccc68b6c", "metadata": { "editable": true }, @@ -5716,7 +5718,7 @@ }, { "cell_type": "markdown", - "id": "06de5d43", + "id": "dae6f381", "metadata": { "editable": true }, @@ -5728,7 +5730,7 @@ }, { "cell_type": "markdown", - "id": "4674fff0", + "id": "497fa67f", "metadata": { "editable": true }, @@ -5747,7 +5749,7 @@ }, { "cell_type": "markdown", - "id": "1a21898f", + "id": "cdc2e331", "metadata": { "editable": true }, @@ -5758,7 +5760,7 @@ }, { "cell_type": "markdown", - "id": "361afe33", + "id": "87f78818", "metadata": { "editable": true }, @@ -5769,7 +5771,7 @@ }, { "cell_type": "markdown", - "id": "0872a8b1", + "id": "6665433c", "metadata": { "editable": true }, @@ -5785,7 +5787,7 @@ }, { "cell_type": "markdown", - "id": "573d56fe", + "id": "9b325ce0", "metadata": { "editable": true }, @@ -5795,7 +5797,7 @@ }, { "cell_type": "markdown", - "id": "371fbbfd", + "id": "1cfa5def", "metadata": { "editable": true }, @@ -5818,7 +5820,7 @@ }, { "cell_type": "markdown", - "id": "7d7972d7", + "id": "36861233", "metadata": { "editable": true }, @@ -5828,7 +5830,7 @@ }, { "cell_type": "markdown", - "id": "7119f580", + "id": "12ae10ab", "metadata": { "editable": true }, @@ -5850,7 +5852,7 @@ }, { "cell_type": "markdown", - "id": "4a3de8bd", + "id": "3631d97f", "metadata": { "editable": true }, @@ -5860,7 +5862,7 @@ }, { "cell_type": "markdown", - "id": "dace2927", + "id": "4d4f96d5", "metadata": { "editable": true }, @@ -5871,7 +5873,7 @@ }, { "cell_type": "markdown", - "id": "fb2465d2", + "id": "266ee021", "metadata": { "editable": true }, @@ -5937,7 +5939,7 @@ }, { "cell_type": "markdown", - "id": "7aca0759", + "id": "f739a26a", "metadata": { "editable": true }, @@ -5948,7 +5950,7 @@ }, { "cell_type": "markdown", - "id": "e4f14771", + "id": "12ef260c", "metadata": { "editable": true }, diff --git a/doc/pub/week9/pdf/week9-beamer.pdf b/doc/pub/week9/pdf/week9-beamer.pdf index 1cc1de25..0fc8f629 100644 Binary files a/doc/pub/week9/pdf/week9-beamer.pdf and b/doc/pub/week9/pdf/week9-beamer.pdf differ diff --git a/doc/pub/week9/pdf/week9.pdf b/doc/pub/week9/pdf/week9.pdf index a0d81b42..e7f2de84 100644 Binary files a/doc/pub/week9/pdf/week9.pdf and b/doc/pub/week9/pdf/week9.pdf differ diff --git a/doc/src/week9/._week9-bs000.html b/doc/src/week9/._week9-bs000.html new file mode 100644 index 00000000..eb84da37 --- /dev/null +++ b/doc/src/week9/._week9-bs000.html @@ -0,0 +1,710 @@ + + + + + + + +
+
+
+
+
+
+
+ + +
+
+
+
+
+
+Morten Hjorth-Jensen Email morten.hjorth-jensen@fys.uio.no [1, 2]
+
+
+
+[1] Department of Physics and Center fo Computing in Science Education, University of Oslo, Oslo, Norway
+
+
+[2] Department of Physics and Astronomy and Facility for Rare Ion Beams, Michigan State University, East Lansing, Michigan, USA
+
+
+
+
+
+ + + + + + +
+
+Week 11, March 11-15: Resampling Techniques, Bootstrap and Blocking
++
March 11-15
++ + + + + + +
+ +
+ +
+
+
+
+
+
+
+ + +
Overview of week 11, March 11-15
+
+
+
+
+
+
+
+-
+
- Reminder from last week about statistical observables, the central limit theorem and bootstrapping, see notes from last week +
- Resampling Techniques, emphasis on Blocking +
- Discussion of onebody densities (whiteboard notes) +
- Start discussion on optimization and parallelization for Python and C++ + + +
Note, these notes contain additional material om optimization and parallelization. Parts of this material will be discussed this week.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Why resampling methods ?
+
+
+
+
+
+
+
+-
+
- Our simulations can be treated as computer experiments. This is particularly the case for Monte Carlo methods +
- The results can be analysed with the same statistical tools as we would use analysing experimental data. +
- As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors. +
+ +
+ +
+
+
+
+
+
+
+ + +
Statistical analysis
+
+
+
+
+
+
+
+-
+
- As in other experiments, many numerical experiments have two classes of errors:
+
- Statistical errors +
- Systematical errors + +
- Statistical errors can be estimated using standard tools from statistics +
- Systematical errors are method specific and must be treated differently from case to case. +
+ +
+ +
+
+
+
+
+
+
+ + +
And why do we use such methods?
+ +As you will see below, due to correlations between various +measurements, we need to evaluate the so-called covariance in order to +establish a proper evaluation of the total variance and the thereby +the standard deviation of a given expectation value. +
+ +The covariance however, leads to an evaluation of a double sum over the various stochastic variables. This becomes computationally too expensive to evaluate. +Methods like the Bootstrap, the Jackknife and/or Blocking allow us to circumvent this problem. +
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Central limit theorem
+ +Last week we derived the central limit theorem with the following assumptions:
+ +
+
+
+
+
+
+
+We assumed that each individual measurement \( x_{ij} \) is represented by stochastic variables which independent and identically distributed (iid). +This defined the sample mean of of experiment \( i \) with \( n \) samples as +
+$$ +\overline{x}_i=\frac{1}{n}\sum_{j} x_{ij}. +$$ + +and the sample variance
+$$ +\sigma^2_i=\frac{1}{n}\sum_{j} \left(x_{ij}-\overline{x}_i\right)^2. +$$ ++ +
+ +
+
+
+
+
+
+
+ + +
Further remarks
+ +Note that we use \( n \) instead of \( n-1 \) in the definition of +variance. The sample variance and the sample mean are not necessarily equal to +the exact values we would get if we knew the corresponding probability +distribution. +
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Running many measurements
+ +
+
+
+
+
+
+With the assumption that the average measurements \( i \) are also defined as iid stochastic variables and have the same probability function \( p \), +we defined the total average over \( m \) experiments as +
+$$ +\overline{X}=\frac{1}{m}\sum_{i} \overline{x}_{i}. +$$ + +and the total variance
+$$ +\sigma^2_{m}=\frac{1}{m}\sum_{i} \left( \overline{x}_{i}-\overline{X}\right)^2. +$$ +These are the quantities we used in showing that if the individual mean values are iid stochastic variables, then in the limit \( m\rightarrow \infty \), the distribution for \( \overline{X} \) is given by a Gaussian distribution with variance \( \sigma^2_m \).
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Adding more definitions
+ +The total sample variance over the \( mn \) measurements is defined as
+$$ +\sigma^2=\frac{1}{mn}\sum_{i=1}^{m} \sum_{j=1}^{n}\left(x_{ij}-\overline{X}\right)^2. +$$ + +We have from the equation for \( \sigma_m^2 \)
+$$ +\overline{x}_i-\overline{X}=\frac{1}{n}\sum_{j=1}^{n}\left(x_{i}-\overline{X}\right), +$$ + +and introducing the centered value \( \tilde{x}_{ij}=x_{ij}-\overline{X} \), we can rewrite \( \sigma_m^2 \) as
+$$ +\sigma^2_{m}=\frac{1}{m}\sum_{i} \left( \overline{x}_{i}-\overline{X}\right)^2=\frac{1}{m}\sum_{i=1}^{m}\left[ \frac{i}{n}\sum_{j=1}^{n}\tilde{x}_{ij}\right]^2. +$$ + + ++ +
+ +
+
+
+
+
+
+
+ + +
Further rewriting
+ +We can rewrite the latter in terms of a sum over diagonal elements only and another sum which contains the non-diagonal elements
+$$ +\begin{align*} +\sigma^2_{m}& =\frac{1}{m}\sum_{i=1}^{m}\left[ \frac{i}{n}\sum_{j=1}^{n}\tilde{x}_{ij}\right]^2 \\ + & = \frac{1}{mn^2}\sum_{i=1}^{m} \sum_{j=1}^{n}\tilde{x}_{ij}^2+\frac{2}{mn^2}\sum_{i=1}^{m} \sum_{j < k}^{n}\tilde{x}_{ij}\tilde{x}_{ik}. +\end{align*} +$$ + +The first term on the last rhs is nothing but the total sample variance \( \sigma^2 \) divided by \( m \). The second term represents the covariance.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
The covariance term
+ +Using the definition of the total sample variance we have
+$$ +\begin{align*} +\sigma^2_{m}& = \frac{\sigma^2}{m}+\frac{2}{mn^2}\sum_{i=1}^{m} \sum_{j < k}^{n}\tilde{x}_{ij}\tilde{x}_{ik}. +\end{align*} +$$ + +The first term is what we have used till now in order to estimate the +standard deviation. However, the second term which gives us a measure +of the correlations between different stochastic events, can result in +contributions which give rise to a larger standard deviation and +variance \( \sigma_m^2 \). Note also the evaluation of the second term +leads to a double sum over all events. If we run a VMC calculation +with say \( 10^9 \) Monte carlo samples, the latter term would lead to +\( 10^{18} \) function evaluations. We don't want to, by obvious reasons, to venture into that many evaluations. +
+ +Note also that if our stochastic events are iid then the covariance terms is zero.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Rewriting the covariance term
+ +We introduce now a variable \( d=\vert j-k\vert \) and rewrite
+$$ +\frac{2}{mn^2}\sum_{i=1}^{m} \sum_{j < k}^{n}\tilde{x}_{ij}\tilde{x}_{ik}, +$$ + +in terms of a function
+$$ +f_d=\frac{2}{mn}\sum_{i=1}^{m} \sum_{k=1}^{n-d}\tilde{x}_{ik}\tilde{x}_{i(k+d)}. +$$ + +We note that for \( d=0 \) we have
+$$ +f_0=\frac{2}{mn}\sum_{i=1}^{m} \sum_{k=1}^{n}\tilde{x}_{ik}\tilde{x}_{i(k)}=\sigma^2! +$$ + + ++ +
+ +
+
+
+
+
+
+
+ + +
Introducing the correlation function
+ +We introduce then a correlation function \( \kappa_d=f_d/\sigma^2 \). Note that \( \kappa_0 =1 \). We rewrite the variance \( \sigma_m^2 \) as
+$$ +\begin{align*} +\sigma^2_{m}& = \frac{\sigma^2}{m}\left[1+2\sum_{d=1}^{n-1} \kappa_d\right]. +\end{align*} +$$ + +The code here shows the evolution of \( \kappa_d \) as a function of \( d \) for a series of random numbers. We see that the function \( \kappa_d \) approaches \( 0 \) as \( d\rightarrow \infty \).
+ +Note: code will be inserted here later.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Resampling methods: Blocking
+ +The blocking method was made popular by Flyvbjerg and Pedersen (1989) +and has become one of the standard ways to estimate the variance +\( \mathrm{var}(\widehat{\theta}) \) for exactly one estimator \( \widehat{\theta} \), namely +\( \widehat{\theta} = \overline{X} \), the mean value. +
+ +Assume \( n = 2^d \) for some integer \( d>1 \) and \( X_1,X_2,\cdots, X_n \) is a stationary time series to begin with. +Moreover, assume that the series is asymptotically uncorrelated. We switch to vector notation by arranging \( X_1,X_2,\cdots,X_n \) in an \( n \)-tuple. Define: +
+$$ +\begin{align*} +\hat{X} = (X_1,X_2,\cdots,X_n). +\end{align*} +$$ + + ++ +
+ +
+
+
+
+
+
+
+ + +
Why blocking?
+ +The strength of the blocking method is when the number of +observations, \( n \) is large. For large \( n \), the complexity of dependent +bootstrapping scales poorly, but the blocking method does not, +moreover, it becomes more accurate the larger \( n \) is. +
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Blocking Transformations
+We now define the blocking transformations. The idea is to take the mean of subsequent +pair of elements from \( \boldsymbol{X} \) and form a new vector +\( \boldsymbol{X}_1 \). Continuing in the same way by taking the mean of +subsequent pairs of elements of \( \boldsymbol{X}_1 \) we obtain \( \boldsymbol{X}_2 \), and +so on. +Define \( \boldsymbol{X}_i \) recursively by: +
+ +$$ +\begin{align} +(\boldsymbol{X}_0)_k &\equiv (\boldsymbol{X})_k \nonumber \\ +(\boldsymbol{X}_{i+1})_k &\equiv \frac{1}{2}\Big( (\boldsymbol{X}_i)_{2k-1} + +(\boldsymbol{X}_i)_{2k} \Big) \qquad \text{for all} \qquad 1 \leq i \leq d-1 +\tag{1} +\end{align} +$$ + + ++ +
+ +
+
+
+
+
+
+
+ + +
Blocking transformations
+ +The quantity \( \boldsymbol{X}_k \) is +subject to \( k \) blocking transformations. We now have \( d \) vectors +\( \boldsymbol{X}_0, \boldsymbol{X}_1,\cdots,\vec X_{d-1} \) containing the subsequent +averages of observations. It turns out that if the components of +\( \boldsymbol{X} \) is a stationary time series, then the components of +\( \boldsymbol{X}_i \) is a stationary time series for all \( 0 \leq i \leq d-1 \) +
+ +We can then compute the autocovariance, the variance, sample mean, and +number of observations for each \( i \). +Let \( \gamma_i, \sigma_i^2, +\overline{X}_i \) denote the covariance, variance and average of the +elements of \( \boldsymbol{X}_i \) and let \( n_i \) be the number of elements of +\( \boldsymbol{X}_i \). It follows by induction that \( n_i = n/2^i \). +
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Blocking Transformations
+ +Using the +definition of the blocking transformation and the distributive +property of the covariance, it is clear that since \( h =|i-j| \) +we can define +
+$$ +\begin{align} +\gamma_{k+1}(h) &= cov\left( ({X}_{k+1})_{i}, ({X}_{k+1})_{j} \right) \nonumber \\ +&= \frac{1}{4}cov\left( ({X}_{k})_{2i-1} + ({X}_{k})_{2i}, ({X}_{k})_{2j-1} + ({X}_{k})_{2j} \right) \nonumber \\ +&= \frac{1}{2}\gamma_{k}(2h) + \frac{1}{2}\gamma_k(2h+1) \hspace{0.1cm} \mathrm{h = 0} +\tag{2}\\ +&=\frac{1}{4}\gamma_k(2h-1) + \frac{1}{2}\gamma_k(2h) + \frac{1}{4}\gamma_k(2h+1) \quad \mathrm{else} +\tag{3} +\end{align} +$$ + +The quantity \( \hat{X} \) is asymptotically uncorrelated by assumption, \( \hat{X}_k \) is also asymptotic uncorrelated. Let's turn our attention to the variance of the sample +mean \( \mathrm{var}(\overline{X}) \). +
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Blocking Transformations, getting there
+We have
+$$ +\begin{align} +\mathrm{var}(\overline{X}_k) = \frac{\sigma_k^2}{n_k} + \underbrace{\frac{2}{n_k} \sum_{h=1}^{n_k-1}\left( 1 - \frac{h}{n_k} \right)\gamma_k(h)}_{\equiv e_k} = \frac{\sigma^2_k}{n_k} + e_k \quad \text{if} \quad \gamma_k(0) = \sigma_k^2. +\tag{4} +\end{align} +$$ + +The term \( e_k \) is called the truncation error:
+$$ +\begin{equation} +e_k = \frac{2}{n_k} \sum_{h=1}^{n_k-1}\left( 1 - \frac{h}{n_k} \right)\gamma_k(h). +\tag{5} +\end{equation} +$$ + +We can show that \( \mathrm{var}(\overline{X}_i) = \mathrm{var}(\overline{X}_j) \) for all \( 0 \leq i \leq d-1 \) and \( 0 \leq j \leq d-1 \).
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Blocking Transformations, final expressions
+ +We can then wrap up
+$$ +\begin{align} +n_{j+1} \overline{X}_{j+1} &= \sum_{i=1}^{n_{j+1}} (\hat{X}_{j+1})_i = \frac{1}{2}\sum_{i=1}^{n_{j}/2} (\hat{X}_{j})_{2i-1} + (\hat{X}_{j})_{2i} \nonumber \\ +&= \frac{1}{2}\left[ (\hat{X}_j)_1 + (\hat{X}_j)_2 + \cdots + (\hat{X}_j)_{n_j} \right] = \underbrace{\frac{n_j}{2}}_{=n_{j+1}} \overline{X}_j = n_{j+1}\overline{X}_j. +\tag{6} +\end{align} +$$ + +By repeated use of this equation we get \( \mathrm{var}(\overline{X}_i) = \mathrm{var}(\overline{X}_0) = \mathrm{var}(\overline{X}) \) for all \( 0 \leq i \leq d-1 \). This has the consequence that
+$$ +\begin{align} +\mathrm{var}(\overline{X}) = \frac{\sigma_k^2}{n_k} + e_k \qquad \text{for all} \qquad 0 \leq k \leq d-1. \tag{7} +\end{align} +$$ + + ++ +
+ +
+
+
+
+
+
+
+ + +
More on the blocking method
+ +Flyvbjerg and Petersen demonstrated that the sequence +\( \{e_k\}_{k=0}^{d-1} \) is decreasing, and conjecture that the term +\( e_k \) can be made as small as we would like by making \( k \) (and hence +\( d \)) sufficiently large. The sequence is decreasing. +It means we can apply blocking transformations until +\( e_k \) is sufficiently small, and then estimate \( \mathrm{var}(\overline{X}) \) by +\( \widehat{\sigma}^2_k/n_k \). +
+ +For an elegant solution and proof of the blocking method, see the recent article of Marius Jonsson (former MSc student of the Computational Physics group).
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Example code form last week
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ # 2-electron VMC code for 2dim quantum dot with importance sampling +# Using gaussian rng for new positions and Metropolis- Hastings +# Added energy minimization +from math import exp, sqrt +from random import random, seed, normalvariate +import numpy as np +import matplotlib.pyplot as plt +from mpl_toolkits.mplot3d import Axes3D +from matplotlib import cm +from matplotlib.ticker import LinearLocator, FormatStrFormatter +from scipy.optimize import minimize +import sys +import os + +# Where to save data files +PROJECT_ROOT_DIR = "Results" +DATA_ID = "Results/EnergyMin" + +if not os.path.exists(PROJECT_ROOT_DIR): + os.mkdir(PROJECT_ROOT_DIR) + +if not os.path.exists(DATA_ID): + os.makedirs(DATA_ID) + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +outfile = open(data_path("Energies.dat"),'w') + + +# Trial wave function for the 2-electron quantum dot in two dims +def WaveFunction(r,alpha,beta): + r1 = r[0,0]**2 + r[0,1]**2 + r2 = r[1,0]**2 + r[1,1]**2 + r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2) + deno = r12/(1+beta*r12) + return exp(-0.5*alpha*(r1+r2)+deno) + +# Local energy for the 2-electron quantum dot in two dims, using analytical local energy +def LocalEnergy(r,alpha,beta): + + r1 = (r[0,0]**2 + r[0,1]**2) + r2 = (r[1,0]**2 + r[1,1]**2) + r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2) + deno = 1.0/(1+beta*r12) + deno2 = deno*deno + return 0.5*(1-alpha*alpha)*(r1 + r2) +2.0*alpha + 1.0/r12+deno2*(alpha*r12-deno2+2*beta*deno-1.0/r12) + +# Derivate of wave function ansatz as function of variational parameters +def DerivativeWFansatz(r,alpha,beta): + + WfDer = np.zeros((2), np.double) + r1 = (r[0,0]**2 + r[0,1]**2) + r2 = (r[1,0]**2 + r[1,1]**2) + r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2) + deno = 1.0/(1+beta*r12) + deno2 = deno*deno + WfDer[0] = -0.5*(r1+r2) + WfDer[1] = -r12*r12*deno2 + return WfDer + +# Setting up the quantum force for the two-electron quantum dot, recall that it is a vector +def QuantumForce(r,alpha,beta): + + qforce = np.zeros((NumberParticles,Dimension), np.double) + r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2) + deno = 1.0/(1+beta*r12) + qforce[0,:] = -2*r[0,:]*alpha*(r[0,:]-r[1,:])*deno*deno/r12 + qforce[1,:] = -2*r[1,:]*alpha*(r[1,:]-r[0,:])*deno*deno/r12 + return qforce + + +# Computing the derivative of the energy and the energy +def EnergyDerivative(x0): + + + # Parameters in the Fokker-Planck simulation of the quantum force + D = 0.5 + TimeStep = 0.05 + # positions + PositionOld = np.zeros((NumberParticles,Dimension), np.double) + PositionNew = np.zeros((NumberParticles,Dimension), np.double) + # Quantum force + QuantumForceOld = np.zeros((NumberParticles,Dimension), np.double) + QuantumForceNew = np.zeros((NumberParticles,Dimension), np.double) + + energy = 0.0 + DeltaE = 0.0 + alpha = x0[0] + beta = x0[1] + EnergyDer = 0.0 + DeltaPsi = 0.0 + DerivativePsiE = 0.0 + #Initial position + for i in range(NumberParticles): + for j in range(Dimension): + PositionOld[i,j] = normalvariate(0.0,1.0)*sqrt(TimeStep) + wfold = WaveFunction(PositionOld,alpha,beta) + QuantumForceOld = QuantumForce(PositionOld,alpha, beta) + + #Loop over MC MCcycles + for MCcycle in range(NumberMCcycles): + #Trial position moving one particle at the time + for i in range(NumberParticles): + for j in range(Dimension): + PositionNew[i,j] = PositionOld[i,j]+normalvariate(0.0,1.0)*sqrt(TimeStep)+\ + QuantumForceOld[i,j]*TimeStep*D + wfnew = WaveFunction(PositionNew,alpha,beta) + QuantumForceNew = QuantumForce(PositionNew,alpha, beta) + GreensFunction = 0.0 + for j in range(Dimension): + GreensFunction += 0.5*(QuantumForceOld[i,j]+QuantumForceNew[i,j])*\ + (D*TimeStep*0.5*(QuantumForceOld[i,j]-QuantumForceNew[i,j])-\ + PositionNew[i,j]+PositionOld[i,j]) + + GreensFunction = exp(GreensFunction) + ProbabilityRatio = GreensFunction*wfnew**2/wfold**2 + #Metropolis-Hastings test to see whether we accept the move + if random() <= ProbabilityRatio: + for j in range(Dimension): + PositionOld[i,j] = PositionNew[i,j] + QuantumForceOld[i,j] = QuantumForceNew[i,j] + wfold = wfnew + DeltaE = LocalEnergy(PositionOld,alpha,beta) + DerPsi = DerivativeWFansatz(PositionOld,alpha,beta) + DeltaPsi += DerPsi + energy += DeltaE + DerivativePsiE += DerPsi*DeltaE + + # We calculate mean values + energy /= NumberMCcycles + DerivativePsiE /= NumberMCcycles + DeltaPsi /= NumberMCcycles + EnergyDer = 2*(DerivativePsiE-DeltaPsi*energy) + return EnergyDer + + +# Computing the expectation value of the local energy +def Energy(x0): + # Parameters in the Fokker-Planck simulation of the quantum force + D = 0.5 + TimeStep = 0.05 + # positions + PositionOld = np.zeros((NumberParticles,Dimension), np.double) + PositionNew = np.zeros((NumberParticles,Dimension), np.double) + # Quantum force + QuantumForceOld = np.zeros((NumberParticles,Dimension), np.double) + QuantumForceNew = np.zeros((NumberParticles,Dimension), np.double) + + energy = 0.0 + DeltaE = 0.0 + alpha = x0[0] + beta = x0[1] + #Initial position + for i in range(NumberParticles): + for j in range(Dimension): + PositionOld[i,j] = normalvariate(0.0,1.0)*sqrt(TimeStep) + wfold = WaveFunction(PositionOld,alpha,beta) + QuantumForceOld = QuantumForce(PositionOld,alpha, beta) + + #Loop over MC MCcycles + for MCcycle in range(NumberMCcycles): + #Trial position moving one particle at the time + for i in range(NumberParticles): + for j in range(Dimension): + PositionNew[i,j] = PositionOld[i,j]+normalvariate(0.0,1.0)*sqrt(TimeStep)+\ + QuantumForceOld[i,j]*TimeStep*D + wfnew = WaveFunction(PositionNew,alpha,beta) + QuantumForceNew = QuantumForce(PositionNew,alpha, beta) + GreensFunction = 0.0 + for j in range(Dimension): + GreensFunction += 0.5*(QuantumForceOld[i,j]+QuantumForceNew[i,j])*\ + (D*TimeStep*0.5*(QuantumForceOld[i,j]-QuantumForceNew[i,j])-\ + PositionNew[i,j]+PositionOld[i,j]) + + GreensFunction = exp(GreensFunction) + ProbabilityRatio = GreensFunction*wfnew**2/wfold**2 + #Metropolis-Hastings test to see whether we accept the move + if random() <= ProbabilityRatio: + for j in range(Dimension): + PositionOld[i,j] = PositionNew[i,j] + QuantumForceOld[i,j] = QuantumForceNew[i,j] + wfold = wfnew + DeltaE = LocalEnergy(PositionOld,alpha,beta) + energy += DeltaE + if Printout: + outfile.write('%f\n' %(energy/(MCcycle+1.0))) + # We calculate mean values + energy /= NumberMCcycles + return energy + +#Here starts the main program with variable declarations +NumberParticles = 2 +Dimension = 2 +# seed for rng generator +seed() +# Monte Carlo cycles for parameter optimization +Printout = False +NumberMCcycles= 10000 +# guess for variational parameters +x0 = np.array([0.9,0.2]) +# Using Broydens method to find optimal parameters +res = minimize(Energy, x0, method='BFGS', jac=EnergyDerivative, options={'gtol': 1e-4,'disp': True}) +x0 = res.x +# Compute the energy again with the optimal parameters and increased number of Monte Cycles +NumberMCcycles= 2**19 +Printout = True +FinalEnergy = Energy(x0) +EResult = np.array([FinalEnergy,FinalEnergy]) +outfile.close() +#nice printout with Pandas +import pandas as pd +from pandas import DataFrame +data ={'Optimal Parameters':x0, 'Final Energy':EResult} +frame = pd.DataFrame(data) +print(frame) ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Resampling analysis
+ +The next step is then to use the above data sets and perform a +resampling analysis using the blocking method +The blocking code, based on the article of Marius Jonsson is given here +
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+ # Common imports +import os + +# Where to save the figures and data files +DATA_ID = "Results/EnergyMin" + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +infile = open(data_path("Energies.dat"),'r') + +from numpy import log2, zeros, mean, var, sum, loadtxt, arange, array, cumsum, dot, transpose, diagonal, sqrt +from numpy.linalg import inv + +def block(x): + # preliminaries + n = len(x) + d = int(log2(n)) + s, gamma = zeros(d), zeros(d) + mu = mean(x) + + # estimate the auto-covariance and variances + # for each blocking transformation + for i in arange(0,d): + n = len(x) + # estimate autocovariance of x + gamma[i] = (n)**(-1)*sum( (x[0:(n-1)]-mu)*(x[1:n]-mu) ) + # estimate variance of x + s[i] = var(x) + # perform blocking transformation + x = 0.5*(x[0::2] + x[1::2]) + + # generate the test observator M_k from the theorem + M = (cumsum( ((gamma/s)**2*2**arange(1,d+1)[::-1])[::-1] ) )[::-1] + + # we need a list of magic numbers + q =array([6.634897,9.210340, 11.344867, 13.276704, 15.086272, 16.811894, 18.475307, 20.090235, 21.665994, 23.209251, 24.724970, 26.216967, 27.688250, 29.141238, 30.577914, 31.999927, 33.408664, 34.805306, 36.190869, 37.566235, 38.932173, 40.289360, 41.638398, 42.979820, 44.314105, 45.641683, 46.962942, 48.278236, 49.587884, 50.892181]) + + # use magic to determine when we should have stopped blocking + for k in arange(0,d): + if(M[k] < q[k]): + break + if (k >= d-1): + print("Warning: Use more data") + return mu, s[k]/2**(d-k) + + +x = loadtxt(infile) +(mean, var) = block(x) +std = sqrt(var) +import pandas as pd +from pandas import DataFrame +data ={'Mean':[mean], 'STDev':[std]} +frame = pd.DataFrame(data,index=['Values']) +print(frame) ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Content
+-
+
- Simple compiler options +
- Tools to benchmark your code +
- Machine architectures +
- What is vectorization? +
- How to measure code performance +
- Parallelization with OpenMP +
- Parallelization with MPI +
- Vectorization and parallelization, examples +
+ +
+ +
+
+
+
+
+
+
+ + +
Optimization and profiling
+
+
+
+
+
+
+
+Till now we have not paid much attention to speed and possible optimization possibilities +inherent in the various compilers. We have compiled and linked as +
+ + +
+
+
+
+
+
+
+
+
+
+
+ c++ -c mycode.cpp +c++ -o mycode.exe mycode.o ++
+
+
+
+
+
+
+ For Fortran replace with for example gfortran or ifort. +This is what we call a flat compiler option and should be used when we develop the code. +It produces normally a very large and slow code when translated to machine instructions. +We use this option for debugging and for establishing the correct program output because +every operation is done precisely as the user specified it. +
+ +It is instructive to look up the compiler manual for further instructions by writing
+ + +
+
+
+
+
+
+
+
+
+
+ man c++ ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
More on optimization
+
+
+
+
+
+
+We have additional compiler options for optimization. These may include procedure inlining where +performance may be improved, moving constants inside loops outside the loop, +identify potential parallelism, include automatic vectorization or replace a division with a reciprocal +and a multiplication if this speeds up the code. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ c++ -O3 -c mycode.cpp +c++ -O3 -o mycode.exe mycode.o ++
+
+
+
+
+
+
+ This (other options are -O2 or -Ofast) is the recommended option.
++ +
+ +
+
+
+
+
+
+
+ + +
Optimization and profiling
+
+
+
+
+
+
+
+It is also useful to profile your program under the development stage. +You would then compile with +
+ + +
+
+
+
+
+
+
+
+
+
+
+ c++ -pg -O3 -c mycode.cpp +c++ -pg -O3 -o mycode.exe mycode.o ++
+
+
+
+
+
+
+ After you have run the code you can obtain the profiling information via
+ + +
+
+
+
+
+
+
+
+
+
+
+ gprof mycode.exe > ProfileOutput ++
+
+
+
+
+
+
+ When you have profiled properly your code, you must take out this option as it +slows down performance. +For memory tests use valgrind. An excellent environment for all these aspects, and much more, is Qt creator. +
++ +
+ +
+
+
+
+
+
+
+ + +
Optimization and debugging
+
+
+
+
+
+
+Adding debugging options is a very useful alternative under the development stage of a program. +You would then compile with +
+ + +
+
+
+
+
+
+
+
+
+
+
+ c++ -g -O0 -c mycode.cpp +c++ -g -O0 -o mycode.exe mycode.o ++
+
+
+
+
+
+
+ This option generates debugging information allowing you to trace for example if an array is properly allocated. Some compilers work best with the no optimization option -O0.
+
+
+
+
+
+
+
+Depending on the compiler, one can add flags which generate code that catches integer overflow errors. +The flag -ftrapv does this for the CLANG compiler on OS X operating systems. +
++ +
+ +
+
+
+
+
+
+
+ + +
Other hints
+
+
+
+
+
+
+
+In general, irrespective of compiler options, it is useful to
+-
+
- avoid if tests or call to functions inside loops, if possible. +
- avoid multiplication with constants inside loops if possible +
Here is an example of a part of a program where specific operations lead to a slower code
+ + +
+
+
+
+
+
+
+
+
+
+
+ k = n-1; +for (i = 0; i < n; i++){ + a[i] = b[i] +c*d; + e = g[k]; +} ++
+
+
+
+
+
+
+ A better code is
+ + +
+
+
+
+
+
+
+
+
+
+
+ temp = c*d; +for (i = 0; i < n; i++){ + a[i] = b[i] + temp; +} +e = g[n-1]; ++
+
+
+
+
+
+
+ Here we avoid a repeated multiplication inside a loop. +Most compilers, depending on compiler flags, identify and optimize such bottlenecks on their own, without requiring any particular action by the programmer. However, it is always useful to single out and avoid code examples like the first one discussed here. +
++ +
+ +
+
+
+
+
+
+
+ + +
Vectorization and the basic idea behind parallel computing
+
+
+
+
+
+
+
+Present CPUs are highly parallel processors with varying levels of parallelism. The typical situation can be described via the following three statements.
+-
+
- Pursuit of shorter computation time and larger simulation size gives rise to parallel computing. +
- Multiple processors are involved to solve a global problem. +
- The essence is to divide the entire computation evenly among collaborative processors. Divide and conquer. +
Before we proceed with a more detailed discussion of topics like vectorization and parallelization, we need to remind ourselves about some basic features of different hardware models.
++ +
+ +
+
+
+
+
+
+
+ + +
A rough classification of hardware models
+
+
+
+
+
+
+
+-
+
- Conventional single-processor computers are named SISD (single-instruction-single-data) machines. +
- SIMD (single-instruction-multiple-data) machines incorporate the idea of parallel processing, using a large number of processing units to execute the same instruction on different data. +
- Modern parallel computers are so-called MIMD (multiple-instruction-multiple-data) machines and can execute different instruction streams in parallel on different data. +
+ +
+ +
+
+
+
+
+
+
+ + +
Shared memory and distributed memory
+
+
+
+
+
+
+
+One way of categorizing modern parallel computers is to look at the memory configuration.
+-
+
- In shared memory systems the CPUs share the same address space. Any CPU can access any data in the global memory. +
- In distributed memory systems each CPU has its own memory. +
The CPUs are connected by some network and may exchange messages.
++ +
+ +
+
+
+
+
+
+
+ + +
Different parallel programming paradigms
+
+
+
+
+
+
+
+-
+
- Task parallelism: the work of a global problem can be divided into a number of independent tasks, which rarely need to synchronize. Monte Carlo simulations represent a typical situation. Integration is another. However this paradigm is of limited use. +
- Data parallelism: use of multiple threads (e.g. one or more threads per processor) to dissect loops over arrays etc. Communication and synchronization between processors are often hidden, thus easy to program. However, the user surrenders much control to a specialized compiler. Examples of data parallelism are compiler-based parallelization and OpenMP directives. +
+ +
+ +
+
+
+
+
+
+
+ + +
Different parallel programming paradigms
+
+
+
+
+
+
+
+
+-
+
- Message passing: all involved processors have an independent memory address space. The user is responsible for partitioning the data/work of a global problem and distributing the subproblems to the processors. Collaboration between processors is achieved by explicit message passing, which is used for data transfer plus synchronization. +
- This paradigm is the most general one where the user has full control. Better parallel efficiency is usually achieved by explicit message passing. However, message-passing programming is more difficult. +
+ +
+ +
+
+
+
+
+
+
+ + +
What is vectorization?
+Vectorization is a special +case of Single Instructions Multiple Data (SIMD) to denote a single +instruction stream capable of operating on multiple data elements in +parallel. +We can think of vectorization as the unrolling of loops accompanied with SIMD instructions. +
+ +Vectorization is the process of converting an algorithm that performs scalar operations +(typically one operation at the time) to vector operations where a single operation can refer to many simultaneous operations. +Consider the following example +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (i = 0; i < n; i++){ + a[i] = b[i] + c[i]; +} ++
+
+
+
+
+
+
+ If the code is not vectorized, the compiler will simply start with the first element and +then perform subsequent additions operating on one address in memory at the time. +
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Number of elements that can acted upon
+A SIMD instruction can operate on multiple data elements in one single instruction. +It uses the so-called 128-bit SIMD floating-point register. +In this sense, vectorization adds some form of parallelism since one instruction is applied +to many parts of say a vector. +
+ +The number of elements which can be operated on in parallel +range from four single-precision floating point data elements in so-called +Streaming SIMD Extensions and two double-precision floating-point data +elements in Streaming SIMD Extensions 2 to sixteen byte operations in +a 128-bit register in Streaming SIMD Extensions 2. Thus, vector-length +ranges from 2 to 16, depending on the instruction extensions used and +on the data type. +
+ +IN summary, our instructions operate on 128 bit (16 byte) operands
+-
+
- 4 floats or ints +
- 2 doubles +
- Data paths 128 bits vide for vector unit +
+ +
+ +
+
+
+
+
+
+
+ + +
Number of elements that can acted upon, examples
+We start with the simple scalar operations given by
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (i = 0; i < n; i++){ + a[i] = b[i] + c[i]; +} ++
+
+
+
+
+
+
+ If the code is not vectorized and we have a 128-bit register to store a 32 bits floating point number, +it means that we have \( 3\times 32 \) bits that are not used. +
+ +We have thus unused space in our SIMD registers. These registers could hold three additional integers.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Operation counts for scalar operation
+The code
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (i = 0; i < n; i++){ + a[i] = b[i] + c[i]; +} ++
+
+
+
+
+
+
+ has for \( n \) repeats
+-
+
- one load for \( c[i] \) in address 1 +
- one load for \( b[i] \) in address 2 +
- add \( c[i] \) and \( b[i] \) to give \( a[i] \) +
- store \( a[i] \) in address 2 +
+ +
+ +
+
+
+
+
+
+
+ + +
Number of elements that can acted upon, examples
+If we vectorize the code, we can perform, with a 128-bit register four simultaneous operations, that is +we have +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (i = 0; i < n; i+=4){ + a[i] = b[i] + c[i]; + a[i+1] = b[i+1] + c[i+1]; + a[i+2] = b[i+2] + c[i+2]; + a[i+3] = b[i+3] + c[i+3]; +} ++
+
+
+
+
+
+
+ Four additions are now done in a single step.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Number of operations when vectorized
+For \( n/4 \) repeats assuming floats or integers
+-
+
- one vector load for \( c[i] \) in address 1 +
- one load for \( b[i] \) in address 2 +
- add \( c[i] \) and \( b[i] \) to give \( a[i] \) +
- store \( a[i] \) in address 2 +
+ +
+ +
+
+
+
+
+
+
+ + +
A simple test case with and without vectorization
+We implement these operations in a simple c++ program that computes at the end the norm of a vector.
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+ #include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include "time.h"
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of square matrix
+ int n = atoi(argv[1]);
+ double s = 1.0/sqrt( (double) n);
+ double *a, *b, *c;
+ // Start timing
+ clock_t start, finish;
+ start = clock();
+// Allocate space for the vectors to be used
+ a = new double [n]; b = new double [n]; c = new double [n];
+ // Define parallel region
+ // Set up values for vectors a and b
+ for (int i = 0; i < n; i++){
+ double angle = 2.0*M_PI*i/ (( double ) n);
+ a[i] = s*(sin(angle) + cos(angle));
+ b[i] = s*sin(2.0*angle);
+ c[i] = 0.0;
+ }
+ // Then perform the vector addition
+ for (int i = 0; i < n; i++){
+ c[i] += a[i]+b[i];
+ }
+ // Compute now the norm-2
+ double Norm2 = 0.0;
+ for (int i = 0; i < n; i++){
+ Norm2 += c[i]*c[i];
+ }
+ finish = clock();
+ double timeused = (double) (finish - start)/(CLOCKS_PER_SEC );
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for norm computation=" << timeused << endl;
+ cout << " Norm-2 = " << Norm2 << endl;
+ // Free up space
+ delete[] a;
+ delete[] b;
+ delete[] c;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Compiling with and without vectorization
+We can compile and link without vectorization using the clang c++ compiler
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang -o novec.x vecexample.cpp ++
+
+
+
+
+
+
+ and with vectorization (and additional optimizations)
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang++ -O3 -Rpass=loop-vectorize -o vec.x vecexample.cpp ++
+
+
+
+
+
+
+ The speedup depends on the size of the vectors. In the example here we have run with \( 10^7 \) elements. +The example here was run on an IMac17.1 with OSX El Capitan (10.11.4) as operating system and an Intel i5 3.3 GHz CPU. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ Compphys:~ hjensen$ ./vec.x 10000000 +Time used for norm computation=0.04720500000 +Compphys:~ hjensen$ ./novec.x 10000000 +Time used for norm computation=0.03311700000 ++
+
+
+
+
+
+
+ This particular C++ compiler speeds up the above loop operations with a factor of 1.5 +Performing the same operations for \( 10^9 \) elements results in a smaller speedup since reading from main memory is required. The non-vectorized code is seemingly faster. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ Compphys:~ hjensen$ ./vec.x 1000000000 +Time used for norm computation=58.41391100 +Compphys:~ hjensen$ ./novec.x 1000000000 +Time used for norm computation=46.51295300 ++
+
+
+
+
+
+
+ We will discuss these issues further in the next slides.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Compiling with and without vectorization using clang
+We can compile and link without vectorization with clang compiler
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang++ -o -fno-vectorize novec.x vecexample.cpp ++
+
+
+
+
+
+
+ and with vectorization
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang++ -O3 -Rpass=loop-vectorize -o vec.x vecexample.cpp ++
+
+
+
+
+
+
+ We can also add vectorization analysis, see for example
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang++ -O3 -Rpass-analysis=loop-vectorize -o vec.x vecexample.cpp ++
+
+
+
+
+
+
+ or figure out if vectorization was missed
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ clang++ -O3 -Rpass-missed=loop-vectorize -o vec.x vecexample.cpp ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Automatic vectorization and vectorization inhibitors, criteria
+ +Not all loops can be vectorized, as discussed in Intel's guide to vectorization
+ +An important criteria is that the loop counter \( n \) is known at the entry of the loop.
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int j = 0; j < n; j++) { + a[j] = cos(j*1.0); + } ++
+
+
+
+
+
+
+ The variable \( n \) does need to be known at compile time. However, this variable must stay the same for the entire duration of the loop. It implies that an exit statement inside the loop cannot be data dependent.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Automatic vectorization and vectorization inhibitors, exit criteria
+ +An exit statement should in general be avoided. +If the exit statement contains data-dependent conditions, the loop cannot be vectorized. +The following is an example of a non-vectorizable loop +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int j = 0; j < n; j++) { + a[j] = cos(j*1.0); + if (a[j] < 0 ) break; + } ++
+
+
+
+
+
+
+ Avoid loop termination conditions and opt for a single entry loop variable \( n \). The lower and upper bounds have to be kept fixed within the loop.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Automatic vectorization and vectorization inhibitors, straight-line code
+ +SIMD instructions perform the same type of operations multiple times. +A switch statement leads thus to a non-vectorizable loop since different statemens cannot branch. +The following code can however be vectorized since the if statement is implemented as a masked assignment. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int j = 0; j < n; j++) { + double x = cos(j*1.0); + if (x > 0 ) { + a[j] = x*sin(j*2.0); + } + else { + a[j] = 0.0; + } + } ++
+
+
+
+
+
+
+ These operations can be performed for all data elements but only those elements which the mask evaluates as true are stored. In general, one should avoid branches such as switch, go to, or return statements or if constructs that cannot be treated as masked assignments.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Automatic vectorization and vectorization inhibitors, nested loops
+ +Only the innermost loop of the following example is vectorized
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int i = 0; i < n; i++) { + for (int j = 0; j < n; j++) { + a[i][j] += b[i][j]; + } + } ++
+
+
+
+
+
+
+ The exception is if an original outer loop is transformed into an inner loop as the result of compiler optimizations.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Automatic vectorization and vectorization inhibitors, function calls
+ +Calls to programmer defined functions ruin vectorization. However, calls to intrinsic functions like +\( \sin{x} \), \( \cos{x} \), \( \exp{x} \) etc are allowed since they are normally efficiently vectorized. +The following example is fully vectorizable +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int i = 0; i < n; i++) { + a[i] = log10(i)*cos(i); + } ++
+
+
+
+
+
+
+ Similarly, inline functions defined by the programmer, allow for vectorization since the function statements are glued into the actual place where the function is called.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Automatic vectorization and vectorization inhibitors, data dependencies
+ +One has to keep in mind that vectorization changes the order of operations inside a loop. A so-called +read-after-write statement with an explicit flow dependency cannot be vectorized. The following code +
+ + +
+
+
+
+
+
+
+
+
+
+
+ double b = 15.; + for (int i = 1; i < n; i++) { + a[i] = a[i-1] + b; + } ++
+
+
+
+
+
+
+ is an example of flow dependency and results in wrong numerical results if vectorized. For a scalar operation, the value \( a[i-1] \) computed during the iteration is loaded into the right-hand side and the results are fine. In vector mode however, with a vector length of four, the values \( a[0] \), \( a[1] \), \( a[2] \) and \( a[3] \) from the previous loop will be loaded into the right-hand side and produce wrong results. That is, we have
+ + +
+
+
+
+
+
+
+
+
+
+
+ a[1] = a[0] + b; + a[2] = a[1] + b; + a[3] = a[2] + b; + a[4] = a[3] + b; ++
+
+
+
+
+
+
+ and if the two first iterations are executed at the same by the SIMD instruction, the value of say \( a[1] \) could be used by the second iteration before it has been calculated by the first iteration, leading thereby to wrong results.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Automatic vectorization and vectorization inhibitors, more data dependencies
+ +On the other hand, a so-called +write-after-read statement can be vectorized. The following code +
+ + +
+
+
+
+
+
+
+
+
+
+
+ double b = 15.; + for (int i = 1; i < n; i++) { + a[i-1] = a[i] + b; + } ++
+
+
+
+
+
+
+ is an example of flow dependency that can be vectorized since no iteration with a higher value of \( i \) +can complete before an iteration with a lower value of \( i \). However, such code leads to problems with parallelization. +
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Automatic vectorization and vectorization inhibitors, memory stride
+ +For C++ programmers it is also worth keeping in mind that an array notation is preferred to the more compact use of pointers to access array elements. The compiler can often not tell if it is safe to vectorize the code.
+ +When dealing with arrays, you should also avoid memory stride, since this slows down considerably vectorization. When you access array element, write for example the inner loop to vectorize using unit stride, that is, access successively the next array element in memory, as shown here
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ for (int i = 0; i < n; i++) { + for (int j = 0; j < n; j++) { + a[i][j] += b[i][j]; + } + } ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Memory management
+The main memory contains the program data
+-
+
- Cache memory contains a copy of the main memory data +
- Cache is faster but consumes more space and power. It is normally assumed to be much faster than main memory +
- Registers contain working data only +
- Modern CPUs perform most or all operations only on data in register +
- Multiple Cache memories contain a copy of the main memory data +
- Cache items accessed by their address in main memory +
- L1 cache is the fastest but has the least capacity +
- L2, L3 provide intermediate performance/size tradeoffs +
-
+
-
+
Loads and stores to memory can be as important as floating point operations when we measure performance.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Memory and communication
+ +-
+
- Most communication in a computer is carried out in chunks, blocks of bytes of data that move together +
- In the memory hierarchy, data moves between memory and cache, and between different levels of cache, in groups called lines +
- Lines are typically 64-128 bytes, or 8-16 double precision words +
- Even if you do not use the data, it is moved and occupies space in the cache +
-
+
Many of these performance features are not captured in most programming languages.
+ ++ +
+ +
+
+
+
+
+
+
+ + +
Measuring performance
+ +How do we measure performance? What is wrong with this code to time a loop?
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ clock_t start, finish;
+ start = clock();
+ for (int j = 0; j < i; j++) {
+ a[j] = b[j]+b[j]*c[j];
+ }
+ finish = clock();
+ double timeused = (double) (finish - start)/(CLOCKS_PER_SEC );
+
+
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Problems with measuring time
+-
+
- Timers are not infinitely accurate +
- All clocks have a granularity, the minimum time that they can measure +
- The error in a time measurement, even if everything is perfect, may be the size of this granularity (sometimes called a clock tick) +
- Always know what your clock granularity is +
- Ensure that your measurement is for a long enough duration (say 100 times the tick) +
+ +
+ +
+
+
+
+
+
+
+ + +
Problems with cold start
+ +What happens when the code is executed? The assumption is that the code is ready to +execute. But +
+-
+
- Code may still be on disk, and not even read into memory. +
- Data may be in slow memory rather than fast (which may be wrong or right for what you are measuring) +
- Multiple tests often necessary to ensure that cold start effects are not present +
- Special effort often required to ensure data in the intended part of the memory hierarchy. +
+ +
+ +
+
+
+
+
+
+
+ + +
Problems with smart compilers
+ +-
+
- If the result of the computation is not used, the compiler may eliminate the code +
- Performance will look impossibly fantastic +
- Even worse, eliminate some of the code so the performance looks plausible +
- Ensure that the results are (or may be) used. +
+ +
+ +
+
+
+
+
+
+
+ + +
Problems with interference
+-
+
- Other activities are sharing your processor +
- Operating system, system demons, other users +
- Some parts of the hardware do not always perform with exactly the same performance +
- Make multiple tests and report +
- Easy choices include +
- Average tests represent what users might observe over time +
-
+
-
+
+ +
+ +
+
+
+
+
+
+
+ + +
Problems with measuring performance
+-
+
- Accurate, reproducible performance measurement is hard +
- Think carefully about your experiment: +
- What is it, precisely, that you want to measure? +
- How representative is your test to the situation that you are trying to measure? +
+ +
+ +
+
+
+
+
+
+
+ + +
Thomas algorithm for tridiagonal linear algebra equations
+
+
+
+
+
+
+$$
+\left( \begin{array}{ccccc}
+ b_0 & c_0 & & & \\
+ a_0 & b_1 & c_1 & & \\
+ & & \ddots & & \\
+ & & a_{m-3} & b_{m-2} & c_{m-2} \\
+ & & & a_{m-2} & b_{m-1}
+ \end{array} \right)
+\left( \begin{array}{c}
+ x_0 \\
+ x_1 \\
+ \vdots \\
+ x_{m-2} \\
+ x_{m-1}
+ \end{array} \right)=\left( \begin{array}{c}
+ f_0 \\
+ f_1 \\
+ \vdots \\
+ f_{m-2} \\
+ f_{m-1} \\
+ \end{array} \right)
+$$
+
++ +
+ +
+
+
+
+
+
+
+ + +
Thomas algorithm, forward substitution
+
+
+
+
+
+
+
+The first step is to multiply the first row by \( a_0/b_0 \) and subtract it from the second row. This is known as the forward substitution step. We obtain then
+$$ + a_i = 0, +$$ + + +$$ + b_i = b_i - \frac{a_{i-1}}{b_{i-1}}c_{i-1}, +$$ + +and
+$$ + f_i = f_i - \frac{a_{i-1}}{b_{i-1}}f_{i-1}. +$$ + +At this point the simplified equation, with only an upper triangular matrix takes the form
+$$ +\left( \begin{array}{ccccc} + b_0 & c_0 & & & \\ + & b_1 & c_1 & & \\ + & & \ddots & & \\ + & & & b_{m-2} & c_{m-2} \\ + & & & & b_{m-1} + \end{array} \right)\left( \begin{array}{c} + x_0 \\ + x_1 \\ + \vdots \\ + x_{m-2} \\ + x_{m-1} + \end{array} \right)=\left( \begin{array}{c} + f_0 \\ + f_1 \\ + \vdots \\ + f_{m-2} \\ + f_{m-1} \\ + \end{array} \right) +$$ ++ +
+ +
+
+
+
+
+
+
+ + +
Thomas algorithm, backward substitution
+
+
+
+
+
+
+
+The next step is the backward substitution step. The last row is multiplied by \( c_{N-3}/b_{N-2} \) and subtracted from the second to last row, thus eliminating \( c_{N-3} \) from the last row. The general backward substitution procedure is
+$$ + c_i = 0, +$$ + +and
+$$ + f_{i-1} = f_{i-1} - \frac{c_{i-1}}{b_i}f_i +$$ + +All that ramains to be computed is the solution, which is the very straight forward process of
+$$ +x_i = \frac{f_i}{b_i} +$$ ++ +
+ +
+
+
+
+
+
+
+ + +
Thomas algorithm and counting of operations (floating point and memory)
+
+
+
+
+
+
+
+
+We have in specific case the following operations with the floating operations
+ +-
+
- Memory Reads: \( 14(N-2) \); +
- Memory Writes: \( 4(N-2) \); +
- Subtractions: \( 3(N-2) \); +
- Multiplications: \( 3(N-2) \); +
- Divisions: \( 4(N-2) \). +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ // Forward substitution +// Note that we can simplify by precalculating a[i-1]/b[i-1] + for (int i=1; i < n; i++) { + b[i] = b[i] - (a[i-1]*c[i-1])/b[i-1]; + f[i] = g[i] - (a[i-1]*f[i-1])/b[i-1]; + } + x[n-1] = f[n-1] / b[n-1]; + // Backwards substitution + for (int i = n-2; i >= 0; i--) { + f[i] = f[i] - c[i]*f[i+1]/b[i+1]; + x[i] = f[i]/b[i]; + } ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Example: Transpose of a matrix
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+ #include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include "time.h"
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of square matrix
+ int n = atoi(argv[1]);
+ double **A, **B;
+ // Allocate space for the two matrices
+ A = new double*[n]; B = new double*[n];
+ for (int i = 0; i < n; i++){
+ A[i] = new double[n];
+ B[i] = new double[n];
+ }
+ // Set up values for matrix A
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ A[i][j] = cos(i*1.0)*sin(j*3.0);
+ }
+ }
+ clock_t start, finish;
+ start = clock();
+ // Then compute the transpose
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ B[i][j]= A[j][i];
+ }
+ }
+
+ finish = clock();
+ double timeused = (double) (finish - start)/(CLOCKS_PER_SEC );
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for setting up transpose of matrix=" << timeused << endl;
+
+ // Free up space
+ for (int i = 0; i < n; i++){
+ delete[] A[i];
+ delete[] B[i];
+ }
+ delete[] A;
+ delete[] B;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Matrix-matrix multiplication
+This the matrix-matrix multiplication code with plain c++ memory allocation. It computes at the end the Frobenius norm.
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+ #include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include "time.h"
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of square matrix
+ int n = atoi(argv[1]);
+ double s = 1.0/sqrt( (double) n);
+ double **A, **B, **C;
+ // Start timing
+ clock_t start, finish;
+ start = clock();
+ // Allocate space for the two matrices
+ A = new double*[n]; B = new double*[n]; C = new double*[n];
+ for (int i = 0; i < n; i++){
+ A[i] = new double[n];
+ B[i] = new double[n];
+ C[i] = new double[n];
+ }
+ // Set up values for matrix A and B and zero matrix C
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ double angle = 2.0*M_PI*i*j/ (( double ) n);
+ A[i][j] = s * ( sin ( angle ) + cos ( angle ) );
+ B[j][i] = A[i][j];
+ }
+ }
+ // Then perform the matrix-matrix multiplication
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ double sum = 0.0;
+ for (int k = 0; k < n; k++) {
+ sum += B[i][k]*A[k][j];
+ }
+ C[i][j] = sum;
+ }
+ }
+ // Compute now the Frobenius norm
+ double Fsum = 0.0;
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ Fsum += C[i][j]*C[i][j];
+ }
+ }
+ Fsum = sqrt(Fsum);
+ finish = clock();
+ double timeused = (double) (finish - start)/(CLOCKS_PER_SEC );
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for matrix-matrix multiplication=" << timeused << endl;
+ cout << " Frobenius norm = " << Fsum << endl;
+ // Free up space
+ for (int i = 0; i < n; i++){
+ delete[] A[i];
+ delete[] B[i];
+ delete[] C[i];
+ }
+ delete[] A;
+ delete[] B;
+ delete[] C;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
How do we define speedup? Simplest form
+
+
+
+
+
+
+
+-
+
- Speedup measures the ratio of performance between two objects +
- Versions of same code, with different number of processors +
- Serial and vector versions +
- Try different programing languages, c++ and Fortran +
- Two algorithms computing the same result +
+ +
+ +
+
+
+
+
+
+
+ + +
How do we define speedup? Correct baseline
+
+
+
+
+
+
+
+The key is choosing the correct baseline for comparison
+-
+
- For our serial vs. vectorization examples, using compiler-provided vectorization, the baseline is simple; the same code, with vectorization turned off +
- For parallel applications, this is much harder: +
- Choice of algorithm, decomposition, performance of baseline case etc. +
-
+
-
+
+ +
+ +
+
+
+
+
+
+
+ + +
Parallel speedup
+
+
+
+
+
+
+
+For parallel applications, speedup is typically defined as
+-
+
- Speedup \( =T_1/T_p \) +
Here \( T_1 \) is the time on one processor and \( T_p \) is the time using \( p \) processors.
+-
+
- Can the speedup become larger than \( p \)? That means using \( p \) processors is more than \( p \) times faster than using one processor. +
+ +
+ +
+
+
+
+
+
+
+ + +
Speedup and memory
+
+
+
+
+
+
+
+The speedup on \( p \) processors can +be greater than \( p \) if memory usage is optimal! +Consider the case of a memorybound computation with \( M \) words of memory +
+-
+
- If \( M/p \) fits into cache while \( M \) does not, the time to access memory will be different in the two cases: +
- \( T_1 \) uses the main memory bandwidth +
- \( T_p \) uses the appropriate cache bandwidth +
+ +
+ +
+
+
+
+
+
+
+ + +
Upper bounds on speedup
+
+
+
+
+
+
+
+Assume that almost all parts of a code are perfectly +parallelizable (fraction \( f \)). The remainder, +fraction \( (1-f) \) cannot be parallelized at all. +
+ +That is, there is work that takes time \( W \) on one process; a fraction \( f \) of that work will take +time \( Wf/p \) on \( p \) processors. +
+-
+
- What is the maximum possible speedup as a function of \( f \)? +
+ +
+ +
+
+
+
+
+
+
+ + +
Amdahl's law
+
+
+
+
+
+
+
+On one processor we have
+$$ +T_1 = (1-f)W + fW = W +$$ + +On \( p \) processors we have
+$$ +T_p = (1-f)W + \frac{fW}{p}, +$$ + +resulting in a speedup of
+$$ +\frac{T_1}{T_p} = \frac{W}{(1-f)W+fW/p} +$$ + +As \( p \) goes to infinity, \( fW/p \) goes to zero, and the maximum speedup is
+$$ +\frac{1}{1-f}, +$$ + +meaning that if +if \( f = 0.99 \) (all but \( 1\% \) parallelizable), the maximum speedup +is \( 1/(1-.99)=100 \)! +
++ +
+ +
+
+
+
+
+
+
+ + +
How much is parallelizable
+
+
+
+
+
+
+
+If any non-parallel code slips into the +application, the parallel +performance is limited. +
+ +In many simulations, however, the fraction of non-parallelizable work +is \( 10^{-6} \) or less due to large arrays or objects that are perfectly parallelizable. +
++ +
+ +
+
+
+
+
+
+
+ + +
Today's situation of parallel computing
+
+
+
+
+
+
+
+
+-
+
- Distributed memory is the dominant hardware configuration. There is a large diversity in these machines, from MPP (massively parallel processing) systems to clusters of off-the-shelf PCs, which are very cost-effective. +
- Message-passing is a mature programming paradigm and widely accepted. It often provides an efficient match to the hardware. It is primarily used for the distributed memory systems, but can also be used on shared memory systems. +
- Modern nodes have nowadays several cores, which makes it interesting to use both shared memory (the given node) and distributed memory (several nodes with communication). This leads often to codes which use both MPI and OpenMP. +
Our lectures will focus on both MPI and OpenMP.
++ +
+ +
+
+
+
+
+
+
+ + +
Overhead present in parallel computing
+
+
+
+
+
+
+
+
+-
+
- Uneven load balance: not all the processors can perform useful work at all time. +
- Overhead of synchronization +
- Overhead of communication +
- Extra computation due to parallelization +
Due to the above overhead and that certain parts of a sequential +algorithm cannot be parallelized we may not achieve an optimal parallelization. +
++ +
+ +
+
+
+
+
+
+
+ + +
Parallelizing a sequential algorithm
+
+
+
+
+
+
+
+
+-
+
- Identify the part(s) of a sequential algorithm that can be executed in parallel. This is the difficult part, +
- Distribute the global work and data among \( P \) processors. +
+ +
+ +
+
+
+
+
+
+
+ + +
Strategies
+
+
+
+
+
+
+
+-
+
- Develop codes locally, run with some few processes and test your codes. Do benchmarking, timing and so forth on local nodes, for example your laptop or PC. +
- When you are convinced that your codes run correctly, you can start your production runs on available supercomputers. +
+ +
+ +
+
+
+
+
+
+
+ + +
How do I run MPI on a PC/Laptop? MPI
+
+
+
+
+
+
+
+To install MPI is rather easy on hardware running unix/linux as operating systems, follow simply the instructions from the OpenMPI website. See also subsequent slides. +When you have made sure you have installed MPI on your PC/laptop, +
+-
+
- Compile with mpicxx/mpic++ or mpif90 +
+
+
+
+
+
+
+
+
+
+ # Compile and link + mpic++ -O3 -o nameofprog.x nameofprog.cpp + # run code with for example 8 processes using mpirun/mpiexec + mpiexec -n 8 ./nameofprog.x ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Can I do it on my own PC/laptop? OpenMP installation
+
+
+
+
+
+
+
+If you wish to install MPI and OpenMP +on your laptop/PC, we recommend the following: +
+ +-
+
- For OpenMP, the compile option -fopenmp is included automatically in recent versions of the C++ compiler and Fortran compilers. For users of different Linux distributions, simply use the available C++ or Fortran compilers and add the above compiler instructions, see also code examples below. +
- For OS X users however, install libomp +
+
+
+
+
+
+
+
+
+
+
+ brew install libomp ++
+
+
+
+
+
+
+ and compile and link as
+ + +
+
+
+
+
+
+
+
+
+
+ c++ -o <name executable> <name program.cpp> -lomp ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Installing MPI
+
+
+
+
+
+
+
+For linux/ubuntu users, you need to install two packages (alternatively use the synaptic package manager)
+ + +
+
+
+
+
+
+
+
+
+
+
+ sudo apt-get install libopenmpi-dev + sudo apt-get install openmpi-bin ++
+
+
+
+
+
+
+ For OS X users, install brew (after having installed xcode and gcc, needed for the +gfortran compiler of openmpi) and then install with brew +
+ + +
+
+
+
+
+
+
+
+
+
+
+ brew install openmpi ++
+
+
+
+
+
+
+ When running an executable (code.x), run as
+ + +
+
+
+
+
+
+
+
+
+
+
+ mpirun -n 10 ./code.x ++
+
+
+
+
+
+
+ where we indicate that we want the number of processes to be 10.
++ +
+ +
+
+
+
+
+
+
+ + +
Installing MPI and using Qt
+
+
+
+
+
+
+
+With openmpi installed, when using Qt, add to your .pro file the instructions here
+ +You may need to tell Qt where openmpi is stored.
++ +
+ +
+
+
+
+
+
+
+ + +
What is Message Passing Interface (MPI)?
+
+
+
+
+
+
+
+MPI is a library, not a language. It specifies the names, calling sequences and results of functions +or subroutines to be called from C/C++ or Fortran programs, and the classes and methods that make up the MPI C++ +library. The programs that users write in Fortran, C or C++ are compiled with ordinary compilers and linked +with the MPI library. +
+ +MPI programs should be able to run +on all possible machines and run all MPI implementetations without change. +
+ +An MPI computation is a collection of processes communicating with messages.
++ +
+ +
+
+
+
+
+
+
+ + +
Going Parallel with MPI
+
+
+
+
+
+
+
+Task parallelism: the work of a global problem can be divided +into a number of independent tasks, which rarely need to synchronize. +Monte Carlo simulations or numerical integration are examples of this. +
+ +MPI is a message-passing library where all the routines +have corresponding C/C++-binding +
+ + +
+
+
+
+
+
+
+
+
+
+
+ MPI_Command_name
+
+
+
+
+
+
+
+
+ and Fortran-binding (routine names are in uppercase, but can also be in lower case)
+ + +
+
+
+
+
+
+
+
+
+
+ MPI_COMMAND_NAME ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
MPI is a library
+
+
+
+
+
+
+
+MPI is a library specification for the message passing interface, +proposed as a standard. +
+ +-
+
- independent of hardware; +
- not a language or compiler specification; +
- not a specific implementation or product. +
A message passing standard for portability and ease-of-use. +Designed for high performance. +
+ +Insert communication and synchronization functions where necessary.
++ +
+ +
+
+
+
+
+
+
+ + +
Bindings to MPI routines
+
+
+
+
+
+
+
+
+MPI is a message-passing library where all the routines +have corresponding C/C++-binding +
+ + +
+
+
+
+
+
+
+
+
+
+
+ MPI_Command_name
+
+
+
+
+
+
+
+
+ and Fortran-binding (routine names are in uppercase, but can also be in lower case)
+ + +
+
+
+
+
+
+
+
+
+
+
+ MPI_COMMAND_NAME ++
+
+
+
+
+
+
+ The discussion in these slides focuses on the C++ binding.
++ +
+ +
+
+
+
+
+
+
+ + +
Communicator
+
+
+
+
+
+
+
+-
+
- A group of MPI processes with a name (context). +
- Any process is identified by its rank. The rank is only meaningful within a particular communicator. +
- By default the communicator contains all the MPI processes. +
+
+
+
+
+
+
+
+
+
+
+ MPI_COMM_WORLD ++
+
+
+
+
+
+
+ -
+
- Mechanism to identify subset of processes. +
- Promotes modular design of parallel libraries. +
+ +
+ +
+
+
+
+
+
+
+ + +
Some of the most important MPI functions
+
+
+
+
+
+
+
+
+-
+
- \( MPI\_Init \) - initiate an MPI computation +
- \( MPI\_Finalize \) - terminate the MPI computation and clean up +
- \( MPI\_Comm\_size \) - how many processes participate in a given MPI communicator? +
- \( MPI\_Comm\_rank \) - which one am I? (A number between 0 and size-1.) +
- \( MPI\_Send \) - send a message to a particular process within an MPI communicator +
- \( MPI\_Recv \) - receive a message from a particular process within an MPI communicator +
- \( MPI\_reduce \) or \( MPI\_Allreduce \), send and receive messages +
+ +
+ +
+
+
+
+
+
+
+ + +
The first MPI C/C++ program
+
+
+
+
+
+
+
+
+Let every process write "Hello world" (oh not this program again!!) on the standard output.
+ + +
+
+
+
+
+
+
+
+
+
+ using namespace std; +#include <mpi.h> +#include <iostream> +int main (int nargs, char* args[]) +{ +int numprocs, my_rank; +// MPI initializations +MPI_Init (&nargs, &args); +MPI_Comm_size (MPI_COMM_WORLD, &numprocs); +MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); +cout << "Hello world, I have rank " << my_rank << " out of " + << numprocs << endl; +// End MPI +MPI_Finalize (); ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
The Fortran program
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ PROGRAM hello +INCLUDE "mpif.h" +INTEGER:: size, my_rank, ierr + +CALL MPI_INIT(ierr) +CALL MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierr) +CALL MPI_COMM_RANK(MPI_COMM_WORLD, my_rank, ierr) +WRITE(*,*)"Hello world, I've rank ",my_rank," out of ",size +CALL MPI_FINALIZE(ierr) + +END PROGRAM hello ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Note 1
+
+
+
+
+
+
+
+
+-
+
- The output to screen is not ordered since all processes are trying to write to screen simultaneously. +
- It is the operating system which opts for an ordering. +
- If we wish to have an organized output, starting from the first process, we may rewrite our program as in the next example. +
+ +
+ +
+
+
+
+
+
+
+ + +
Ordered output with MPIBarrier
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ int main (int nargs, char* args[]) +{ + int numprocs, my_rank, i; + MPI_Init (&nargs, &args); + MPI_Comm_size (MPI_COMM_WORLD, &numprocs); + MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); + for (i = 0; i < numprocs; i++) {} + MPI_Barrier (MPI_COMM_WORLD); + if (i == my_rank) { + cout << "Hello world, I have rank " << my_rank << + " out of " << numprocs << endl;} + MPI_Finalize (); ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Note 2
+
+
+
+
+
+
+
+-
+
- Here we have used the \( MPI\_Barrier \) function to ensure that that every process has completed its set of instructions in a particular order. +
- A barrier is a special collective operation that does not allow the processes to continue until all processes in the communicator (here \( MPI\_COMM\_WORLD \)) have called \( MPI\_Barrier \). +
- The barriers make sure that all processes have reached the same point in the code. Many of the collective operations like \( MPI\_ALLREDUCE \) to be discussed later, have the same property; that is, no process can exit the operation until all processes have started. +
However, this is slightly more time-consuming since the processes synchronize between themselves as many times as there +are processes. In the next Hello world example we use the send and receive functions in order to a have a synchronized +action. +
++ +
+ +
+
+
+
+
+
+
+ + +
Ordered output
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ ..... +int numprocs, my_rank, flag; +MPI_Status status; +MPI_Init (&nargs, &args); +MPI_Comm_size (MPI_COMM_WORLD, &numprocs); +MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); +if (my_rank > 0) +MPI_Recv (&flag, 1, MPI_INT, my_rank-1, 100, + MPI_COMM_WORLD, &status); +cout << "Hello world, I have rank " << my_rank << " out of " +<< numprocs << endl; +if (my_rank < numprocs-1) +MPI_Send (&my_rank, 1, MPI_INT, my_rank+1, + 100, MPI_COMM_WORLD); +MPI_Finalize (); ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Note 3
+
+
+
+
+
+
+
+
+The basic sending of messages is given by the function \( MPI\_SEND \), which in C/C++ +is defined as +
+ + +
+
+
+
+
+
+
+
+
+
+
+ int MPI_Send(void *buf, int count, + MPI_Datatype datatype, + int dest, int tag, MPI_Comm comm)} ++
+
+
+
+
+
+
+ This single command allows the passing of any kind of variable, even a large array, to any group of tasks. +The variable buf is the variable we wish to send while count +is the number of variables we are passing. If we are passing only a single value, this should be 1. +
+ +If we transfer an array, it is the overall size of the array. +For example, if we want to send a 10 by 10 array, count would be \( 10\times 10=100 \) +since we are actually passing 100 values. +
++ +
+ +
+
+
+
+
+
+
+ + +
Note 4
+
+
+
+
+
+
+
+
+Once you have sent a message, you must receive it on another task. The function \( MPI\_RECV \) +is similar to the send call. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ int MPI_Recv( void *buf, int count, MPI_Datatype datatype, + int source, + int tag, MPI_Comm comm, MPI_Status *status ) ++
+
+
+
+
+
+
+ The arguments that are different from those in MPI\_SEND are +buf which is the name of the variable where you will be storing the received data, +source which replaces the destination in the send command. This is the return ID of the sender. +
+ +Finally, we have used \( MPI\_Status\_status \), +where one can check if the receive was completed. +
+ +The output of this code is the same as the previous example, but now +process 0 sends a message to process 1, which forwards it further +to process 2, and so forth. +
++ +
+ +
+
+
+
+
+
+
+ + +
Numerical integration in parallel
+
+
+
+
+
+
+
+
+-
+
- The code example computes \( \pi \) using the trapezoidal rules. +
- The trapezoidal rule +
Click on this link for the full program.
++ +
+ +
+
+
+
+
+
+
+ + +
Dissection of trapezoidal rule with \( MPI\_reduce \)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ // Trapezoidal rule and numerical integration usign MPI +using namespace std; +#include <mpi.h> +#include <iostream> + +// Here we define various functions called by the main program + +double int_function(double ); +double trapezoidal_rule(double , double , int , double (*)(double)); + +// Main function begins here +int main (int nargs, char* args[]) +{ + int n, local_n, numprocs, my_rank; + double a, b, h, local_a, local_b, total_sum, local_sum; + double time_start, time_end, total_time; ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Dissection of trapezoidal rule
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ // MPI initializations + MPI_Init (&nargs, &args); + MPI_Comm_size (MPI_COMM_WORLD, &numprocs); + MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); + time_start = MPI_Wtime(); + // Fixed values for a, b and n + a = 0.0 ; b = 1.0; n = 1000; + h = (b-a)/n; // h is the same for all processes + local_n = n/numprocs; + // make sure n > numprocs, else integer division gives zero + // Length of each process' interval of + // integration = local_n*h. + local_a = a + my_rank*local_n*h; + local_b = local_a + local_n*h; ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Integrating with MPI
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ total_sum = 0.0; + local_sum = trapezoidal_rule(local_a, local_b, local_n, + &int_function); + MPI_Reduce(&local_sum, &total_sum, 1, MPI_DOUBLE, + MPI_SUM, 0, MPI_COMM_WORLD); + time_end = MPI_Wtime(); + total_time = time_end-time_start; + if ( my_rank == 0) { + cout << "Trapezoidal rule = " << total_sum << endl; + cout << "Time = " << total_time + << " on number of processors: " << numprocs << endl; + } + // End MPI + MPI_Finalize (); + return 0; +} // end of main program ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
How do I use \( MPI\_reduce \)?
+
+
+
+
+
+
+
+
+Here we have used
+ + +
+
+
+
+
+
+
+
+
+
+
+ MPI_reduce( void *senddata, void* resultdata, int count, + MPI_Datatype datatype, MPI_Op, int root, MPI_Comm comm) ++
+
+
+
+
+
+
+ The two variables \( senddata \) and \( resultdata \) are obvious, besides the fact that one sends the address +of the variable or the first element of an array. If they are arrays they need to have the same size. +The variable \( count \) represents the total dimensionality, 1 in case of just one variable, +while \( MPI\_Datatype \) +defines the type of variable which is sent and received. +
+ +The new feature is \( MPI\_Op \). It defines the type +of operation we want to do. +
++ +
+ +
+
+
+
+
+
+
+ + +
More on \( MPI\_Reduce \)
+
+
+
+
+
+
+
+In our case, since we are summing +the rectangle contributions from every process we define \( MPI\_Op = MPI\_SUM \). +If we have an array or matrix we can search for the largest og smallest element by sending either \( MPI\_MAX \) or +\( MPI\_MIN \). If we want the location as well (which array element) we simply transfer +\( MPI\_MAXLOC \) or \( MPI\_MINOC \). If we want the product we write \( MPI\_PROD \). +
+ +\( MPI\_Allreduce \) is defined as
+ + +
+
+
+
+
+
+
+
+
+
+ MPI_Allreduce( void *senddata, void* resultdata, int count, + MPI_Datatype datatype, MPI_Op, MPI_Comm comm) ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Dissection of trapezoidal rule
+
+
+
+
+
+
+
+
+We use \( MPI\_reduce \) to collect data from each process. Note also the use of the function +\( MPI\_Wtime \). +
+ + +
+
+
+
+
+
+
+
+
+
+ // this function defines the function to integrate +double int_function(double x) +{ + double value = 4./(1.+x*x); + return value; +} // end of function to evaluate ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Dissection of trapezoidal rule
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ // this function defines the trapezoidal rule +double trapezoidal_rule(double a, double b, int n, + double (*func)(double)) +{ + double trapez_sum; + double fa, fb, x, step; + int j; + step=(b-a)/((double) n); + fa=(*func)(a)/2. ; + fb=(*func)(b)/2. ; + trapez_sum=0.; + for (j=1; j <= n-1; j++){ + x=j*step+a; + trapez_sum+=(*func)(x); + } + trapez_sum=(trapez_sum+fb+fa)*step; + return trapez_sum; +} // end trapezoidal_rule ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
The quantum dot program for two electrons
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ // Variational Monte Carlo for atoms with importance sampling, slater det +// Test case for 2-electron quantum dot, no classes using Mersenne-Twister RNG +#include "mpi.h" +#include <cmath> +#include <random> +#include <string> +#include <iostream> +#include <fstream> +#include <iomanip> +#include "vectormatrixclass.h" + +using namespace std; +// output file as global variable +ofstream ofile; +// the step length and its squared inverse for the second derivative +// Here we define global variables used in various functions +// These can be changed by using classes +int Dimension = 2; +int NumberParticles = 2; // we fix also the number of electrons to be 2 + +// declaration of functions + +// The Mc sampling for the variational Monte Carlo +void MonteCarloSampling(int, double &, double &, Vector &); + +// The variational wave function +double WaveFunction(Matrix &, Vector &); + +// The local energy +double LocalEnergy(Matrix &, Vector &); + +// The quantum force +void QuantumForce(Matrix &, Matrix &, Vector &); + + +// inline function for single-particle wave function +inline double SPwavefunction(double r, double alpha) { + return exp(-alpha*r*0.5); +} + +// inline function for derivative of single-particle wave function +inline double DerivativeSPwavefunction(double r, double alpha) { + return -r*alpha; +} + +// function for absolute value of relative distance +double RelativeDistance(Matrix &r, int i, int j) { + double r_ij = 0; + for (int k = 0; k < Dimension; k++) { + r_ij += (r(i,k)-r(j,k))*(r(i,k)-r(j,k)); + } + return sqrt(r_ij); +} + +// inline function for derivative of Jastrow factor +inline double JastrowDerivative(Matrix &r, double beta, int i, int j, int k){ + return (r(i,k)-r(j,k))/(RelativeDistance(r, i, j)*pow(1.0+beta*RelativeDistance(r, i, j),2)); +} + +// function for square of position of single particle +double singleparticle_pos2(Matrix &r, int i) { + double r_single_particle = 0; + for (int j = 0; j < Dimension; j++) { + r_single_particle += r(i,j)*r(i,j); + } + return r_single_particle; +} + +void lnsrch(int n, Vector &xold, double fold, Vector &g, Vector &p, Vector &x, + double *f, double stpmax, int *check, double (*func)(Vector &p)); + +void dfpmin(Vector &p, int n, double gtol, int *iter, double *fret, + double(*func)(Vector &p), void (*dfunc)(Vector &p, Vector &g)); + +static double sqrarg; +#define SQR(a) ((sqrarg=(a)) == 0.0 ? 0.0 : sqrarg*sqrarg) + + +static double maxarg1,maxarg2; +#define FMAX(a,b) (maxarg1=(a),maxarg2=(b),(maxarg1) > (maxarg2) ?\ + (maxarg1) : (maxarg2)) + + +// Begin of main program + +int main(int argc, char* argv[]) +{ + + // MPI initializations + int NumberProcesses, MyRank, NumberMCsamples; + MPI_Init (&argc, &argv); + MPI_Comm_size (MPI_COMM_WORLD, &NumberProcesses); + MPI_Comm_rank (MPI_COMM_WORLD, &MyRank); + double StartTime = MPI_Wtime(); + if (MyRank == 0 && argc <= 1) { + cout << "Bad Usage: " << argv[0] << + " Read also output file on same line and number of Monte Carlo cycles" << endl; + } + // Read filename and number of Monte Carlo cycles from the command line + if (MyRank == 0 && argc > 2) { + string filename = argv[1]; // first command line argument after name of program + NumberMCsamples = atoi(argv[2]); + string fileout = filename; + string argument = to_string(NumberMCsamples); + // Final filename as filename+NumberMCsamples + fileout.append(argument); + ofile.open(fileout); + } + // broadcast the number of Monte Carlo samples + MPI_Bcast (&NumberMCsamples, 1, MPI_INT, 0, MPI_COMM_WORLD); + // Two variational parameters only + Vector VariationalParameters(2); + int TotalNumberMCsamples = NumberMCsamples*NumberProcesses; + // Loop over variational parameters + for (double alpha = 0.5; alpha <= 1.5; alpha +=0.1){ + for (double beta = 0.1; beta <= 0.5; beta +=0.05){ + VariationalParameters(0) = alpha; // value of alpha + VariationalParameters(1) = beta; // value of beta + // Do the mc sampling and accumulate data with MPI_Reduce + double TotalEnergy, TotalEnergySquared, LocalProcessEnergy, LocalProcessEnergy2; + LocalProcessEnergy = LocalProcessEnergy2 = 0.0; + MonteCarloSampling(NumberMCsamples, LocalProcessEnergy, LocalProcessEnergy2, VariationalParameters); + // Collect data in total averages + MPI_Reduce(&LocalProcessEnergy, &TotalEnergy, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); + MPI_Reduce(&LocalProcessEnergy2, &TotalEnergySquared, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); + // Print out results in case of Master node, set to MyRank = 0 + if ( MyRank == 0) { + double Energy = TotalEnergy/( (double)NumberProcesses); + double Variance = TotalEnergySquared/( (double)NumberProcesses)-Energy*Energy; + double StandardDeviation = sqrt(Variance/((double)TotalNumberMCsamples)); // over optimistic error + ofile << setiosflags(ios::showpoint | ios::uppercase); + ofile << setw(15) << setprecision(8) << VariationalParameters(0); + ofile << setw(15) << setprecision(8) << VariationalParameters(1); + ofile << setw(15) << setprecision(8) << Energy; + ofile << setw(15) << setprecision(8) << Variance; + ofile << setw(15) << setprecision(8) << StandardDeviation << endl; + } + } + } + double EndTime = MPI_Wtime(); + double TotalTime = EndTime-StartTime; + if ( MyRank == 0 ) cout << "Time = " << TotalTime << " on number of processors: " << NumberProcesses << endl; + if (MyRank == 0) ofile.close(); // close output file + // End MPI + MPI_Finalize (); + return 0; +} // end of main function + + +// Monte Carlo sampling with the Metropolis algorithm + +void MonteCarloSampling(int NumberMCsamples, double &cumulative_e, double &cumulative_e2, Vector &VariationalParameters) +{ + + // Initialize the seed and call the Mersienne algo + std::random_device rd; + std::mt19937_64 gen(rd()); + // Set up the uniform distribution for x \in [[0, 1] + std::uniform_real_distribution<double> UniformNumberGenerator(0.0,1.0); + std::normal_distribution<double> Normaldistribution(0.0,1.0); + // diffusion constant from Schroedinger equation + double D = 0.5; + double timestep = 0.05; // we fix the time step for the gaussian deviate + // allocate matrices which contain the position of the particles + Matrix OldPosition( NumberParticles, Dimension), NewPosition( NumberParticles, Dimension); + Matrix OldQuantumForce(NumberParticles, Dimension), NewQuantumForce(NumberParticles, Dimension); + double Energy = 0.0; double EnergySquared = 0.0; double DeltaE = 0.0; + // initial trial positions + for (int i = 0; i < NumberParticles; i++) { + for (int j = 0; j < Dimension; j++) { + OldPosition(i,j) = Normaldistribution(gen)*sqrt(timestep); + } + } + double OldWaveFunction = WaveFunction(OldPosition, VariationalParameters); + QuantumForce(OldPosition, OldQuantumForce, VariationalParameters); + // loop over monte carlo cycles + for (int cycles = 1; cycles <= NumberMCsamples; cycles++){ + // new position + for (int i = 0; i < NumberParticles; i++) { + for (int j = 0; j < Dimension; j++) { + // gaussian deviate to compute new positions using a given timestep + NewPosition(i,j) = OldPosition(i,j) + Normaldistribution(gen)*sqrt(timestep)+OldQuantumForce(i,j)*timestep*D; + // NewPosition(i,j) = OldPosition(i,j) + gaussian_deviate(&idum)*sqrt(timestep)+OldQuantumForce(i,j)*timestep*D; + } + // for the other particles we need to set the position to the old position since + // we move only one particle at the time + for (int k = 0; k < NumberParticles; k++) { + if ( k != i) { + for (int j = 0; j < Dimension; j++) { + NewPosition(k,j) = OldPosition(k,j); + } + } + } + double NewWaveFunction = WaveFunction(NewPosition, VariationalParameters); + QuantumForce(NewPosition, NewQuantumForce, VariationalParameters); + // we compute the log of the ratio of the greens functions to be used in the + // Metropolis-Hastings algorithm + double GreensFunction = 0.0; + for (int j = 0; j < Dimension; j++) { + GreensFunction += 0.5*(OldQuantumForce(i,j)+NewQuantumForce(i,j))* + (D*timestep*0.5*(OldQuantumForce(i,j)-NewQuantumForce(i,j))-NewPosition(i,j)+OldPosition(i,j)); + } + GreensFunction = exp(GreensFunction); + // The Metropolis test is performed by moving one particle at the time + if(UniformNumberGenerator(gen) <= GreensFunction*NewWaveFunction*NewWaveFunction/OldWaveFunction/OldWaveFunction ) { + for (int j = 0; j < Dimension; j++) { + OldPosition(i,j) = NewPosition(i,j); + OldQuantumForce(i,j) = NewQuantumForce(i,j); + } + OldWaveFunction = NewWaveFunction; + } + } // end of loop over particles + // compute local energy + double DeltaE = LocalEnergy(OldPosition, VariationalParameters); + // update energies + Energy += DeltaE; + EnergySquared += DeltaE*DeltaE; + } // end of loop over MC trials + // update the energy average and its squared + cumulative_e = Energy/NumberMCsamples; + cumulative_e2 = EnergySquared/NumberMCsamples; +} // end MonteCarloSampling function + + +// Function to compute the squared wave function and the quantum force + +double WaveFunction(Matrix &r, Vector &VariationalParameters) +{ + double wf = 0.0; + // full Slater determinant for two particles, replace with Slater det for more particles + wf = SPwavefunction(singleparticle_pos2(r, 0), VariationalParameters(0))*SPwavefunction(singleparticle_pos2(r, 1),VariationalParameters(0)); + // contribution from Jastrow factor + for (int i = 0; i < NumberParticles-1; i++) { + for (int j = i+1; j < NumberParticles; j++) { + wf *= exp(RelativeDistance(r, i, j)/((1.0+VariationalParameters(1)*RelativeDistance(r, i, j)))); + } + } + return wf; +} + +// Function to calculate the local energy without numerical derivation of kinetic energy + +double LocalEnergy(Matrix &r, Vector &VariationalParameters) +{ + + // compute the kinetic and potential energy from the single-particle part + // for a many-electron system this has to be replaced by a Slater determinant + // The absolute value of the interparticle length + Matrix length( NumberParticles, NumberParticles); + // Set up interparticle distance + for (int i = 0; i < NumberParticles-1; i++) { + for(int j = i+1; j < NumberParticles; j++){ + length(i,j) = RelativeDistance(r, i, j); + length(j,i) = length(i,j); + } + } + double KineticEnergy = 0.0; + // Set up kinetic energy from Slater and Jastrow terms + for (int i = 0; i < NumberParticles; i++) { + for (int k = 0; k < Dimension; k++) { + double sum1 = 0.0; + for(int j = 0; j < NumberParticles; j++){ + if ( j != i) { + sum1 += JastrowDerivative(r, VariationalParameters(1), i, j, k); + } + } + KineticEnergy += (sum1+DerivativeSPwavefunction(r(i,k),VariationalParameters(0)))*(sum1+DerivativeSPwavefunction(r(i,k),VariationalParameters(0))); + } + } + KineticEnergy += -2*VariationalParameters(0)*NumberParticles; + for (int i = 0; i < NumberParticles-1; i++) { + for (int j = i+1; j < NumberParticles; j++) { + KineticEnergy += 2.0/(pow(1.0 + VariationalParameters(1)*length(i,j),2))*(1.0/length(i,j)-2*VariationalParameters(1)/(1+VariationalParameters(1)*length(i,j)) ); + } + } + KineticEnergy *= -0.5; + // Set up potential energy, external potential + eventual electron-electron repulsion + double PotentialEnergy = 0; + for (int i = 0; i < NumberParticles; i++) { + double DistanceSquared = singleparticle_pos2(r, i); + PotentialEnergy += 0.5*DistanceSquared; // sp energy HO part, note it has the oscillator frequency set to 1! + } + // Add the electron-electron repulsion + for (int i = 0; i < NumberParticles-1; i++) { + for (int j = i+1; j < NumberParticles; j++) { + PotentialEnergy += 1.0/length(i,j); + } + } + double LocalE = KineticEnergy+PotentialEnergy; + return LocalE; +} + +// Compute the analytical expression for the quantum force +void QuantumForce(Matrix &r, Matrix &qforce, Vector &VariationalParameters) +{ + // compute the first derivative + for (int i = 0; i < NumberParticles; i++) { + for (int k = 0; k < Dimension; k++) { + // single-particle part, replace with Slater det for larger systems + double sppart = DerivativeSPwavefunction(r(i,k),VariationalParameters(0)); + // Jastrow factor contribution + double Jsum = 0.0; + for (int j = 0; j < NumberParticles; j++) { + if ( j != i) { + Jsum += JastrowDerivative(r, VariationalParameters(1), i, j, k); + } + } + qforce(i,k) = 2.0*(Jsum+sppart); + } + } +} // end of QuantumForce function + + +#define ITMAX 200 +#define EPS 3.0e-8 +#define TOLX (4*EPS) +#define STPMX 100.0 + +void dfpmin(Vector &p, int n, double gtol, int *iter, double *fret, + double(*func)(Vector &p), void (*dfunc)(Vector &p, Vector &g)) +{ + + int check,i,its,j; + double den,fac,fad,fae,fp,stpmax,sum=0.0,sumdg,sumxi,temp,test; + Vector dg(n), g(n), hdg(n), pnew(n), xi(n); + Matrix hessian(n,n); + + fp=(*func)(p); + (*dfunc)(p,g); + for (i = 0;i < n;i++) { + for (j = 0; j< n;j++) hessian(i,j)=0.0; + hessian(i,i)=1.0; + xi(i) = -g(i); + sum += p(i)*p(i); + } + stpmax=STPMX*FMAX(sqrt(sum),(double)n); + for (its=1;its<=ITMAX;its++) { + *iter=its; + lnsrch(n,p,fp,g,xi,pnew,fret,stpmax,&check,func); + fp = *fret; + for (i = 0; i< n;i++) { + xi(i)=pnew(i)-p(i); + p(i)=pnew(i); + } + test=0.0; + for (i = 0;i< n;i++) { + temp=fabs(xi(i))/FMAX(fabs(p(i)),1.0); + if (temp > test) test=temp; + } + if (test < TOLX) { + return; + } + for (i=0;i<n;i++) dg(i)=g(i); + (*dfunc)(p,g); + test=0.0; + den=FMAX(*fret,1.0); + for (i=0;i<n;i++) { + temp=fabs(g(i))*FMAX(fabs(p(i)),1.0)/den; + if (temp > test) test=temp; + } + if (test < gtol) { + return; + } + for (i=0;i<n;i++) dg(i)=g(i)-dg(i); + for (i=0;i<n;i++) { + hdg(i)=0.0; + for (j=0;j<n;j++) hdg(i) += hessian(i,j)*dg(j); + } + fac=fae=sumdg=sumxi=0.0; + for (i=0;i<n;i++) { + fac += dg(i)*xi(i); + fae += dg(i)*hdg(i); + sumdg += SQR(dg(i)); + sumxi += SQR(xi(i)); + } + if (fac*fac > EPS*sumdg*sumxi) { + fac=1.0/fac; + fad=1.0/fae; + for (i=0;i<n;i++) dg(i)=fac*xi(i)-fad*hdg(i); + for (i=0;i<n;i++) { + for (j=0;j<n;j++) { + hessian(i,j) += fac*xi(i)*xi(j) + -fad*hdg(i)*hdg(j)+fae*dg(i)*dg(j); + } + } + } + for (i=0;i<n;i++) { + xi(i)=0.0; + for (j=0;j<n;j++) xi(i) -= hessian(i,j)*g(j); + } + } + cout << "too many iterations in dfpmin" << endl; +} +#undef ITMAX +#undef EPS +#undef TOLX +#undef STPMX + +#define ALF 1.0e-4 +#define TOLX 1.0e-7 + +void lnsrch(int n, Vector &xold, double fold, Vector &g, Vector &p, Vector &x, + double *f, double stpmax, int *check, double (*func)(Vector &p)) +{ + int i; + double a,alam,alam2,alamin,b,disc,f2,fold2,rhs1,rhs2,slope,sum,temp, + test,tmplam; + + *check=0; + for (sum=0.0,i=0;i<n;i++) sum += p(i)*p(i); + sum=sqrt(sum); + if (sum > stpmax) + for (i=0;i<n;i++) p(i) *= stpmax/sum; + for (slope=0.0,i=0;i<n;i++) + slope += g(i)*p(i); + test=0.0; + for (i=0;i<n;i++) { + temp=fabs(p(i))/FMAX(fabs(xold(i)),1.0); + if (temp > test) test=temp; + } + alamin=TOLX/test; + alam=1.0; + for (;;) { + for (i=0;i<n;i++) x(i)=xold(i)+alam*p(i); + *f=(*func)(x); + if (alam < alamin) { + for (i=0;i<n;i++) x(i)=xold(i); + *check=1; + return; + } else if (*f <= fold+ALF*alam*slope) return; + else { + if (alam == 1.0) + tmplam = -slope/(2.0*(*f-fold-slope)); + else { + rhs1 = *f-fold-alam*slope; + rhs2=f2-fold2-alam2*slope; + a=(rhs1/(alam*alam)-rhs2/(alam2*alam2))/(alam-alam2); + b=(-alam2*rhs1/(alam*alam)+alam*rhs2/(alam2*alam2))/(alam-alam2); + if (a == 0.0) tmplam = -slope/(2.0*b); + else { + disc=b*b-3.0*a*slope; + if (disc<0.0) cout << "Roundoff problem in lnsrch." << endl; + else tmplam=(-b+sqrt(disc))/(3.0*a); + } + if (tmplam>0.5*alam) + tmplam=0.5*alam; + } + } + alam2=alam; + f2 = *f; + fold2=fold; + alam=FMAX(tmplam,0.1*alam); + } +} +#undef ALF +#undef TOLX ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
What is OpenMP
+
+
+
+
+
+
+
+-
+
- OpenMP provides high-level thread programming +
- Multiple cooperating threads are allowed to run simultaneously +
- Threads are created and destroyed dynamically in a fork-join pattern +
- An OpenMP program consists of a number of parallel regions +
- Between two parallel regions there is only one master thread +
- In the beginning of a parallel region, a team of new threads is spawned +
- The newly spawned threads work simultaneously with the master thread +
- At the end of a parallel region, the new threads are destroyed +
-
+
Many good tutorials online and excellent textbook
+-
+
- Using OpenMP, by B. Chapman, G. Jost, and A. van der Pas +
- Many tutorials online like OpenMP official site +
+ +
+ +
+
+
+
+
+
+
+ + +
Getting started, things to remember
+
+
+
+
+
+
+
+-
+
- Remember the header file +
+
+
+
+
+
+
+
+
+
+
+ #include <omp.h> ++
+
+
+
+
+
+
+ -
+
- Insert compiler directives in C++ syntax as +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp...
+
+
+
+
+
+
+
+
+ -
+
- Compile with for example c++ -fopenmp code.cpp +
- Execute +
- Remember to assign the environment variable OMP NUM THREADS +
- It specifies the total number of threads inside a parallel region, if not otherwise overwritten +
-
+
+ +
+ +
+
+
+
+
+
+
+ + +
OpenMP syntax
+-
+
- Mostly directives +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp construct [ clause ...]
+
+
+
+
+
+
+
+
+ -
+
- Some functions and types +
+
+
+
+
+
+
+
+
+
+
+ #include <omp.h> ++
+
+
+
+
+
+
+ -
+
- Most apply to a block of code +
- Specifically, a structured block +
- Enter at top, exit at bottom only, exit(), abort() permitted +
+ +
+ +
+
+
+
+
+
+
+ + +
Different OpenMP styles of parallelism
+OpenMP supports several different ways to specify thread parallelism
+ +-
+
- General parallel regions: All threads execute the code, roughly as if you made a routine of that region and created a thread to run that code +
- Parallel loops: Special case for loops, simplifies data parallel code +
- Task parallelism, new in OpenMP 3 +
- Several ways to manage thread coordination, including Master regions and Locks +
- Memory model for shared data +
+ +
+ +
+
+
+
+
+
+
+ + +
General code structure
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ #include <omp.h> +main () +{ +int var1, var2, var3; +/* serial code */ +/* ... */ +/* start of a parallel region */ +#pragma omp parallel private(var1, var2) shared(var3) +{ +/* ... */ +} +/* more serial code */ +/* ... */ +/* another parallel region */ +#pragma omp parallel +{ +/* ... */ +} +} ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Parallel region
+
+
+
+
+
+
+
+-
+
- A parallel region is a block of code that is executed by a team of threads +
- The following compiler directive creates a parallel region +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel { ... }
+
+
+
+
+
+
+
+
+ -
+
- Clauses can be added at the end of the directive +
- Most often used clauses: +
- default(shared) or default(none) +
- public(list of variables) +
- private(list of variables) +
-
+
+ +
+ +
+
+
+
+
+
+
+ + +
Hello world, not again, please!
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ #include <omp.h> +#include <cstdio> +int main (int argc, char *argv[]) +{ +int th_id, nthreads; +#pragma omp parallel private(th_id) shared(nthreads) +{ +th_id = omp_get_thread_num(); +printf("Hello World from thread %d\n", th_id); +#pragma omp barrier +if ( th_id == 0 ) { +nthreads = omp_get_num_threads(); +printf("There are %d threads\n",nthreads); +} +} +return 0; +} ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Hello world, yet another variant
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ #include <cstdio> +#include <omp.h> +int main(int argc, char *argv[]) +{ + omp_set_num_threads(4); +#pragma omp parallel + { + int id = omp_get_thread_num(); + int nproc = omp_get_num_threads(); + cout << "Hello world with id number and processes " << id << nproc << endl; + } +return 0; +} ++
+
+
+
+
+
+
+ Variables declared outside of the parallel region are shared by all threads +If a variable like id is declared outside of the +
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel,
+
+
+
+
+
+
+
+
+ it would have been shared by various the threads, possibly causing erroneous output
+-
+
- Why? What would go wrong? Why do we add possibly? +
+ +
+ +
+
+
+
+
+
+
+ + +
Important OpenMP library routines
+
+
+
+
+
+
+
+
+-
+
- int omp get num threads (), returns the number of threads inside a parallel region +
- int omp get thread num (), returns the a thread for each thread inside a parallel region +
- void omp set num threads (int), sets the number of threads to be used +
- void omp set nested (int), turns nested parallelism on/off +
+ +
+ +
+
+
+
+
+
+
+ + +
Private variables
+
+
+
+
+
+
+
+Private clause can be used to make thread- private versions of such variables:
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel private(id) +{ + int id = omp_get_thread_num(); + cout << "My thread num" << id << endl; +} ++
+
+
+
+
+
+
+ -
+
- What is their value on entry? Exit? +
- OpenMP provides ways to control that +
- Can use default(none) to require the sharing of each variable to be described +
+ +
+ +
+
+
+
+
+
+
+ + +
Master region
+
+
+
+
+
+
+
+It is often useful to have only one thread execute some of the code in a parallel region. I/O statements are a common example
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel +{ + #pragma omp master + { + int id = omp_get_thread_num(); + cout << "My thread num" << id << endl; + } +} ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Parallel for loop
+
+
+
+
+
+
+
+-
+
- Inside a parallel region, the following compiler directive can be used to parallelize a for-loop: +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp for
+
+
+
+
+
+
+
+
+ -
+
- Clauses can be added, such as +
- schedule(static, chunk size) +
- schedule(dynamic, chunk size) +
- schedule(guided, chunk size) (non-deterministic allocation) +
- schedule(runtime) +
- private(list of variables) +
- reduction(operator:variable) +
- nowait +
-
+
+ +
+ +
+
+
+
+
+
+
+ + +
Parallel computations and loops
+ +
+
+
+
+
+
+
+OpenMP provides an easy way to parallelize a loop
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for + for (i=0; i<n; i++) c[i] = a[i]; ++
+
+
+
+
+
+
+ OpenMP handles index variable (no need to declare in for loop or make private)
+ +Which thread does which values? Several options.
++ +
+ +
+
+
+
+
+
+
+ + +
Scheduling of loop computations
+ +
+
+
+
+
+
+
+We can let the OpenMP runtime decide. The decision is about how the loop iterates are scheduled +and OpenMP defines three choices of loop scheduling: +
+-
+
- Static: Predefined at compile time. Lowest overhead, predictable +
- Dynamic: Selection made at runtime +
- Guided: Special case of dynamic; attempts to reduce overhead +
+ +
+ +
+
+
+
+
+
+
+ + +
Example code for loop scheduling
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ #include <omp.h> +#define CHUNKSIZE 100 +#define N 1000 +int main (int argc, char *argv[]) +{ +int i, chunk; +float a[N], b[N], c[N]; +for (i=0; i < N; i++) a[i] = b[i] = i * 1.0; +chunk = CHUNKSIZE; +#pragma omp parallel shared(a,b,c,chunk) private(i) +{ +#pragma omp for schedule(dynamic,chunk) +for (i=0; i < N; i++) c[i] = a[i] + b[i]; +} /* end of parallel region */ +} ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Example code for loop scheduling, guided instead of dynamic
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ #include <omp.h> +#define CHUNKSIZE 100 +#define N 1000 +int main (int argc, char *argv[]) +{ +int i, chunk; +float a[N], b[N], c[N]; +for (i=0; i < N; i++) a[i] = b[i] = i * 1.0; +chunk = CHUNKSIZE; +#pragma omp parallel shared(a,b,c,chunk) private(i) +{ +#pragma omp for schedule(guided,chunk) +for (i=0; i < N; i++) c[i] = a[i] + b[i]; +} /* end of parallel region */ +} ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
More on Parallel for loop
+
+
+
+
+
+
+
+-
+
- The number of loop iterations cannot be non-deterministic; break, return, exit, goto not allowed inside the for-loop +
- The loop index is private to each thread +
- A reduction variable is special +
- During the for-loop there is a local private copy in each thread +
- At the end of the for-loop, all the local copies are combined together by the reduction operation +
- Unless the nowait clause is used, an implicit barrier synchronization will be added at the end by the compiler +
-
+
+
+
+
+
+
+
+
+
+
+
+ // #pragma omp parallel and #pragma omp for
+
+
+
+
+
+
+
+
+ can be combined into
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for
+
+
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
What can happen with this loop?
+ +
+
+
+
+
+
+
+What happens with code like this
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for +for (i=0; i<n; i++) sum += a[i]*a[i]; ++
+
+
+
+
+
+
+ All threads can access the sum variable, but the addition is not atomic! It is important to avoid race between threads. So-called reductions in OpenMP are thus important for performance and for obtaining correct results. OpenMP lets us indicate that a variable is used for a reduction with a particular operator. The above code becomes
+ + +
+
+
+
+
+
+
+
+
+
+ sum = 0.0; +#pragma omp parallel for reduction(+:sum) +for (i=0; i<n; i++) sum += a[i]*a[i]; ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Inner product
+
+
+
+
+
+
+$$
+\sum_{i=0}^{n-1} a_ib_i
+$$
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ int i; +double sum = 0.; +/* allocating and initializing arrays */ +/* ... */ +#pragma omp parallel for default(shared) private(i) reduction(+:sum) + for (i=0; i<N; i++) sum += a[i]*b[i]; +} ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Different threads do different tasks
+
+
+
+
+
+
+
+
+Different threads do different tasks independently, each section is executed by one thread.
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel +{ +#pragma omp sections +{ +#pragma omp section +funcA (); +#pragma omp section +funcB (); +#pragma omp section +funcC (); +} +} ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Single execution
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ #pragma omp single { ... }
+
+
+
+
+
+
+
+
+ The code is executed by one thread only, no guarantee which thread
+ +Can introduce an implicit barrier at the end
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp master { ... }
+
+
+
+
+
+
+
+
+ Code executed by the master thread, guaranteed and no implicit barrier at the end.
++ +
+ +
+
+
+
+
+
+
+ + +
Coordination and synchronization
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ #pragma omp barrier
+
+
+
+
+
+
+
+
+ Synchronization, must be encountered by all threads in a team (or none)
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp ordered { a block of codes }
+
+
+
+
+
+
+
+
+ is another form of synchronization (in sequential order). +The form +
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp critical { a block of codes }
+
+
+
+
+
+
+
+
+ and
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp atomic { single assignment statement }
+
+
+
+
+
+
+
+
+ is more efficient than
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp critical
+
+
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Data scope
+
+
+
+
+
+
+
+-
+
- OpenMP data scope attribute clauses: +
- shared +
- private +
- firstprivate +
- lastprivate +
- reduction +
-
+
What are the purposes of these attributes
+-
+
- define how and which variables are transferred to a parallel region (and back) +
- define which variables are visible to all threads in a parallel region, and which variables are privately allocated to each thread +
+ +
+ +
+
+
+
+
+
+
+ + +
Some remarks
+
+
+
+
+
+
+
+
+-
+
- When entering a parallel region, the private clause ensures each thread having its own new variable instances. The new variables are assumed to be uninitialized. +
- A shared variable exists in only one memory location and all threads can read and write to that address. It is the programmer's responsibility to ensure that multiple threads properly access a shared variable. +
- The firstprivate clause combines the behavior of the private clause with automatic initialization. +
- The lastprivate clause combines the behavior of the private clause with a copy back (from the last loop iteration or section) to the original variable outside the parallel region. +
+ +
+ +
+
+
+
+
+
+
+ + +
Parallelizing nested for-loops
+
+
+
+
+
+
+
+
+-
+
- Serial code +
+
+
+
+
+
+
+
+
+
+
+
+ for (i=0; i<100; i++) + for (j=0; j<100; j++) + a[i][j] = b[i][j] + c[i][j]; + } +} ++
+
+
+
+
+
+
+ -
+
- Parallelization +
+
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for private(j) +for (i=0; i<100; i++) + for (j=0; j<100; j++) + a[i][j] = b[i][j] + c[i][j]; + } +} ++
+
+
+
+
+
+
+ -
+
- Why not parallelize the inner loop? to save overhead of repeated thread forks-joins +
- Why must j be private? To avoid race condition among the threads +
+ +
+ +
+
+
+
+
+
+
+ + +
Nested parallelism
+
+
+
+
+
+
+
+When a thread in a parallel region encounters another parallel construct, it +may create a new team of threads and become the master of the new +team. +
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel num_threads(4) +{ +/* .... */ +#pragma omp parallel num_threads(2) +{ +// +} +} ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Parallel tasks
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ #pragma omp task +#pragma omp parallel shared(p_vec) private(i) +{ +#pragma omp single +{ +for (i=0; i<N; i++) { + double r = random_number(); + if (p_vec[i] > r) { +#pragma omp task + do_work (p_vec[i]); ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Common mistakes
+
+
+
+
+
+
+
+Race condition
+ + +
+
+
+
+
+
+
+
+
+
+
+ int nthreads; +#pragma omp parallel shared(nthreads) +{ +nthreads = omp_get_num_threads(); +} ++
+
+
+
+
+
+
+ Deadlock
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel +{ +... +#pragma omp critical +{ +... +#pragma omp barrier +} +} ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Not all computations are simple
+
+
+
+
+
+
+
+Not all computations are simple loops where the data can be evenly +divided among threads without any dependencies between threads +
+ +An example is finding the location and value of the largest element in an array
+ + +
+
+
+
+
+
+
+
+
+
+ for (i=0; i<n; i++) { + if (x[i] > maxval) { + maxval = x[i]; + maxloc = i; + } +} ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Not all computations are simple, competing threads
+
+
+
+
+
+
+
+All threads are potentially accessing and changing the same values, maxloc and maxval.
+-
+
- OpenMP provides several ways to coordinate access to shared values +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp atomic
+
+
+
+
+
+
+
+
+ -
+
- Only one thread at a time can execute the following statement (not block). We can use the critical option +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp critical
+
+
+
+
+
+
+
+
+ -
+
- Only one thread at a time can execute the following block +
Atomic may be faster than critical but depends on hardware
++ +
+ +
+
+
+
+
+
+
+ + +
How to find the max value using OpenMP
+
+
+
+
+
+
+
+Write down the simplest algorithm and look carefully for race conditions. How would you handle them? +The first step would be to parallelize as +
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for + for (i=0; i<n; i++) { + if (x[i] > maxval) { + maxval = x[i]; + maxloc = i; + } +} ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Then deal with the race conditions
+
+
+
+
+
+
+Write down the simplest algorithm and look carefully for race conditions. How would you handle them? +The first step would be to parallelize as +
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for + for (i=0; i<n; i++) { +#pragma omp critical + { + if (x[i] > maxval) { + maxval = x[i]; + maxloc = i; + } + } +} ++
+
+
+
+
+
+
+ Exercise: write a code which implements this and give an estimate on performance. Perform several runs, +with a serial code only with and without vectorization and compare the serial code with the one that uses OpenMP. Run on different archictectures if you can. +
++ +
+ +
+
+
+
+
+
+
+ + +
What can slow down OpenMP performance?
+
+
+
+
+
+
+
+Performance poor because we insisted on keeping track of the maxval and location during the execution of the loop.
+-
+
- We do not care about the value during the execution of the loop, just the value at the end. +
This is a common source of performance issues, namely the description of the method used to compute a value imposes additional, unnecessary requirements or properties
+ +Idea: Have each thread find the maxloc in its own data, then combine and use temporary arrays indexed by thread number to hold the values found by each thread ++ +
+ +
+
+
+
+
+
+
+ + +
Find the max location for each thread
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ int maxloc[MAX_THREADS], mloc; +double maxval[MAX_THREADS], mval; +#pragma omp parallel shared(maxval,maxloc) +{ + int id = omp_get_thread_num(); + maxval[id] = -1.0e30; +#pragma omp for + for (int i=0; i<n; i++) { + if (x[i] > maxval[id]) { + maxloc[id] = i; + maxval[id] = x[i]; + } + } +} ++
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Combine the values from each thread
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ #pragma omp flush (maxloc,maxval) +#pragma omp master + { + int nt = omp_get_num_threads(); + mloc = maxloc[0]; + mval = maxval[0]; + for (int i=1; i<nt; i++) { + if (maxval[i] > mval) { + mval = maxval[i]; + mloc = maxloc[i]; + } + } + } ++
+
+
+
+
+
+
+ Note that we let the master process perform the last operation.
++ +
+ +
+
+
+
+
+
+
+ + +
Matrix-matrix multiplication
+This code computes the norm of a vector using OpenMp
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ // OpenMP program to compute vector norm by adding two other vectors
+#include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include <omp.h>
+# include <ctime>
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of vector
+ int n = atoi(argv[1]);
+ double *a, *b, *c;
+ int i;
+ int thread_num;
+ double wtime, Norm2, s, angle;
+ cout << " Perform addition of two vectors and compute the norm-2." << endl;
+ omp_set_num_threads(4);
+ thread_num = omp_get_max_threads ();
+ cout << " The number of processors available = " << omp_get_num_procs () << endl ;
+ cout << " The number of threads available = " << thread_num << endl;
+ cout << " The matrix order n = " << n << endl;
+
+ s = 1.0/sqrt( (double) n);
+ wtime = omp_get_wtime ( );
+ // Allocate space for the vectors to be used
+ a = new double [n]; b = new double [n]; c = new double [n];
+ // Define parallel region
+# pragma omp parallel for default(shared) private (angle, i) reduction(+:Norm2)
+ // Set up values for vectors a and b
+ for (i = 0; i < n; i++){
+ angle = 2.0*M_PI*i/ (( double ) n);
+ a[i] = s*(sin(angle) + cos(angle));
+ b[i] = s*sin(2.0*angle);
+ c[i] = 0.0;
+ }
+ // Then perform the vector addition
+ for (i = 0; i < n; i++){
+ c[i] += a[i]+b[i];
+ }
+ // Compute now the norm-2
+ Norm2 = 0.0;
+ for (i = 0; i < n; i++){
+ Norm2 += c[i]*c[i];
+ }
+// end parallel region
+ wtime = omp_get_wtime ( ) - wtime;
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for norm-2 computation=" << wtime << endl;
+ cout << " Norm-2 = " << Norm2 << endl;
+ // Free up space
+ delete[] a;
+ delete[] b;
+ delete[] c;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ + +
+ +
+
+
+
+
+
+
+ + +
Matrix-matrix multiplication
+This the matrix-matrix multiplication code with plain c++ memory allocation using OpenMP
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+ // Matrix-matrix multiplication and Frobenius norm of a matrix with OpenMP
+#include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include <omp.h>
+# include <ctime>
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of square matrix
+ int n = atoi(argv[1]);
+ double **A, **B, **C;
+ int i, j, k;
+ int thread_num;
+ double wtime, Fsum, s, angle;
+ cout << " Compute matrix product C = A * B and Frobenius norm." << endl;
+ omp_set_num_threads(4);
+ thread_num = omp_get_max_threads ();
+ cout << " The number of processors available = " << omp_get_num_procs () << endl ;
+ cout << " The number of threads available = " << thread_num << endl;
+ cout << " The matrix order n = " << n << endl;
+
+ s = 1.0/sqrt( (double) n);
+ wtime = omp_get_wtime ( );
+ // Allocate space for the two matrices
+ A = new double*[n]; B = new double*[n]; C = new double*[n];
+ for (i = 0; i < n; i++){
+ A[i] = new double[n];
+ B[i] = new double[n];
+ C[i] = new double[n];
+ }
+ // Define parallel region
+# pragma omp parallel for default(shared) private (angle, i, j, k) reduction(+:Fsum)
+ // Set up values for matrix A and B and zero matrix C
+ for (i = 0; i < n; i++){
+ for (j = 0; j < n; j++) {
+ angle = 2.0*M_PI*i*j/ (( double ) n);
+ A[i][j] = s * ( sin ( angle ) + cos ( angle ) );
+ B[j][i] = A[i][j];
+ }
+ }
+ // Then perform the matrix-matrix multiplication
+ for (i = 0; i < n; i++){
+ for (j = 0; j < n; j++) {
+ C[i][j] = 0.0;
+ for (k = 0; k < n; k++) {
+ C[i][j] += A[i][k]*B[k][j];
+ }
+ }
+ }
+ // Compute now the Frobenius norm
+ Fsum = 0.0;
+ for (i = 0; i < n; i++){
+ for (j = 0; j < n; j++) {
+ Fsum += C[i][j]*C[i][j];
+ }
+ }
+ Fsum = sqrt(Fsum);
+// end parallel region and letting only one thread perform I/O
+ wtime = omp_get_wtime ( ) - wtime;
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for matrix-matrix multiplication=" << wtime << endl;
+ cout << " Frobenius norm = " << Fsum << endl;
+ // Free up space
+ for (int i = 0; i < n; i++){
+ delete[] A[i];
+ delete[] B[i];
+ delete[] C[i];
+ }
+ delete[] A;
+ delete[] B;
+ delete[] C;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ + +
+ +
+
+
+```
+
+### Markdown
+
+It's possible to write your slides using Markdown. To enable Markdown, add the ```data-markdown``` attribute to your ``` elements
+ { src: 'plugin/highlight/highlight.js', async: true, callback: function() { hljs.initHighlightingOnLoad(); } },
+
+ // Zoom in and out with Alt+click
+ { src: 'plugin/zoom-js/zoom.js', async: true },
+
+ // Speaker notes
+ { src: 'plugin/notes/notes.js', async: true },
+
+ // Remote control your reveal.js presentation using a touch device
+ { src: 'plugin/remotes/remotes.js', async: true },
+
+ // MathJax
+ { src: 'plugin/math/math.js', async: true }
+ ]
+});
+```
+
+You can add your own extensions using the same syntax. The following properties are available for each dependency object:
+- **src**: Path to the script to load
+- **async**: [optional] Flags if the script should load after reveal.js has started, defaults to false
+- **callback**: [optional] Function to execute when the script has loaded
+- **condition**: [optional] Function which must return true for the script to be loaded
+
+
+### Ready Event
+
+A 'ready' event is fired when reveal.js has loaded all non-async dependencies and is ready to start navigating. To check if reveal.js is already 'ready' you can call `Reveal.isReady()`.
+
+```javascript
+Reveal.addEventListener( 'ready', function( event ) {
+ // event.currentSlide, event.indexh, event.indexv
+} );
+```
+
+
+### Presentation Size
+
+All presentations have a normal size, that is the resolution at which they are authored. The framework will automatically scale presentations uniformly based on this size to ensure that everything fits on any given display or viewport.
+
+See below for a list of configuration options related to sizing, including default values:
+
+```javascript
+Reveal.initialize({
+
+ ...
+
+ // The "normal" size of the presentation, aspect ratio will be preserved
+ // when the presentation is scaled to fit different resolutions. Can be
+ // specified using percentage units.
+ width: 960,
+ height: 700,
+
+ // Factor of the display size that should remain empty around the content
+ margin: 0.1,
+
+ // Bounds for smallest/largest possible scale to apply to content
+ minScale: 0.2,
+ maxScale: 1.5
+
+});
+```
+
+
+### Auto-sliding
+
+Presentations can be configured to progress through slides automatically, without any user input. To enable this you will need to tell the framework how many milliseconds it should wait between slides:
+
+```javascript
+// Slide every five seconds
+Reveal.configure({
+ autoSlide: 5000
+});
+```
+When this is turned on a control element will appear that enables users to pause and resume auto-sliding. Alternatively, sliding can be paused or resumed by pressing »a« on the keyboard. Sliding is paused automatically as soon as the user starts navigating. You can disable these controls by specifying ```autoSlideStoppable: false``` in your reveal.js config.
+
+You can also override the slide duration for individual slides and fragments by using the ```data-autoslide``` attribute:
+
+```html
+
+ After 2 seconds the first fragment will be shown.
+ After 10 seconds the next fragment will be shown.
+ Now, the fragment is displayed for 2 seconds before the next slide is shown.
+
+```
+
+Whenever the auto-slide mode is resumed or paused the ```autoslideresumed``` and ```autoslidepaused``` events are fired.
+
+
+### Keyboard Bindings
+
+If you're unhappy with any of the default keyboard bindings you can override them using the ```keyboard``` config option:
+
+```javascript
+Reveal.configure({
+ keyboard: {
+ 13: 'next', // go to the next slide when the ENTER key is pressed
+ 27: function() {}, // do something custom when ESC is pressed
+ 32: null // don't do anything when SPACE is pressed (i.e. disable a reveal.js default binding)
+ }
+});
+```
+
+### Lazy Loading
+
+When working on presentation with a lot of media or iframe content it's important to load lazily. Lazy loading means that reveal.js will only load content for the few slides nearest to the current slide. The number of slides that are preloaded is determined by the `viewDistance` configuration option.
+
+To enable lazy loading all you need to do is change your "src" attributes to "data-src" as shown below. This is supported for image, video, audio and iframe elements. Lazy loaded iframes will also unload when the containing slide is no longer visible.
+
+```html
+
+ ![]()
+
+
+
+```
+
+
+### API
+
+The ``Reveal`` object exposes a JavaScript API for controlling navigation and reading state:
+
+```javascript
+// Navigation
+Reveal.slide( indexh, indexv, indexf );
+Reveal.left();
+Reveal.right();
+Reveal.up();
+Reveal.down();
+Reveal.prev();
+Reveal.next();
+Reveal.prevFragment();
+Reveal.nextFragment();
+
+// Toggle presentation states, optionally pass true/false to force on/off
+Reveal.toggleOverview();
+Reveal.togglePause();
+Reveal.toggleAutoSlide();
+
+// Change a config value at runtime
+Reveal.configure({ controls: true });
+
+// Returns the present configuration options
+Reveal.getConfig();
+
+// Fetch the current scale of the presentation
+Reveal.getScale();
+
+// Retrieves the previous and current slide elements
+Reveal.getPreviousSlide();
+Reveal.getCurrentSlide();
+
+Reveal.getIndices(); // { h: 0, v: 0 } }
+Reveal.getProgress(); // 0-1
+Reveal.getTotalSlides();
+
+// State checks
+Reveal.isFirstSlide();
+Reveal.isLastSlide();
+Reveal.isOverview();
+Reveal.isPaused();
+Reveal.isAutoSliding();
+```
+
+### Slide Changed Event
+
+A 'slidechanged' event is fired each time the slide is changed (regardless of state). The event object holds the index values of the current slide as well as a reference to the previous and current slide HTML nodes.
+
+Some libraries, like MathJax (see [#226](https://github.com/hakimel/reveal.js/issues/226#issuecomment-10261609)), get confused by the transforms and display states of slides. Often times, this can be fixed by calling their update or render function from this callback.
+
+```javascript
+Reveal.addEventListener( 'slidechanged', function( event ) {
+ // event.previousSlide, event.currentSlide, event.indexh, event.indexv
+} );
+```
+
+### Presentation State
+
+The presentation's current state can be fetched by using the `getState` method. A state object contains all of the information required to put the presentation back as it was when `getState` was first called. Sort of like a snapshot. It's a simple object that can easily be stringified and persisted or sent over the wire.
+
+```javascript
+Reveal.slide( 1 );
+// we're on slide 1
+
+var state = Reveal.getState();
+
+Reveal.slide( 3 );
+// we're on slide 3
+
+Reveal.setState( state );
+// we're back on slide 1
+```
+
+### Slide States
+
+If you set ``data-state="somestate"`` on a slide ````, "somestate" will be applied as a class on the document element when that slide is opened. This allows you to apply broad style changes to the page based on the active slide.
+
+Furthermore you can also listen to these changes in state via JavaScript:
+
+```javascript
+Reveal.addEventListener( 'somestate', function() {
+ // TODO: Sprinkle magic
+}, false );
+```
+
+### Slide Backgrounds
+
+Slides are contained within a limited portion of the screen by default to allow them to fit any display and scale uniformly. You can apply full page backgrounds outside of the slide area by adding a ```data-background``` attribute to your `````` elements. Four different types of backgrounds are supported: color, image, video and iframe. Below are a few examples.
+
+```html
+
+ All CSS color formats are supported, like rgba() or hsl().
+
+
+ This slide will have a full-size background image.
+
+
+ This background image will be sized to 100px and repeated.
+
+
+ Video. Multiple sources can be defined using a comma separated list. Video will loop when the data-background-video-loop attribute is provided.
+
+
+ Embeds a web page as a background. Note that the page won't be interactive.
+
+```
+
+Backgrounds transition using a fade animation by default. This can be changed to a linear sliding transition by passing ```backgroundTransition: 'slide'``` to the ```Reveal.initialize()``` call. Alternatively you can set ```data-background-transition``` on any section with a background to override that specific transition.
+
+
+### Parallax Background
+
+If you want to use a parallax scrolling background, set the first two config properties below when initializing reveal.js (the other two are optional).
+
+```javascript
+Reveal.initialize({
+
+ // Parallax background image
+ parallaxBackgroundImage: '', // e.g. "https://s3.amazonaws.com/hakim-static/reveal-js/reveal-parallax-1.jpg"
+
+ // Parallax background size
+ parallaxBackgroundSize: '', // CSS syntax, e.g. "2100px 900px" - currently only pixels are supported (don't use % or auto)
+
+ // Amount of pixels to move the parallax background per slide step,
+ // a value of 0 disables movement along the given axis
+ // These are optional, if they aren't specified they'll be calculated automatically
+ parallaxBackgroundHorizontal: 200,
+ parallaxBackgroundVertical: 50
+
+});
+```
+
+Make sure that the background size is much bigger than screen size to allow for some scrolling. [View example](http://lab.hakim.se/reveal-js/?parallaxBackgroundImage=https%3A%2F%2Fs3.amazonaws.com%2Fhakim-static%2Freveal-js%2Freveal-parallax-1.jpg¶llaxBackgroundSize=2100px%20900px).
+
+
+
+### Slide Transitions
+The global presentation transition is set using the ```transition``` config value. You can override the global transition for a specific slide by using the ```data-transition``` attribute:
+
+```html
+
+ This slide will override the presentation transition and zoom!
+
+
+
+ Choose from three transition speeds: default, fast or slow!
+
+```
+
+You can also use different in and out transitions for the same slide:
+
+```html
+
+ The train goes on …
+
+
+ and on …
+
+
+ and stops.
+
+
+ (Passengers entering and leaving)
+
+
+ And it starts again.
+
+```
+
+
+Note that this does not work with the page and cube transitions.
+
+
+### Internal links
+
+It's easy to link between slides. The first example below targets the index of another slide whereas the second targets a slide with an ID attribute (``````):
+
+```html
+Link
+Link
+```
+
+You can also add relative navigation links, similar to the built in reveal.js controls, by appending one of the following classes on any element. Note that each element is automatically given an ```enabled``` class when it's a valid navigation route based on the current slide.
+
+```html
+
+
+
+
+
+
+```
+
+
+### Fragments
+Fragments are used to highlight individual elements on a slide. Every element with the class ```fragment``` will be stepped through before moving on to the next slide. Here's an example: http://lab.hakim.se/reveal-js/#/fragments
+
+The default fragment style is to start out invisible and fade in. This style can be changed by appending a different class to the fragment:
+
+```html
+
+ grow
+ shrink
+ fade-out
+ visible only once
+ blue only once
+ highlight-red
+ highlight-green
+ highlight-blue
+
+```
+
+Multiple fragments can be applied to the same element sequentially by wrapping it, this will fade in the text on the first step and fade it back out on the second.
+
+```html
+
+
+ I'll fade in, then out
+
+
+```
+
+The display order of fragments can be controlled using the ```data-fragment-index``` attribute.
+
+```html
+
+ Appears last
+ Appears first
+ Appears second
+
+```
+
+### Fragment events
+
+When a slide fragment is either shown or hidden reveal.js will dispatch an event.
+
+Some libraries, like MathJax (see #505), get confused by the initially hidden fragment elements. Often times this can be fixed by calling their update or render function from this callback.
+
+```javascript
+Reveal.addEventListener( 'fragmentshown', function( event ) {
+ // event.fragment = the fragment DOM element
+} );
+Reveal.addEventListener( 'fragmenthidden', function( event ) {
+ // event.fragment = the fragment DOM element
+} );
+```
+
+### Code syntax highlighting
+
+By default, Reveal is configured with [highlight.js](http://softwaremaniacs.org/soft/highlight/en/) for code syntax highlighting. Below is an example with clojure code that will be syntax highlighted. When the `data-trim` attribute is present surrounding whitespace is automatically removed.
+
+```html
+
+
+(def lazy-fib
+ (concat
+ [0 1]
+ ((fn rfib [a b]
+ (lazy-cons (+ a b) (rfib b (+ a b)))) 0 1)))
+
+
+```
+
+### Slide number
+If you would like to display the page number of the current slide you can do so using the ```slideNumber``` configuration value.
+
+```javascript
+// Shows the slide number using default formatting
+Reveal.configure({ slideNumber: true });
+
+// Slide number formatting can be configured using these variables:
+// h: current slide's horizontal index
+// v: current slide's vertical index
+// c: current slide index (flattened)
+// t: total number of slides (flattened)
+Reveal.configure({ slideNumber: 'c / t' });
+
+```
+
+
+### Overview mode
+
+Press "Esc" or "o" keys to toggle the overview mode on and off. While you're in this mode, you can still navigate between slides,
+as if you were at 1,000 feet above your presentation. The overview mode comes with a few API hooks:
+
+```javascript
+Reveal.addEventListener( 'overviewshown', function( event ) { /* ... */ } );
+Reveal.addEventListener( 'overviewhidden', function( event ) { /* ... */ } );
+
+// Toggle the overview mode programmatically
+Reveal.toggleOverview();
+```
+
+### Fullscreen mode
+Just press »F« on your keyboard to show your presentation in fullscreen mode. Press the »ESC« key to exit fullscreen mode.
+
+
+### Embedded media
+Embedded HTML5 ` 
+
+
+
+
+
+
+
+
+
+ + +
+
+
+
+
+
+Morten Hjorth-Jensen Email morten.hjorth-jensen@fys.uio.no [1, 2]
+
+
+
+[1] Department of Physics and Center fo Computing in Science Education, University of Oslo, Oslo, Norway
+
+
+[2] Department of Physics and Astronomy and Facility for Rare Ion Beams, Michigan State University, East Lansing, Michigan, USA
+
+
+
+
+
+ + + + + + +
+
+Week 11, March 11-15: Resampling Techniques, Bootstrap and Blocking
++
March 11-15
++ + + + + + +
+ +
+ +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/doc/src/week9/week9-solarized.html b/doc/src/week9/week9-solarized.html
new file mode 100644
index 00000000..354c9093
--- /dev/null
+++ b/doc/src/week9/week9-solarized.html
@@ -0,0 +1,6276 @@
+
+
+
+
+
+
+
+Week 11, March 11-15: Resampling Techniques, Bootstrap and Blocking
++
March 11-15
++ +
+
Overview of week 11, March 11-15
+
+Topics
+
+
+
++
-
+
- Reminder from last week about statistical observables, the central limit theorem and bootstrapping, see notes from last week +
- Resampling Techniques, emphasis on Blocking +
- Discussion of onebody densities (whiteboard notes) +
- Start discussion on optimization and parallelization for Python and C++ + + +
Note, these notes contain additional material om optimization and parallelization. Parts of this material will be discussed this week.
+ ++
Why resampling methods ?
+
+Statistical analysis
+
+
+
++
-
+
- Our simulations can be treated as computer experiments. This is particularly the case for Monte Carlo methods +
- The results can be analysed with the same statistical tools as we would use analysing experimental data. +
- As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors. +
+
Statistical analysis
+
+
+
+
+
++
-
+
- As in other experiments, many numerical experiments have two classes of errors:
+
- Statistical errors +
- Systematical errors + +
- Statistical errors can be estimated using standard tools from statistics +
- Systematical errors are method specific and must be treated differently from case to case. +
+
And why do we use such methods?
+ +As you will see below, due to correlations between various +measurements, we need to evaluate the so-called covariance in order to +establish a proper evaluation of the total variance and the thereby +the standard deviation of a given expectation value. +
+ +The covariance however, leads to an evaluation of a double sum over the various stochastic variables. This becomes computationally too expensive to evaluate. +Methods like the Bootstrap, the Jackknife and/or Blocking allow us to circumvent this problem. +
+ ++
Central limit theorem
+ +Last week we derived the central limit theorem with the following assumptions:
+ +
+Measurement \( i \)
+
+
+
++
We assumed that each individual measurement \( x_{ij} \) is represented by stochastic variables which independent and identically distributed (iid). +This defined the sample mean of of experiment \( i \) with \( n \) samples as +
+$$ +\overline{x}_i=\frac{1}{n}\sum_{j} x_{ij}. +$$ + +and the sample variance
+$$ +\sigma^2_i=\frac{1}{n}\sum_{j} \left(x_{ij}-\overline{x}_i\right)^2. +$$ ++
Further remarks
+ +Note that we use \( n \) instead of \( n-1 \) in the definition of +variance. The sample variance and the sample mean are not necessarily equal to +the exact values we would get if we knew the corresponding probability +distribution. +
+ ++
Running many measurements
+ +
+Adding \( m \) measurements \( i \)
+
+
++
With the assumption that the average measurements \( i \) are also defined as iid stochastic variables and have the same probability function \( p \), +we defined the total average over \( m \) experiments as +
+$$ +\overline{X}=\frac{1}{m}\sum_{i} \overline{x}_{i}. +$$ + +and the total variance
+$$ +\sigma^2_{m}=\frac{1}{m}\sum_{i} \left( \overline{x}_{i}-\overline{X}\right)^2. +$$ +These are the quantities we used in showing that if the individual mean values are iid stochastic variables, then in the limit \( m\rightarrow \infty \), the distribution for \( \overline{X} \) is given by a Gaussian distribution with variance \( \sigma^2_m \).
+ ++
Adding more definitions
+ +The total sample variance over the \( mn \) measurements is defined as
+$$ +\sigma^2=\frac{1}{mn}\sum_{i=1}^{m} \sum_{j=1}^{n}\left(x_{ij}-\overline{X}\right)^2. +$$ + +We have from the equation for \( \sigma_m^2 \)
+$$ +\overline{x}_i-\overline{X}=\frac{1}{n}\sum_{j=1}^{n}\left(x_{i}-\overline{X}\right), +$$ + +and introducing the centered value \( \tilde{x}_{ij}=x_{ij}-\overline{X} \), we can rewrite \( \sigma_m^2 \) as
+$$ +\sigma^2_{m}=\frac{1}{m}\sum_{i} \left( \overline{x}_{i}-\overline{X}\right)^2=\frac{1}{m}\sum_{i=1}^{m}\left[ \frac{i}{n}\sum_{j=1}^{n}\tilde{x}_{ij}\right]^2. +$$ + + ++
Further rewriting
+ +We can rewrite the latter in terms of a sum over diagonal elements only and another sum which contains the non-diagonal elements
+$$ +\begin{align*} +\sigma^2_{m}& =\frac{1}{m}\sum_{i=1}^{m}\left[ \frac{i}{n}\sum_{j=1}^{n}\tilde{x}_{ij}\right]^2 \\ + & = \frac{1}{mn^2}\sum_{i=1}^{m} \sum_{j=1}^{n}\tilde{x}_{ij}^2+\frac{2}{mn^2}\sum_{i=1}^{m} \sum_{j < k}^{n}\tilde{x}_{ij}\tilde{x}_{ik}. +\end{align*} +$$ + +The first term on the last rhs is nothing but the total sample variance \( \sigma^2 \) divided by \( m \). The second term represents the covariance.
+ ++
The covariance term
+ +Using the definition of the total sample variance we have
+$$ +\begin{align*} +\sigma^2_{m}& = \frac{\sigma^2}{m}+\frac{2}{mn^2}\sum_{i=1}^{m} \sum_{j < k}^{n}\tilde{x}_{ij}\tilde{x}_{ik}. +\end{align*} +$$ + +The first term is what we have used till now in order to estimate the +standard deviation. However, the second term which gives us a measure +of the correlations between different stochastic events, can result in +contributions which give rise to a larger standard deviation and +variance \( \sigma_m^2 \). Note also the evaluation of the second term +leads to a double sum over all events. If we run a VMC calculation +with say \( 10^9 \) Monte carlo samples, the latter term would lead to +\( 10^{18} \) function evaluations. We don't want to, by obvious reasons, to venture into that many evaluations. +
+ +Note also that if our stochastic events are iid then the covariance terms is zero.
+ ++
Rewriting the covariance term
+ +We introduce now a variable \( d=\vert j-k\vert \) and rewrite
+$$ +\frac{2}{mn^2}\sum_{i=1}^{m} \sum_{j < k}^{n}\tilde{x}_{ij}\tilde{x}_{ik}, +$$ + +in terms of a function
+$$ +f_d=\frac{2}{mn}\sum_{i=1}^{m} \sum_{k=1}^{n-d}\tilde{x}_{ik}\tilde{x}_{i(k+d)}. +$$ + +We note that for \( d=0 \) we have
+$$ +f_0=\frac{2}{mn}\sum_{i=1}^{m} \sum_{k=1}^{n}\tilde{x}_{ik}\tilde{x}_{i(k)}=\sigma^2! +$$ + + ++
Introducing the correlation function
+ +We introduce then a correlation function \( \kappa_d=f_d/\sigma^2 \). Note that \( \kappa_0 =1 \). We rewrite the variance \( \sigma_m^2 \) as
+$$ +\begin{align*} +\sigma^2_{m}& = \frac{\sigma^2}{m}\left[1+2\sum_{d=1}^{n-1} \kappa_d\right]. +\end{align*} +$$ + +The code here shows the evolution of \( \kappa_d \) as a function of \( d \) for a series of random numbers. We see that the function \( \kappa_d \) approaches \( 0 \) as \( d\rightarrow \infty \).
+ +Note: code will be inserted here later.
+ ++
Resampling methods: Blocking
+ +The blocking method was made popular by Flyvbjerg and Pedersen (1989) +and has become one of the standard ways to estimate the variance +\( \mathrm{var}(\widehat{\theta}) \) for exactly one estimator \( \widehat{\theta} \), namely +\( \widehat{\theta} = \overline{X} \), the mean value. +
+ +Assume \( n = 2^d \) for some integer \( d>1 \) and \( X_1,X_2,\cdots, X_n \) is a stationary time series to begin with. +Moreover, assume that the series is asymptotically uncorrelated. We switch to vector notation by arranging \( X_1,X_2,\cdots,X_n \) in an \( n \)-tuple. Define: +
+$$ +\begin{align*} +\hat{X} = (X_1,X_2,\cdots,X_n). +\end{align*} +$$ + + ++
Why blocking?
+ +The strength of the blocking method is when the number of +observations, \( n \) is large. For large \( n \), the complexity of dependent +bootstrapping scales poorly, but the blocking method does not, +moreover, it becomes more accurate the larger \( n \) is. +
+ ++
Blocking Transformations
+We now define the blocking transformations. The idea is to take the mean of subsequent +pair of elements from \( \boldsymbol{X} \) and form a new vector +\( \boldsymbol{X}_1 \). Continuing in the same way by taking the mean of +subsequent pairs of elements of \( \boldsymbol{X}_1 \) we obtain \( \boldsymbol{X}_2 \), and +so on. +Define \( \boldsymbol{X}_i \) recursively by: +
+ +$$ +\begin{align} +(\boldsymbol{X}_0)_k &\equiv (\boldsymbol{X})_k \nonumber \\ +(\boldsymbol{X}_{i+1})_k &\equiv \frac{1}{2}\Big( (\boldsymbol{X}_i)_{2k-1} + +(\boldsymbol{X}_i)_{2k} \Big) \qquad \text{for all} \qquad 1 \leq i \leq d-1 +\label{_auto1} +\end{align} +$$ + + ++
Blocking transformations
+ +The quantity \( \boldsymbol{X}_k \) is +subject to \( k \) blocking transformations. We now have \( d \) vectors +\( \boldsymbol{X}_0, \boldsymbol{X}_1,\cdots,\vec X_{d-1} \) containing the subsequent +averages of observations. It turns out that if the components of +\( \boldsymbol{X} \) is a stationary time series, then the components of +\( \boldsymbol{X}_i \) is a stationary time series for all \( 0 \leq i \leq d-1 \) +
+ +We can then compute the autocovariance, the variance, sample mean, and +number of observations for each \( i \). +Let \( \gamma_i, \sigma_i^2, +\overline{X}_i \) denote the covariance, variance and average of the +elements of \( \boldsymbol{X}_i \) and let \( n_i \) be the number of elements of +\( \boldsymbol{X}_i \). It follows by induction that \( n_i = n/2^i \). +
+ ++
Blocking Transformations
+ +Using the +definition of the blocking transformation and the distributive +property of the covariance, it is clear that since \( h =|i-j| \) +we can define +
+$$ +\begin{align} +\gamma_{k+1}(h) &= cov\left( ({X}_{k+1})_{i}, ({X}_{k+1})_{j} \right) \nonumber \\ +&= \frac{1}{4}cov\left( ({X}_{k})_{2i-1} + ({X}_{k})_{2i}, ({X}_{k})_{2j-1} + ({X}_{k})_{2j} \right) \nonumber \\ +&= \frac{1}{2}\gamma_{k}(2h) + \frac{1}{2}\gamma_k(2h+1) \hspace{0.1cm} \mathrm{h = 0} +\label{_auto2}\\ +&=\frac{1}{4}\gamma_k(2h-1) + \frac{1}{2}\gamma_k(2h) + \frac{1}{4}\gamma_k(2h+1) \quad \mathrm{else} +\label{_auto3} +\end{align} +$$ + +The quantity \( \hat{X} \) is asymptotically uncorrelated by assumption, \( \hat{X}_k \) is also asymptotic uncorrelated. Let's turn our attention to the variance of the sample +mean \( \mathrm{var}(\overline{X}) \). +
+ ++
Blocking Transformations, getting there
+We have
+$$ +\begin{align} +\mathrm{var}(\overline{X}_k) = \frac{\sigma_k^2}{n_k} + \underbrace{\frac{2}{n_k} \sum_{h=1}^{n_k-1}\left( 1 - \frac{h}{n_k} \right)\gamma_k(h)}_{\equiv e_k} = \frac{\sigma^2_k}{n_k} + e_k \quad \text{if} \quad \gamma_k(0) = \sigma_k^2. +\label{_auto4} +\end{align} +$$ + +The term \( e_k \) is called the truncation error:
+$$ +\begin{equation} +e_k = \frac{2}{n_k} \sum_{h=1}^{n_k-1}\left( 1 - \frac{h}{n_k} \right)\gamma_k(h). +\label{_auto5} +\end{equation} +$$ + +We can show that \( \mathrm{var}(\overline{X}_i) = \mathrm{var}(\overline{X}_j) \) for all \( 0 \leq i \leq d-1 \) and \( 0 \leq j \leq d-1 \).
+ ++
Blocking Transformations, final expressions
+ +We can then wrap up
+$$ +\begin{align} +n_{j+1} \overline{X}_{j+1} &= \sum_{i=1}^{n_{j+1}} (\hat{X}_{j+1})_i = \frac{1}{2}\sum_{i=1}^{n_{j}/2} (\hat{X}_{j})_{2i-1} + (\hat{X}_{j})_{2i} \nonumber \\ +&= \frac{1}{2}\left[ (\hat{X}_j)_1 + (\hat{X}_j)_2 + \cdots + (\hat{X}_j)_{n_j} \right] = \underbrace{\frac{n_j}{2}}_{=n_{j+1}} \overline{X}_j = n_{j+1}\overline{X}_j. +\label{_auto6} +\end{align} +$$ + +By repeated use of this equation we get \( \mathrm{var}(\overline{X}_i) = \mathrm{var}(\overline{X}_0) = \mathrm{var}(\overline{X}) \) for all \( 0 \leq i \leq d-1 \). This has the consequence that
+$$ +\begin{align} +\mathrm{var}(\overline{X}) = \frac{\sigma_k^2}{n_k} + e_k \qquad \text{for all} \qquad 0 \leq k \leq d-1. \label{eq:convergence} +\end{align} +$$ + + ++
More on the blocking method
+ +Flyvbjerg and Petersen demonstrated that the sequence +\( \{e_k\}_{k=0}^{d-1} \) is decreasing, and conjecture that the term +\( e_k \) can be made as small as we would like by making \( k \) (and hence +\( d \)) sufficiently large. The sequence is decreasing. +It means we can apply blocking transformations until +\( e_k \) is sufficiently small, and then estimate \( \mathrm{var}(\overline{X}) \) by +\( \widehat{\sigma}^2_k/n_k \). +
+ +For an elegant solution and proof of the blocking method, see the recent article of Marius Jonsson (former MSc student of the Computational Physics group).
+ ++
Example code form last week
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ # 2-electron VMC code for 2dim quantum dot with importance sampling +# Using gaussian rng for new positions and Metropolis- Hastings +# Added energy minimization +from math import exp, sqrt +from random import random, seed, normalvariate +import numpy as np +import matplotlib.pyplot as plt +from mpl_toolkits.mplot3d import Axes3D +from matplotlib import cm +from matplotlib.ticker import LinearLocator, FormatStrFormatter +from scipy.optimize import minimize +import sys +import os + +# Where to save data files +PROJECT_ROOT_DIR = "Results" +DATA_ID = "Results/EnergyMin" + +if not os.path.exists(PROJECT_ROOT_DIR): + os.mkdir(PROJECT_ROOT_DIR) + +if not os.path.exists(DATA_ID): + os.makedirs(DATA_ID) + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +outfile = open(data_path("Energies.dat"),'w') + + +# Trial wave function for the 2-electron quantum dot in two dims +def WaveFunction(r,alpha,beta): + r1 = r[0,0]**2 + r[0,1]**2 + r2 = r[1,0]**2 + r[1,1]**2 + r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2) + deno = r12/(1+beta*r12) + return exp(-0.5*alpha*(r1+r2)+deno) + +# Local energy for the 2-electron quantum dot in two dims, using analytical local energy +def LocalEnergy(r,alpha,beta): + + r1 = (r[0,0]**2 + r[0,1]**2) + r2 = (r[1,0]**2 + r[1,1]**2) + r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2) + deno = 1.0/(1+beta*r12) + deno2 = deno*deno + return 0.5*(1-alpha*alpha)*(r1 + r2) +2.0*alpha + 1.0/r12+deno2*(alpha*r12-deno2+2*beta*deno-1.0/r12) + +# Derivate of wave function ansatz as function of variational parameters +def DerivativeWFansatz(r,alpha,beta): + + WfDer = np.zeros((2), np.double) + r1 = (r[0,0]**2 + r[0,1]**2) + r2 = (r[1,0]**2 + r[1,1]**2) + r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2) + deno = 1.0/(1+beta*r12) + deno2 = deno*deno + WfDer[0] = -0.5*(r1+r2) + WfDer[1] = -r12*r12*deno2 + return WfDer + +# Setting up the quantum force for the two-electron quantum dot, recall that it is a vector +def QuantumForce(r,alpha,beta): + + qforce = np.zeros((NumberParticles,Dimension), np.double) + r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2) + deno = 1.0/(1+beta*r12) + qforce[0,:] = -2*r[0,:]*alpha*(r[0,:]-r[1,:])*deno*deno/r12 + qforce[1,:] = -2*r[1,:]*alpha*(r[1,:]-r[0,:])*deno*deno/r12 + return qforce + + +# Computing the derivative of the energy and the energy +def EnergyDerivative(x0): + + + # Parameters in the Fokker-Planck simulation of the quantum force + D = 0.5 + TimeStep = 0.05 + # positions + PositionOld = np.zeros((NumberParticles,Dimension), np.double) + PositionNew = np.zeros((NumberParticles,Dimension), np.double) + # Quantum force + QuantumForceOld = np.zeros((NumberParticles,Dimension), np.double) + QuantumForceNew = np.zeros((NumberParticles,Dimension), np.double) + + energy = 0.0 + DeltaE = 0.0 + alpha = x0[0] + beta = x0[1] + EnergyDer = 0.0 + DeltaPsi = 0.0 + DerivativePsiE = 0.0 + #Initial position + for i in range(NumberParticles): + for j in range(Dimension): + PositionOld[i,j] = normalvariate(0.0,1.0)*sqrt(TimeStep) + wfold = WaveFunction(PositionOld,alpha,beta) + QuantumForceOld = QuantumForce(PositionOld,alpha, beta) + + #Loop over MC MCcycles + for MCcycle in range(NumberMCcycles): + #Trial position moving one particle at the time + for i in range(NumberParticles): + for j in range(Dimension): + PositionNew[i,j] = PositionOld[i,j]+normalvariate(0.0,1.0)*sqrt(TimeStep)+\ + QuantumForceOld[i,j]*TimeStep*D + wfnew = WaveFunction(PositionNew,alpha,beta) + QuantumForceNew = QuantumForce(PositionNew,alpha, beta) + GreensFunction = 0.0 + for j in range(Dimension): + GreensFunction += 0.5*(QuantumForceOld[i,j]+QuantumForceNew[i,j])*\ + (D*TimeStep*0.5*(QuantumForceOld[i,j]-QuantumForceNew[i,j])-\ + PositionNew[i,j]+PositionOld[i,j]) + + GreensFunction = exp(GreensFunction) + ProbabilityRatio = GreensFunction*wfnew**2/wfold**2 + #Metropolis-Hastings test to see whether we accept the move + if random() <= ProbabilityRatio: + for j in range(Dimension): + PositionOld[i,j] = PositionNew[i,j] + QuantumForceOld[i,j] = QuantumForceNew[i,j] + wfold = wfnew + DeltaE = LocalEnergy(PositionOld,alpha,beta) + DerPsi = DerivativeWFansatz(PositionOld,alpha,beta) + DeltaPsi += DerPsi + energy += DeltaE + DerivativePsiE += DerPsi*DeltaE + + # We calculate mean values + energy /= NumberMCcycles + DerivativePsiE /= NumberMCcycles + DeltaPsi /= NumberMCcycles + EnergyDer = 2*(DerivativePsiE-DeltaPsi*energy) + return EnergyDer + + +# Computing the expectation value of the local energy +def Energy(x0): + # Parameters in the Fokker-Planck simulation of the quantum force + D = 0.5 + TimeStep = 0.05 + # positions + PositionOld = np.zeros((NumberParticles,Dimension), np.double) + PositionNew = np.zeros((NumberParticles,Dimension), np.double) + # Quantum force + QuantumForceOld = np.zeros((NumberParticles,Dimension), np.double) + QuantumForceNew = np.zeros((NumberParticles,Dimension), np.double) + + energy = 0.0 + DeltaE = 0.0 + alpha = x0[0] + beta = x0[1] + #Initial position + for i in range(NumberParticles): + for j in range(Dimension): + PositionOld[i,j] = normalvariate(0.0,1.0)*sqrt(TimeStep) + wfold = WaveFunction(PositionOld,alpha,beta) + QuantumForceOld = QuantumForce(PositionOld,alpha, beta) + + #Loop over MC MCcycles + for MCcycle in range(NumberMCcycles): + #Trial position moving one particle at the time + for i in range(NumberParticles): + for j in range(Dimension): + PositionNew[i,j] = PositionOld[i,j]+normalvariate(0.0,1.0)*sqrt(TimeStep)+\ + QuantumForceOld[i,j]*TimeStep*D + wfnew = WaveFunction(PositionNew,alpha,beta) + QuantumForceNew = QuantumForce(PositionNew,alpha, beta) + GreensFunction = 0.0 + for j in range(Dimension): + GreensFunction += 0.5*(QuantumForceOld[i,j]+QuantumForceNew[i,j])*\ + (D*TimeStep*0.5*(QuantumForceOld[i,j]-QuantumForceNew[i,j])-\ + PositionNew[i,j]+PositionOld[i,j]) + + GreensFunction = exp(GreensFunction) + ProbabilityRatio = GreensFunction*wfnew**2/wfold**2 + #Metropolis-Hastings test to see whether we accept the move + if random() <= ProbabilityRatio: + for j in range(Dimension): + PositionOld[i,j] = PositionNew[i,j] + QuantumForceOld[i,j] = QuantumForceNew[i,j] + wfold = wfnew + DeltaE = LocalEnergy(PositionOld,alpha,beta) + energy += DeltaE + if Printout: + outfile.write('%f\n' %(energy/(MCcycle+1.0))) + # We calculate mean values + energy /= NumberMCcycles + return energy + +#Here starts the main program with variable declarations +NumberParticles = 2 +Dimension = 2 +# seed for rng generator +seed() +# Monte Carlo cycles for parameter optimization +Printout = False +NumberMCcycles= 10000 +# guess for variational parameters +x0 = np.array([0.9,0.2]) +# Using Broydens method to find optimal parameters +res = minimize(Energy, x0, method='BFGS', jac=EnergyDerivative, options={'gtol': 1e-4,'disp': True}) +x0 = res.x +# Compute the energy again with the optimal parameters and increased number of Monte Cycles +NumberMCcycles= 2**19 +Printout = True +FinalEnergy = Energy(x0) +EResult = np.array([FinalEnergy,FinalEnergy]) +outfile.close() +#nice printout with Pandas +import pandas as pd +from pandas import DataFrame +data ={'Optimal Parameters':x0, 'Final Energy':EResult} +frame = pd.DataFrame(data) +print(frame) ++
+
+
+
+
+
+
+ +
Resampling analysis
+ +The next step is then to use the above data sets and perform a +resampling analysis using the blocking method +The blocking code, based on the article of Marius Jonsson is given here +
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+ # Common imports +import os + +# Where to save the figures and data files +DATA_ID = "Results/EnergyMin" + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +infile = open(data_path("Energies.dat"),'r') + +from numpy import log2, zeros, mean, var, sum, loadtxt, arange, array, cumsum, dot, transpose, diagonal, sqrt +from numpy.linalg import inv + +def block(x): + # preliminaries + n = len(x) + d = int(log2(n)) + s, gamma = zeros(d), zeros(d) + mu = mean(x) + + # estimate the auto-covariance and variances + # for each blocking transformation + for i in arange(0,d): + n = len(x) + # estimate autocovariance of x + gamma[i] = (n)**(-1)*sum( (x[0:(n-1)]-mu)*(x[1:n]-mu) ) + # estimate variance of x + s[i] = var(x) + # perform blocking transformation + x = 0.5*(x[0::2] + x[1::2]) + + # generate the test observator M_k from the theorem + M = (cumsum( ((gamma/s)**2*2**arange(1,d+1)[::-1])[::-1] ) )[::-1] + + # we need a list of magic numbers + q =array([6.634897,9.210340, 11.344867, 13.276704, 15.086272, 16.811894, 18.475307, 20.090235, 21.665994, 23.209251, 24.724970, 26.216967, 27.688250, 29.141238, 30.577914, 31.999927, 33.408664, 34.805306, 36.190869, 37.566235, 38.932173, 40.289360, 41.638398, 42.979820, 44.314105, 45.641683, 46.962942, 48.278236, 49.587884, 50.892181]) + + # use magic to determine when we should have stopped blocking + for k in arange(0,d): + if(M[k] < q[k]): + break + if (k >= d-1): + print("Warning: Use more data") + return mu, s[k]/2**(d-k) + + +x = loadtxt(infile) +(mean, var) = block(x) +std = sqrt(var) +import pandas as pd +from pandas import DataFrame +data ={'Mean':[mean], 'STDev':[std]} +frame = pd.DataFrame(data,index=['Values']) +print(frame) ++
+
+
+
+
+
+
+ +
Content
+-
+
- Simple compiler options +
- Tools to benchmark your code +
- Machine architectures +
- What is vectorization? +
- How to measure code performance +
- Parallelization with OpenMP +
- Parallelization with MPI +
- Vectorization and parallelization, examples +
+
Optimization and profiling
+
+
+
+
++ +
Till now we have not paid much attention to speed and possible optimization possibilities +inherent in the various compilers. We have compiled and linked as +
+ + +
+
+
+
+
+
+
+
+
+
+
+ c++ -c mycode.cpp +c++ -o mycode.exe mycode.o ++
+
+
+
+
+
+
+ For Fortran replace with for example gfortran or ifort. +This is what we call a flat compiler option and should be used when we develop the code. +It produces normally a very large and slow code when translated to machine instructions. +We use this option for debugging and for establishing the correct program output because +every operation is done precisely as the user specified it. +
+ +It is instructive to look up the compiler manual for further instructions by writing
+ + +
+
+
+
+
+
+
+
+
+
+ man c++
+
+
+
+
+
+
+
+
+ +
More on optimization
+
+
+
+
++
We have additional compiler options for optimization. These may include procedure inlining where +performance may be improved, moving constants inside loops outside the loop, +identify potential parallelism, include automatic vectorization or replace a division with a reciprocal +and a multiplication if this speeds up the code. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ c++ -O3 -c mycode.cpp +c++ -O3 -o mycode.exe mycode.o ++
+
+
+
+
+
+
+ This (other options are -O2 or -Ofast) is the recommended option.
++
Optimization and profiling
+
+
+
+
+
++
It is also useful to profile your program under the development stage. +You would then compile with +
+ + +
+
+
+
+
+
+
+
+
+
+
+ c++ -pg -O3 -c mycode.cpp +c++ -pg -O3 -o mycode.exe mycode.o ++
+
+
+
+
+
+
+ After you have run the code you can obtain the profiling information via
+ + +
+
+
+
+
+
+
+
+
+
+
+ gprof mycode.exe > ProfileOutput ++
+
+
+
+
+
+
+ When you have profiled properly your code, you must take out this option as it +slows down performance. +For memory tests use valgrind. An excellent environment for all these aspects, and much more, is Qt creator. +
++
Optimization and debugging
+
+
+
+
++
Adding debugging options is a very useful alternative under the development stage of a program. +You would then compile with +
+ + +
+
+
+
+
+
+
+
+
+
+
+ c++ -g -O0 -c mycode.cpp +c++ -g -O0 -o mycode.exe mycode.o ++
+
+
+
+
+
+
+ This option generates debugging information allowing you to trace for example if an array is properly allocated. Some compilers work best with the no optimization option -O0.
+
+Other optimization flags
+
+
+
++
Depending on the compiler, one can add flags which generate code that catches integer overflow errors. +The flag -ftrapv does this for the CLANG compiler on OS X operating systems. +
++
Other hints
+
+
+
+
+
++
In general, irrespective of compiler options, it is useful to
+-
+
- avoid if tests or call to functions inside loops, if possible. +
- avoid multiplication with constants inside loops if possible +
Here is an example of a part of a program where specific operations lead to a slower code
+ + +
+
+
+
+
+
+
+
+
+
+
+ k = n-1; +for (i = 0; i < n; i++){ + a[i] = b[i] +c*d; + e = g[k]; +} ++
+
+
+
+
+
+
+ A better code is
+ + +
+
+
+
+
+
+
+
+
+
+
+ temp = c*d; +for (i = 0; i < n; i++){ + a[i] = b[i] + temp; +} +e = g[n-1]; ++
+
+
+
+
+
+
+ Here we avoid a repeated multiplication inside a loop. +Most compilers, depending on compiler flags, identify and optimize such bottlenecks on their own, without requiring any particular action by the programmer. However, it is always useful to single out and avoid code examples like the first one discussed here. +
++
Vectorization and the basic idea behind parallel computing
+
+
+
+
+
++
Present CPUs are highly parallel processors with varying levels of parallelism. The typical situation can be described via the following three statements.
+-
+
- Pursuit of shorter computation time and larger simulation size gives rise to parallel computing. +
- Multiple processors are involved to solve a global problem. +
- The essence is to divide the entire computation evenly among collaborative processors. Divide and conquer. +
Before we proceed with a more detailed discussion of topics like vectorization and parallelization, we need to remind ourselves about some basic features of different hardware models.
++
A rough classification of hardware models
+
+
+
+
++ +
-
+
- Conventional single-processor computers are named SISD (single-instruction-single-data) machines. +
- SIMD (single-instruction-multiple-data) machines incorporate the idea of parallel processing, using a large number of processing units to execute the same instruction on different data. +
- Modern parallel computers are so-called MIMD (multiple-instruction-multiple-data) machines and can execute different instruction streams in parallel on different data. +
+
Shared memory and distributed memory
+
+
+
+
+
++
One way of categorizing modern parallel computers is to look at the memory configuration.
+-
+
- In shared memory systems the CPUs share the same address space. Any CPU can access any data in the global memory. +
- In distributed memory systems each CPU has its own memory. +
The CPUs are connected by some network and may exchange messages.
++
Different parallel programming paradigms
+
+
+
+
++ +
-
+
- Task parallelism: the work of a global problem can be divided into a number of independent tasks, which rarely need to synchronize. Monte Carlo simulations represent a typical situation. Integration is another. However this paradigm is of limited use. +
- Data parallelism: use of multiple threads (e.g. one or more threads per processor) to dissect loops over arrays etc. Communication and synchronization between processors are often hidden, thus easy to program. However, the user surrenders much control to a specialized compiler. Examples of data parallelism are compiler-based parallelization and OpenMP directives. +
+
Different parallel programming paradigms
+
+
+
+
+
+
++ +
-
+
- Message passing: all involved processors have an independent memory address space. The user is responsible for partitioning the data/work of a global problem and distributing the subproblems to the processors. Collaboration between processors is achieved by explicit message passing, which is used for data transfer plus synchronization. +
- This paradigm is the most general one where the user has full control. Better parallel efficiency is usually achieved by explicit message passing. However, message-passing programming is more difficult. +
What is vectorization?
+Vectorization is a special +case of Single Instructions Multiple Data (SIMD) to denote a single +instruction stream capable of operating on multiple data elements in +parallel. +We can think of vectorization as the unrolling of loops accompanied with SIMD instructions. +
+ +Vectorization is the process of converting an algorithm that performs scalar operations +(typically one operation at the time) to vector operations where a single operation can refer to many simultaneous operations. +Consider the following example +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (i = 0; i < n; i++){ + a[i] = b[i] + c[i]; +} ++
+
+
+
+
+
+
+ If the code is not vectorized, the compiler will simply start with the first element and +then perform subsequent additions operating on one address in memory at the time. +
+ + +Number of elements that can acted upon
+A SIMD instruction can operate on multiple data elements in one single instruction. +It uses the so-called 128-bit SIMD floating-point register. +In this sense, vectorization adds some form of parallelism since one instruction is applied +to many parts of say a vector. +
+ +The number of elements which can be operated on in parallel +range from four single-precision floating point data elements in so-called +Streaming SIMD Extensions and two double-precision floating-point data +elements in Streaming SIMD Extensions 2 to sixteen byte operations in +a 128-bit register in Streaming SIMD Extensions 2. Thus, vector-length +ranges from 2 to 16, depending on the instruction extensions used and +on the data type. +
+ +IN summary, our instructions operate on 128 bit (16 byte) operands
+-
+
- 4 floats or ints +
- 2 doubles +
- Data paths 128 bits vide for vector unit +
Number of elements that can acted upon, examples
+We start with the simple scalar operations given by
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (i = 0; i < n; i++){ + a[i] = b[i] + c[i]; +} ++
+
+
+
+
+
+
+ If the code is not vectorized and we have a 128-bit register to store a 32 bits floating point number, +it means that we have \( 3\times 32 \) bits that are not used. +
+ +We have thus unused space in our SIMD registers. These registers could hold three additional integers.
+ + +Operation counts for scalar operation
+The code
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (i = 0; i < n; i++){ + a[i] = b[i] + c[i]; +} ++
+
+
+
+
+
+
+ has for \( n \) repeats
+-
+
- one load for \( c[i] \) in address 1 +
- one load for \( b[i] \) in address 2 +
- add \( c[i] \) and \( b[i] \) to give \( a[i] \) +
- store \( a[i] \) in address 2 +
Number of elements that can acted upon, examples
+If we vectorize the code, we can perform, with a 128-bit register four simultaneous operations, that is +we have +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (i = 0; i < n; i+=4){ + a[i] = b[i] + c[i]; + a[i+1] = b[i+1] + c[i+1]; + a[i+2] = b[i+2] + c[i+2]; + a[i+3] = b[i+3] + c[i+3]; +} ++
+
+
+
+
+
+
+ Four additions are now done in a single step.
+ + +Number of operations when vectorized
+For \( n/4 \) repeats assuming floats or integers
+-
+
- one vector load for \( c[i] \) in address 1 +
- one load for \( b[i] \) in address 2 +
- add \( c[i] \) and \( b[i] \) to give \( a[i] \) +
- store \( a[i] \) in address 2 +
+
A simple test case with and without vectorization
+We implement these operations in a simple c++ program that computes at the end the norm of a vector.
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+
+ #include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include "time.h"
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of square matrix
+ int n = atoi(argv[1]);
+ double s = 1.0/sqrt( (double) n);
+ double *a, *b, *c;
+ // Start timing
+ clock_t start, finish;
+ start = clock();
+// Allocate space for the vectors to be used
+ a = new double [n]; b = new double [n]; c = new double [n];
+ // Define parallel region
+ // Set up values for vectors a and b
+ for (int i = 0; i < n; i++){
+ double angle = 2.0*M_PI*i/ (( double ) n);
+ a[i] = s*(sin(angle) + cos(angle));
+ b[i] = s*sin(2.0*angle);
+ c[i] = 0.0;
+ }
+ // Then perform the vector addition
+ for (int i = 0; i < n; i++){
+ c[i] += a[i]+b[i];
+ }
+ // Compute now the norm-2
+ double Norm2 = 0.0;
+ for (int i = 0; i < n; i++){
+ Norm2 += c[i]*c[i];
+ }
+ finish = clock();
+ double timeused = (double) (finish - start)/(CLOCKS_PER_SEC );
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for norm computation=" << timeused << endl;
+ cout << " Norm-2 = " << Norm2 << endl;
+ // Free up space
+ delete[] a;
+ delete[] b;
+ delete[] c;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ Compiling with and without vectorization
+We can compile and link without vectorization using the clang c++ compiler
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang -o novec.x vecexample.cpp ++
+
+
+
+
+
+
+ and with vectorization (and additional optimizations)
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang++ -O3 -Rpass=loop-vectorize -o vec.x vecexample.cpp ++
+
+
+
+
+
+
+ The speedup depends on the size of the vectors. In the example here we have run with \( 10^7 \) elements. +The example here was run on an IMac17.1 with OSX El Capitan (10.11.4) as operating system and an Intel i5 3.3 GHz CPU. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ Compphys:~ hjensen$ ./vec.x 10000000 +Time used for norm computation=0.04720500000 +Compphys:~ hjensen$ ./novec.x 10000000 +Time used for norm computation=0.03311700000 ++
+
+
+
+
+
+
+ This particular C++ compiler speeds up the above loop operations with a factor of 1.5 +Performing the same operations for \( 10^9 \) elements results in a smaller speedup since reading from main memory is required. The non-vectorized code is seemingly faster. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ Compphys:~ hjensen$ ./vec.x 1000000000 +Time used for norm computation=58.41391100 +Compphys:~ hjensen$ ./novec.x 1000000000 +Time used for norm computation=46.51295300 ++
+
+
+
+
+
+
+ We will discuss these issues further in the next slides.
+ + +Compiling with and without vectorization using clang
+We can compile and link without vectorization with clang compiler
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang++ -o -fno-vectorize novec.x vecexample.cpp ++
+
+
+
+
+
+
+ and with vectorization
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang++ -O3 -Rpass=loop-vectorize -o vec.x vecexample.cpp ++
+
+
+
+
+
+
+ We can also add vectorization analysis, see for example
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang++ -O3 -Rpass-analysis=loop-vectorize -o vec.x vecexample.cpp ++
+
+
+
+
+
+
+ or figure out if vectorization was missed
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ clang++ -O3 -Rpass-missed=loop-vectorize -o vec.x vecexample.cpp ++
+
+
+
+
+
+
+ +
Automatic vectorization and vectorization inhibitors, criteria
+ +Not all loops can be vectorized, as discussed in Intel's guide to vectorization
+ +An important criteria is that the loop counter \( n \) is known at the entry of the loop.
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int j = 0; j < n; j++) { + a[j] = cos(j*1.0); + } ++
+
+
+
+
+
+
+ The variable \( n \) does need to be known at compile time. However, this variable must stay the same for the entire duration of the loop. It implies that an exit statement inside the loop cannot be data dependent.
+ ++
Automatic vectorization and vectorization inhibitors, exit criteria
+ +An exit statement should in general be avoided. +If the exit statement contains data-dependent conditions, the loop cannot be vectorized. +The following is an example of a non-vectorizable loop +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int j = 0; j < n; j++) { + a[j] = cos(j*1.0); + if (a[j] < 0 ) break; + } ++
+
+
+
+
+
+
+ Avoid loop termination conditions and opt for a single entry loop variable \( n \). The lower and upper bounds have to be kept fixed within the loop.
+ ++
Automatic vectorization and vectorization inhibitors, straight-line code
+ +SIMD instructions perform the same type of operations multiple times. +A switch statement leads thus to a non-vectorizable loop since different statemens cannot branch. +The following code can however be vectorized since the if statement is implemented as a masked assignment. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int j = 0; j < n; j++) { + double x = cos(j*1.0); + if (x > 0 ) { + a[j] = x*sin(j*2.0); + } + else { + a[j] = 0.0; + } + } ++
+
+
+
+
+
+
+ These operations can be performed for all data elements but only those elements which the mask evaluates as true are stored. In general, one should avoid branches such as switch, go to, or return statements or if constructs that cannot be treated as masked assignments.
+ ++
Automatic vectorization and vectorization inhibitors, nested loops
+ +Only the innermost loop of the following example is vectorized
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int i = 0; i < n; i++) { + for (int j = 0; j < n; j++) { + a[i][j] += b[i][j]; + } + } ++
+
+
+
+
+
+
+ The exception is if an original outer loop is transformed into an inner loop as the result of compiler optimizations.
+ ++
Automatic vectorization and vectorization inhibitors, function calls
+ +Calls to programmer defined functions ruin vectorization. However, calls to intrinsic functions like +\( \sin{x} \), \( \cos{x} \), \( \exp{x} \) etc are allowed since they are normally efficiently vectorized. +The following example is fully vectorizable +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int i = 0; i < n; i++) { + a[i] = log10(i)*cos(i); + } ++
+
+
+
+
+
+
+ Similarly, inline functions defined by the programmer, allow for vectorization since the function statements are glued into the actual place where the function is called.
+ ++
Automatic vectorization and vectorization inhibitors, data dependencies
+ +One has to keep in mind that vectorization changes the order of operations inside a loop. A so-called +read-after-write statement with an explicit flow dependency cannot be vectorized. The following code +
+ + +
+
+
+
+
+
+
+
+
+
+
+ double b = 15.; + for (int i = 1; i < n; i++) { + a[i] = a[i-1] + b; + } ++
+
+
+
+
+
+
+ is an example of flow dependency and results in wrong numerical results if vectorized. For a scalar operation, the value \( a[i-1] \) computed during the iteration is loaded into the right-hand side and the results are fine. In vector mode however, with a vector length of four, the values \( a[0] \), \( a[1] \), \( a[2] \) and \( a[3] \) from the previous loop will be loaded into the right-hand side and produce wrong results. That is, we have
+ + +
+
+
+
+
+
+
+
+
+
+
+ a[1] = a[0] + b; + a[2] = a[1] + b; + a[3] = a[2] + b; + a[4] = a[3] + b; ++
+
+
+
+
+
+
+ and if the two first iterations are executed at the same by the SIMD instruction, the value of say \( a[1] \) could be used by the second iteration before it has been calculated by the first iteration, leading thereby to wrong results.
+ ++
Automatic vectorization and vectorization inhibitors, more data dependencies
+ +On the other hand, a so-called +write-after-read statement can be vectorized. The following code +
+ + +
+
+
+
+
+
+
+
+
+
+
+ double b = 15.; + for (int i = 1; i < n; i++) { + a[i-1] = a[i] + b; + } ++
+
+
+
+
+
+
+ is an example of flow dependency that can be vectorized since no iteration with a higher value of \( i \) +can complete before an iteration with a lower value of \( i \). However, such code leads to problems with parallelization. +
+ ++
Automatic vectorization and vectorization inhibitors, memory stride
+ +For C++ programmers it is also worth keeping in mind that an array notation is preferred to the more compact use of pointers to access array elements. The compiler can often not tell if it is safe to vectorize the code.
+ +When dealing with arrays, you should also avoid memory stride, since this slows down considerably vectorization. When you access array element, write for example the inner loop to vectorize using unit stride, that is, access successively the next array element in memory, as shown here
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ for (int i = 0; i < n; i++) { + for (int j = 0; j < n; j++) { + a[i][j] += b[i][j]; + } + } ++
+
+
+
+
+
+
+ +
Memory management
+The main memory contains the program data
+-
+
- Cache memory contains a copy of the main memory data +
- Cache is faster but consumes more space and power. It is normally assumed to be much faster than main memory +
- Registers contain working data only +
- Modern CPUs perform most or all operations only on data in register +
- Multiple Cache memories contain a copy of the main memory data +
- Cache items accessed by their address in main memory +
- L1 cache is the fastest but has the least capacity +
- L2, L3 provide intermediate performance/size tradeoffs +
-
+
-
+
Loads and stores to memory can be as important as floating point operations when we measure performance.
+ ++
Memory and communication
+ +-
+
- Most communication in a computer is carried out in chunks, blocks of bytes of data that move together +
- In the memory hierarchy, data moves between memory and cache, and between different levels of cache, in groups called lines +
- Lines are typically 64-128 bytes, or 8-16 double precision words +
- Even if you do not use the data, it is moved and occupies space in the cache +
-
+
Many of these performance features are not captured in most programming languages.
+ ++
Measuring performance
+ +How do we measure performance? What is wrong with this code to time a loop?
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ clock_t start, finish;
+ start = clock();
+ for (int j = 0; j < i; j++) {
+ a[j] = b[j]+b[j]*c[j];
+ }
+ finish = clock();
+ double timeused = (double) (finish - start)/(CLOCKS_PER_SEC );
+
+
+
+
+
+
+
+
+ +
Problems with measuring time
+-
+
- Timers are not infinitely accurate +
- All clocks have a granularity, the minimum time that they can measure +
- The error in a time measurement, even if everything is perfect, may be the size of this granularity (sometimes called a clock tick) +
- Always know what your clock granularity is +
- Ensure that your measurement is for a long enough duration (say 100 times the tick) +
+
Problems with cold start
+ +What happens when the code is executed? The assumption is that the code is ready to +execute. But +
+-
+
- Code may still be on disk, and not even read into memory. +
- Data may be in slow memory rather than fast (which may be wrong or right for what you are measuring) +
- Multiple tests often necessary to ensure that cold start effects are not present +
- Special effort often required to ensure data in the intended part of the memory hierarchy. +
+
Problems with smart compilers
+ +-
+
- If the result of the computation is not used, the compiler may eliminate the code +
- Performance will look impossibly fantastic +
- Even worse, eliminate some of the code so the performance looks plausible +
- Ensure that the results are (or may be) used. +
+
Problems with interference
+-
+
- Other activities are sharing your processor +
- Operating system, system demons, other users +
- Some parts of the hardware do not always perform with exactly the same performance +
- Make multiple tests and report +
- Easy choices include +
- Average tests represent what users might observe over time +
-
+
-
+
+
Problems with measuring performance
+-
+
- Accurate, reproducible performance measurement is hard +
- Think carefully about your experiment: +
- What is it, precisely, that you want to measure? +
- How representative is your test to the situation that you are trying to measure? +
+
Thomas algorithm for tridiagonal linear algebra equations
+
+
+
+
+
++$$ +\left( \begin{array}{ccccc} + b_0 & c_0 & & & \\ + a_0 & b_1 & c_1 & & \\ + & & \ddots & & \\ + & & a_{m-3} & b_{m-2} & c_{m-2} \\ + & & & a_{m-2} & b_{m-1} + \end{array} \right) +\left( \begin{array}{c} + x_0 \\ + x_1 \\ + \vdots \\ + x_{m-2} \\ + x_{m-1} + \end{array} \right)=\left( \begin{array}{c} + f_0 \\ + f_1 \\ + \vdots \\ + f_{m-2} \\ + f_{m-1} \\ + \end{array} \right) +$$ +
+
Thomas algorithm, forward substitution
+
+
+
+
+
++
The first step is to multiply the first row by \( a_0/b_0 \) and subtract it from the second row. This is known as the forward substitution step. We obtain then
+$$ + a_i = 0, +$$ + + +$$ + b_i = b_i - \frac{a_{i-1}}{b_{i-1}}c_{i-1}, +$$ + +and
+$$ + f_i = f_i - \frac{a_{i-1}}{b_{i-1}}f_{i-1}. +$$ + +At this point the simplified equation, with only an upper triangular matrix takes the form
+$$ +\left( \begin{array}{ccccc} + b_0 & c_0 & & & \\ + & b_1 & c_1 & & \\ + & & \ddots & & \\ + & & & b_{m-2} & c_{m-2} \\ + & & & & b_{m-1} + \end{array} \right)\left( \begin{array}{c} + x_0 \\ + x_1 \\ + \vdots \\ + x_{m-2} \\ + x_{m-1} + \end{array} \right)=\left( \begin{array}{c} + f_0 \\ + f_1 \\ + \vdots \\ + f_{m-2} \\ + f_{m-1} \\ + \end{array} \right) +$$ ++
Thomas algorithm, backward substitution
+
+
+
+
+
++
The next step is the backward substitution step. The last row is multiplied by \( c_{N-3}/b_{N-2} \) and subtracted from the second to last row, thus eliminating \( c_{N-3} \) from the last row. The general backward substitution procedure is
+$$ + c_i = 0, +$$ + +and
+$$ + f_{i-1} = f_{i-1} - \frac{c_{i-1}}{b_i}f_i +$$ + +All that ramains to be computed is the solution, which is the very straight forward process of
+$$ +x_i = \frac{f_i}{b_i} +$$ ++
Thomas algorithm and counting of operations (floating point and memory)
+
+
+
+
+
++ +
We have in specific case the following operations with the floating operations
+ +-
+
- Memory Reads: \( 14(N-2) \); +
- Memory Writes: \( 4(N-2) \); +
- Subtractions: \( 3(N-2) \); +
- Multiplications: \( 3(N-2) \); +
- Divisions: \( 4(N-2) \). +
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ // Forward substitution +// Note that we can simplify by precalculating a[i-1]/b[i-1] + for (int i=1; i < n; i++) { + b[i] = b[i] - (a[i-1]*c[i-1])/b[i-1]; + f[i] = g[i] - (a[i-1]*f[i-1])/b[i-1]; + } + x[n-1] = f[n-1] / b[n-1]; + // Backwards substitution + for (int i = n-2; i >= 0; i--) { + f[i] = f[i] - c[i]*f[i+1]/b[i+1]; + x[i] = f[i]/b[i]; + } ++
+
+
+
+
+
+
+ +
Example: Transpose of a matrix
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+ #include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include "time.h"
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of square matrix
+ int n = atoi(argv[1]);
+ double **A, **B;
+ // Allocate space for the two matrices
+ A = new double*[n]; B = new double*[n];
+ for (int i = 0; i < n; i++){
+ A[i] = new double[n];
+ B[i] = new double[n];
+ }
+ // Set up values for matrix A
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ A[i][j] = cos(i*1.0)*sin(j*3.0);
+ }
+ }
+ clock_t start, finish;
+ start = clock();
+ // Then compute the transpose
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ B[i][j]= A[j][i];
+ }
+ }
+
+ finish = clock();
+ double timeused = (double) (finish - start)/(CLOCKS_PER_SEC );
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for setting up transpose of matrix=" << timeused << endl;
+
+ // Free up space
+ for (int i = 0; i < n; i++){
+ delete[] A[i];
+ delete[] B[i];
+ }
+ delete[] A;
+ delete[] B;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ +
Matrix-matrix multiplication
+This the matrix-matrix multiplication code with plain c++ memory allocation. It computes at the end the Frobenius norm.
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+ #include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include "time.h"
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of square matrix
+ int n = atoi(argv[1]);
+ double s = 1.0/sqrt( (double) n);
+ double **A, **B, **C;
+ // Start timing
+ clock_t start, finish;
+ start = clock();
+ // Allocate space for the two matrices
+ A = new double*[n]; B = new double*[n]; C = new double*[n];
+ for (int i = 0; i < n; i++){
+ A[i] = new double[n];
+ B[i] = new double[n];
+ C[i] = new double[n];
+ }
+ // Set up values for matrix A and B and zero matrix C
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ double angle = 2.0*M_PI*i*j/ (( double ) n);
+ A[i][j] = s * ( sin ( angle ) + cos ( angle ) );
+ B[j][i] = A[i][j];
+ }
+ }
+ // Then perform the matrix-matrix multiplication
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ double sum = 0.0;
+ for (int k = 0; k < n; k++) {
+ sum += B[i][k]*A[k][j];
+ }
+ C[i][j] = sum;
+ }
+ }
+ // Compute now the Frobenius norm
+ double Fsum = 0.0;
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ Fsum += C[i][j]*C[i][j];
+ }
+ }
+ Fsum = sqrt(Fsum);
+ finish = clock();
+ double timeused = (double) (finish - start)/(CLOCKS_PER_SEC );
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for matrix-matrix multiplication=" << timeused << endl;
+ cout << " Frobenius norm = " << Fsum << endl;
+ // Free up space
+ for (int i = 0; i < n; i++){
+ delete[] A[i];
+ delete[] B[i];
+ delete[] C[i];
+ }
+ delete[] A;
+ delete[] B;
+ delete[] C;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ +
How do we define speedup? Simplest form
+
+
+
+
+
++
-
+
- Speedup measures the ratio of performance between two objects +
- Versions of same code, with different number of processors +
- Serial and vector versions +
- Try different programing languages, c++ and Fortran +
- Two algorithms computing the same result +
+
How do we define speedup? Correct baseline
+
+
+
+
+
++
The key is choosing the correct baseline for comparison
+-
+
- For our serial vs. vectorization examples, using compiler-provided vectorization, the baseline is simple; the same code, with vectorization turned off +
- For parallel applications, this is much harder: +
- Choice of algorithm, decomposition, performance of baseline case etc. +
-
+
-
+
+
Parallel speedup
+
+
+
+
+
++
For parallel applications, speedup is typically defined as
+-
+
- Speedup \( =T_1/T_p \) +
Here \( T_1 \) is the time on one processor and \( T_p \) is the time using \( p \) processors.
+-
+
- Can the speedup become larger than \( p \)? That means using \( p \) processors is more than \( p \) times faster than using one processor. +
+
Speedup and memory
+
+
+
+
+
++
The speedup on \( p \) processors can +be greater than \( p \) if memory usage is optimal! +Consider the case of a memorybound computation with \( M \) words of memory +
+-
+
- If \( M/p \) fits into cache while \( M \) does not, the time to access memory will be different in the two cases: +
- \( T_1 \) uses the main memory bandwidth +
- \( T_p \) uses the appropriate cache bandwidth +
+
Upper bounds on speedup
+
+
+
+
+
++
Assume that almost all parts of a code are perfectly +parallelizable (fraction \( f \)). The remainder, +fraction \( (1-f) \) cannot be parallelized at all. +
+ +That is, there is work that takes time \( W \) on one process; a fraction \( f \) of that work will take +time \( Wf/p \) on \( p \) processors. +
+-
+
- What is the maximum possible speedup as a function of \( f \)? +
+
Amdahl's law
+
+
+
+
+
++
On one processor we have
+$$ +T_1 = (1-f)W + fW = W +$$ + +On \( p \) processors we have
+$$ +T_p = (1-f)W + \frac{fW}{p}, +$$ + +resulting in a speedup of
+$$ +\frac{T_1}{T_p} = \frac{W}{(1-f)W+fW/p} +$$ + +As \( p \) goes to infinity, \( fW/p \) goes to zero, and the maximum speedup is
+$$ +\frac{1}{1-f}, +$$ + +meaning that if +if \( f = 0.99 \) (all but \( 1\% \) parallelizable), the maximum speedup +is \( 1/(1-.99)=100 \)! +
++
How much is parallelizable
+
+
+
+
+
++
If any non-parallel code slips into the +application, the parallel +performance is limited. +
+ +In many simulations, however, the fraction of non-parallelizable work +is \( 10^{-6} \) or less due to large arrays or objects that are perfectly parallelizable. +
++
Today's situation of parallel computing
+
+
+
+
+
++ +
-
+
- Distributed memory is the dominant hardware configuration. There is a large diversity in these machines, from MPP (massively parallel processing) systems to clusters of off-the-shelf PCs, which are very cost-effective. +
- Message-passing is a mature programming paradigm and widely accepted. It often provides an efficient match to the hardware. It is primarily used for the distributed memory systems, but can also be used on shared memory systems. +
- Modern nodes have nowadays several cores, which makes it interesting to use both shared memory (the given node) and distributed memory (several nodes with communication). This leads often to codes which use both MPI and OpenMP. +
Our lectures will focus on both MPI and OpenMP.
++
Overhead present in parallel computing
+
+
+
+
+
++ +
-
+
- Uneven load balance: not all the processors can perform useful work at all time. +
- Overhead of synchronization +
- Overhead of communication +
- Extra computation due to parallelization +
Due to the above overhead and that certain parts of a sequential +algorithm cannot be parallelized we may not achieve an optimal parallelization. +
++
Parallelizing a sequential algorithm
+
+
+
+
+
++ +
-
+
- Identify the part(s) of a sequential algorithm that can be executed in parallel. This is the difficult part, +
- Distribute the global work and data among \( P \) processors. +
+
Strategies
+
+
+
+
+
++
-
+
- Develop codes locally, run with some few processes and test your codes. Do benchmarking, timing and so forth on local nodes, for example your laptop or PC. +
- When you are convinced that your codes run correctly, you can start your production runs on available supercomputers. +
+
How do I run MPI on a PC/Laptop? MPI
+
+
+
+
+
++
To install MPI is rather easy on hardware running unix/linux as operating systems, follow simply the instructions from the OpenMPI website. See also subsequent slides. +When you have made sure you have installed MPI on your PC/laptop, +
+-
+
- Compile with mpicxx/mpic++ or mpif90 +
+
+
+
+
+
+
+
+
+
+ # Compile and link + mpic++ -O3 -o nameofprog.x nameofprog.cpp + # run code with for example 8 processes using mpirun/mpiexec + mpiexec -n 8 ./nameofprog.x ++
+
+
+
+
+
+
+ +
Can I do it on my own PC/laptop? OpenMP installation
+
+
+
+
+
++
If you wish to install MPI and OpenMP +on your laptop/PC, we recommend the following: +
+ +-
+
- For OpenMP, the compile option -fopenmp is included automatically in recent versions of the C++ compiler and Fortran compilers. For users of different Linux distributions, simply use the available C++ or Fortran compilers and add the above compiler instructions, see also code examples below. +
- For OS X users however, install libomp +
+
+
+
+
+
+
+
+
+
+
+ brew install libomp ++
+
+
+
+
+
+
+ and compile and link as
+ + +
+
+
+
+
+
+
+
+
+
+ c++ -o <name executable> <name program.cpp> -lomp ++
+
+
+
+
+
+
+ +
Installing MPI
+
+
+
+
+
++
For linux/ubuntu users, you need to install two packages (alternatively use the synaptic package manager)
+ + +
+
+
+
+
+
+
+
+
+
+
+ sudo apt-get install libopenmpi-dev + sudo apt-get install openmpi-bin ++
+
+
+
+
+
+
+ For OS X users, install brew (after having installed xcode and gcc, needed for the +gfortran compiler of openmpi) and then install with brew +
+ + +
+
+
+
+
+
+
+
+
+
+
+ brew install openmpi ++
+
+
+
+
+
+
+ When running an executable (code.x), run as
+ + +
+
+
+
+
+
+
+
+
+
+
+ mpirun -n 10 ./code.x ++
+
+
+
+
+
+
+ where we indicate that we want the number of processes to be 10.
++
Installing MPI and using Qt
+
+
+
+
+
++
With openmpi installed, when using Qt, add to your .pro file the instructions here
+ +You may need to tell Qt where openmpi is stored.
++
What is Message Passing Interface (MPI)?
+
+
+
+
++ +
MPI is a library, not a language. It specifies the names, calling sequences and results of functions +or subroutines to be called from C/C++ or Fortran programs, and the classes and methods that make up the MPI C++ +library. The programs that users write in Fortran, C or C++ are compiled with ordinary compilers and linked +with the MPI library. +
+ +MPI programs should be able to run +on all possible machines and run all MPI implementetations without change. +
+ +An MPI computation is a collection of processes communicating with messages.
++
Going Parallel with MPI
+
+
+
+
+
++
Task parallelism: the work of a global problem can be divided +into a number of independent tasks, which rarely need to synchronize. +Monte Carlo simulations or numerical integration are examples of this. +
+ +MPI is a message-passing library where all the routines +have corresponding C/C++-binding +
+ + +
+
+
+
+
+
+
+
+
+
+
+ MPI_Command_name
+
+
+
+
+
+
+
+
+ and Fortran-binding (routine names are in uppercase, but can also be in lower case)
+ + +
+
+
+
+
+
+
+
+
+
+ MPI_COMMAND_NAME ++
+
+
+
+
+
+
+ +
MPI is a library
+
+
+
+
+
++
MPI is a library specification for the message passing interface, +proposed as a standard. +
+ +-
+
- independent of hardware; +
- not a language or compiler specification; +
- not a specific implementation or product. +
A message passing standard for portability and ease-of-use. +Designed for high performance. +
+ +Insert communication and synchronization functions where necessary.
++
Bindings to MPI routines
+
+
+
+
+
++ +
MPI is a message-passing library where all the routines +have corresponding C/C++-binding +
+ + +
+
+
+
+
+
+
+
+
+
+
+ MPI_Command_name
+
+
+
+
+
+
+
+
+ and Fortran-binding (routine names are in uppercase, but can also be in lower case)
+ + +
+
+
+
+
+
+
+
+
+
+
+ MPI_COMMAND_NAME ++
+
+
+
+
+
+
+ The discussion in these slides focuses on the C++ binding.
++
Communicator
+
+
+
+
+
++
-
+
- A group of MPI processes with a name (context). +
- Any process is identified by its rank. The rank is only meaningful within a particular communicator. +
- By default the communicator contains all the MPI processes. +
+
+
+
+
+
+
+
+
+
+
+ MPI_COMM_WORLD ++
+
+
+
+
+
+
+ -
+
- Mechanism to identify subset of processes. +
- Promotes modular design of parallel libraries. +
+
Some of the most important MPI functions
+
+
+
+
+
++ +
-
+
- \( MPI\_Init \) - initiate an MPI computation +
- \( MPI\_Finalize \) - terminate the MPI computation and clean up +
- \( MPI\_Comm\_size \) - how many processes participate in a given MPI communicator? +
- \( MPI\_Comm\_rank \) - which one am I? (A number between 0 and size-1.) +
- \( MPI\_Send \) - send a message to a particular process within an MPI communicator +
- \( MPI\_Recv \) - receive a message from a particular process within an MPI communicator +
- \( MPI\_reduce \) or \( MPI\_Allreduce \), send and receive messages +
+
The first MPI C/C++ program
+
+
+
+
+
++ +
Let every process write "Hello world" (oh not this program again!!) on the standard output.
+ + +
+
+
+
+
+
+
+
+
+
+ using namespace std; +#include <mpi.h> +#include <iostream> +int main (int nargs, char* args[]) +{ +int numprocs, my_rank; +// MPI initializations +MPI_Init (&nargs, &args); +MPI_Comm_size (MPI_COMM_WORLD, &numprocs); +MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); +cout << "Hello world, I have rank " << my_rank << " out of " + << numprocs << endl; +// End MPI +MPI_Finalize (); ++
+
+
+
+
+
+
+ +
The Fortran program
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ PROGRAM hello +INCLUDE "mpif.h" +INTEGER:: size, my_rank, ierr + +CALL MPI_INIT(ierr) +CALL MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierr) +CALL MPI_COMM_RANK(MPI_COMM_WORLD, my_rank, ierr) +WRITE(*,*)"Hello world, I've rank ",my_rank," out of ",size +CALL MPI_FINALIZE(ierr) + +END PROGRAM hello ++
+
+
+
+
+
+
+ +
Note 1
+
+
+
+
+
++ +
-
+
- The output to screen is not ordered since all processes are trying to write to screen simultaneously. +
- It is the operating system which opts for an ordering. +
- If we wish to have an organized output, starting from the first process, we may rewrite our program as in the next example. +
+
Ordered output with MPIBarrier
+
+
+
+
+
++ + + +
+
+
+
+
+
+
+
+
+
+ int main (int nargs, char* args[]) +{ + int numprocs, my_rank, i; + MPI_Init (&nargs, &args); + MPI_Comm_size (MPI_COMM_WORLD, &numprocs); + MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); + for (i = 0; i < numprocs; i++) {} + MPI_Barrier (MPI_COMM_WORLD); + if (i == my_rank) { + cout << "Hello world, I have rank " << my_rank << + " out of " << numprocs << endl;} + MPI_Finalize (); ++
+
+
+
+
+
+
+ +
Note 2
+
+
+
+
+
++
-
+
- Here we have used the \( MPI\_Barrier \) function to ensure that that every process has completed its set of instructions in a particular order. +
- A barrier is a special collective operation that does not allow the processes to continue until all processes in the communicator (here \( MPI\_COMM\_WORLD \)) have called \( MPI\_Barrier \). +
- The barriers make sure that all processes have reached the same point in the code. Many of the collective operations like \( MPI\_ALLREDUCE \) to be discussed later, have the same property; that is, no process can exit the operation until all processes have started. +
However, this is slightly more time-consuming since the processes synchronize between themselves as many times as there +are processes. In the next Hello world example we use the send and receive functions in order to a have a synchronized +action. +
++
Ordered output
+
+
+
+
+
++ + + +
+
+
+
+
+
+
+
+
+
+ ..... +int numprocs, my_rank, flag; +MPI_Status status; +MPI_Init (&nargs, &args); +MPI_Comm_size (MPI_COMM_WORLD, &numprocs); +MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); +if (my_rank > 0) +MPI_Recv (&flag, 1, MPI_INT, my_rank-1, 100, + MPI_COMM_WORLD, &status); +cout << "Hello world, I have rank " << my_rank << " out of " +<< numprocs << endl; +if (my_rank < numprocs-1) +MPI_Send (&my_rank, 1, MPI_INT, my_rank+1, + 100, MPI_COMM_WORLD); +MPI_Finalize (); ++
+
+
+
+
+
+
+ +
Note 3
+
+
+
+
+
++ +
The basic sending of messages is given by the function \( MPI\_SEND \), which in C/C++ +is defined as +
+ + +
+
+
+
+
+
+
+
+
+
+
+ int MPI_Send(void *buf, int count, + MPI_Datatype datatype, + int dest, int tag, MPI_Comm comm)} ++
+
+
+
+
+
+
+ This single command allows the passing of any kind of variable, even a large array, to any group of tasks. +The variable buf is the variable we wish to send while count +is the number of variables we are passing. If we are passing only a single value, this should be 1. +
+ +If we transfer an array, it is the overall size of the array. +For example, if we want to send a 10 by 10 array, count would be \( 10\times 10=100 \) +since we are actually passing 100 values. +
++
Note 4
+
+
+
+
+
++ +
Once you have sent a message, you must receive it on another task. The function \( MPI\_RECV \) +is similar to the send call. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ int MPI_Recv( void *buf, int count, MPI_Datatype datatype, + int source, + int tag, MPI_Comm comm, MPI_Status *status ) ++
+
+
+
+
+
+
+ The arguments that are different from those in MPI\_SEND are +buf which is the name of the variable where you will be storing the received data, +source which replaces the destination in the send command. This is the return ID of the sender. +
+ +Finally, we have used \( MPI\_Status\_status \), +where one can check if the receive was completed. +
+ +The output of this code is the same as the previous example, but now +process 0 sends a message to process 1, which forwards it further +to process 2, and so forth. +
++
Numerical integration in parallel
+
+Integrating \( \pi \)
+
+
+
++ +
-
+
- The code example computes \( \pi \) using the trapezoidal rules. +
- The trapezoidal rule +
Click on this link for the full program.
++
Dissection of trapezoidal rule with \( MPI\_reduce \)
+
+
+
+
+
++ + + +
+
+
+
+
+
+
+
+
+
+ // Trapezoidal rule and numerical integration usign MPI +using namespace std; +#include <mpi.h> +#include <iostream> + +// Here we define various functions called by the main program + +double int_function(double ); +double trapezoidal_rule(double , double , int , double (*)(double)); + +// Main function begins here +int main (int nargs, char* args[]) +{ + int n, local_n, numprocs, my_rank; + double a, b, h, local_a, local_b, total_sum, local_sum; + double time_start, time_end, total_time; ++
+
+
+
+
+
+
+ +
Dissection of trapezoidal rule
+
+
+
+
+
++ + + +
+
+
+
+
+
+
+
+
+
+ // MPI initializations + MPI_Init (&nargs, &args); + MPI_Comm_size (MPI_COMM_WORLD, &numprocs); + MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); + time_start = MPI_Wtime(); + // Fixed values for a, b and n + a = 0.0 ; b = 1.0; n = 1000; + h = (b-a)/n; // h is the same for all processes + local_n = n/numprocs; + // make sure n > numprocs, else integer division gives zero + // Length of each process' interval of + // integration = local_n*h. + local_a = a + my_rank*local_n*h; + local_b = local_a + local_n*h; ++
+
+
+
+
+
+
+ +
Integrating with MPI
+
+
+
+
+
++ + + +
+
+
+
+
+
+
+
+
+
+ total_sum = 0.0; + local_sum = trapezoidal_rule(local_a, local_b, local_n, + &int_function); + MPI_Reduce(&local_sum, &total_sum, 1, MPI_DOUBLE, + MPI_SUM, 0, MPI_COMM_WORLD); + time_end = MPI_Wtime(); + total_time = time_end-time_start; + if ( my_rank == 0) { + cout << "Trapezoidal rule = " << total_sum << endl; + cout << "Time = " << total_time + << " on number of processors: " << numprocs << endl; + } + // End MPI + MPI_Finalize (); + return 0; +} // end of main program ++
+
+
+
+
+
+
+ +
How do I use \( MPI\_reduce \)?
+
+
+
+
+
++ +
Here we have used
+ + +
+
+
+
+
+
+
+
+
+
+
+ MPI_reduce( void *senddata, void* resultdata, int count, + MPI_Datatype datatype, MPI_Op, int root, MPI_Comm comm) ++
+
+
+
+
+
+
+ The two variables \( senddata \) and \( resultdata \) are obvious, besides the fact that one sends the address +of the variable or the first element of an array. If they are arrays they need to have the same size. +The variable \( count \) represents the total dimensionality, 1 in case of just one variable, +while \( MPI\_Datatype \) +defines the type of variable which is sent and received. +
+ +The new feature is \( MPI\_Op \). It defines the type +of operation we want to do. +
++
More on \( MPI\_Reduce \)
+
+
+
+
+
++
In our case, since we are summing +the rectangle contributions from every process we define \( MPI\_Op = MPI\_SUM \). +If we have an array or matrix we can search for the largest og smallest element by sending either \( MPI\_MAX \) or +\( MPI\_MIN \). If we want the location as well (which array element) we simply transfer +\( MPI\_MAXLOC \) or \( MPI\_MINOC \). If we want the product we write \( MPI\_PROD \). +
+ +\( MPI\_Allreduce \) is defined as
+ + +
+
+
+
+
+
+
+
+
+
+ MPI_Allreduce( void *senddata, void* resultdata, int count, + MPI_Datatype datatype, MPI_Op, MPI_Comm comm) ++
+
+
+
+
+
+
+ +
Dissection of trapezoidal rule
+
+
+
+
+
++ +
We use \( MPI\_reduce \) to collect data from each process. Note also the use of the function +\( MPI\_Wtime \). +
+ + +
+
+
+
+
+
+
+
+
+
+ // this function defines the function to integrate +double int_function(double x) +{ + double value = 4./(1.+x*x); + return value; +} // end of function to evaluate ++
+
+
+
+
+
+
+ +
Dissection of trapezoidal rule
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ // this function defines the trapezoidal rule +double trapezoidal_rule(double a, double b, int n, + double (*func)(double)) +{ + double trapez_sum; + double fa, fb, x, step; + int j; + step=(b-a)/((double) n); + fa=(*func)(a)/2. ; + fb=(*func)(b)/2. ; + trapez_sum=0.; + for (j=1; j <= n-1; j++){ + x=j*step+a; + trapez_sum+=(*func)(x); + } + trapez_sum=(trapez_sum+fb+fa)*step; + return trapez_sum; +} // end trapezoidal_rule ++
+
+
+
+
+
+
+ +
The quantum dot program for two electrons
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ // Variational Monte Carlo for atoms with importance sampling, slater det +// Test case for 2-electron quantum dot, no classes using Mersenne-Twister RNG +#include "mpi.h" +#include <cmath> +#include <random> +#include <string> +#include <iostream> +#include <fstream> +#include <iomanip> +#include "vectormatrixclass.h" + +using namespace std; +// output file as global variable +ofstream ofile; +// the step length and its squared inverse for the second derivative +// Here we define global variables used in various functions +// These can be changed by using classes +int Dimension = 2; +int NumberParticles = 2; // we fix also the number of electrons to be 2 + +// declaration of functions + +// The Mc sampling for the variational Monte Carlo +void MonteCarloSampling(int, double &, double &, Vector &); + +// The variational wave function +double WaveFunction(Matrix &, Vector &); + +// The local energy +double LocalEnergy(Matrix &, Vector &); + +// The quantum force +void QuantumForce(Matrix &, Matrix &, Vector &); + + +// inline function for single-particle wave function +inline double SPwavefunction(double r, double alpha) { + return exp(-alpha*r*0.5); +} + +// inline function for derivative of single-particle wave function +inline double DerivativeSPwavefunction(double r, double alpha) { + return -r*alpha; +} + +// function for absolute value of relative distance +double RelativeDistance(Matrix &r, int i, int j) { + double r_ij = 0; + for (int k = 0; k < Dimension; k++) { + r_ij += (r(i,k)-r(j,k))*(r(i,k)-r(j,k)); + } + return sqrt(r_ij); +} + +// inline function for derivative of Jastrow factor +inline double JastrowDerivative(Matrix &r, double beta, int i, int j, int k){ + return (r(i,k)-r(j,k))/(RelativeDistance(r, i, j)*pow(1.0+beta*RelativeDistance(r, i, j),2)); +} + +// function for square of position of single particle +double singleparticle_pos2(Matrix &r, int i) { + double r_single_particle = 0; + for (int j = 0; j < Dimension; j++) { + r_single_particle += r(i,j)*r(i,j); + } + return r_single_particle; +} + +void lnsrch(int n, Vector &xold, double fold, Vector &g, Vector &p, Vector &x, + double *f, double stpmax, int *check, double (*func)(Vector &p)); + +void dfpmin(Vector &p, int n, double gtol, int *iter, double *fret, + double(*func)(Vector &p), void (*dfunc)(Vector &p, Vector &g)); + +static double sqrarg; +#define SQR(a) ((sqrarg=(a)) == 0.0 ? 0.0 : sqrarg*sqrarg) + + +static double maxarg1,maxarg2; +#define FMAX(a,b) (maxarg1=(a),maxarg2=(b),(maxarg1) > (maxarg2) ?\ + (maxarg1) : (maxarg2)) + + +// Begin of main program + +int main(int argc, char* argv[]) +{ + + // MPI initializations + int NumberProcesses, MyRank, NumberMCsamples; + MPI_Init (&argc, &argv); + MPI_Comm_size (MPI_COMM_WORLD, &NumberProcesses); + MPI_Comm_rank (MPI_COMM_WORLD, &MyRank); + double StartTime = MPI_Wtime(); + if (MyRank == 0 && argc <= 1) { + cout << "Bad Usage: " << argv[0] << + " Read also output file on same line and number of Monte Carlo cycles" << endl; + } + // Read filename and number of Monte Carlo cycles from the command line + if (MyRank == 0 && argc > 2) { + string filename = argv[1]; // first command line argument after name of program + NumberMCsamples = atoi(argv[2]); + string fileout = filename; + string argument = to_string(NumberMCsamples); + // Final filename as filename+NumberMCsamples + fileout.append(argument); + ofile.open(fileout); + } + // broadcast the number of Monte Carlo samples + MPI_Bcast (&NumberMCsamples, 1, MPI_INT, 0, MPI_COMM_WORLD); + // Two variational parameters only + Vector VariationalParameters(2); + int TotalNumberMCsamples = NumberMCsamples*NumberProcesses; + // Loop over variational parameters + for (double alpha = 0.5; alpha <= 1.5; alpha +=0.1){ + for (double beta = 0.1; beta <= 0.5; beta +=0.05){ + VariationalParameters(0) = alpha; // value of alpha + VariationalParameters(1) = beta; // value of beta + // Do the mc sampling and accumulate data with MPI_Reduce + double TotalEnergy, TotalEnergySquared, LocalProcessEnergy, LocalProcessEnergy2; + LocalProcessEnergy = LocalProcessEnergy2 = 0.0; + MonteCarloSampling(NumberMCsamples, LocalProcessEnergy, LocalProcessEnergy2, VariationalParameters); + // Collect data in total averages + MPI_Reduce(&LocalProcessEnergy, &TotalEnergy, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); + MPI_Reduce(&LocalProcessEnergy2, &TotalEnergySquared, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); + // Print out results in case of Master node, set to MyRank = 0 + if ( MyRank == 0) { + double Energy = TotalEnergy/( (double)NumberProcesses); + double Variance = TotalEnergySquared/( (double)NumberProcesses)-Energy*Energy; + double StandardDeviation = sqrt(Variance/((double)TotalNumberMCsamples)); // over optimistic error + ofile << setiosflags(ios::showpoint | ios::uppercase); + ofile << setw(15) << setprecision(8) << VariationalParameters(0); + ofile << setw(15) << setprecision(8) << VariationalParameters(1); + ofile << setw(15) << setprecision(8) << Energy; + ofile << setw(15) << setprecision(8) << Variance; + ofile << setw(15) << setprecision(8) << StandardDeviation << endl; + } + } + } + double EndTime = MPI_Wtime(); + double TotalTime = EndTime-StartTime; + if ( MyRank == 0 ) cout << "Time = " << TotalTime << " on number of processors: " << NumberProcesses << endl; + if (MyRank == 0) ofile.close(); // close output file + // End MPI + MPI_Finalize (); + return 0; +} // end of main function + + +// Monte Carlo sampling with the Metropolis algorithm + +void MonteCarloSampling(int NumberMCsamples, double &cumulative_e, double &cumulative_e2, Vector &VariationalParameters) +{ + + // Initialize the seed and call the Mersienne algo + std::random_device rd; + std::mt19937_64 gen(rd()); + // Set up the uniform distribution for x \in [[0, 1] + std::uniform_real_distribution<double> UniformNumberGenerator(0.0,1.0); + std::normal_distribution<double> Normaldistribution(0.0,1.0); + // diffusion constant from Schroedinger equation + double D = 0.5; + double timestep = 0.05; // we fix the time step for the gaussian deviate + // allocate matrices which contain the position of the particles + Matrix OldPosition( NumberParticles, Dimension), NewPosition( NumberParticles, Dimension); + Matrix OldQuantumForce(NumberParticles, Dimension), NewQuantumForce(NumberParticles, Dimension); + double Energy = 0.0; double EnergySquared = 0.0; double DeltaE = 0.0; + // initial trial positions + for (int i = 0; i < NumberParticles; i++) { + for (int j = 0; j < Dimension; j++) { + OldPosition(i,j) = Normaldistribution(gen)*sqrt(timestep); + } + } + double OldWaveFunction = WaveFunction(OldPosition, VariationalParameters); + QuantumForce(OldPosition, OldQuantumForce, VariationalParameters); + // loop over monte carlo cycles + for (int cycles = 1; cycles <= NumberMCsamples; cycles++){ + // new position + for (int i = 0; i < NumberParticles; i++) { + for (int j = 0; j < Dimension; j++) { + // gaussian deviate to compute new positions using a given timestep + NewPosition(i,j) = OldPosition(i,j) + Normaldistribution(gen)*sqrt(timestep)+OldQuantumForce(i,j)*timestep*D; + // NewPosition(i,j) = OldPosition(i,j) + gaussian_deviate(&idum)*sqrt(timestep)+OldQuantumForce(i,j)*timestep*D; + } + // for the other particles we need to set the position to the old position since + // we move only one particle at the time + for (int k = 0; k < NumberParticles; k++) { + if ( k != i) { + for (int j = 0; j < Dimension; j++) { + NewPosition(k,j) = OldPosition(k,j); + } + } + } + double NewWaveFunction = WaveFunction(NewPosition, VariationalParameters); + QuantumForce(NewPosition, NewQuantumForce, VariationalParameters); + // we compute the log of the ratio of the greens functions to be used in the + // Metropolis-Hastings algorithm + double GreensFunction = 0.0; + for (int j = 0; j < Dimension; j++) { + GreensFunction += 0.5*(OldQuantumForce(i,j)+NewQuantumForce(i,j))* + (D*timestep*0.5*(OldQuantumForce(i,j)-NewQuantumForce(i,j))-NewPosition(i,j)+OldPosition(i,j)); + } + GreensFunction = exp(GreensFunction); + // The Metropolis test is performed by moving one particle at the time + if(UniformNumberGenerator(gen) <= GreensFunction*NewWaveFunction*NewWaveFunction/OldWaveFunction/OldWaveFunction ) { + for (int j = 0; j < Dimension; j++) { + OldPosition(i,j) = NewPosition(i,j); + OldQuantumForce(i,j) = NewQuantumForce(i,j); + } + OldWaveFunction = NewWaveFunction; + } + } // end of loop over particles + // compute local energy + double DeltaE = LocalEnergy(OldPosition, VariationalParameters); + // update energies + Energy += DeltaE; + EnergySquared += DeltaE*DeltaE; + } // end of loop over MC trials + // update the energy average and its squared + cumulative_e = Energy/NumberMCsamples; + cumulative_e2 = EnergySquared/NumberMCsamples; +} // end MonteCarloSampling function + + +// Function to compute the squared wave function and the quantum force + +double WaveFunction(Matrix &r, Vector &VariationalParameters) +{ + double wf = 0.0; + // full Slater determinant for two particles, replace with Slater det for more particles + wf = SPwavefunction(singleparticle_pos2(r, 0), VariationalParameters(0))*SPwavefunction(singleparticle_pos2(r, 1),VariationalParameters(0)); + // contribution from Jastrow factor + for (int i = 0; i < NumberParticles-1; i++) { + for (int j = i+1; j < NumberParticles; j++) { + wf *= exp(RelativeDistance(r, i, j)/((1.0+VariationalParameters(1)*RelativeDistance(r, i, j)))); + } + } + return wf; +} + +// Function to calculate the local energy without numerical derivation of kinetic energy + +double LocalEnergy(Matrix &r, Vector &VariationalParameters) +{ + + // compute the kinetic and potential energy from the single-particle part + // for a many-electron system this has to be replaced by a Slater determinant + // The absolute value of the interparticle length + Matrix length( NumberParticles, NumberParticles); + // Set up interparticle distance + for (int i = 0; i < NumberParticles-1; i++) { + for(int j = i+1; j < NumberParticles; j++){ + length(i,j) = RelativeDistance(r, i, j); + length(j,i) = length(i,j); + } + } + double KineticEnergy = 0.0; + // Set up kinetic energy from Slater and Jastrow terms + for (int i = 0; i < NumberParticles; i++) { + for (int k = 0; k < Dimension; k++) { + double sum1 = 0.0; + for(int j = 0; j < NumberParticles; j++){ + if ( j != i) { + sum1 += JastrowDerivative(r, VariationalParameters(1), i, j, k); + } + } + KineticEnergy += (sum1+DerivativeSPwavefunction(r(i,k),VariationalParameters(0)))*(sum1+DerivativeSPwavefunction(r(i,k),VariationalParameters(0))); + } + } + KineticEnergy += -2*VariationalParameters(0)*NumberParticles; + for (int i = 0; i < NumberParticles-1; i++) { + for (int j = i+1; j < NumberParticles; j++) { + KineticEnergy += 2.0/(pow(1.0 + VariationalParameters(1)*length(i,j),2))*(1.0/length(i,j)-2*VariationalParameters(1)/(1+VariationalParameters(1)*length(i,j)) ); + } + } + KineticEnergy *= -0.5; + // Set up potential energy, external potential + eventual electron-electron repulsion + double PotentialEnergy = 0; + for (int i = 0; i < NumberParticles; i++) { + double DistanceSquared = singleparticle_pos2(r, i); + PotentialEnergy += 0.5*DistanceSquared; // sp energy HO part, note it has the oscillator frequency set to 1! + } + // Add the electron-electron repulsion + for (int i = 0; i < NumberParticles-1; i++) { + for (int j = i+1; j < NumberParticles; j++) { + PotentialEnergy += 1.0/length(i,j); + } + } + double LocalE = KineticEnergy+PotentialEnergy; + return LocalE; +} + +// Compute the analytical expression for the quantum force +void QuantumForce(Matrix &r, Matrix &qforce, Vector &VariationalParameters) +{ + // compute the first derivative + for (int i = 0; i < NumberParticles; i++) { + for (int k = 0; k < Dimension; k++) { + // single-particle part, replace with Slater det for larger systems + double sppart = DerivativeSPwavefunction(r(i,k),VariationalParameters(0)); + // Jastrow factor contribution + double Jsum = 0.0; + for (int j = 0; j < NumberParticles; j++) { + if ( j != i) { + Jsum += JastrowDerivative(r, VariationalParameters(1), i, j, k); + } + } + qforce(i,k) = 2.0*(Jsum+sppart); + } + } +} // end of QuantumForce function + + +#define ITMAX 200 +#define EPS 3.0e-8 +#define TOLX (4*EPS) +#define STPMX 100.0 + +void dfpmin(Vector &p, int n, double gtol, int *iter, double *fret, + double(*func)(Vector &p), void (*dfunc)(Vector &p, Vector &g)) +{ + + int check,i,its,j; + double den,fac,fad,fae,fp,stpmax,sum=0.0,sumdg,sumxi,temp,test; + Vector dg(n), g(n), hdg(n), pnew(n), xi(n); + Matrix hessian(n,n); + + fp=(*func)(p); + (*dfunc)(p,g); + for (i = 0;i < n;i++) { + for (j = 0; j< n;j++) hessian(i,j)=0.0; + hessian(i,i)=1.0; + xi(i) = -g(i); + sum += p(i)*p(i); + } + stpmax=STPMX*FMAX(sqrt(sum),(double)n); + for (its=1;its<=ITMAX;its++) { + *iter=its; + lnsrch(n,p,fp,g,xi,pnew,fret,stpmax,&check,func); + fp = *fret; + for (i = 0; i< n;i++) { + xi(i)=pnew(i)-p(i); + p(i)=pnew(i); + } + test=0.0; + for (i = 0;i< n;i++) { + temp=fabs(xi(i))/FMAX(fabs(p(i)),1.0); + if (temp > test) test=temp; + } + if (test < TOLX) { + return; + } + for (i=0;i<n;i++) dg(i)=g(i); + (*dfunc)(p,g); + test=0.0; + den=FMAX(*fret,1.0); + for (i=0;i<n;i++) { + temp=fabs(g(i))*FMAX(fabs(p(i)),1.0)/den; + if (temp > test) test=temp; + } + if (test < gtol) { + return; + } + for (i=0;i<n;i++) dg(i)=g(i)-dg(i); + for (i=0;i<n;i++) { + hdg(i)=0.0; + for (j=0;j<n;j++) hdg(i) += hessian(i,j)*dg(j); + } + fac=fae=sumdg=sumxi=0.0; + for (i=0;i<n;i++) { + fac += dg(i)*xi(i); + fae += dg(i)*hdg(i); + sumdg += SQR(dg(i)); + sumxi += SQR(xi(i)); + } + if (fac*fac > EPS*sumdg*sumxi) { + fac=1.0/fac; + fad=1.0/fae; + for (i=0;i<n;i++) dg(i)=fac*xi(i)-fad*hdg(i); + for (i=0;i<n;i++) { + for (j=0;j<n;j++) { + hessian(i,j) += fac*xi(i)*xi(j) + -fad*hdg(i)*hdg(j)+fae*dg(i)*dg(j); + } + } + } + for (i=0;i<n;i++) { + xi(i)=0.0; + for (j=0;j<n;j++) xi(i) -= hessian(i,j)*g(j); + } + } + cout << "too many iterations in dfpmin" << endl; +} +#undef ITMAX +#undef EPS +#undef TOLX +#undef STPMX + +#define ALF 1.0e-4 +#define TOLX 1.0e-7 + +void lnsrch(int n, Vector &xold, double fold, Vector &g, Vector &p, Vector &x, + double *f, double stpmax, int *check, double (*func)(Vector &p)) +{ + int i; + double a,alam,alam2,alamin,b,disc,f2,fold2,rhs1,rhs2,slope,sum,temp, + test,tmplam; + + *check=0; + for (sum=0.0,i=0;i<n;i++) sum += p(i)*p(i); + sum=sqrt(sum); + if (sum > stpmax) + for (i=0;i<n;i++) p(i) *= stpmax/sum; + for (slope=0.0,i=0;i<n;i++) + slope += g(i)*p(i); + test=0.0; + for (i=0;i<n;i++) { + temp=fabs(p(i))/FMAX(fabs(xold(i)),1.0); + if (temp > test) test=temp; + } + alamin=TOLX/test; + alam=1.0; + for (;;) { + for (i=0;i<n;i++) x(i)=xold(i)+alam*p(i); + *f=(*func)(x); + if (alam < alamin) { + for (i=0;i<n;i++) x(i)=xold(i); + *check=1; + return; + } else if (*f <= fold+ALF*alam*slope) return; + else { + if (alam == 1.0) + tmplam = -slope/(2.0*(*f-fold-slope)); + else { + rhs1 = *f-fold-alam*slope; + rhs2=f2-fold2-alam2*slope; + a=(rhs1/(alam*alam)-rhs2/(alam2*alam2))/(alam-alam2); + b=(-alam2*rhs1/(alam*alam)+alam*rhs2/(alam2*alam2))/(alam-alam2); + if (a == 0.0) tmplam = -slope/(2.0*b); + else { + disc=b*b-3.0*a*slope; + if (disc<0.0) cout << "Roundoff problem in lnsrch." << endl; + else tmplam=(-b+sqrt(disc))/(3.0*a); + } + if (tmplam>0.5*alam) + tmplam=0.5*alam; + } + } + alam2=alam; + f2 = *f; + fold2=fold; + alam=FMAX(tmplam,0.1*alam); + } +} +#undef ALF +#undef TOLX ++
+
+
+
+
+
+
+ +
What is OpenMP
+
+
+
+
+
++
-
+
- OpenMP provides high-level thread programming +
- Multiple cooperating threads are allowed to run simultaneously +
- Threads are created and destroyed dynamically in a fork-join pattern +
- An OpenMP program consists of a number of parallel regions +
- Between two parallel regions there is only one master thread +
- In the beginning of a parallel region, a team of new threads is spawned +
- The newly spawned threads work simultaneously with the master thread +
- At the end of a parallel region, the new threads are destroyed +
-
+
Many good tutorials online and excellent textbook
+-
+
- Using OpenMP, by B. Chapman, G. Jost, and A. van der Pas +
- Many tutorials online like OpenMP official site +
+
Getting started, things to remember
+
+
+
+
+
++
-
+
- Remember the header file +
+
+
+
+
+
+
+
+
+
+
+ #include <omp.h> ++
+
+
+
+
+
+
+ -
+
- Insert compiler directives in C++ syntax as +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp...
+
+
+
+
+
+
+
+
+ -
+
- Compile with for example c++ -fopenmp code.cpp +
- Execute +
- Remember to assign the environment variable OMP NUM THREADS +
- It specifies the total number of threads inside a parallel region, if not otherwise overwritten +
-
+
+
OpenMP syntax
+-
+
- Mostly directives +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp construct [ clause ...]
+
+
+
+
+
+
+
+
+ -
+
- Some functions and types +
+
+
+
+
+
+
+
+
+
+
+ #include <omp.h> ++
+
+
+
+
+
+
+ -
+
- Most apply to a block of code +
- Specifically, a structured block +
- Enter at top, exit at bottom only, exit(), abort() permitted +
+
Different OpenMP styles of parallelism
+OpenMP supports several different ways to specify thread parallelism
+ +-
+
- General parallel regions: All threads execute the code, roughly as if you made a routine of that region and created a thread to run that code +
- Parallel loops: Special case for loops, simplifies data parallel code +
- Task parallelism, new in OpenMP 3 +
- Several ways to manage thread coordination, including Master regions and Locks +
- Memory model for shared data +
+
General code structure
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ #include <omp.h> +main () +{ +int var1, var2, var3; +/* serial code */ +/* ... */ +/* start of a parallel region */ +#pragma omp parallel private(var1, var2) shared(var3) +{ +/* ... */ +} +/* more serial code */ +/* ... */ +/* another parallel region */ +#pragma omp parallel +{ +/* ... */ +} +} ++
+
+
+
+
+
+
+ +
Parallel region
+
+
+
+
+
++
-
+
- A parallel region is a block of code that is executed by a team of threads +
- The following compiler directive creates a parallel region +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel { ... }
+
+
+
+
+
+
+
+
+ -
+
- Clauses can be added at the end of the directive +
- Most often used clauses: +
- default(shared) or default(none) +
- public(list of variables) +
- private(list of variables) +
-
+
+
Hello world, not again, please!
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ #include <omp.h> +#include <cstdio> +int main (int argc, char *argv[]) +{ +int th_id, nthreads; +#pragma omp parallel private(th_id) shared(nthreads) +{ +th_id = omp_get_thread_num(); +printf("Hello World from thread %d\n", th_id); +#pragma omp barrier +if ( th_id == 0 ) { +nthreads = omp_get_num_threads(); +printf("There are %d threads\n",nthreads); +} +} +return 0; +} ++
+
+
+
+
+
+
+ +
Hello world, yet another variant
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+
+ #include <cstdio> +#include <omp.h> +int main(int argc, char *argv[]) +{ + omp_set_num_threads(4); +#pragma omp parallel + { + int id = omp_get_thread_num(); + int nproc = omp_get_num_threads(); + cout << "Hello world with id number and processes " << id << nproc << endl; + } +return 0; +} ++
+
+
+
+
+
+
+ Variables declared outside of the parallel region are shared by all threads +If a variable like id is declared outside of the +
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel,
+
+
+
+
+
+
+
+
+ it would have been shared by various the threads, possibly causing erroneous output
+-
+
- Why? What would go wrong? Why do we add possibly? +
+
Important OpenMP library routines
+
+
+
+
+
++ +
-
+
- int omp get num threads (), returns the number of threads inside a parallel region +
- int omp get thread num (), returns the a thread for each thread inside a parallel region +
- void omp set num threads (int), sets the number of threads to be used +
- void omp set nested (int), turns nested parallelism on/off +
+
Private variables
+
+
+
+
+
++
Private clause can be used to make thread- private versions of such variables:
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel private(id) +{ + int id = omp_get_thread_num(); + cout << "My thread num" << id << endl; +} ++
+
+
+
+
+
+
+ -
+
- What is their value on entry? Exit? +
- OpenMP provides ways to control that +
- Can use default(none) to require the sharing of each variable to be described +
+
Master region
+
+
+
+
+
++
It is often useful to have only one thread execute some of the code in a parallel region. I/O statements are a common example
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel +{ + #pragma omp master + { + int id = omp_get_thread_num(); + cout << "My thread num" << id << endl; + } +} ++
+
+
+
+
+
+
+ +
Parallel for loop
+
+
+
+
+
++
-
+
- Inside a parallel region, the following compiler directive can be used to parallelize a for-loop: +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp for
+
+
+
+
+
+
+
+
+ -
+
- Clauses can be added, such as +
- schedule(static, chunk size) +
- schedule(dynamic, chunk size) +
- schedule(guided, chunk size) (non-deterministic allocation) +
- schedule(runtime) +
- private(list of variables) +
- reduction(operator:variable) +
- nowait +
-
+
+
Parallel computations and loops
+ +
+
+
+
+
++
OpenMP provides an easy way to parallelize a loop
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for + for (i=0; i<n; i++) c[i] = a[i]; ++
+
+
+
+
+
+
+ OpenMP handles index variable (no need to declare in for loop or make private)
+ +Which thread does which values? Several options.
++
Scheduling of loop computations
+ +
+
+
+
+
++
We can let the OpenMP runtime decide. The decision is about how the loop iterates are scheduled +and OpenMP defines three choices of loop scheduling: +
+-
+
- Static: Predefined at compile time. Lowest overhead, predictable +
- Dynamic: Selection made at runtime +
- Guided: Special case of dynamic; attempts to reduce overhead +
+
Example code for loop scheduling
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ #include <omp.h> +#define CHUNKSIZE 100 +#define N 1000 +int main (int argc, char *argv[]) +{ +int i, chunk; +float a[N], b[N], c[N]; +for (i=0; i < N; i++) a[i] = b[i] = i * 1.0; +chunk = CHUNKSIZE; +#pragma omp parallel shared(a,b,c,chunk) private(i) +{ +#pragma omp for schedule(dynamic,chunk) +for (i=0; i < N; i++) c[i] = a[i] + b[i]; +} /* end of parallel region */ +} ++
+
+
+
+
+
+
+ +
Example code for loop scheduling, guided instead of dynamic
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ #include <omp.h> +#define CHUNKSIZE 100 +#define N 1000 +int main (int argc, char *argv[]) +{ +int i, chunk; +float a[N], b[N], c[N]; +for (i=0; i < N; i++) a[i] = b[i] = i * 1.0; +chunk = CHUNKSIZE; +#pragma omp parallel shared(a,b,c,chunk) private(i) +{ +#pragma omp for schedule(guided,chunk) +for (i=0; i < N; i++) c[i] = a[i] + b[i]; +} /* end of parallel region */ +} ++
+
+
+
+
+
+
+ +
More on Parallel for loop
+
+
+
+
+
++
-
+
- The number of loop iterations cannot be non-deterministic; break, return, exit, goto not allowed inside the for-loop +
- The loop index is private to each thread +
- A reduction variable is special +
- During the for-loop there is a local private copy in each thread +
- At the end of the for-loop, all the local copies are combined together by the reduction operation +
- Unless the nowait clause is used, an implicit barrier synchronization will be added at the end by the compiler +
-
+
+
+
+
+
+
+
+
+
+
+
+ // #pragma omp parallel and #pragma omp for
+
+
+
+
+
+
+
+
+ can be combined into
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for
+
+
+
+
+
+
+
+
+ +
What can happen with this loop?
+ +
+
+
+
+
++
What happens with code like this
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for +for (i=0; i<n; i++) sum += a[i]*a[i]; ++
+
+
+
+
+
+
+ All threads can access the sum variable, but the addition is not atomic! It is important to avoid race between threads. So-called reductions in OpenMP are thus important for performance and for obtaining correct results. OpenMP lets us indicate that a variable is used for a reduction with a particular operator. The above code becomes
+ + +
+
+
+
+
+
+
+
+
+
+ sum = 0.0; +#pragma omp parallel for reduction(+:sum) +for (i=0; i<n; i++) sum += a[i]*a[i]; ++
+
+
+
+
+
+
+ +
Inner product
+
+
+
+
+
++$$ +\sum_{i=0}^{n-1} a_ib_i +$$ + + + +
+
+
+
+
+
+
+
+
+
+ int i; +double sum = 0.; +/* allocating and initializing arrays */ +/* ... */ +#pragma omp parallel for default(shared) private(i) reduction(+:sum) + for (i=0; i<N; i++) sum += a[i]*b[i]; +} ++
+
+
+
+
+
+
+ +
Different threads do different tasks
+
+
+
+
+
++ +
Different threads do different tasks independently, each section is executed by one thread.
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel +{ +#pragma omp sections +{ +#pragma omp section +funcA (); +#pragma omp section +funcB (); +#pragma omp section +funcC (); +} +} ++
+
+
+
+
+
+
+ +
Single execution
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp single { ... }
+
+
+
+
+
+
+
+
+ The code is executed by one thread only, no guarantee which thread
+ +Can introduce an implicit barrier at the end
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp master { ... }
+
+
+
+
+
+
+
+
+ Code executed by the master thread, guaranteed and no implicit barrier at the end.
++
Coordination and synchronization
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp barrier
+
+
+
+
+
+
+
+
+ Synchronization, must be encountered by all threads in a team (or none)
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp ordered { a block of codes }
+
+
+
+
+
+
+
+
+ is another form of synchronization (in sequential order). +The form +
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp critical { a block of codes }
+
+
+
+
+
+
+
+
+ and
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp atomic { single assignment statement }
+
+
+
+
+
+
+
+
+ is more efficient than
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp critical
+
+
+
+
+
+
+
+
+ +
Data scope
+
+
+
+
+
++
-
+
- OpenMP data scope attribute clauses: +
- shared +
- private +
- firstprivate +
- lastprivate +
- reduction +
-
+
What are the purposes of these attributes
+-
+
- define how and which variables are transferred to a parallel region (and back) +
- define which variables are visible to all threads in a parallel region, and which variables are privately allocated to each thread +
+
Some remarks
+
+
+
+
+
++ +
-
+
- When entering a parallel region, the private clause ensures each thread having its own new variable instances. The new variables are assumed to be uninitialized. +
- A shared variable exists in only one memory location and all threads can read and write to that address. It is the programmer's responsibility to ensure that multiple threads properly access a shared variable. +
- The firstprivate clause combines the behavior of the private clause with automatic initialization. +
- The lastprivate clause combines the behavior of the private clause with a copy back (from the last loop iteration or section) to the original variable outside the parallel region. +
+
Parallelizing nested for-loops
+
+
+
+
+
++ +
-
+
- Serial code +
+
+
+
+
+
+
+
+
+
+
+
+ for (i=0; i<100; i++) + for (j=0; j<100; j++) + a[i][j] = b[i][j] + c[i][j]; + } +} ++
+
+
+
+
+
+
+ -
+
- Parallelization +
+
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for private(j) +for (i=0; i<100; i++) + for (j=0; j<100; j++) + a[i][j] = b[i][j] + c[i][j]; + } +} ++
+
+
+
+
+
+
+ -
+
- Why not parallelize the inner loop? to save overhead of repeated thread forks-joins +
- Why must j be private? To avoid race condition among the threads +
+
Nested parallelism
+
+
+
+
+
++
When a thread in a parallel region encounters another parallel construct, it +may create a new team of threads and become the master of the new +team. +
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel num_threads(4) +{ +/* .... */ +#pragma omp parallel num_threads(2) +{ +// +} +} ++
+
+
+
+
+
+
+ +
Parallel tasks
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp task +#pragma omp parallel shared(p_vec) private(i) +{ +#pragma omp single +{ +for (i=0; i<N; i++) { + double r = random_number(); + if (p_vec[i] > r) { +#pragma omp task + do_work (p_vec[i]); ++
+
+
+
+
+
+
+ +
Common mistakes
+
+
+
+
+
+
++
Race condition
+ + +
+
+
+
+
+
+
+
+
+
+
+ int nthreads; +#pragma omp parallel shared(nthreads) +{ +nthreads = omp_get_num_threads(); +} ++
+
+
+
+
+
+
+ Deadlock
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel +{ +... +#pragma omp critical +{ +... +#pragma omp barrier +} +} ++
+
+
+
+
+
+
+ Not all computations are simple
+
+
+
+
+
+
++
Not all computations are simple loops where the data can be evenly +divided among threads without any dependencies between threads +
+ +An example is finding the location and value of the largest element in an array
+ + +
+
+
+
+
+
+
+
+
+
+ for (i=0; i<n; i++) { + if (x[i] > maxval) { + maxval = x[i]; + maxloc = i; + } +} ++
+
+
+
+
+
+
+ Not all computations are simple, competing threads
+
+
+
+
+
++
All threads are potentially accessing and changing the same values, maxloc and maxval.
+-
+
- OpenMP provides several ways to coordinate access to shared values +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp atomic
+
+
+
+
+
+
+
+
+ -
+
- Only one thread at a time can execute the following statement (not block). We can use the critical option +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp critical
+
+
+
+
+
+
+
+
+ -
+
- Only one thread at a time can execute the following block +
Atomic may be faster than critical but depends on hardware
++
How to find the max value using OpenMP
+
+
+
+
+
++
Write down the simplest algorithm and look carefully for race conditions. How would you handle them? +The first step would be to parallelize as +
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for + for (i=0; i<n; i++) { + if (x[i] > maxval) { + maxval = x[i]; + maxloc = i; + } +} ++
+
+
+
+
+
+
+ +
Then deal with the race conditions
+
+
+
+
++
Write down the simplest algorithm and look carefully for race conditions. How would you handle them? +The first step would be to parallelize as +
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for + for (i=0; i<n; i++) { +#pragma omp critical + { + if (x[i] > maxval) { + maxval = x[i]; + maxloc = i; + } + } +} ++
+
+
+
+
+
+
+ Exercise: write a code which implements this and give an estimate on performance. Perform several runs, +with a serial code only with and without vectorization and compare the serial code with the one that uses OpenMP. Run on different archictectures if you can. +
++
What can slow down OpenMP performance?
+Give it a thought!
+ ++
What can slow down OpenMP performance?
+
+
+
+
+
++
Performance poor because we insisted on keeping track of the maxval and location during the execution of the loop.
+-
+
- We do not care about the value during the execution of the loop, just the value at the end. +
This is a common source of performance issues, namely the description of the method used to compute a value imposes additional, unnecessary requirements or properties
+ +Idea: Have each thread find the maxloc in its own data, then combine and use temporary arrays indexed by thread number to hold the values found by each thread ++
Find the max location for each thread
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ int maxloc[MAX_THREADS], mloc; +double maxval[MAX_THREADS], mval; +#pragma omp parallel shared(maxval,maxloc) +{ + int id = omp_get_thread_num(); + maxval[id] = -1.0e30; +#pragma omp for + for (int i=0; i<n; i++) { + if (x[i] > maxval[id]) { + maxloc[id] = i; + maxval[id] = x[i]; + } + } +} ++
+
+
+
+
+
+
+ +
Combine the values from each thread
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp flush (maxloc,maxval) +#pragma omp master + { + int nt = omp_get_num_threads(); + mloc = maxloc[0]; + mval = maxval[0]; + for (int i=1; i<nt; i++) { + if (maxval[i] > mval) { + mval = maxval[i]; + mloc = maxloc[i]; + } + } + } ++
+
+
+
+
+
+
+ Note that we let the master process perform the last operation.
++
Matrix-matrix multiplication
+This code computes the norm of a vector using OpenMp
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ // OpenMP program to compute vector norm by adding two other vectors
+#include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include <omp.h>
+# include <ctime>
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of vector
+ int n = atoi(argv[1]);
+ double *a, *b, *c;
+ int i;
+ int thread_num;
+ double wtime, Norm2, s, angle;
+ cout << " Perform addition of two vectors and compute the norm-2." << endl;
+ omp_set_num_threads(4);
+ thread_num = omp_get_max_threads ();
+ cout << " The number of processors available = " << omp_get_num_procs () << endl ;
+ cout << " The number of threads available = " << thread_num << endl;
+ cout << " The matrix order n = " << n << endl;
+
+ s = 1.0/sqrt( (double) n);
+ wtime = omp_get_wtime ( );
+ // Allocate space for the vectors to be used
+ a = new double [n]; b = new double [n]; c = new double [n];
+ // Define parallel region
+# pragma omp parallel for default(shared) private (angle, i) reduction(+:Norm2)
+ // Set up values for vectors a and b
+ for (i = 0; i < n; i++){
+ angle = 2.0*M_PI*i/ (( double ) n);
+ a[i] = s*(sin(angle) + cos(angle));
+ b[i] = s*sin(2.0*angle);
+ c[i] = 0.0;
+ }
+ // Then perform the vector addition
+ for (i = 0; i < n; i++){
+ c[i] += a[i]+b[i];
+ }
+ // Compute now the norm-2
+ Norm2 = 0.0;
+ for (i = 0; i < n; i++){
+ Norm2 += c[i]*c[i];
+ }
+// end parallel region
+ wtime = omp_get_wtime ( ) - wtime;
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for norm-2 computation=" << wtime << endl;
+ cout << " Norm-2 = " << Norm2 << endl;
+ // Free up space
+ delete[] a;
+ delete[] b;
+ delete[] c;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ +
Matrix-matrix multiplication
+This the matrix-matrix multiplication code with plain c++ memory allocation using OpenMP
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+
+ // Matrix-matrix multiplication and Frobenius norm of a matrix with OpenMP
+#include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include <omp.h>
+# include <ctime>
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of square matrix
+ int n = atoi(argv[1]);
+ double **A, **B, **C;
+ int i, j, k;
+ int thread_num;
+ double wtime, Fsum, s, angle;
+ cout << " Compute matrix product C = A * B and Frobenius norm." << endl;
+ omp_set_num_threads(4);
+ thread_num = omp_get_max_threads ();
+ cout << " The number of processors available = " << omp_get_num_procs () << endl ;
+ cout << " The number of threads available = " << thread_num << endl;
+ cout << " The matrix order n = " << n << endl;
+
+ s = 1.0/sqrt( (double) n);
+ wtime = omp_get_wtime ( );
+ // Allocate space for the two matrices
+ A = new double*[n]; B = new double*[n]; C = new double*[n];
+ for (i = 0; i < n; i++){
+ A[i] = new double[n];
+ B[i] = new double[n];
+ C[i] = new double[n];
+ }
+ // Define parallel region
+# pragma omp parallel for default(shared) private (angle, i, j, k) reduction(+:Fsum)
+ // Set up values for matrix A and B and zero matrix C
+ for (i = 0; i < n; i++){
+ for (j = 0; j < n; j++) {
+ angle = 2.0*M_PI*i*j/ (( double ) n);
+ A[i][j] = s * ( sin ( angle ) + cos ( angle ) );
+ B[j][i] = A[i][j];
+ }
+ }
+ // Then perform the matrix-matrix multiplication
+ for (i = 0; i < n; i++){
+ for (j = 0; j < n; j++) {
+ C[i][j] = 0.0;
+ for (k = 0; k < n; k++) {
+ C[i][j] += A[i][k]*B[k][j];
+ }
+ }
+ }
+ // Compute now the Frobenius norm
+ Fsum = 0.0;
+ for (i = 0; i < n; i++){
+ for (j = 0; j < n; j++) {
+ Fsum += C[i][j]*C[i][j];
+ }
+ }
+ Fsum = sqrt(Fsum);
+// end parallel region and letting only one thread perform I/O
+ wtime = omp_get_wtime ( ) - wtime;
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for matrix-matrix multiplication=" << wtime << endl;
+ cout << " Frobenius norm = " << Fsum << endl;
+ // Free up space
+ for (int i = 0; i < n; i++){
+ delete[] A[i];
+ delete[] B[i];
+ delete[] C[i];
+ }
+ delete[] A;
+ delete[] B;
+ delete[] C;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ Week 11, March 11-15: Resampling Techniques, Bootstrap and Blocking
++
March 11-15
++ +
+
Overview of week 11, March 11-15
+
+Topics
+
+
+
++
-
+
- Reminder from last week about statistical observables, the central limit theorem and bootstrapping, see notes from last week +
- Resampling Techniques, emphasis on Blocking +
- Discussion of onebody densities (whiteboard notes) +
- Start discussion on optimization and parallelization for Python and C++ + + +
Note, these notes contain additional material om optimization and parallelization. Parts of this material will be discussed this week.
+ ++
Why resampling methods ?
+
+Statistical analysis
+
+
+
++
-
+
- Our simulations can be treated as computer experiments. This is particularly the case for Monte Carlo methods +
- The results can be analysed with the same statistical tools as we would use analysing experimental data. +
- As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors. +
+
Statistical analysis
+
+
+
+
+
++
-
+
- As in other experiments, many numerical experiments have two classes of errors:
+
- Statistical errors +
- Systematical errors + +
- Statistical errors can be estimated using standard tools from statistics +
- Systematical errors are method specific and must be treated differently from case to case. +
+
And why do we use such methods?
+ +As you will see below, due to correlations between various +measurements, we need to evaluate the so-called covariance in order to +establish a proper evaluation of the total variance and the thereby +the standard deviation of a given expectation value. +
+ +The covariance however, leads to an evaluation of a double sum over the various stochastic variables. This becomes computationally too expensive to evaluate. +Methods like the Bootstrap, the Jackknife and/or Blocking allow us to circumvent this problem. +
+ ++
Central limit theorem
+ +Last week we derived the central limit theorem with the following assumptions:
+ +
+Measurement \( i \)
+
+
+
++
We assumed that each individual measurement \( x_{ij} \) is represented by stochastic variables which independent and identically distributed (iid). +This defined the sample mean of of experiment \( i \) with \( n \) samples as +
+$$ +\overline{x}_i=\frac{1}{n}\sum_{j} x_{ij}. +$$ + +and the sample variance
+$$ +\sigma^2_i=\frac{1}{n}\sum_{j} \left(x_{ij}-\overline{x}_i\right)^2. +$$ ++
Further remarks
+ +Note that we use \( n \) instead of \( n-1 \) in the definition of +variance. The sample variance and the sample mean are not necessarily equal to +the exact values we would get if we knew the corresponding probability +distribution. +
+ ++
Running many measurements
+ +
+Adding \( m \) measurements \( i \)
+
+
++
With the assumption that the average measurements \( i \) are also defined as iid stochastic variables and have the same probability function \( p \), +we defined the total average over \( m \) experiments as +
+$$ +\overline{X}=\frac{1}{m}\sum_{i} \overline{x}_{i}. +$$ + +and the total variance
+$$ +\sigma^2_{m}=\frac{1}{m}\sum_{i} \left( \overline{x}_{i}-\overline{X}\right)^2. +$$ +These are the quantities we used in showing that if the individual mean values are iid stochastic variables, then in the limit \( m\rightarrow \infty \), the distribution for \( \overline{X} \) is given by a Gaussian distribution with variance \( \sigma^2_m \).
+ ++
Adding more definitions
+ +The total sample variance over the \( mn \) measurements is defined as
+$$ +\sigma^2=\frac{1}{mn}\sum_{i=1}^{m} \sum_{j=1}^{n}\left(x_{ij}-\overline{X}\right)^2. +$$ + +We have from the equation for \( \sigma_m^2 \)
+$$ +\overline{x}_i-\overline{X}=\frac{1}{n}\sum_{j=1}^{n}\left(x_{i}-\overline{X}\right), +$$ + +and introducing the centered value \( \tilde{x}_{ij}=x_{ij}-\overline{X} \), we can rewrite \( \sigma_m^2 \) as
+$$ +\sigma^2_{m}=\frac{1}{m}\sum_{i} \left( \overline{x}_{i}-\overline{X}\right)^2=\frac{1}{m}\sum_{i=1}^{m}\left[ \frac{i}{n}\sum_{j=1}^{n}\tilde{x}_{ij}\right]^2. +$$ + + ++
Further rewriting
+ +We can rewrite the latter in terms of a sum over diagonal elements only and another sum which contains the non-diagonal elements
+$$ +\begin{align*} +\sigma^2_{m}& =\frac{1}{m}\sum_{i=1}^{m}\left[ \frac{i}{n}\sum_{j=1}^{n}\tilde{x}_{ij}\right]^2 \\ + & = \frac{1}{mn^2}\sum_{i=1}^{m} \sum_{j=1}^{n}\tilde{x}_{ij}^2+\frac{2}{mn^2}\sum_{i=1}^{m} \sum_{j < k}^{n}\tilde{x}_{ij}\tilde{x}_{ik}. +\end{align*} +$$ + +The first term on the last rhs is nothing but the total sample variance \( \sigma^2 \) divided by \( m \). The second term represents the covariance.
+ ++
The covariance term
+ +Using the definition of the total sample variance we have
+$$ +\begin{align*} +\sigma^2_{m}& = \frac{\sigma^2}{m}+\frac{2}{mn^2}\sum_{i=1}^{m} \sum_{j < k}^{n}\tilde{x}_{ij}\tilde{x}_{ik}. +\end{align*} +$$ + +The first term is what we have used till now in order to estimate the +standard deviation. However, the second term which gives us a measure +of the correlations between different stochastic events, can result in +contributions which give rise to a larger standard deviation and +variance \( \sigma_m^2 \). Note also the evaluation of the second term +leads to a double sum over all events. If we run a VMC calculation +with say \( 10^9 \) Monte carlo samples, the latter term would lead to +\( 10^{18} \) function evaluations. We don't want to, by obvious reasons, to venture into that many evaluations. +
+ +Note also that if our stochastic events are iid then the covariance terms is zero.
+ ++
Rewriting the covariance term
+ +We introduce now a variable \( d=\vert j-k\vert \) and rewrite
+$$ +\frac{2}{mn^2}\sum_{i=1}^{m} \sum_{j < k}^{n}\tilde{x}_{ij}\tilde{x}_{ik}, +$$ + +in terms of a function
+$$ +f_d=\frac{2}{mn}\sum_{i=1}^{m} \sum_{k=1}^{n-d}\tilde{x}_{ik}\tilde{x}_{i(k+d)}. +$$ + +We note that for \( d=0 \) we have
+$$ +f_0=\frac{2}{mn}\sum_{i=1}^{m} \sum_{k=1}^{n}\tilde{x}_{ik}\tilde{x}_{i(k)}=\sigma^2! +$$ + + ++
Introducing the correlation function
+ +We introduce then a correlation function \( \kappa_d=f_d/\sigma^2 \). Note that \( \kappa_0 =1 \). We rewrite the variance \( \sigma_m^2 \) as
+$$ +\begin{align*} +\sigma^2_{m}& = \frac{\sigma^2}{m}\left[1+2\sum_{d=1}^{n-1} \kappa_d\right]. +\end{align*} +$$ + +The code here shows the evolution of \( \kappa_d \) as a function of \( d \) for a series of random numbers. We see that the function \( \kappa_d \) approaches \( 0 \) as \( d\rightarrow \infty \).
+ +Note: code will be inserted here later.
+ ++
Resampling methods: Blocking
+ +The blocking method was made popular by Flyvbjerg and Pedersen (1989) +and has become one of the standard ways to estimate the variance +\( \mathrm{var}(\widehat{\theta}) \) for exactly one estimator \( \widehat{\theta} \), namely +\( \widehat{\theta} = \overline{X} \), the mean value. +
+ +Assume \( n = 2^d \) for some integer \( d>1 \) and \( X_1,X_2,\cdots, X_n \) is a stationary time series to begin with. +Moreover, assume that the series is asymptotically uncorrelated. We switch to vector notation by arranging \( X_1,X_2,\cdots,X_n \) in an \( n \)-tuple. Define: +
+$$ +\begin{align*} +\hat{X} = (X_1,X_2,\cdots,X_n). +\end{align*} +$$ + + ++
Why blocking?
+ +The strength of the blocking method is when the number of +observations, \( n \) is large. For large \( n \), the complexity of dependent +bootstrapping scales poorly, but the blocking method does not, +moreover, it becomes more accurate the larger \( n \) is. +
+ ++
Blocking Transformations
+We now define the blocking transformations. The idea is to take the mean of subsequent +pair of elements from \( \boldsymbol{X} \) and form a new vector +\( \boldsymbol{X}_1 \). Continuing in the same way by taking the mean of +subsequent pairs of elements of \( \boldsymbol{X}_1 \) we obtain \( \boldsymbol{X}_2 \), and +so on. +Define \( \boldsymbol{X}_i \) recursively by: +
+ +$$ +\begin{align} +(\boldsymbol{X}_0)_k &\equiv (\boldsymbol{X})_k \nonumber \\ +(\boldsymbol{X}_{i+1})_k &\equiv \frac{1}{2}\Big( (\boldsymbol{X}_i)_{2k-1} + +(\boldsymbol{X}_i)_{2k} \Big) \qquad \text{for all} \qquad 1 \leq i \leq d-1 +\label{_auto1} +\end{align} +$$ + + ++
Blocking transformations
+ +The quantity \( \boldsymbol{X}_k \) is +subject to \( k \) blocking transformations. We now have \( d \) vectors +\( \boldsymbol{X}_0, \boldsymbol{X}_1,\cdots,\vec X_{d-1} \) containing the subsequent +averages of observations. It turns out that if the components of +\( \boldsymbol{X} \) is a stationary time series, then the components of +\( \boldsymbol{X}_i \) is a stationary time series for all \( 0 \leq i \leq d-1 \) +
+ +We can then compute the autocovariance, the variance, sample mean, and +number of observations for each \( i \). +Let \( \gamma_i, \sigma_i^2, +\overline{X}_i \) denote the covariance, variance and average of the +elements of \( \boldsymbol{X}_i \) and let \( n_i \) be the number of elements of +\( \boldsymbol{X}_i \). It follows by induction that \( n_i = n/2^i \). +
+ ++
Blocking Transformations
+ +Using the +definition of the blocking transformation and the distributive +property of the covariance, it is clear that since \( h =|i-j| \) +we can define +
+$$ +\begin{align} +\gamma_{k+1}(h) &= cov\left( ({X}_{k+1})_{i}, ({X}_{k+1})_{j} \right) \nonumber \\ +&= \frac{1}{4}cov\left( ({X}_{k})_{2i-1} + ({X}_{k})_{2i}, ({X}_{k})_{2j-1} + ({X}_{k})_{2j} \right) \nonumber \\ +&= \frac{1}{2}\gamma_{k}(2h) + \frac{1}{2}\gamma_k(2h+1) \hspace{0.1cm} \mathrm{h = 0} +\label{_auto2}\\ +&=\frac{1}{4}\gamma_k(2h-1) + \frac{1}{2}\gamma_k(2h) + \frac{1}{4}\gamma_k(2h+1) \quad \mathrm{else} +\label{_auto3} +\end{align} +$$ + +The quantity \( \hat{X} \) is asymptotically uncorrelated by assumption, \( \hat{X}_k \) is also asymptotic uncorrelated. Let's turn our attention to the variance of the sample +mean \( \mathrm{var}(\overline{X}) \). +
+ ++
Blocking Transformations, getting there
+We have
+$$ +\begin{align} +\mathrm{var}(\overline{X}_k) = \frac{\sigma_k^2}{n_k} + \underbrace{\frac{2}{n_k} \sum_{h=1}^{n_k-1}\left( 1 - \frac{h}{n_k} \right)\gamma_k(h)}_{\equiv e_k} = \frac{\sigma^2_k}{n_k} + e_k \quad \text{if} \quad \gamma_k(0) = \sigma_k^2. +\label{_auto4} +\end{align} +$$ + +The term \( e_k \) is called the truncation error:
+$$ +\begin{equation} +e_k = \frac{2}{n_k} \sum_{h=1}^{n_k-1}\left( 1 - \frac{h}{n_k} \right)\gamma_k(h). +\label{_auto5} +\end{equation} +$$ + +We can show that \( \mathrm{var}(\overline{X}_i) = \mathrm{var}(\overline{X}_j) \) for all \( 0 \leq i \leq d-1 \) and \( 0 \leq j \leq d-1 \).
+ ++
Blocking Transformations, final expressions
+ +We can then wrap up
+$$ +\begin{align} +n_{j+1} \overline{X}_{j+1} &= \sum_{i=1}^{n_{j+1}} (\hat{X}_{j+1})_i = \frac{1}{2}\sum_{i=1}^{n_{j}/2} (\hat{X}_{j})_{2i-1} + (\hat{X}_{j})_{2i} \nonumber \\ +&= \frac{1}{2}\left[ (\hat{X}_j)_1 + (\hat{X}_j)_2 + \cdots + (\hat{X}_j)_{n_j} \right] = \underbrace{\frac{n_j}{2}}_{=n_{j+1}} \overline{X}_j = n_{j+1}\overline{X}_j. +\label{_auto6} +\end{align} +$$ + +By repeated use of this equation we get \( \mathrm{var}(\overline{X}_i) = \mathrm{var}(\overline{X}_0) = \mathrm{var}(\overline{X}) \) for all \( 0 \leq i \leq d-1 \). This has the consequence that
+$$ +\begin{align} +\mathrm{var}(\overline{X}) = \frac{\sigma_k^2}{n_k} + e_k \qquad \text{for all} \qquad 0 \leq k \leq d-1. \label{eq:convergence} +\end{align} +$$ + + ++
More on the blocking method
+ +Flyvbjerg and Petersen demonstrated that the sequence +\( \{e_k\}_{k=0}^{d-1} \) is decreasing, and conjecture that the term +\( e_k \) can be made as small as we would like by making \( k \) (and hence +\( d \)) sufficiently large. The sequence is decreasing. +It means we can apply blocking transformations until +\( e_k \) is sufficiently small, and then estimate \( \mathrm{var}(\overline{X}) \) by +\( \widehat{\sigma}^2_k/n_k \). +
+ +For an elegant solution and proof of the blocking method, see the recent article of Marius Jonsson (former MSc student of the Computational Physics group).
+ ++
Example code form last week
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ # 2-electron VMC code for 2dim quantum dot with importance sampling +# Using gaussian rng for new positions and Metropolis- Hastings +# Added energy minimization +from math import exp, sqrt +from random import random, seed, normalvariate +import numpy as np +import matplotlib.pyplot as plt +from mpl_toolkits.mplot3d import Axes3D +from matplotlib import cm +from matplotlib.ticker import LinearLocator, FormatStrFormatter +from scipy.optimize import minimize +import sys +import os + +# Where to save data files +PROJECT_ROOT_DIR = "Results" +DATA_ID = "Results/EnergyMin" + +if not os.path.exists(PROJECT_ROOT_DIR): + os.mkdir(PROJECT_ROOT_DIR) + +if not os.path.exists(DATA_ID): + os.makedirs(DATA_ID) + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +outfile = open(data_path("Energies.dat"),'w') + + +# Trial wave function for the 2-electron quantum dot in two dims +def WaveFunction(r,alpha,beta): + r1 = r[0,0]**2 + r[0,1]**2 + r2 = r[1,0]**2 + r[1,1]**2 + r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2) + deno = r12/(1+beta*r12) + return exp(-0.5*alpha*(r1+r2)+deno) + +# Local energy for the 2-electron quantum dot in two dims, using analytical local energy +def LocalEnergy(r,alpha,beta): + + r1 = (r[0,0]**2 + r[0,1]**2) + r2 = (r[1,0]**2 + r[1,1]**2) + r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2) + deno = 1.0/(1+beta*r12) + deno2 = deno*deno + return 0.5*(1-alpha*alpha)*(r1 + r2) +2.0*alpha + 1.0/r12+deno2*(alpha*r12-deno2+2*beta*deno-1.0/r12) + +# Derivate of wave function ansatz as function of variational parameters +def DerivativeWFansatz(r,alpha,beta): + + WfDer = np.zeros((2), np.double) + r1 = (r[0,0]**2 + r[0,1]**2) + r2 = (r[1,0]**2 + r[1,1]**2) + r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2) + deno = 1.0/(1+beta*r12) + deno2 = deno*deno + WfDer[0] = -0.5*(r1+r2) + WfDer[1] = -r12*r12*deno2 + return WfDer + +# Setting up the quantum force for the two-electron quantum dot, recall that it is a vector +def QuantumForce(r,alpha,beta): + + qforce = np.zeros((NumberParticles,Dimension), np.double) + r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2) + deno = 1.0/(1+beta*r12) + qforce[0,:] = -2*r[0,:]*alpha*(r[0,:]-r[1,:])*deno*deno/r12 + qforce[1,:] = -2*r[1,:]*alpha*(r[1,:]-r[0,:])*deno*deno/r12 + return qforce + + +# Computing the derivative of the energy and the energy +def EnergyDerivative(x0): + + + # Parameters in the Fokker-Planck simulation of the quantum force + D = 0.5 + TimeStep = 0.05 + # positions + PositionOld = np.zeros((NumberParticles,Dimension), np.double) + PositionNew = np.zeros((NumberParticles,Dimension), np.double) + # Quantum force + QuantumForceOld = np.zeros((NumberParticles,Dimension), np.double) + QuantumForceNew = np.zeros((NumberParticles,Dimension), np.double) + + energy = 0.0 + DeltaE = 0.0 + alpha = x0[0] + beta = x0[1] + EnergyDer = 0.0 + DeltaPsi = 0.0 + DerivativePsiE = 0.0 + #Initial position + for i in range(NumberParticles): + for j in range(Dimension): + PositionOld[i,j] = normalvariate(0.0,1.0)*sqrt(TimeStep) + wfold = WaveFunction(PositionOld,alpha,beta) + QuantumForceOld = QuantumForce(PositionOld,alpha, beta) + + #Loop over MC MCcycles + for MCcycle in range(NumberMCcycles): + #Trial position moving one particle at the time + for i in range(NumberParticles): + for j in range(Dimension): + PositionNew[i,j] = PositionOld[i,j]+normalvariate(0.0,1.0)*sqrt(TimeStep)+\ + QuantumForceOld[i,j]*TimeStep*D + wfnew = WaveFunction(PositionNew,alpha,beta) + QuantumForceNew = QuantumForce(PositionNew,alpha, beta) + GreensFunction = 0.0 + for j in range(Dimension): + GreensFunction += 0.5*(QuantumForceOld[i,j]+QuantumForceNew[i,j])*\ + (D*TimeStep*0.5*(QuantumForceOld[i,j]-QuantumForceNew[i,j])-\ + PositionNew[i,j]+PositionOld[i,j]) + + GreensFunction = exp(GreensFunction) + ProbabilityRatio = GreensFunction*wfnew**2/wfold**2 + #Metropolis-Hastings test to see whether we accept the move + if random() <= ProbabilityRatio: + for j in range(Dimension): + PositionOld[i,j] = PositionNew[i,j] + QuantumForceOld[i,j] = QuantumForceNew[i,j] + wfold = wfnew + DeltaE = LocalEnergy(PositionOld,alpha,beta) + DerPsi = DerivativeWFansatz(PositionOld,alpha,beta) + DeltaPsi += DerPsi + energy += DeltaE + DerivativePsiE += DerPsi*DeltaE + + # We calculate mean values + energy /= NumberMCcycles + DerivativePsiE /= NumberMCcycles + DeltaPsi /= NumberMCcycles + EnergyDer = 2*(DerivativePsiE-DeltaPsi*energy) + return EnergyDer + + +# Computing the expectation value of the local energy +def Energy(x0): + # Parameters in the Fokker-Planck simulation of the quantum force + D = 0.5 + TimeStep = 0.05 + # positions + PositionOld = np.zeros((NumberParticles,Dimension), np.double) + PositionNew = np.zeros((NumberParticles,Dimension), np.double) + # Quantum force + QuantumForceOld = np.zeros((NumberParticles,Dimension), np.double) + QuantumForceNew = np.zeros((NumberParticles,Dimension), np.double) + + energy = 0.0 + DeltaE = 0.0 + alpha = x0[0] + beta = x0[1] + #Initial position + for i in range(NumberParticles): + for j in range(Dimension): + PositionOld[i,j] = normalvariate(0.0,1.0)*sqrt(TimeStep) + wfold = WaveFunction(PositionOld,alpha,beta) + QuantumForceOld = QuantumForce(PositionOld,alpha, beta) + + #Loop over MC MCcycles + for MCcycle in range(NumberMCcycles): + #Trial position moving one particle at the time + for i in range(NumberParticles): + for j in range(Dimension): + PositionNew[i,j] = PositionOld[i,j]+normalvariate(0.0,1.0)*sqrt(TimeStep)+\ + QuantumForceOld[i,j]*TimeStep*D + wfnew = WaveFunction(PositionNew,alpha,beta) + QuantumForceNew = QuantumForce(PositionNew,alpha, beta) + GreensFunction = 0.0 + for j in range(Dimension): + GreensFunction += 0.5*(QuantumForceOld[i,j]+QuantumForceNew[i,j])*\ + (D*TimeStep*0.5*(QuantumForceOld[i,j]-QuantumForceNew[i,j])-\ + PositionNew[i,j]+PositionOld[i,j]) + + GreensFunction = exp(GreensFunction) + ProbabilityRatio = GreensFunction*wfnew**2/wfold**2 + #Metropolis-Hastings test to see whether we accept the move + if random() <= ProbabilityRatio: + for j in range(Dimension): + PositionOld[i,j] = PositionNew[i,j] + QuantumForceOld[i,j] = QuantumForceNew[i,j] + wfold = wfnew + DeltaE = LocalEnergy(PositionOld,alpha,beta) + energy += DeltaE + if Printout: + outfile.write('%f\n' %(energy/(MCcycle+1.0))) + # We calculate mean values + energy /= NumberMCcycles + return energy + +#Here starts the main program with variable declarations +NumberParticles = 2 +Dimension = 2 +# seed for rng generator +seed() +# Monte Carlo cycles for parameter optimization +Printout = False +NumberMCcycles= 10000 +# guess for variational parameters +x0 = np.array([0.9,0.2]) +# Using Broydens method to find optimal parameters +res = minimize(Energy, x0, method='BFGS', jac=EnergyDerivative, options={'gtol': 1e-4,'disp': True}) +x0 = res.x +# Compute the energy again with the optimal parameters and increased number of Monte Cycles +NumberMCcycles= 2**19 +Printout = True +FinalEnergy = Energy(x0) +EResult = np.array([FinalEnergy,FinalEnergy]) +outfile.close() +#nice printout with Pandas +import pandas as pd +from pandas import DataFrame +data ={'Optimal Parameters':x0, 'Final Energy':EResult} +frame = pd.DataFrame(data) +print(frame) ++
+
+
+
+
+
+
+ +
Resampling analysis
+ +The next step is then to use the above data sets and perform a +resampling analysis using the blocking method +The blocking code, based on the article of Marius Jonsson is given here +
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+ # Common imports +import os + +# Where to save the figures and data files +DATA_ID = "Results/EnergyMin" + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +infile = open(data_path("Energies.dat"),'r') + +from numpy import log2, zeros, mean, var, sum, loadtxt, arange, array, cumsum, dot, transpose, diagonal, sqrt +from numpy.linalg import inv + +def block(x): + # preliminaries + n = len(x) + d = int(log2(n)) + s, gamma = zeros(d), zeros(d) + mu = mean(x) + + # estimate the auto-covariance and variances + # for each blocking transformation + for i in arange(0,d): + n = len(x) + # estimate autocovariance of x + gamma[i] = (n)**(-1)*sum( (x[0:(n-1)]-mu)*(x[1:n]-mu) ) + # estimate variance of x + s[i] = var(x) + # perform blocking transformation + x = 0.5*(x[0::2] + x[1::2]) + + # generate the test observator M_k from the theorem + M = (cumsum( ((gamma/s)**2*2**arange(1,d+1)[::-1])[::-1] ) )[::-1] + + # we need a list of magic numbers + q =array([6.634897,9.210340, 11.344867, 13.276704, 15.086272, 16.811894, 18.475307, 20.090235, 21.665994, 23.209251, 24.724970, 26.216967, 27.688250, 29.141238, 30.577914, 31.999927, 33.408664, 34.805306, 36.190869, 37.566235, 38.932173, 40.289360, 41.638398, 42.979820, 44.314105, 45.641683, 46.962942, 48.278236, 49.587884, 50.892181]) + + # use magic to determine when we should have stopped blocking + for k in arange(0,d): + if(M[k] < q[k]): + break + if (k >= d-1): + print("Warning: Use more data") + return mu, s[k]/2**(d-k) + + +x = loadtxt(infile) +(mean, var) = block(x) +std = sqrt(var) +import pandas as pd +from pandas import DataFrame +data ={'Mean':[mean], 'STDev':[std]} +frame = pd.DataFrame(data,index=['Values']) +print(frame) ++
+
+
+
+
+
+
+ +
Content
+-
+
- Simple compiler options +
- Tools to benchmark your code +
- Machine architectures +
- What is vectorization? +
- How to measure code performance +
- Parallelization with OpenMP +
- Parallelization with MPI +
- Vectorization and parallelization, examples +
+
Optimization and profiling
+
+
+
+
++ +
Till now we have not paid much attention to speed and possible optimization possibilities +inherent in the various compilers. We have compiled and linked as +
+ + +
+
+
+
+
+
+
+
+
+
+
+ c++ -c mycode.cpp +c++ -o mycode.exe mycode.o ++
+
+
+
+
+
+
+ For Fortran replace with for example gfortran or ifort. +This is what we call a flat compiler option and should be used when we develop the code. +It produces normally a very large and slow code when translated to machine instructions. +We use this option for debugging and for establishing the correct program output because +every operation is done precisely as the user specified it. +
+ +It is instructive to look up the compiler manual for further instructions by writing
+ + +
+
+
+
+
+
+
+
+
+
+ man c++ ++
+
+
+
+
+
+
+ +
More on optimization
+
+
+
+
++
We have additional compiler options for optimization. These may include procedure inlining where +performance may be improved, moving constants inside loops outside the loop, +identify potential parallelism, include automatic vectorization or replace a division with a reciprocal +and a multiplication if this speeds up the code. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ c++ -O3 -c mycode.cpp +c++ -O3 -o mycode.exe mycode.o ++
+
+
+
+
+
+
+ This (other options are -O2 or -Ofast) is the recommended option.
++
Optimization and profiling
+
+
+
+
+
++
It is also useful to profile your program under the development stage. +You would then compile with +
+ + +
+
+
+
+
+
+
+
+
+
+
+ c++ -pg -O3 -c mycode.cpp +c++ -pg -O3 -o mycode.exe mycode.o ++
+
+
+
+
+
+
+ After you have run the code you can obtain the profiling information via
+ + +
+
+
+
+
+
+
+
+
+
+
+ gprof mycode.exe > ProfileOutput ++
+
+
+
+
+
+
+ When you have profiled properly your code, you must take out this option as it +slows down performance. +For memory tests use valgrind. An excellent environment for all these aspects, and much more, is Qt creator. +
++
Optimization and debugging
+
+
+
+
++
Adding debugging options is a very useful alternative under the development stage of a program. +You would then compile with +
+ + +
+
+
+
+
+
+
+
+
+
+
+ c++ -g -O0 -c mycode.cpp +c++ -g -O0 -o mycode.exe mycode.o ++
+
+
+
+
+
+
+ This option generates debugging information allowing you to trace for example if an array is properly allocated. Some compilers work best with the no optimization option -O0.
+
+Other optimization flags
+
+
+
++
Depending on the compiler, one can add flags which generate code that catches integer overflow errors. +The flag -ftrapv does this for the CLANG compiler on OS X operating systems. +
++
Other hints
+
+
+
+
+
++
In general, irrespective of compiler options, it is useful to
+-
+
- avoid if tests or call to functions inside loops, if possible. +
- avoid multiplication with constants inside loops if possible +
Here is an example of a part of a program where specific operations lead to a slower code
+ + +
+
+
+
+
+
+
+
+
+
+
+ k = n-1; +for (i = 0; i < n; i++){ + a[i] = b[i] +c*d; + e = g[k]; +} ++
+
+
+
+
+
+
+ A better code is
+ + +
+
+
+
+
+
+
+
+
+
+
+ temp = c*d; +for (i = 0; i < n; i++){ + a[i] = b[i] + temp; +} +e = g[n-1]; ++
+
+
+
+
+
+
+ Here we avoid a repeated multiplication inside a loop. +Most compilers, depending on compiler flags, identify and optimize such bottlenecks on their own, without requiring any particular action by the programmer. However, it is always useful to single out and avoid code examples like the first one discussed here. +
++
Vectorization and the basic idea behind parallel computing
+
+
+
+
+
++
Present CPUs are highly parallel processors with varying levels of parallelism. The typical situation can be described via the following three statements.
+-
+
- Pursuit of shorter computation time and larger simulation size gives rise to parallel computing. +
- Multiple processors are involved to solve a global problem. +
- The essence is to divide the entire computation evenly among collaborative processors. Divide and conquer. +
Before we proceed with a more detailed discussion of topics like vectorization and parallelization, we need to remind ourselves about some basic features of different hardware models.
++
A rough classification of hardware models
+
+
+
+
++ +
-
+
- Conventional single-processor computers are named SISD (single-instruction-single-data) machines. +
- SIMD (single-instruction-multiple-data) machines incorporate the idea of parallel processing, using a large number of processing units to execute the same instruction on different data. +
- Modern parallel computers are so-called MIMD (multiple-instruction-multiple-data) machines and can execute different instruction streams in parallel on different data. +
+
Shared memory and distributed memory
+
+
+
+
+
++
One way of categorizing modern parallel computers is to look at the memory configuration.
+-
+
- In shared memory systems the CPUs share the same address space. Any CPU can access any data in the global memory. +
- In distributed memory systems each CPU has its own memory. +
The CPUs are connected by some network and may exchange messages.
++
Different parallel programming paradigms
+
+
+
+
++ +
-
+
- Task parallelism: the work of a global problem can be divided into a number of independent tasks, which rarely need to synchronize. Monte Carlo simulations represent a typical situation. Integration is another. However this paradigm is of limited use. +
- Data parallelism: use of multiple threads (e.g. one or more threads per processor) to dissect loops over arrays etc. Communication and synchronization between processors are often hidden, thus easy to program. However, the user surrenders much control to a specialized compiler. Examples of data parallelism are compiler-based parallelization and OpenMP directives. +
+
Different parallel programming paradigms
+
+
+
+
+
+
++ +
-
+
- Message passing: all involved processors have an independent memory address space. The user is responsible for partitioning the data/work of a global problem and distributing the subproblems to the processors. Collaboration between processors is achieved by explicit message passing, which is used for data transfer plus synchronization. +
- This paradigm is the most general one where the user has full control. Better parallel efficiency is usually achieved by explicit message passing. However, message-passing programming is more difficult. +
What is vectorization?
+Vectorization is a special +case of Single Instructions Multiple Data (SIMD) to denote a single +instruction stream capable of operating on multiple data elements in +parallel. +We can think of vectorization as the unrolling of loops accompanied with SIMD instructions. +
+ +Vectorization is the process of converting an algorithm that performs scalar operations +(typically one operation at the time) to vector operations where a single operation can refer to many simultaneous operations. +Consider the following example +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (i = 0; i < n; i++){ + a[i] = b[i] + c[i]; +} ++
+
+
+
+
+
+
+ If the code is not vectorized, the compiler will simply start with the first element and +then perform subsequent additions operating on one address in memory at the time. +
+ + +Number of elements that can acted upon
+A SIMD instruction can operate on multiple data elements in one single instruction. +It uses the so-called 128-bit SIMD floating-point register. +In this sense, vectorization adds some form of parallelism since one instruction is applied +to many parts of say a vector. +
+ +The number of elements which can be operated on in parallel +range from four single-precision floating point data elements in so-called +Streaming SIMD Extensions and two double-precision floating-point data +elements in Streaming SIMD Extensions 2 to sixteen byte operations in +a 128-bit register in Streaming SIMD Extensions 2. Thus, vector-length +ranges from 2 to 16, depending on the instruction extensions used and +on the data type. +
+ +IN summary, our instructions operate on 128 bit (16 byte) operands
+-
+
- 4 floats or ints +
- 2 doubles +
- Data paths 128 bits vide for vector unit +
Number of elements that can acted upon, examples
+We start with the simple scalar operations given by
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (i = 0; i < n; i++){ + a[i] = b[i] + c[i]; +} ++
+
+
+
+
+
+
+ If the code is not vectorized and we have a 128-bit register to store a 32 bits floating point number, +it means that we have \( 3\times 32 \) bits that are not used. +
+ +We have thus unused space in our SIMD registers. These registers could hold three additional integers.
+ + +Operation counts for scalar operation
+The code
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (i = 0; i < n; i++){ + a[i] = b[i] + c[i]; +} ++
+
+
+
+
+
+
+ has for \( n \) repeats
+-
+
- one load for \( c[i] \) in address 1 +
- one load for \( b[i] \) in address 2 +
- add \( c[i] \) and \( b[i] \) to give \( a[i] \) +
- store \( a[i] \) in address 2 +
Number of elements that can acted upon, examples
+If we vectorize the code, we can perform, with a 128-bit register four simultaneous operations, that is +we have +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (i = 0; i < n; i+=4){ + a[i] = b[i] + c[i]; + a[i+1] = b[i+1] + c[i+1]; + a[i+2] = b[i+2] + c[i+2]; + a[i+3] = b[i+3] + c[i+3]; +} ++
+
+
+
+
+
+
+ Four additions are now done in a single step.
+ + +Number of operations when vectorized
+For \( n/4 \) repeats assuming floats or integers
+-
+
- one vector load for \( c[i] \) in address 1 +
- one load for \( b[i] \) in address 2 +
- add \( c[i] \) and \( b[i] \) to give \( a[i] \) +
- store \( a[i] \) in address 2 +
+
A simple test case with and without vectorization
+We implement these operations in a simple c++ program that computes at the end the norm of a vector.
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+
+ #include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include "time.h"
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of square matrix
+ int n = atoi(argv[1]);
+ double s = 1.0/sqrt( (double) n);
+ double *a, *b, *c;
+ // Start timing
+ clock_t start, finish;
+ start = clock();
+// Allocate space for the vectors to be used
+ a = new double [n]; b = new double [n]; c = new double [n];
+ // Define parallel region
+ // Set up values for vectors a and b
+ for (int i = 0; i < n; i++){
+ double angle = 2.0*M_PI*i/ (( double ) n);
+ a[i] = s*(sin(angle) + cos(angle));
+ b[i] = s*sin(2.0*angle);
+ c[i] = 0.0;
+ }
+ // Then perform the vector addition
+ for (int i = 0; i < n; i++){
+ c[i] += a[i]+b[i];
+ }
+ // Compute now the norm-2
+ double Norm2 = 0.0;
+ for (int i = 0; i < n; i++){
+ Norm2 += c[i]*c[i];
+ }
+ finish = clock();
+ double timeused = (double) (finish - start)/(CLOCKS_PER_SEC );
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for norm computation=" << timeused << endl;
+ cout << " Norm-2 = " << Norm2 << endl;
+ // Free up space
+ delete[] a;
+ delete[] b;
+ delete[] c;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ Compiling with and without vectorization
+We can compile and link without vectorization using the clang c++ compiler
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang -o novec.x vecexample.cpp ++
+
+
+
+
+
+
+ and with vectorization (and additional optimizations)
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang++ -O3 -Rpass=loop-vectorize -o vec.x vecexample.cpp ++
+
+
+
+
+
+
+ The speedup depends on the size of the vectors. In the example here we have run with \( 10^7 \) elements. +The example here was run on an IMac17.1 with OSX El Capitan (10.11.4) as operating system and an Intel i5 3.3 GHz CPU. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ Compphys:~ hjensen$ ./vec.x 10000000 +Time used for norm computation=0.04720500000 +Compphys:~ hjensen$ ./novec.x 10000000 +Time used for norm computation=0.03311700000 ++
+
+
+
+
+
+
+ This particular C++ compiler speeds up the above loop operations with a factor of 1.5 +Performing the same operations for \( 10^9 \) elements results in a smaller speedup since reading from main memory is required. The non-vectorized code is seemingly faster. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ Compphys:~ hjensen$ ./vec.x 1000000000 +Time used for norm computation=58.41391100 +Compphys:~ hjensen$ ./novec.x 1000000000 +Time used for norm computation=46.51295300 ++
+
+
+
+
+
+
+ We will discuss these issues further in the next slides.
+ + +Compiling with and without vectorization using clang
+We can compile and link without vectorization with clang compiler
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang++ -o -fno-vectorize novec.x vecexample.cpp ++
+
+
+
+
+
+
+ and with vectorization
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang++ -O3 -Rpass=loop-vectorize -o vec.x vecexample.cpp ++
+
+
+
+
+
+
+ We can also add vectorization analysis, see for example
+ + +
+
+
+
+
+
+
+
+
+
+
+ clang++ -O3 -Rpass-analysis=loop-vectorize -o vec.x vecexample.cpp ++
+
+
+
+
+
+
+ or figure out if vectorization was missed
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ clang++ -O3 -Rpass-missed=loop-vectorize -o vec.x vecexample.cpp ++
+
+
+
+
+
+
+ +
Automatic vectorization and vectorization inhibitors, criteria
+ +Not all loops can be vectorized, as discussed in Intel's guide to vectorization
+ +An important criteria is that the loop counter \( n \) is known at the entry of the loop.
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int j = 0; j < n; j++) { + a[j] = cos(j*1.0); + } ++
+
+
+
+
+
+
+ The variable \( n \) does need to be known at compile time. However, this variable must stay the same for the entire duration of the loop. It implies that an exit statement inside the loop cannot be data dependent.
+ ++
Automatic vectorization and vectorization inhibitors, exit criteria
+ +An exit statement should in general be avoided. +If the exit statement contains data-dependent conditions, the loop cannot be vectorized. +The following is an example of a non-vectorizable loop +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int j = 0; j < n; j++) { + a[j] = cos(j*1.0); + if (a[j] < 0 ) break; + } ++
+
+
+
+
+
+
+ Avoid loop termination conditions and opt for a single entry loop variable \( n \). The lower and upper bounds have to be kept fixed within the loop.
+ ++
Automatic vectorization and vectorization inhibitors, straight-line code
+ +SIMD instructions perform the same type of operations multiple times. +A switch statement leads thus to a non-vectorizable loop since different statemens cannot branch. +The following code can however be vectorized since the if statement is implemented as a masked assignment. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int j = 0; j < n; j++) { + double x = cos(j*1.0); + if (x > 0 ) { + a[j] = x*sin(j*2.0); + } + else { + a[j] = 0.0; + } + } ++
+
+
+
+
+
+
+ These operations can be performed for all data elements but only those elements which the mask evaluates as true are stored. In general, one should avoid branches such as switch, go to, or return statements or if constructs that cannot be treated as masked assignments.
+ ++
Automatic vectorization and vectorization inhibitors, nested loops
+ +Only the innermost loop of the following example is vectorized
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int i = 0; i < n; i++) { + for (int j = 0; j < n; j++) { + a[i][j] += b[i][j]; + } + } ++
+
+
+
+
+
+
+ The exception is if an original outer loop is transformed into an inner loop as the result of compiler optimizations.
+ ++
Automatic vectorization and vectorization inhibitors, function calls
+ +Calls to programmer defined functions ruin vectorization. However, calls to intrinsic functions like +\( \sin{x} \), \( \cos{x} \), \( \exp{x} \) etc are allowed since they are normally efficiently vectorized. +The following example is fully vectorizable +
+ + +
+
+
+
+
+
+
+
+
+
+
+ for (int i = 0; i < n; i++) { + a[i] = log10(i)*cos(i); + } ++
+
+
+
+
+
+
+ Similarly, inline functions defined by the programmer, allow for vectorization since the function statements are glued into the actual place where the function is called.
+ ++
Automatic vectorization and vectorization inhibitors, data dependencies
+ +One has to keep in mind that vectorization changes the order of operations inside a loop. A so-called +read-after-write statement with an explicit flow dependency cannot be vectorized. The following code +
+ + +
+
+
+
+
+
+
+
+
+
+
+ double b = 15.; + for (int i = 1; i < n; i++) { + a[i] = a[i-1] + b; + } ++
+
+
+
+
+
+
+ is an example of flow dependency and results in wrong numerical results if vectorized. For a scalar operation, the value \( a[i-1] \) computed during the iteration is loaded into the right-hand side and the results are fine. In vector mode however, with a vector length of four, the values \( a[0] \), \( a[1] \), \( a[2] \) and \( a[3] \) from the previous loop will be loaded into the right-hand side and produce wrong results. That is, we have
+ + +
+
+
+
+
+
+
+
+
+
+
+ a[1] = a[0] + b; + a[2] = a[1] + b; + a[3] = a[2] + b; + a[4] = a[3] + b; ++
+
+
+
+
+
+
+ and if the two first iterations are executed at the same by the SIMD instruction, the value of say \( a[1] \) could be used by the second iteration before it has been calculated by the first iteration, leading thereby to wrong results.
+ ++
Automatic vectorization and vectorization inhibitors, more data dependencies
+ +On the other hand, a so-called +write-after-read statement can be vectorized. The following code +
+ + +
+
+
+
+
+
+
+
+
+
+
+ double b = 15.; + for (int i = 1; i < n; i++) { + a[i-1] = a[i] + b; + } ++
+
+
+
+
+
+
+ is an example of flow dependency that can be vectorized since no iteration with a higher value of \( i \) +can complete before an iteration with a lower value of \( i \). However, such code leads to problems with parallelization. +
+ ++
Automatic vectorization and vectorization inhibitors, memory stride
+ +For C++ programmers it is also worth keeping in mind that an array notation is preferred to the more compact use of pointers to access array elements. The compiler can often not tell if it is safe to vectorize the code.
+ +When dealing with arrays, you should also avoid memory stride, since this slows down considerably vectorization. When you access array element, write for example the inner loop to vectorize using unit stride, that is, access successively the next array element in memory, as shown here
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ for (int i = 0; i < n; i++) { + for (int j = 0; j < n; j++) { + a[i][j] += b[i][j]; + } + } ++
+
+
+
+
+
+
+ +
Memory management
+The main memory contains the program data
+-
+
- Cache memory contains a copy of the main memory data +
- Cache is faster but consumes more space and power. It is normally assumed to be much faster than main memory +
- Registers contain working data only +
- Modern CPUs perform most or all operations only on data in register +
- Multiple Cache memories contain a copy of the main memory data +
- Cache items accessed by their address in main memory +
- L1 cache is the fastest but has the least capacity +
- L2, L3 provide intermediate performance/size tradeoffs +
-
+
-
+
Loads and stores to memory can be as important as floating point operations when we measure performance.
+ ++
Memory and communication
+ +-
+
- Most communication in a computer is carried out in chunks, blocks of bytes of data that move together +
- In the memory hierarchy, data moves between memory and cache, and between different levels of cache, in groups called lines +
- Lines are typically 64-128 bytes, or 8-16 double precision words +
- Even if you do not use the data, it is moved and occupies space in the cache +
-
+
Many of these performance features are not captured in most programming languages.
+ ++
Measuring performance
+ +How do we measure performance? What is wrong with this code to time a loop?
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ clock_t start, finish;
+ start = clock();
+ for (int j = 0; j < i; j++) {
+ a[j] = b[j]+b[j]*c[j];
+ }
+ finish = clock();
+ double timeused = (double) (finish - start)/(CLOCKS_PER_SEC );
+
+
+
+
+
+
+
+
+ +
Problems with measuring time
+-
+
- Timers are not infinitely accurate +
- All clocks have a granularity, the minimum time that they can measure +
- The error in a time measurement, even if everything is perfect, may be the size of this granularity (sometimes called a clock tick) +
- Always know what your clock granularity is +
- Ensure that your measurement is for a long enough duration (say 100 times the tick) +
+
Problems with cold start
+ +What happens when the code is executed? The assumption is that the code is ready to +execute. But +
+-
+
- Code may still be on disk, and not even read into memory. +
- Data may be in slow memory rather than fast (which may be wrong or right for what you are measuring) +
- Multiple tests often necessary to ensure that cold start effects are not present +
- Special effort often required to ensure data in the intended part of the memory hierarchy. +
+
Problems with smart compilers
+ +-
+
- If the result of the computation is not used, the compiler may eliminate the code +
- Performance will look impossibly fantastic +
- Even worse, eliminate some of the code so the performance looks plausible +
- Ensure that the results are (or may be) used. +
+
Problems with interference
+-
+
- Other activities are sharing your processor +
- Operating system, system demons, other users +
- Some parts of the hardware do not always perform with exactly the same performance +
- Make multiple tests and report +
- Easy choices include +
- Average tests represent what users might observe over time +
-
+
-
+
+
Problems with measuring performance
+-
+
- Accurate, reproducible performance measurement is hard +
- Think carefully about your experiment: +
- What is it, precisely, that you want to measure? +
- How representative is your test to the situation that you are trying to measure? +
+
Thomas algorithm for tridiagonal linear algebra equations
+
+
+
+
+
++$$ +\left( \begin{array}{ccccc} + b_0 & c_0 & & & \\ + a_0 & b_1 & c_1 & & \\ + & & \ddots & & \\ + & & a_{m-3} & b_{m-2} & c_{m-2} \\ + & & & a_{m-2} & b_{m-1} + \end{array} \right) +\left( \begin{array}{c} + x_0 \\ + x_1 \\ + \vdots \\ + x_{m-2} \\ + x_{m-1} + \end{array} \right)=\left( \begin{array}{c} + f_0 \\ + f_1 \\ + \vdots \\ + f_{m-2} \\ + f_{m-1} \\ + \end{array} \right) +$$ +
+
Thomas algorithm, forward substitution
+
+
+
+
+
++
The first step is to multiply the first row by \( a_0/b_0 \) and subtract it from the second row. This is known as the forward substitution step. We obtain then
+$$ + a_i = 0, +$$ + + +$$ + b_i = b_i - \frac{a_{i-1}}{b_{i-1}}c_{i-1}, +$$ + +and
+$$ + f_i = f_i - \frac{a_{i-1}}{b_{i-1}}f_{i-1}. +$$ + +At this point the simplified equation, with only an upper triangular matrix takes the form
+$$ +\left( \begin{array}{ccccc} + b_0 & c_0 & & & \\ + & b_1 & c_1 & & \\ + & & \ddots & & \\ + & & & b_{m-2} & c_{m-2} \\ + & & & & b_{m-1} + \end{array} \right)\left( \begin{array}{c} + x_0 \\ + x_1 \\ + \vdots \\ + x_{m-2} \\ + x_{m-1} + \end{array} \right)=\left( \begin{array}{c} + f_0 \\ + f_1 \\ + \vdots \\ + f_{m-2} \\ + f_{m-1} \\ + \end{array} \right) +$$ ++
Thomas algorithm, backward substitution
+
+
+
+
+
++
The next step is the backward substitution step. The last row is multiplied by \( c_{N-3}/b_{N-2} \) and subtracted from the second to last row, thus eliminating \( c_{N-3} \) from the last row. The general backward substitution procedure is
+$$ + c_i = 0, +$$ + +and
+$$ + f_{i-1} = f_{i-1} - \frac{c_{i-1}}{b_i}f_i +$$ + +All that ramains to be computed is the solution, which is the very straight forward process of
+$$ +x_i = \frac{f_i}{b_i} +$$ ++
Thomas algorithm and counting of operations (floating point and memory)
+
+
+
+
+
++ +
We have in specific case the following operations with the floating operations
+ +-
+
- Memory Reads: \( 14(N-2) \); +
- Memory Writes: \( 4(N-2) \); +
- Subtractions: \( 3(N-2) \); +
- Multiplications: \( 3(N-2) \); +
- Divisions: \( 4(N-2) \). +
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ // Forward substitution +// Note that we can simplify by precalculating a[i-1]/b[i-1] + for (int i=1; i < n; i++) { + b[i] = b[i] - (a[i-1]*c[i-1])/b[i-1]; + f[i] = g[i] - (a[i-1]*f[i-1])/b[i-1]; + } + x[n-1] = f[n-1] / b[n-1]; + // Backwards substitution + for (int i = n-2; i >= 0; i--) { + f[i] = f[i] - c[i]*f[i+1]/b[i+1]; + x[i] = f[i]/b[i]; + } ++
+
+
+
+
+
+
+ +
Example: Transpose of a matrix
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+ #include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include "time.h"
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of square matrix
+ int n = atoi(argv[1]);
+ double **A, **B;
+ // Allocate space for the two matrices
+ A = new double*[n]; B = new double*[n];
+ for (int i = 0; i < n; i++){
+ A[i] = new double[n];
+ B[i] = new double[n];
+ }
+ // Set up values for matrix A
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ A[i][j] = cos(i*1.0)*sin(j*3.0);
+ }
+ }
+ clock_t start, finish;
+ start = clock();
+ // Then compute the transpose
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ B[i][j]= A[j][i];
+ }
+ }
+
+ finish = clock();
+ double timeused = (double) (finish - start)/(CLOCKS_PER_SEC );
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for setting up transpose of matrix=" << timeused << endl;
+
+ // Free up space
+ for (int i = 0; i < n; i++){
+ delete[] A[i];
+ delete[] B[i];
+ }
+ delete[] A;
+ delete[] B;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ +
Matrix-matrix multiplication
+This the matrix-matrix multiplication code with plain c++ memory allocation. It computes at the end the Frobenius norm.
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+ #include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include "time.h"
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of square matrix
+ int n = atoi(argv[1]);
+ double s = 1.0/sqrt( (double) n);
+ double **A, **B, **C;
+ // Start timing
+ clock_t start, finish;
+ start = clock();
+ // Allocate space for the two matrices
+ A = new double*[n]; B = new double*[n]; C = new double*[n];
+ for (int i = 0; i < n; i++){
+ A[i] = new double[n];
+ B[i] = new double[n];
+ C[i] = new double[n];
+ }
+ // Set up values for matrix A and B and zero matrix C
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ double angle = 2.0*M_PI*i*j/ (( double ) n);
+ A[i][j] = s * ( sin ( angle ) + cos ( angle ) );
+ B[j][i] = A[i][j];
+ }
+ }
+ // Then perform the matrix-matrix multiplication
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ double sum = 0.0;
+ for (int k = 0; k < n; k++) {
+ sum += B[i][k]*A[k][j];
+ }
+ C[i][j] = sum;
+ }
+ }
+ // Compute now the Frobenius norm
+ double Fsum = 0.0;
+ for (int i = 0; i < n; i++){
+ for (int j = 0; j < n; j++) {
+ Fsum += C[i][j]*C[i][j];
+ }
+ }
+ Fsum = sqrt(Fsum);
+ finish = clock();
+ double timeused = (double) (finish - start)/(CLOCKS_PER_SEC );
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for matrix-matrix multiplication=" << timeused << endl;
+ cout << " Frobenius norm = " << Fsum << endl;
+ // Free up space
+ for (int i = 0; i < n; i++){
+ delete[] A[i];
+ delete[] B[i];
+ delete[] C[i];
+ }
+ delete[] A;
+ delete[] B;
+ delete[] C;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ +
How do we define speedup? Simplest form
+
+
+
+
+
++
-
+
- Speedup measures the ratio of performance between two objects +
- Versions of same code, with different number of processors +
- Serial and vector versions +
- Try different programing languages, c++ and Fortran +
- Two algorithms computing the same result +
+
How do we define speedup? Correct baseline
+
+
+
+
+
++
The key is choosing the correct baseline for comparison
+-
+
- For our serial vs. vectorization examples, using compiler-provided vectorization, the baseline is simple; the same code, with vectorization turned off +
- For parallel applications, this is much harder: +
- Choice of algorithm, decomposition, performance of baseline case etc. +
-
+
-
+
+
Parallel speedup
+
+
+
+
+
++
For parallel applications, speedup is typically defined as
+-
+
- Speedup \( =T_1/T_p \) +
Here \( T_1 \) is the time on one processor and \( T_p \) is the time using \( p \) processors.
+-
+
- Can the speedup become larger than \( p \)? That means using \( p \) processors is more than \( p \) times faster than using one processor. +
+
Speedup and memory
+
+
+
+
+
++
The speedup on \( p \) processors can +be greater than \( p \) if memory usage is optimal! +Consider the case of a memorybound computation with \( M \) words of memory +
+-
+
- If \( M/p \) fits into cache while \( M \) does not, the time to access memory will be different in the two cases: +
- \( T_1 \) uses the main memory bandwidth +
- \( T_p \) uses the appropriate cache bandwidth +
+
Upper bounds on speedup
+
+
+
+
+
++
Assume that almost all parts of a code are perfectly +parallelizable (fraction \( f \)). The remainder, +fraction \( (1-f) \) cannot be parallelized at all. +
+ +That is, there is work that takes time \( W \) on one process; a fraction \( f \) of that work will take +time \( Wf/p \) on \( p \) processors. +
+-
+
- What is the maximum possible speedup as a function of \( f \)? +
+
Amdahl's law
+
+
+
+
+
++
On one processor we have
+$$ +T_1 = (1-f)W + fW = W +$$ + +On \( p \) processors we have
+$$ +T_p = (1-f)W + \frac{fW}{p}, +$$ + +resulting in a speedup of
+$$ +\frac{T_1}{T_p} = \frac{W}{(1-f)W+fW/p} +$$ + +As \( p \) goes to infinity, \( fW/p \) goes to zero, and the maximum speedup is
+$$ +\frac{1}{1-f}, +$$ + +meaning that if +if \( f = 0.99 \) (all but \( 1\% \) parallelizable), the maximum speedup +is \( 1/(1-.99)=100 \)! +
++
How much is parallelizable
+
+
+
+
+
++
If any non-parallel code slips into the +application, the parallel +performance is limited. +
+ +In many simulations, however, the fraction of non-parallelizable work +is \( 10^{-6} \) or less due to large arrays or objects that are perfectly parallelizable. +
++
Today's situation of parallel computing
+
+
+
+
+
++ +
-
+
- Distributed memory is the dominant hardware configuration. There is a large diversity in these machines, from MPP (massively parallel processing) systems to clusters of off-the-shelf PCs, which are very cost-effective. +
- Message-passing is a mature programming paradigm and widely accepted. It often provides an efficient match to the hardware. It is primarily used for the distributed memory systems, but can also be used on shared memory systems. +
- Modern nodes have nowadays several cores, which makes it interesting to use both shared memory (the given node) and distributed memory (several nodes with communication). This leads often to codes which use both MPI and OpenMP. +
Our lectures will focus on both MPI and OpenMP.
++
Overhead present in parallel computing
+
+
+
+
+
++ +
-
+
- Uneven load balance: not all the processors can perform useful work at all time. +
- Overhead of synchronization +
- Overhead of communication +
- Extra computation due to parallelization +
Due to the above overhead and that certain parts of a sequential +algorithm cannot be parallelized we may not achieve an optimal parallelization. +
++
Parallelizing a sequential algorithm
+
+
+
+
+
++ +
-
+
- Identify the part(s) of a sequential algorithm that can be executed in parallel. This is the difficult part, +
- Distribute the global work and data among \( P \) processors. +
+
Strategies
+
+
+
+
+
++
-
+
- Develop codes locally, run with some few processes and test your codes. Do benchmarking, timing and so forth on local nodes, for example your laptop or PC. +
- When you are convinced that your codes run correctly, you can start your production runs on available supercomputers. +
+
How do I run MPI on a PC/Laptop? MPI
+
+
+
+
+
++
To install MPI is rather easy on hardware running unix/linux as operating systems, follow simply the instructions from the OpenMPI website. See also subsequent slides. +When you have made sure you have installed MPI on your PC/laptop, +
+-
+
- Compile with mpicxx/mpic++ or mpif90 +
+
+
+
+
+
+
+
+
+
+ # Compile and link + mpic++ -O3 -o nameofprog.x nameofprog.cpp + # run code with for example 8 processes using mpirun/mpiexec + mpiexec -n 8 ./nameofprog.x ++
+
+
+
+
+
+
+ +
Can I do it on my own PC/laptop? OpenMP installation
+
+
+
+
+
++
If you wish to install MPI and OpenMP +on your laptop/PC, we recommend the following: +
+ +-
+
- For OpenMP, the compile option -fopenmp is included automatically in recent versions of the C++ compiler and Fortran compilers. For users of different Linux distributions, simply use the available C++ or Fortran compilers and add the above compiler instructions, see also code examples below. +
- For OS X users however, install libomp +
+
+
+
+
+
+
+
+
+
+
+ brew install libomp ++
+
+
+
+
+
+
+ and compile and link as
+ + +
+
+
+
+
+
+
+
+
+
+ c++ -o <name executable> <name program.cpp> -lomp ++
+
+
+
+
+
+
+ +
Installing MPI
+
+
+
+
+
++
For linux/ubuntu users, you need to install two packages (alternatively use the synaptic package manager)
+ + +
+
+
+
+
+
+
+
+
+
+
+ sudo apt-get install libopenmpi-dev + sudo apt-get install openmpi-bin ++
+
+
+
+
+
+
+ For OS X users, install brew (after having installed xcode and gcc, needed for the +gfortran compiler of openmpi) and then install with brew +
+ + +
+
+
+
+
+
+
+
+
+
+
+ brew install openmpi ++
+
+
+
+
+
+
+ When running an executable (code.x), run as
+ + +
+
+
+
+
+
+
+
+
+
+
+ mpirun -n 10 ./code.x ++
+
+
+
+
+
+
+ where we indicate that we want the number of processes to be 10.
++
Installing MPI and using Qt
+
+
+
+
+
++
With openmpi installed, when using Qt, add to your .pro file the instructions here
+ +You may need to tell Qt where openmpi is stored.
++
What is Message Passing Interface (MPI)?
+
+
+
+
++ +
MPI is a library, not a language. It specifies the names, calling sequences and results of functions +or subroutines to be called from C/C++ or Fortran programs, and the classes and methods that make up the MPI C++ +library. The programs that users write in Fortran, C or C++ are compiled with ordinary compilers and linked +with the MPI library. +
+ +MPI programs should be able to run +on all possible machines and run all MPI implementetations without change. +
+ +An MPI computation is a collection of processes communicating with messages.
++
Going Parallel with MPI
+
+
+
+
+
++
Task parallelism: the work of a global problem can be divided +into a number of independent tasks, which rarely need to synchronize. +Monte Carlo simulations or numerical integration are examples of this. +
+ +MPI is a message-passing library where all the routines +have corresponding C/C++-binding +
+ + +
+
+
+
+
+
+
+
+
+
+
+ MPI_Command_name
+
+
+
+
+
+
+
+
+ and Fortran-binding (routine names are in uppercase, but can also be in lower case)
+ + +
+
+
+
+
+
+
+
+
+
+ MPI_COMMAND_NAME ++
+
+
+
+
+
+
+ +
MPI is a library
+
+
+
+
+
++
MPI is a library specification for the message passing interface, +proposed as a standard. +
+ +-
+
- independent of hardware; +
- not a language or compiler specification; +
- not a specific implementation or product. +
A message passing standard for portability and ease-of-use. +Designed for high performance. +
+ +Insert communication and synchronization functions where necessary.
++
Bindings to MPI routines
+
+
+
+
+
++ +
MPI is a message-passing library where all the routines +have corresponding C/C++-binding +
+ + +
+
+
+
+
+
+
+
+
+
+
+ MPI_Command_name
+
+
+
+
+
+
+
+
+ and Fortran-binding (routine names are in uppercase, but can also be in lower case)
+ + +
+
+
+
+
+
+
+
+
+
+
+ MPI_COMMAND_NAME ++
+
+
+
+
+
+
+ The discussion in these slides focuses on the C++ binding.
++
Communicator
+
+
+
+
+
++
-
+
- A group of MPI processes with a name (context). +
- Any process is identified by its rank. The rank is only meaningful within a particular communicator. +
- By default the communicator contains all the MPI processes. +
+
+
+
+
+
+
+
+
+
+
+ MPI_COMM_WORLD ++
+
+
+
+
+
+
+ -
+
- Mechanism to identify subset of processes. +
- Promotes modular design of parallel libraries. +
+
Some of the most important MPI functions
+
+
+
+
+
++ +
-
+
- \( MPI\_Init \) - initiate an MPI computation +
- \( MPI\_Finalize \) - terminate the MPI computation and clean up +
- \( MPI\_Comm\_size \) - how many processes participate in a given MPI communicator? +
- \( MPI\_Comm\_rank \) - which one am I? (A number between 0 and size-1.) +
- \( MPI\_Send \) - send a message to a particular process within an MPI communicator +
- \( MPI\_Recv \) - receive a message from a particular process within an MPI communicator +
- \( MPI\_reduce \) or \( MPI\_Allreduce \), send and receive messages +
+
The first MPI C/C++ program
+
+
+
+
+
++ +
Let every process write "Hello world" (oh not this program again!!) on the standard output.
+ + +
+
+
+
+
+
+
+
+
+
+ using namespace std; +#include <mpi.h> +#include <iostream> +int main (int nargs, char* args[]) +{ +int numprocs, my_rank; +// MPI initializations +MPI_Init (&nargs, &args); +MPI_Comm_size (MPI_COMM_WORLD, &numprocs); +MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); +cout << "Hello world, I have rank " << my_rank << " out of " + << numprocs << endl; +// End MPI +MPI_Finalize (); ++
+
+
+
+
+
+
+ +
The Fortran program
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ PROGRAM hello +INCLUDE "mpif.h" +INTEGER:: size, my_rank, ierr + +CALL MPI_INIT(ierr) +CALL MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierr) +CALL MPI_COMM_RANK(MPI_COMM_WORLD, my_rank, ierr) +WRITE(*,*)"Hello world, I've rank ",my_rank," out of ",size +CALL MPI_FINALIZE(ierr) + +END PROGRAM hello ++
+
+
+
+
+
+
+ +
Note 1
+
+
+
+
+
++ +
-
+
- The output to screen is not ordered since all processes are trying to write to screen simultaneously. +
- It is the operating system which opts for an ordering. +
- If we wish to have an organized output, starting from the first process, we may rewrite our program as in the next example. +
+
Ordered output with MPIBarrier
+
+
+
+
+
++ + + +
+
+
+
+
+
+
+
+
+
+ int main (int nargs, char* args[]) +{ + int numprocs, my_rank, i; + MPI_Init (&nargs, &args); + MPI_Comm_size (MPI_COMM_WORLD, &numprocs); + MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); + for (i = 0; i < numprocs; i++) {} + MPI_Barrier (MPI_COMM_WORLD); + if (i == my_rank) { + cout << "Hello world, I have rank " << my_rank << + " out of " << numprocs << endl;} + MPI_Finalize (); ++
+
+
+
+
+
+
+ +
Note 2
+
+
+
+
+
++
-
+
- Here we have used the \( MPI\_Barrier \) function to ensure that that every process has completed its set of instructions in a particular order. +
- A barrier is a special collective operation that does not allow the processes to continue until all processes in the communicator (here \( MPI\_COMM\_WORLD \)) have called \( MPI\_Barrier \). +
- The barriers make sure that all processes have reached the same point in the code. Many of the collective operations like \( MPI\_ALLREDUCE \) to be discussed later, have the same property; that is, no process can exit the operation until all processes have started. +
However, this is slightly more time-consuming since the processes synchronize between themselves as many times as there +are processes. In the next Hello world example we use the send and receive functions in order to a have a synchronized +action. +
++
Ordered output
+
+
+
+
+
++ + + +
+
+
+
+
+
+
+
+
+
+ ..... +int numprocs, my_rank, flag; +MPI_Status status; +MPI_Init (&nargs, &args); +MPI_Comm_size (MPI_COMM_WORLD, &numprocs); +MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); +if (my_rank > 0) +MPI_Recv (&flag, 1, MPI_INT, my_rank-1, 100, + MPI_COMM_WORLD, &status); +cout << "Hello world, I have rank " << my_rank << " out of " +<< numprocs << endl; +if (my_rank < numprocs-1) +MPI_Send (&my_rank, 1, MPI_INT, my_rank+1, + 100, MPI_COMM_WORLD); +MPI_Finalize (); ++
+
+
+
+
+
+
+ +
Note 3
+
+
+
+
+
++ +
The basic sending of messages is given by the function \( MPI\_SEND \), which in C/C++ +is defined as +
+ + +
+
+
+
+
+
+
+
+
+
+
+ int MPI_Send(void *buf, int count, + MPI_Datatype datatype, + int dest, int tag, MPI_Comm comm)} ++
+
+
+
+
+
+
+ This single command allows the passing of any kind of variable, even a large array, to any group of tasks. +The variable buf is the variable we wish to send while count +is the number of variables we are passing. If we are passing only a single value, this should be 1. +
+ +If we transfer an array, it is the overall size of the array. +For example, if we want to send a 10 by 10 array, count would be \( 10\times 10=100 \) +since we are actually passing 100 values. +
++
Note 4
+
+
+
+
+
++ +
Once you have sent a message, you must receive it on another task. The function \( MPI\_RECV \) +is similar to the send call. +
+ + +
+
+
+
+
+
+
+
+
+
+
+ int MPI_Recv( void *buf, int count, MPI_Datatype datatype, + int source, + int tag, MPI_Comm comm, MPI_Status *status ) ++
+
+
+
+
+
+
+ The arguments that are different from those in MPI\_SEND are +buf which is the name of the variable where you will be storing the received data, +source which replaces the destination in the send command. This is the return ID of the sender. +
+ +Finally, we have used \( MPI\_Status\_status \), +where one can check if the receive was completed. +
+ +The output of this code is the same as the previous example, but now +process 0 sends a message to process 1, which forwards it further +to process 2, and so forth. +
++
Numerical integration in parallel
+
+Integrating \( \pi \)
+
+
+
++ +
-
+
- The code example computes \( \pi \) using the trapezoidal rules. +
- The trapezoidal rule +
Click on this link for the full program.
++
Dissection of trapezoidal rule with \( MPI\_reduce \)
+
+
+
+
+
++ + + +
+
+
+
+
+
+
+
+
+
+ // Trapezoidal rule and numerical integration usign MPI +using namespace std; +#include <mpi.h> +#include <iostream> + +// Here we define various functions called by the main program + +double int_function(double ); +double trapezoidal_rule(double , double , int , double (*)(double)); + +// Main function begins here +int main (int nargs, char* args[]) +{ + int n, local_n, numprocs, my_rank; + double a, b, h, local_a, local_b, total_sum, local_sum; + double time_start, time_end, total_time; ++
+
+
+
+
+
+
+ +
Dissection of trapezoidal rule
+
+
+
+
+
++ + + +
+
+
+
+
+
+
+
+
+
+ // MPI initializations + MPI_Init (&nargs, &args); + MPI_Comm_size (MPI_COMM_WORLD, &numprocs); + MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); + time_start = MPI_Wtime(); + // Fixed values for a, b and n + a = 0.0 ; b = 1.0; n = 1000; + h = (b-a)/n; // h is the same for all processes + local_n = n/numprocs; + // make sure n > numprocs, else integer division gives zero + // Length of each process' interval of + // integration = local_n*h. + local_a = a + my_rank*local_n*h; + local_b = local_a + local_n*h; ++
+
+
+
+
+
+
+ +
Integrating with MPI
+
+
+
+
+
++ + + +
+
+
+
+
+
+
+
+
+
+ total_sum = 0.0; + local_sum = trapezoidal_rule(local_a, local_b, local_n, + &int_function); + MPI_Reduce(&local_sum, &total_sum, 1, MPI_DOUBLE, + MPI_SUM, 0, MPI_COMM_WORLD); + time_end = MPI_Wtime(); + total_time = time_end-time_start; + if ( my_rank == 0) { + cout << "Trapezoidal rule = " << total_sum << endl; + cout << "Time = " << total_time + << " on number of processors: " << numprocs << endl; + } + // End MPI + MPI_Finalize (); + return 0; +} // end of main program ++
+
+
+
+
+
+
+ +
How do I use \( MPI\_reduce \)?
+
+
+
+
+
++ +
Here we have used
+ + +
+
+
+
+
+
+
+
+
+
+
+ MPI_reduce( void *senddata, void* resultdata, int count, + MPI_Datatype datatype, MPI_Op, int root, MPI_Comm comm) ++
+
+
+
+
+
+
+ The two variables \( senddata \) and \( resultdata \) are obvious, besides the fact that one sends the address +of the variable or the first element of an array. If they are arrays they need to have the same size. +The variable \( count \) represents the total dimensionality, 1 in case of just one variable, +while \( MPI\_Datatype \) +defines the type of variable which is sent and received. +
+ +The new feature is \( MPI\_Op \). It defines the type +of operation we want to do. +
++
More on \( MPI\_Reduce \)
+
+
+
+
+
++
In our case, since we are summing +the rectangle contributions from every process we define \( MPI\_Op = MPI\_SUM \). +If we have an array or matrix we can search for the largest og smallest element by sending either \( MPI\_MAX \) or +\( MPI\_MIN \). If we want the location as well (which array element) we simply transfer +\( MPI\_MAXLOC \) or \( MPI\_MINOC \). If we want the product we write \( MPI\_PROD \). +
+ +\( MPI\_Allreduce \) is defined as
+ + +
+
+
+
+
+
+
+
+
+
+ MPI_Allreduce( void *senddata, void* resultdata, int count, + MPI_Datatype datatype, MPI_Op, MPI_Comm comm) ++
+
+
+
+
+
+
+ +
Dissection of trapezoidal rule
+
+
+
+
+
++ +
We use \( MPI\_reduce \) to collect data from each process. Note also the use of the function +\( MPI\_Wtime \). +
+ + +
+
+
+
+
+
+
+
+
+
+ // this function defines the function to integrate +double int_function(double x) +{ + double value = 4./(1.+x*x); + return value; +} // end of function to evaluate ++
+
+
+
+
+
+
+ +
Dissection of trapezoidal rule
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ // this function defines the trapezoidal rule +double trapezoidal_rule(double a, double b, int n, + double (*func)(double)) +{ + double trapez_sum; + double fa, fb, x, step; + int j; + step=(b-a)/((double) n); + fa=(*func)(a)/2. ; + fb=(*func)(b)/2. ; + trapez_sum=0.; + for (j=1; j <= n-1; j++){ + x=j*step+a; + trapez_sum+=(*func)(x); + } + trapez_sum=(trapez_sum+fb+fa)*step; + return trapez_sum; +} // end trapezoidal_rule ++
+
+
+
+
+
+
+ +
The quantum dot program for two electrons
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ // Variational Monte Carlo for atoms with importance sampling, slater det +// Test case for 2-electron quantum dot, no classes using Mersenne-Twister RNG +#include "mpi.h" +#include <cmath> +#include <random> +#include <string> +#include <iostream> +#include <fstream> +#include <iomanip> +#include "vectormatrixclass.h" + +using namespace std; +// output file as global variable +ofstream ofile; +// the step length and its squared inverse for the second derivative +// Here we define global variables used in various functions +// These can be changed by using classes +int Dimension = 2; +int NumberParticles = 2; // we fix also the number of electrons to be 2 + +// declaration of functions + +// The Mc sampling for the variational Monte Carlo +void MonteCarloSampling(int, double &, double &, Vector &); + +// The variational wave function +double WaveFunction(Matrix &, Vector &); + +// The local energy +double LocalEnergy(Matrix &, Vector &); + +// The quantum force +void QuantumForce(Matrix &, Matrix &, Vector &); + + +// inline function for single-particle wave function +inline double SPwavefunction(double r, double alpha) { + return exp(-alpha*r*0.5); +} + +// inline function for derivative of single-particle wave function +inline double DerivativeSPwavefunction(double r, double alpha) { + return -r*alpha; +} + +// function for absolute value of relative distance +double RelativeDistance(Matrix &r, int i, int j) { + double r_ij = 0; + for (int k = 0; k < Dimension; k++) { + r_ij += (r(i,k)-r(j,k))*(r(i,k)-r(j,k)); + } + return sqrt(r_ij); +} + +// inline function for derivative of Jastrow factor +inline double JastrowDerivative(Matrix &r, double beta, int i, int j, int k){ + return (r(i,k)-r(j,k))/(RelativeDistance(r, i, j)*pow(1.0+beta*RelativeDistance(r, i, j),2)); +} + +// function for square of position of single particle +double singleparticle_pos2(Matrix &r, int i) { + double r_single_particle = 0; + for (int j = 0; j < Dimension; j++) { + r_single_particle += r(i,j)*r(i,j); + } + return r_single_particle; +} + +void lnsrch(int n, Vector &xold, double fold, Vector &g, Vector &p, Vector &x, + double *f, double stpmax, int *check, double (*func)(Vector &p)); + +void dfpmin(Vector &p, int n, double gtol, int *iter, double *fret, + double(*func)(Vector &p), void (*dfunc)(Vector &p, Vector &g)); + +static double sqrarg; +#define SQR(a) ((sqrarg=(a)) == 0.0 ? 0.0 : sqrarg*sqrarg) + + +static double maxarg1,maxarg2; +#define FMAX(a,b) (maxarg1=(a),maxarg2=(b),(maxarg1) > (maxarg2) ?\ + (maxarg1) : (maxarg2)) + + +// Begin of main program + +int main(int argc, char* argv[]) +{ + + // MPI initializations + int NumberProcesses, MyRank, NumberMCsamples; + MPI_Init (&argc, &argv); + MPI_Comm_size (MPI_COMM_WORLD, &NumberProcesses); + MPI_Comm_rank (MPI_COMM_WORLD, &MyRank); + double StartTime = MPI_Wtime(); + if (MyRank == 0 && argc <= 1) { + cout << "Bad Usage: " << argv[0] << + " Read also output file on same line and number of Monte Carlo cycles" << endl; + } + // Read filename and number of Monte Carlo cycles from the command line + if (MyRank == 0 && argc > 2) { + string filename = argv[1]; // first command line argument after name of program + NumberMCsamples = atoi(argv[2]); + string fileout = filename; + string argument = to_string(NumberMCsamples); + // Final filename as filename+NumberMCsamples + fileout.append(argument); + ofile.open(fileout); + } + // broadcast the number of Monte Carlo samples + MPI_Bcast (&NumberMCsamples, 1, MPI_INT, 0, MPI_COMM_WORLD); + // Two variational parameters only + Vector VariationalParameters(2); + int TotalNumberMCsamples = NumberMCsamples*NumberProcesses; + // Loop over variational parameters + for (double alpha = 0.5; alpha <= 1.5; alpha +=0.1){ + for (double beta = 0.1; beta <= 0.5; beta +=0.05){ + VariationalParameters(0) = alpha; // value of alpha + VariationalParameters(1) = beta; // value of beta + // Do the mc sampling and accumulate data with MPI_Reduce + double TotalEnergy, TotalEnergySquared, LocalProcessEnergy, LocalProcessEnergy2; + LocalProcessEnergy = LocalProcessEnergy2 = 0.0; + MonteCarloSampling(NumberMCsamples, LocalProcessEnergy, LocalProcessEnergy2, VariationalParameters); + // Collect data in total averages + MPI_Reduce(&LocalProcessEnergy, &TotalEnergy, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); + MPI_Reduce(&LocalProcessEnergy2, &TotalEnergySquared, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); + // Print out results in case of Master node, set to MyRank = 0 + if ( MyRank == 0) { + double Energy = TotalEnergy/( (double)NumberProcesses); + double Variance = TotalEnergySquared/( (double)NumberProcesses)-Energy*Energy; + double StandardDeviation = sqrt(Variance/((double)TotalNumberMCsamples)); // over optimistic error + ofile << setiosflags(ios::showpoint | ios::uppercase); + ofile << setw(15) << setprecision(8) << VariationalParameters(0); + ofile << setw(15) << setprecision(8) << VariationalParameters(1); + ofile << setw(15) << setprecision(8) << Energy; + ofile << setw(15) << setprecision(8) << Variance; + ofile << setw(15) << setprecision(8) << StandardDeviation << endl; + } + } + } + double EndTime = MPI_Wtime(); + double TotalTime = EndTime-StartTime; + if ( MyRank == 0 ) cout << "Time = " << TotalTime << " on number of processors: " << NumberProcesses << endl; + if (MyRank == 0) ofile.close(); // close output file + // End MPI + MPI_Finalize (); + return 0; +} // end of main function + + +// Monte Carlo sampling with the Metropolis algorithm + +void MonteCarloSampling(int NumberMCsamples, double &cumulative_e, double &cumulative_e2, Vector &VariationalParameters) +{ + + // Initialize the seed and call the Mersienne algo + std::random_device rd; + std::mt19937_64 gen(rd()); + // Set up the uniform distribution for x \in [[0, 1] + std::uniform_real_distribution<double> UniformNumberGenerator(0.0,1.0); + std::normal_distribution<double> Normaldistribution(0.0,1.0); + // diffusion constant from Schroedinger equation + double D = 0.5; + double timestep = 0.05; // we fix the time step for the gaussian deviate + // allocate matrices which contain the position of the particles + Matrix OldPosition( NumberParticles, Dimension), NewPosition( NumberParticles, Dimension); + Matrix OldQuantumForce(NumberParticles, Dimension), NewQuantumForce(NumberParticles, Dimension); + double Energy = 0.0; double EnergySquared = 0.0; double DeltaE = 0.0; + // initial trial positions + for (int i = 0; i < NumberParticles; i++) { + for (int j = 0; j < Dimension; j++) { + OldPosition(i,j) = Normaldistribution(gen)*sqrt(timestep); + } + } + double OldWaveFunction = WaveFunction(OldPosition, VariationalParameters); + QuantumForce(OldPosition, OldQuantumForce, VariationalParameters); + // loop over monte carlo cycles + for (int cycles = 1; cycles <= NumberMCsamples; cycles++){ + // new position + for (int i = 0; i < NumberParticles; i++) { + for (int j = 0; j < Dimension; j++) { + // gaussian deviate to compute new positions using a given timestep + NewPosition(i,j) = OldPosition(i,j) + Normaldistribution(gen)*sqrt(timestep)+OldQuantumForce(i,j)*timestep*D; + // NewPosition(i,j) = OldPosition(i,j) + gaussian_deviate(&idum)*sqrt(timestep)+OldQuantumForce(i,j)*timestep*D; + } + // for the other particles we need to set the position to the old position since + // we move only one particle at the time + for (int k = 0; k < NumberParticles; k++) { + if ( k != i) { + for (int j = 0; j < Dimension; j++) { + NewPosition(k,j) = OldPosition(k,j); + } + } + } + double NewWaveFunction = WaveFunction(NewPosition, VariationalParameters); + QuantumForce(NewPosition, NewQuantumForce, VariationalParameters); + // we compute the log of the ratio of the greens functions to be used in the + // Metropolis-Hastings algorithm + double GreensFunction = 0.0; + for (int j = 0; j < Dimension; j++) { + GreensFunction += 0.5*(OldQuantumForce(i,j)+NewQuantumForce(i,j))* + (D*timestep*0.5*(OldQuantumForce(i,j)-NewQuantumForce(i,j))-NewPosition(i,j)+OldPosition(i,j)); + } + GreensFunction = exp(GreensFunction); + // The Metropolis test is performed by moving one particle at the time + if(UniformNumberGenerator(gen) <= GreensFunction*NewWaveFunction*NewWaveFunction/OldWaveFunction/OldWaveFunction ) { + for (int j = 0; j < Dimension; j++) { + OldPosition(i,j) = NewPosition(i,j); + OldQuantumForce(i,j) = NewQuantumForce(i,j); + } + OldWaveFunction = NewWaveFunction; + } + } // end of loop over particles + // compute local energy + double DeltaE = LocalEnergy(OldPosition, VariationalParameters); + // update energies + Energy += DeltaE; + EnergySquared += DeltaE*DeltaE; + } // end of loop over MC trials + // update the energy average and its squared + cumulative_e = Energy/NumberMCsamples; + cumulative_e2 = EnergySquared/NumberMCsamples; +} // end MonteCarloSampling function + + +// Function to compute the squared wave function and the quantum force + +double WaveFunction(Matrix &r, Vector &VariationalParameters) +{ + double wf = 0.0; + // full Slater determinant for two particles, replace with Slater det for more particles + wf = SPwavefunction(singleparticle_pos2(r, 0), VariationalParameters(0))*SPwavefunction(singleparticle_pos2(r, 1),VariationalParameters(0)); + // contribution from Jastrow factor + for (int i = 0; i < NumberParticles-1; i++) { + for (int j = i+1; j < NumberParticles; j++) { + wf *= exp(RelativeDistance(r, i, j)/((1.0+VariationalParameters(1)*RelativeDistance(r, i, j)))); + } + } + return wf; +} + +// Function to calculate the local energy without numerical derivation of kinetic energy + +double LocalEnergy(Matrix &r, Vector &VariationalParameters) +{ + + // compute the kinetic and potential energy from the single-particle part + // for a many-electron system this has to be replaced by a Slater determinant + // The absolute value of the interparticle length + Matrix length( NumberParticles, NumberParticles); + // Set up interparticle distance + for (int i = 0; i < NumberParticles-1; i++) { + for(int j = i+1; j < NumberParticles; j++){ + length(i,j) = RelativeDistance(r, i, j); + length(j,i) = length(i,j); + } + } + double KineticEnergy = 0.0; + // Set up kinetic energy from Slater and Jastrow terms + for (int i = 0; i < NumberParticles; i++) { + for (int k = 0; k < Dimension; k++) { + double sum1 = 0.0; + for(int j = 0; j < NumberParticles; j++){ + if ( j != i) { + sum1 += JastrowDerivative(r, VariationalParameters(1), i, j, k); + } + } + KineticEnergy += (sum1+DerivativeSPwavefunction(r(i,k),VariationalParameters(0)))*(sum1+DerivativeSPwavefunction(r(i,k),VariationalParameters(0))); + } + } + KineticEnergy += -2*VariationalParameters(0)*NumberParticles; + for (int i = 0; i < NumberParticles-1; i++) { + for (int j = i+1; j < NumberParticles; j++) { + KineticEnergy += 2.0/(pow(1.0 + VariationalParameters(1)*length(i,j),2))*(1.0/length(i,j)-2*VariationalParameters(1)/(1+VariationalParameters(1)*length(i,j)) ); + } + } + KineticEnergy *= -0.5; + // Set up potential energy, external potential + eventual electron-electron repulsion + double PotentialEnergy = 0; + for (int i = 0; i < NumberParticles; i++) { + double DistanceSquared = singleparticle_pos2(r, i); + PotentialEnergy += 0.5*DistanceSquared; // sp energy HO part, note it has the oscillator frequency set to 1! + } + // Add the electron-electron repulsion + for (int i = 0; i < NumberParticles-1; i++) { + for (int j = i+1; j < NumberParticles; j++) { + PotentialEnergy += 1.0/length(i,j); + } + } + double LocalE = KineticEnergy+PotentialEnergy; + return LocalE; +} + +// Compute the analytical expression for the quantum force +void QuantumForce(Matrix &r, Matrix &qforce, Vector &VariationalParameters) +{ + // compute the first derivative + for (int i = 0; i < NumberParticles; i++) { + for (int k = 0; k < Dimension; k++) { + // single-particle part, replace with Slater det for larger systems + double sppart = DerivativeSPwavefunction(r(i,k),VariationalParameters(0)); + // Jastrow factor contribution + double Jsum = 0.0; + for (int j = 0; j < NumberParticles; j++) { + if ( j != i) { + Jsum += JastrowDerivative(r, VariationalParameters(1), i, j, k); + } + } + qforce(i,k) = 2.0*(Jsum+sppart); + } + } +} // end of QuantumForce function + + +#define ITMAX 200 +#define EPS 3.0e-8 +#define TOLX (4*EPS) +#define STPMX 100.0 + +void dfpmin(Vector &p, int n, double gtol, int *iter, double *fret, + double(*func)(Vector &p), void (*dfunc)(Vector &p, Vector &g)) +{ + + int check,i,its,j; + double den,fac,fad,fae,fp,stpmax,sum=0.0,sumdg,sumxi,temp,test; + Vector dg(n), g(n), hdg(n), pnew(n), xi(n); + Matrix hessian(n,n); + + fp=(*func)(p); + (*dfunc)(p,g); + for (i = 0;i < n;i++) { + for (j = 0; j< n;j++) hessian(i,j)=0.0; + hessian(i,i)=1.0; + xi(i) = -g(i); + sum += p(i)*p(i); + } + stpmax=STPMX*FMAX(sqrt(sum),(double)n); + for (its=1;its<=ITMAX;its++) { + *iter=its; + lnsrch(n,p,fp,g,xi,pnew,fret,stpmax,&check,func); + fp = *fret; + for (i = 0; i< n;i++) { + xi(i)=pnew(i)-p(i); + p(i)=pnew(i); + } + test=0.0; + for (i = 0;i< n;i++) { + temp=fabs(xi(i))/FMAX(fabs(p(i)),1.0); + if (temp > test) test=temp; + } + if (test < TOLX) { + return; + } + for (i=0;i<n;i++) dg(i)=g(i); + (*dfunc)(p,g); + test=0.0; + den=FMAX(*fret,1.0); + for (i=0;i<n;i++) { + temp=fabs(g(i))*FMAX(fabs(p(i)),1.0)/den; + if (temp > test) test=temp; + } + if (test < gtol) { + return; + } + for (i=0;i<n;i++) dg(i)=g(i)-dg(i); + for (i=0;i<n;i++) { + hdg(i)=0.0; + for (j=0;j<n;j++) hdg(i) += hessian(i,j)*dg(j); + } + fac=fae=sumdg=sumxi=0.0; + for (i=0;i<n;i++) { + fac += dg(i)*xi(i); + fae += dg(i)*hdg(i); + sumdg += SQR(dg(i)); + sumxi += SQR(xi(i)); + } + if (fac*fac > EPS*sumdg*sumxi) { + fac=1.0/fac; + fad=1.0/fae; + for (i=0;i<n;i++) dg(i)=fac*xi(i)-fad*hdg(i); + for (i=0;i<n;i++) { + for (j=0;j<n;j++) { + hessian(i,j) += fac*xi(i)*xi(j) + -fad*hdg(i)*hdg(j)+fae*dg(i)*dg(j); + } + } + } + for (i=0;i<n;i++) { + xi(i)=0.0; + for (j=0;j<n;j++) xi(i) -= hessian(i,j)*g(j); + } + } + cout << "too many iterations in dfpmin" << endl; +} +#undef ITMAX +#undef EPS +#undef TOLX +#undef STPMX + +#define ALF 1.0e-4 +#define TOLX 1.0e-7 + +void lnsrch(int n, Vector &xold, double fold, Vector &g, Vector &p, Vector &x, + double *f, double stpmax, int *check, double (*func)(Vector &p)) +{ + int i; + double a,alam,alam2,alamin,b,disc,f2,fold2,rhs1,rhs2,slope,sum,temp, + test,tmplam; + + *check=0; + for (sum=0.0,i=0;i<n;i++) sum += p(i)*p(i); + sum=sqrt(sum); + if (sum > stpmax) + for (i=0;i<n;i++) p(i) *= stpmax/sum; + for (slope=0.0,i=0;i<n;i++) + slope += g(i)*p(i); + test=0.0; + for (i=0;i<n;i++) { + temp=fabs(p(i))/FMAX(fabs(xold(i)),1.0); + if (temp > test) test=temp; + } + alamin=TOLX/test; + alam=1.0; + for (;;) { + for (i=0;i<n;i++) x(i)=xold(i)+alam*p(i); + *f=(*func)(x); + if (alam < alamin) { + for (i=0;i<n;i++) x(i)=xold(i); + *check=1; + return; + } else if (*f <= fold+ALF*alam*slope) return; + else { + if (alam == 1.0) + tmplam = -slope/(2.0*(*f-fold-slope)); + else { + rhs1 = *f-fold-alam*slope; + rhs2=f2-fold2-alam2*slope; + a=(rhs1/(alam*alam)-rhs2/(alam2*alam2))/(alam-alam2); + b=(-alam2*rhs1/(alam*alam)+alam*rhs2/(alam2*alam2))/(alam-alam2); + if (a == 0.0) tmplam = -slope/(2.0*b); + else { + disc=b*b-3.0*a*slope; + if (disc<0.0) cout << "Roundoff problem in lnsrch." << endl; + else tmplam=(-b+sqrt(disc))/(3.0*a); + } + if (tmplam>0.5*alam) + tmplam=0.5*alam; + } + } + alam2=alam; + f2 = *f; + fold2=fold; + alam=FMAX(tmplam,0.1*alam); + } +} +#undef ALF +#undef TOLX ++
+
+
+
+
+
+
+ +
What is OpenMP
+
+
+
+
+
++
-
+
- OpenMP provides high-level thread programming +
- Multiple cooperating threads are allowed to run simultaneously +
- Threads are created and destroyed dynamically in a fork-join pattern +
- An OpenMP program consists of a number of parallel regions +
- Between two parallel regions there is only one master thread +
- In the beginning of a parallel region, a team of new threads is spawned +
- The newly spawned threads work simultaneously with the master thread +
- At the end of a parallel region, the new threads are destroyed +
-
+
Many good tutorials online and excellent textbook
+-
+
- Using OpenMP, by B. Chapman, G. Jost, and A. van der Pas +
- Many tutorials online like OpenMP official site +
+
Getting started, things to remember
+
+
+
+
+
++
-
+
- Remember the header file +
+
+
+
+
+
+
+
+
+
+
+ #include <omp.h> ++
+
+
+
+
+
+
+ -
+
- Insert compiler directives in C++ syntax as +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp...
+
+
+
+
+
+
+
+
+ -
+
- Compile with for example c++ -fopenmp code.cpp +
- Execute +
- Remember to assign the environment variable OMP NUM THREADS +
- It specifies the total number of threads inside a parallel region, if not otherwise overwritten +
-
+
+
OpenMP syntax
+-
+
- Mostly directives +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp construct [ clause ...]
+
+
+
+
+
+
+
+
+ -
+
- Some functions and types +
+
+
+
+
+
+
+
+
+
+
+ #include <omp.h> ++
+
+
+
+
+
+
+ -
+
- Most apply to a block of code +
- Specifically, a structured block +
- Enter at top, exit at bottom only, exit(), abort() permitted +
+
Different OpenMP styles of parallelism
+OpenMP supports several different ways to specify thread parallelism
+ +-
+
- General parallel regions: All threads execute the code, roughly as if you made a routine of that region and created a thread to run that code +
- Parallel loops: Special case for loops, simplifies data parallel code +
- Task parallelism, new in OpenMP 3 +
- Several ways to manage thread coordination, including Master regions and Locks +
- Memory model for shared data +
+
General code structure
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ #include <omp.h> +main () +{ +int var1, var2, var3; +/* serial code */ +/* ... */ +/* start of a parallel region */ +#pragma omp parallel private(var1, var2) shared(var3) +{ +/* ... */ +} +/* more serial code */ +/* ... */ +/* another parallel region */ +#pragma omp parallel +{ +/* ... */ +} +} ++
+
+
+
+
+
+
+ +
Parallel region
+
+
+
+
+
++
-
+
- A parallel region is a block of code that is executed by a team of threads +
- The following compiler directive creates a parallel region +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel { ... }
+
+
+
+
+
+
+
+
+ -
+
- Clauses can be added at the end of the directive +
- Most often used clauses: +
- default(shared) or default(none) +
- public(list of variables) +
- private(list of variables) +
-
+
+
Hello world, not again, please!
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ #include <omp.h> +#include <cstdio> +int main (int argc, char *argv[]) +{ +int th_id, nthreads; +#pragma omp parallel private(th_id) shared(nthreads) +{ +th_id = omp_get_thread_num(); +printf("Hello World from thread %d\n", th_id); +#pragma omp barrier +if ( th_id == 0 ) { +nthreads = omp_get_num_threads(); +printf("There are %d threads\n",nthreads); +} +} +return 0; +} ++
+
+
+
+
+
+
+ +
Hello world, yet another variant
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+
+ #include <cstdio> +#include <omp.h> +int main(int argc, char *argv[]) +{ + omp_set_num_threads(4); +#pragma omp parallel + { + int id = omp_get_thread_num(); + int nproc = omp_get_num_threads(); + cout << "Hello world with id number and processes " << id << nproc << endl; + } +return 0; +} ++
+
+
+
+
+
+
+ Variables declared outside of the parallel region are shared by all threads +If a variable like id is declared outside of the +
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel,
+
+
+
+
+
+
+
+
+ it would have been shared by various the threads, possibly causing erroneous output
+-
+
- Why? What would go wrong? Why do we add possibly? +
+
Important OpenMP library routines
+
+
+
+
+
++ +
-
+
- int omp get num threads (), returns the number of threads inside a parallel region +
- int omp get thread num (), returns the a thread for each thread inside a parallel region +
- void omp set num threads (int), sets the number of threads to be used +
- void omp set nested (int), turns nested parallelism on/off +
+
Private variables
+
+
+
+
+
++
Private clause can be used to make thread- private versions of such variables:
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel private(id) +{ + int id = omp_get_thread_num(); + cout << "My thread num" << id << endl; +} ++
+
+
+
+
+
+
+ -
+
- What is their value on entry? Exit? +
- OpenMP provides ways to control that +
- Can use default(none) to require the sharing of each variable to be described +
+
Master region
+
+
+
+
+
++
It is often useful to have only one thread execute some of the code in a parallel region. I/O statements are a common example
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel +{ + #pragma omp master + { + int id = omp_get_thread_num(); + cout << "My thread num" << id << endl; + } +} ++
+
+
+
+
+
+
+ +
Parallel for loop
+
+
+
+
+
++
-
+
- Inside a parallel region, the following compiler directive can be used to parallelize a for-loop: +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp for
+
+
+
+
+
+
+
+
+ -
+
- Clauses can be added, such as +
- schedule(static, chunk size) +
- schedule(dynamic, chunk size) +
- schedule(guided, chunk size) (non-deterministic allocation) +
- schedule(runtime) +
- private(list of variables) +
- reduction(operator:variable) +
- nowait +
-
+
+
Parallel computations and loops
+ +
+
+
+
+
++
OpenMP provides an easy way to parallelize a loop
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for + for (i=0; i<n; i++) c[i] = a[i]; ++
+
+
+
+
+
+
+ OpenMP handles index variable (no need to declare in for loop or make private)
+ +Which thread does which values? Several options.
++
Scheduling of loop computations
+ +
+
+
+
+
++
We can let the OpenMP runtime decide. The decision is about how the loop iterates are scheduled +and OpenMP defines three choices of loop scheduling: +
+-
+
- Static: Predefined at compile time. Lowest overhead, predictable +
- Dynamic: Selection made at runtime +
- Guided: Special case of dynamic; attempts to reduce overhead +
+
Example code for loop scheduling
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ #include <omp.h> +#define CHUNKSIZE 100 +#define N 1000 +int main (int argc, char *argv[]) +{ +int i, chunk; +float a[N], b[N], c[N]; +for (i=0; i < N; i++) a[i] = b[i] = i * 1.0; +chunk = CHUNKSIZE; +#pragma omp parallel shared(a,b,c,chunk) private(i) +{ +#pragma omp for schedule(dynamic,chunk) +for (i=0; i < N; i++) c[i] = a[i] + b[i]; +} /* end of parallel region */ +} ++
+
+
+
+
+
+
+ +
Example code for loop scheduling, guided instead of dynamic
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ #include <omp.h> +#define CHUNKSIZE 100 +#define N 1000 +int main (int argc, char *argv[]) +{ +int i, chunk; +float a[N], b[N], c[N]; +for (i=0; i < N; i++) a[i] = b[i] = i * 1.0; +chunk = CHUNKSIZE; +#pragma omp parallel shared(a,b,c,chunk) private(i) +{ +#pragma omp for schedule(guided,chunk) +for (i=0; i < N; i++) c[i] = a[i] + b[i]; +} /* end of parallel region */ +} ++
+
+
+
+
+
+
+ +
More on Parallel for loop
+
+
+
+
+
++
-
+
- The number of loop iterations cannot be non-deterministic; break, return, exit, goto not allowed inside the for-loop +
- The loop index is private to each thread +
- A reduction variable is special +
- During the for-loop there is a local private copy in each thread +
- At the end of the for-loop, all the local copies are combined together by the reduction operation +
- Unless the nowait clause is used, an implicit barrier synchronization will be added at the end by the compiler +
-
+
+
+
+
+
+
+
+
+
+
+
+ // #pragma omp parallel and #pragma omp for
+
+
+
+
+
+
+
+
+ can be combined into
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for
+
+
+
+
+
+
+
+
+ +
What can happen with this loop?
+ +
+
+
+
+
++
What happens with code like this
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for +for (i=0; i<n; i++) sum += a[i]*a[i]; ++
+
+
+
+
+
+
+ All threads can access the sum variable, but the addition is not atomic! It is important to avoid race between threads. So-called reductions in OpenMP are thus important for performance and for obtaining correct results. OpenMP lets us indicate that a variable is used for a reduction with a particular operator. The above code becomes
+ + +
+
+
+
+
+
+
+
+
+
+ sum = 0.0; +#pragma omp parallel for reduction(+:sum) +for (i=0; i<n; i++) sum += a[i]*a[i]; ++
+
+
+
+
+
+
+ +
Inner product
+
+
+
+
+
++$$ +\sum_{i=0}^{n-1} a_ib_i +$$ + + + +
+
+
+
+
+
+
+
+
+
+ int i; +double sum = 0.; +/* allocating and initializing arrays */ +/* ... */ +#pragma omp parallel for default(shared) private(i) reduction(+:sum) + for (i=0; i<N; i++) sum += a[i]*b[i]; +} ++
+
+
+
+
+
+
+ +
Different threads do different tasks
+
+
+
+
+
++ +
Different threads do different tasks independently, each section is executed by one thread.
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel +{ +#pragma omp sections +{ +#pragma omp section +funcA (); +#pragma omp section +funcB (); +#pragma omp section +funcC (); +} +} ++
+
+
+
+
+
+
+ +
Single execution
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp single { ... }
+
+
+
+
+
+
+
+
+ The code is executed by one thread only, no guarantee which thread
+ +Can introduce an implicit barrier at the end
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp master { ... }
+
+
+
+
+
+
+
+
+ Code executed by the master thread, guaranteed and no implicit barrier at the end.
++
Coordination and synchronization
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp barrier
+
+
+
+
+
+
+
+
+ Synchronization, must be encountered by all threads in a team (or none)
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp ordered { a block of codes }
+
+
+
+
+
+
+
+
+ is another form of synchronization (in sequential order). +The form +
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp critical { a block of codes }
+
+
+
+
+
+
+
+
+ and
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp atomic { single assignment statement }
+
+
+
+
+
+
+
+
+ is more efficient than
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp critical
+
+
+
+
+
+
+
+
+ +
Data scope
+
+
+
+
+
++
-
+
- OpenMP data scope attribute clauses: +
- shared +
- private +
- firstprivate +
- lastprivate +
- reduction +
-
+
What are the purposes of these attributes
+-
+
- define how and which variables are transferred to a parallel region (and back) +
- define which variables are visible to all threads in a parallel region, and which variables are privately allocated to each thread +
+
Some remarks
+
+
+
+
+
++ +
-
+
- When entering a parallel region, the private clause ensures each thread having its own new variable instances. The new variables are assumed to be uninitialized. +
- A shared variable exists in only one memory location and all threads can read and write to that address. It is the programmer's responsibility to ensure that multiple threads properly access a shared variable. +
- The firstprivate clause combines the behavior of the private clause with automatic initialization. +
- The lastprivate clause combines the behavior of the private clause with a copy back (from the last loop iteration or section) to the original variable outside the parallel region. +
+
Parallelizing nested for-loops
+
+
+
+
+
++ +
-
+
- Serial code +
+
+
+
+
+
+
+
+
+
+
+
+ for (i=0; i<100; i++) + for (j=0; j<100; j++) + a[i][j] = b[i][j] + c[i][j]; + } +} ++
+
+
+
+
+
+
+ -
+
- Parallelization +
+
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for private(j) +for (i=0; i<100; i++) + for (j=0; j<100; j++) + a[i][j] = b[i][j] + c[i][j]; + } +} ++
+
+
+
+
+
+
+ -
+
- Why not parallelize the inner loop? to save overhead of repeated thread forks-joins +
- Why must j be private? To avoid race condition among the threads +
+
Nested parallelism
+
+
+
+
+
++
When a thread in a parallel region encounters another parallel construct, it +may create a new team of threads and become the master of the new +team. +
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel num_threads(4) +{ +/* .... */ +#pragma omp parallel num_threads(2) +{ +// +} +} ++
+
+
+
+
+
+
+ +
Parallel tasks
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp task +#pragma omp parallel shared(p_vec) private(i) +{ +#pragma omp single +{ +for (i=0; i<N; i++) { + double r = random_number(); + if (p_vec[i] > r) { +#pragma omp task + do_work (p_vec[i]); ++
+
+
+
+
+
+
+ +
Common mistakes
+
+
+
+
+
+
++
Race condition
+ + +
+
+
+
+
+
+
+
+
+
+
+ int nthreads; +#pragma omp parallel shared(nthreads) +{ +nthreads = omp_get_num_threads(); +} ++
+
+
+
+
+
+
+ Deadlock
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel +{ +... +#pragma omp critical +{ +... +#pragma omp barrier +} +} ++
+
+
+
+
+
+
+ Not all computations are simple
+
+
+
+
+
+
++
Not all computations are simple loops where the data can be evenly +divided among threads without any dependencies between threads +
+ +An example is finding the location and value of the largest element in an array
+ + +
+
+
+
+
+
+
+
+
+
+ for (i=0; i<n; i++) { + if (x[i] > maxval) { + maxval = x[i]; + maxloc = i; + } +} ++
+
+
+
+
+
+
+ Not all computations are simple, competing threads
+
+
+
+
+
++
All threads are potentially accessing and changing the same values, maxloc and maxval.
+-
+
- OpenMP provides several ways to coordinate access to shared values +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp atomic
+
+
+
+
+
+
+
+
+ -
+
- Only one thread at a time can execute the following statement (not block). We can use the critical option +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp critical
+
+
+
+
+
+
+
+
+ -
+
- Only one thread at a time can execute the following block +
Atomic may be faster than critical but depends on hardware
++
How to find the max value using OpenMP
+
+
+
+
+
++
Write down the simplest algorithm and look carefully for race conditions. How would you handle them? +The first step would be to parallelize as +
+ + +
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for + for (i=0; i<n; i++) { + if (x[i] > maxval) { + maxval = x[i]; + maxloc = i; + } +} ++
+
+
+
+
+
+
+ +
Then deal with the race conditions
+
+
+
+
++
Write down the simplest algorithm and look carefully for race conditions. How would you handle them? +The first step would be to parallelize as +
+ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp parallel for + for (i=0; i<n; i++) { +#pragma omp critical + { + if (x[i] > maxval) { + maxval = x[i]; + maxloc = i; + } + } +} ++
+
+
+
+
+
+
+ Exercise: write a code which implements this and give an estimate on performance. Perform several runs, +with a serial code only with and without vectorization and compare the serial code with the one that uses OpenMP. Run on different archictectures if you can. +
++
What can slow down OpenMP performance?
+Give it a thought!
+ ++
What can slow down OpenMP performance?
+
+
+
+
+
++
Performance poor because we insisted on keeping track of the maxval and location during the execution of the loop.
+-
+
- We do not care about the value during the execution of the loop, just the value at the end. +
This is a common source of performance issues, namely the description of the method used to compute a value imposes additional, unnecessary requirements or properties
+ +Idea: Have each thread find the maxloc in its own data, then combine and use temporary arrays indexed by thread number to hold the values found by each thread ++
Find the max location for each thread
+
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+ int maxloc[MAX_THREADS], mloc; +double maxval[MAX_THREADS], mval; +#pragma omp parallel shared(maxval,maxloc) +{ + int id = omp_get_thread_num(); + maxval[id] = -1.0e30; +#pragma omp for + for (int i=0; i<n; i++) { + if (x[i] > maxval[id]) { + maxloc[id] = i; + maxval[id] = x[i]; + } + } +} ++
+
+
+
+
+
+
+ +
Combine the values from each thread
+
+
+
+
++ + +
+
+
+
+
+
+
+
+
+
+
+ #pragma omp flush (maxloc,maxval) +#pragma omp master + { + int nt = omp_get_num_threads(); + mloc = maxloc[0]; + mval = maxval[0]; + for (int i=1; i<nt; i++) { + if (maxval[i] > mval) { + mval = maxval[i]; + mloc = maxloc[i]; + } + } + } ++
+
+
+
+
+
+
+ Note that we let the master process perform the last operation.
++
Matrix-matrix multiplication
+This code computes the norm of a vector using OpenMp
+ + +
+
+
+
+
+
+
+
+
+
+
+
+ // OpenMP program to compute vector norm by adding two other vectors
+#include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include <omp.h>
+# include <ctime>
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of vector
+ int n = atoi(argv[1]);
+ double *a, *b, *c;
+ int i;
+ int thread_num;
+ double wtime, Norm2, s, angle;
+ cout << " Perform addition of two vectors and compute the norm-2." << endl;
+ omp_set_num_threads(4);
+ thread_num = omp_get_max_threads ();
+ cout << " The number of processors available = " << omp_get_num_procs () << endl ;
+ cout << " The number of threads available = " << thread_num << endl;
+ cout << " The matrix order n = " << n << endl;
+
+ s = 1.0/sqrt( (double) n);
+ wtime = omp_get_wtime ( );
+ // Allocate space for the vectors to be used
+ a = new double [n]; b = new double [n]; c = new double [n];
+ // Define parallel region
+# pragma omp parallel for default(shared) private (angle, i) reduction(+:Norm2)
+ // Set up values for vectors a and b
+ for (i = 0; i < n; i++){
+ angle = 2.0*M_PI*i/ (( double ) n);
+ a[i] = s*(sin(angle) + cos(angle));
+ b[i] = s*sin(2.0*angle);
+ c[i] = 0.0;
+ }
+ // Then perform the vector addition
+ for (i = 0; i < n; i++){
+ c[i] += a[i]+b[i];
+ }
+ // Compute now the norm-2
+ Norm2 = 0.0;
+ for (i = 0; i < n; i++){
+ Norm2 += c[i]*c[i];
+ }
+// end parallel region
+ wtime = omp_get_wtime ( ) - wtime;
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for norm-2 computation=" << wtime << endl;
+ cout << " Norm-2 = " << Norm2 << endl;
+ // Free up space
+ delete[] a;
+ delete[] b;
+ delete[] c;
+ return 0;
+}
+
+
+
+
+
+
+
+
+ +
Matrix-matrix multiplication
+This the matrix-matrix multiplication code with plain c++ memory allocation using OpenMP
+ + + +
+
+
+
+
+
+
+
+
+
+
+
+
+ // Matrix-matrix multiplication and Frobenius norm of a matrix with OpenMP
+#include <cstdlib>
+#include <iostream>
+#include <cmath>
+#include <iomanip>
+#include <omp.h>
+# include <ctime>
+
+using namespace std; // note use of namespace
+int main (int argc, char* argv[])
+{
+ // read in dimension of square matrix
+ int n = atoi(argv[1]);
+ double **A, **B, **C;
+ int i, j, k;
+ int thread_num;
+ double wtime, Fsum, s, angle;
+ cout << " Compute matrix product C = A * B and Frobenius norm." << endl;
+ omp_set_num_threads(4);
+ thread_num = omp_get_max_threads ();
+ cout << " The number of processors available = " << omp_get_num_procs () << endl ;
+ cout << " The number of threads available = " << thread_num << endl;
+ cout << " The matrix order n = " << n << endl;
+
+ s = 1.0/sqrt( (double) n);
+ wtime = omp_get_wtime ( );
+ // Allocate space for the two matrices
+ A = new double*[n]; B = new double*[n]; C = new double*[n];
+ for (i = 0; i < n; i++){
+ A[i] = new double[n];
+ B[i] = new double[n];
+ C[i] = new double[n];
+ }
+ // Define parallel region
+# pragma omp parallel for default(shared) private (angle, i, j, k) reduction(+:Fsum)
+ // Set up values for matrix A and B and zero matrix C
+ for (i = 0; i < n; i++){
+ for (j = 0; j < n; j++) {
+ angle = 2.0*M_PI*i*j/ (( double ) n);
+ A[i][j] = s * ( sin ( angle ) + cos ( angle ) );
+ B[j][i] = A[i][j];
+ }
+ }
+ // Then perform the matrix-matrix multiplication
+ for (i = 0; i < n; i++){
+ for (j = 0; j < n; j++) {
+ C[i][j] = 0.0;
+ for (k = 0; k < n; k++) {
+ C[i][j] += A[i][k]*B[k][j];
+ }
+ }
+ }
+ // Compute now the Frobenius norm
+ Fsum = 0.0;
+ for (i = 0; i < n; i++){
+ for (j = 0; j < n; j++) {
+ Fsum += C[i][j]*C[i][j];
+ }
+ }
+ Fsum = sqrt(Fsum);
+// end parallel region and letting only one thread perform I/O
+ wtime = omp_get_wtime ( ) - wtime;
+ cout << setiosflags(ios::showpoint | ios::uppercase);
+ cout << setprecision(10) << setw(20) << "Time used for matrix-matrix multiplication=" << wtime << endl;
+ cout << " Frobenius norm = " << Fsum << endl;
+ // Free up space
+ for (int i = 0; i < n; i++){
+ delete[] A[i];
+ delete[] B[i];
+ delete[] C[i];
+ }
+ delete[] A;
+ delete[] B;
+ delete[] C;
+ return 0;
+}
+
+
+
+
+
+
+
+
+