Adding Ember documentation

src/sst/elements/ember/Makefile.am

@@ -12,6 +12,8 @@ compdir = $(pkglibdir)
 comp_LTLIBRARIES = libember.la
 
 libember_la_SOURCES = \
+	example/motifs/emberexample.h
+	example/motifs/emberexample.cc
 	embergen.h \
 	embergen.cc	\
 	embermap.h \

src/sst/elements/ember/example/CreatingMotifs.md

@@ -0,0 +1,116 @@
+# C++ Ember Motifs
+## Ember
+
+[Ember](http://sst-simulator.org/SSTPages/SSTElementEmber/) is a library representing various network communications. It is often used in conjunction with motifs.
+
+## Motif
+
+Motifs are condensed, efficient generators for simulating different kinds of communication/computation patterns.
+The motif presented here does no real work, but more detailed motifs can be found in sst-elements/src/sst/elements/ember/mpi/motifs/.
+
+Motifs are executed as follows:
+
+1) The motif generator is initialized (The constructor)
+2) The generate function is invoked, places events on the event queue, and returns either true or false
+3) The events on the event queue are processed.
+4) If the generate function in step 2 returns false, go to step 2. Otherwise, the motif is complete.
+
+Motifs are intended to be run as a 'job submission.'
+The generate function models an entire iteration of an application, using the event queue to mix compute and MPI operations in every iteration.
+
+### Register Subcomponent
+
+Ember motifs need to be registered as an SST subcomponent. This typically happens in the header file.
+For example:
+```
+SST_ELI_REGISTER_SUBCOMPONENT_DERIVED(
+        Example,
+        "ember",
+        "ExampleMotif",
+        SST_ELI_ELEMENT_VERSION(1,0,0),
+        "Performs an idle on the node. No traffic can be generated.",
+        SST::Ember::EmberGenerator
+    );
+```
+
+The parameters of this function are:
+
+1) class associated with this Motif
+3) Aspect of elements being used, in the case of Motifs, "ember."
+3) Identifier of the Motif. Note that the end of this parameter must always be Motif
+4) SST elements version,
+5) Comment describing what the motif does
+6) EmberGenerator
+
+### Writing a constructor
+```
+EmberExample::EmberExample(SST::ComponentId_t id, Params& params) :
+	EmberMessagePassingGenerator(id, params, "Example"),
+
+```
+
+
+The constructor also reads parameters from the python script. 
+The params are passed through the python file in the form: `ep.addMotif("Example firstParam=100 secondParam=200")`. Note no space is allowed before or after the = operator. Parameters read from the python file will be prepended with "arg." before being passed to the C++ file. i.e. "firstParam" becomes "arg.firstParam".
+The constructor can be used to perform the setup operations necessary for the generating function.
+
+The params can be parsed in the C++ file with `firstParam = params.find<uint32_t>("arg.firstParam", 100);` where the name of the parameter is "firstParam".
+
+### Writing a generate() function
+```
+bool Example::generate( std::queue<EmberEvent*>& evQ)
+```
+The generate function is the 'main' function of a Motif. 
+If the python file invokes an `addMotif`, the generating function will be invoked until the generate function returns true.
+Once the generate function returns, the events queued in the evQ variable will be performed. 
+
+The generate function takes an event queue as a parameter. The event queue allows the user to include computation operations and MPI events in the simulation. Motifs are intended to be designed so that every call to generate() mimics an entire iteration of the application. The events that can be added to the event queue are listed in subsequent sections. 
+
+.  
+
+# User-defined events
+
+These functions allow the user to control how the simulation estimates computation time.
+
+* `enQ_compute(Queue&, uint64_t nanoSecondDelay)`   -- The delay by the expected cost of the compute operation in nanoSeconds (without actually computing it)
+* `enQ_compute( Queue& q, std::function<uint64_t()> func)`; -- A function is passed as a parameter and invoked. It returns a 64-bit unsigned integer that indicates the simulation delay associated with invoking the function (in nanoseconds).
+* `enQ_getTime(evQ, &m_startTime)` -- sets m_startTime to the current time
+
+There are two types of compute functions that can be enqueued.
+
+The first computation operation takes a 64-bit integer as a parameter. This parameter is the amount of time needed in nanoseconds to perform the computation. The simulator then delays by that number of nanoseconds as if the actual computation had taken place. 
+The second compute takes a function as a parameter. When the event is processed, the function is invoked. The function returns the amount of time needed to perform the compute operation. The simulation is delayed by the return value of the function. The time delay could be estimated through a heuristic or measured through representative computation. 
+
+
+# MPI Events
+
+
+We list the supported MPI events below. For more detail, see MPI API documentation.
+
+
+* `enQ_init(evQ)`  --  MPI  initialize
+* `enQ_fini(evQ)` -- MPI finalize
+* `enQ_rank(evQ, m_newComm[0], &m_new_rank)` -- MPI rank
+* `enQ_size(evQ, m_newComm[0], &m_new_size)` --   MPI size
+* `enQ_send(evQ, x_neg, x_xferSize, 0, GroupWorld)`  --  MPI send
+* `enQ_recv(evQ, x_neg, x_xferSize, 0, GroupWorld)`  --  MPI  recv
+* `enQ_isend(evQ, next_comm_rank, itemsThisPhase, 0, GroupWorld, &requests[next_request])` --  MPI isend
+* `enQ_irecv(evQ, xface_down, items_per_cell * sizeof_cell * ny * nz, 0, GroupWorld, &requests[nextRequest])` --  MPI irecv
+* `enQ_cancel(evQ, m_req[i])` -- MPI cancel
+* `enQ_sendrecv(evQ, m_sendBuf, m_messageSize, DATA_TYPE, destRank(), 0xdeadbeef, m_recvBuf, m_messageSize, DATA_TYPE, srcRank(),  0xdeadbeef, GroupWorld, &m_resp)`  --  MPI send or recv
+* `enQ_wait(evQ,  req)`  --  MPI wait
+* `enQ_waitany(evQ, m_req.size(), &m_req[0], &m_indx, &m_resp)` --  MPI waitany
+* `enQ_waitall(evQ, 1, &m_req, (MessageResponse**)&m_resp)` --  MPI waitall
+* `enQ_barrier(evQ, GroupWorld)` -- MPI barrier
+* `enQ_bcast(evQ, m_sendBuf, m_count, DOUBLE, m_root, GroupWorld)` -- MPI broadcast
+* `enQ_scatter(evQ, m_sendBuf, m_sendCnt, m_sendDsp.data(), LONG, m_recvBuf, m_count, LONG, m_root, GroupWorld)` 
+* `enQ_scatterv(evQ, m_sendBuf, &m_sendCnts[0], m_sendDsp.data(), LONG, m_recvBuf, m_count, LONG, m_root, GroupWorld)` -- MPI Scatter variable message size
+* `enQ_reduce( evQ, m_sendBuf, m_recvBuf, m_count, DOUBLE, MP::SUM, m_redRoot, GroupWorld)`
+* `enQ_allreduce(evQ, m_sendBuf, m_recvBuf, m_count, DOUBLE, m_op, GroupWorld)` -- MPI allreduce function
+* `enQ_alltoall(evQ, m_sendBuf, m_colSendCnt, &m_colSendDsp_f[0], DOUBLE, m_colComm)` -- MPI alltoall with constant message size
+* `enQ_alltoallv(evQ, m_sendBuf, &m_colSendCnts_f[0], &m_colSendDsp_f[0], DOUBLE, m_colComm)` -- MPI alltoall with varying message sizes
+* `enQ_allgather(evQ, m_sendBuf, m_count, INT, m_recvBuf, m_count, INT, GroupWorld)` --  MPI allgather with messages of constant size
+* `enQ_allgatherv(evQ, m_sendBuf, m_sendCnt, INT, m_recvBuf, &m_recvCnts[0], &m_recvDsp[0], INT, GroupWorld)` --  MPI allgather with messages of varying sizes
+* `enQ_commSplit(evQ, *comm, color, key, newComm)` -- Splits the MPI comm
+* `enQ_commCreate(evQ, GroupWorld, m_rowGrpRanks, &m_rowComm)` -- Creates MPI Com
+* `enQ_commDestroy(evQ, m_rowComm)` --Destroys MPI comm

src/sst/elements/ember/example/QuickStart.md

@@ -0,0 +1,217 @@
+# What is an Ember Motif?
+
+Ember Motifs provide representative workloads and then estimate performance on hardware using SST simulations.
+This guide provides a directions on how to:
+
+*   Implement a empty motif.
+*   Create and run motif in a SST simulation.
+
+Implementing and running Ember Motifs requires three different files:
+
+1.  Python File
+    Provides information on the hardware to be simulated when running an Ember Motif
+    This directs SST on the available hardware components and how they are set up.
+2.  C++ file
+    Decides how the computation is structured and computed.
+3.  C++ header file
+    The header file contains primarily naming information
+
+# Quick Start Guide
+
+This guide gives a simple example of how to create and run a simple Motif.
+
+First ensure SST-core and SST-elements are properly installed on your machine. For download and installation see [SST-Downloads](http://sst-simulator.org/SSTPages/SSTMainDownloads/).
+
+Navigate to `sst-elements/src/elements/ember` inside the sst-elements library.
+First, create a new directory to hold our example Motif.
+
+`mkdir ExampleMotif`
+
+Next, create a `ExampleMotif/exampleMotif.h` file
+```
+    #ifndef _H_EMBER_EXAMPLE
+    #define _H_EMBER_EXAMPLE
+
+    #include "../../mpi/embermpigen.h"
+
+    namespace SST {
+    namespace Ember {
+
+    class ExampleGenerator : public EmberMessagePassingGenerator {
+       public:
+
+        SST_ELI_REGISTER_SUBCOMPONENT_DERIVED(
+          Example,
+          "ember",
+          "ExampleMotif",
+           SST_ELI_ELEMENT_VERSION(1,0,0),
+          "Performs an idle on the node, no traffic can be generated.",
+          SST::Ember::EmberGenerator
+    );
+
+        SST_ELI_DOCUMENT_PARAMS(
+        );
+
+        SST_ELI_DOCUMENT_STATISTICS(
+          { "time-Init", "Time spent in Init event",             "ns", 0},
+          { "time-Finalize", "Time spent in Finalize event",     "ns", 0},
+          { "time-Rank", "Time spent in Rank event",             "ns", 0},
+          { "time-Size", "Time spent in Size event",             "ns", 0},
+          { "time-Send", "Time spent in Recv event",             "ns", 0},
+          { "time-Recv", "Time spent in Recv event",             "ns", 0},
+          { "time-Irecv", "Time spent in Irecv event",           "ns", 0},
+          { "time-Isend", "Time spent in Isend event",           "ns", 0},
+          { "time-Wait", "Time spent in Wait event",             "ns", 0},
+          { "time-Waitall", "Time spent in Waitall event",       "ns", 0},
+          { "time-Waitany", "Time spent in Waitany event",       "ns", 0},
+          { "time-Compute", "Time spent in Compute event",       "ns", 0},
+          { "time-Barrier", "Time spent in Barrier event",       "ns", 0},
+          { "time-Alltoallv", "Time spent in Alltoallv event",   "ns", 0},
+          { "time-Alltoall", "Time spent in Alltoall event",     "ns", 0},
+          { "time-Allreduce", "Time spent in Allreduce event",   "ns", 0},
+          { "time-Reduce", "Time spent in Reduce event",         "ns", 0},
+          { "time-Bcast", "Time spent in Bcast event",           "ns", 0},
+          { "time-Gettime", "Time spent in Gettime event",       "ns", 0},
+          { "time-Commsplit", "Time spent in Commsplit event",   "ns", 0},
+          { "time-Commcreate", "Time spent in Commcreate event", "ns", 0},
+        );
+
+
+    };
+    }
+    }
+
+    #endif /* _H_EMBER_EXAMPLE */
+```
+This creates a Motif generator class called ExampleGenerator that inherits from EmberMessagePassingGenerator. The Python code we provide later creates and invokes the generate function.
+The SST Document Statistics provides tracking for initialization and various MPI function calls.
+
+Motifs are executed as follows:
+
+
+
+1) The motif generator is initialized (The contructor)
+2) The generate function is invoked and returns either true or false
+3) The events on the eventQueue are processed.
+4) If the generate function in step 2 returned false, return to step 2, otherwise the motif is complete.
+
+Here is a motif that returns without doing any additional work.
+
+Then we create a file `ExampleMotif/Example.cc` with the contents:
+
+```
+    #include <sst_config.h>
+    #include "Example.h"
+
+    using namespace SST:Ember;
+
+    ExampleGenerator::ExampleGenerator(SST::ComponentId_t id, Params& params) :
+        	EmberMessagePassingGenerator(id, params, "Null" )
+    {
+    }
+
+    bool ExampleGenerator::generate( std::queue<EmberEvent*>& evQ)
+    { 
+        // Code here is repeated until true is returned. 
+        return true;
+    }
+```
+
+
+Motifs are intended to be ran as a 'job submission'.
+The generate function models an entire iteration of an application. The event queue allows the motif to mix compute and MPI operations in every iteration.
+
+
+
+The C++ file loads the SST:Ember namespace giving it access to other Motifs and EmberEvents. 
+Each Motif generator has an constructor and generate function. 
+
+
+Then we add the `.cc` and `.h` file to `Makefile.am`
+
+i.e.
+```
+     embermemoryev.h \
+     ExampleMotif/example.h \
+     ExampleMotif/example.cc \
+     libs/emberLib.h \
+```
+
+Finally, a python file `example.py` needs to be created:
+
+```
+    from email.mime import base
+    import sst
+    from sst.merlin.base import *
+    from sst.merlin.endpoint import *
+    from sst.merlin.interface import *
+    from sst.merlin.topology import *
+
+    from sst.ember import *
+
+    def example():
+        PlatformDefinition.setCurrentPlatform("firefly-defaults")
+
+        ### Setup the topology
+        topo = topoDragonFly()
+        topo.hosts_per_router = 2
+        topo.routers_per_group = 4
+        topo.intergroup_links = 2
+        topo.num_groups = 4
+        topo.algorithm = ["minimal","adaptive-local"]
+
+        # Set up the routers
+        router = hr_router()
+        router.link_bw = "4GB/s"
+        router.flit_size = "8B"
+        router.xbar_bw = "6GB/s"
+        router.input_latency = "20ns"
+        router.output_latency = "20ns"
+        router.input_buf_size = "4kB"
+        router.output_buf_size = "4kB"
+        router.num_vns = 2
+        router.xbar_arb = "merlin.xbar_arb_lru"
+
+        topo.router = router
+        topo.link_latency = "20ns"
+
+        ### set up the endpoint
+        networkif = ReorderLinkControl()
+        networkif.link_bw = "4GB/s"
+        networkif.input_buf_size = "1kB"
+        networkif.output_buf_size = "1kB"
+
+        ep = EmberMPIJob(0,topo.getNumNodes())
+        ep.network_interface = networkif
+        ep.addMotif("Example")
+        ep.nic.nic2host_lat= "100ns"
+
+        system = System()
+        system.setTopology(topo)
+        system.allocateNodes(ep,"linear")
+
+        system.build()
+
+    if __name__ == "__main__":
+        example()
+```
+
+This follows common python syntax.
+The hardware variables (topology, router and network interface) are created through assignment of a constructor
+The fields of the variables are accessed using a dot operator.
+Topology, router, and network interface variables need to be created for a simulation. More detailed descriptions of configuration options is in RunningMotifs.md. 
+Next an endpoint is created and given the motif to be used using a ep.addMotif("MotifName")call
+Note multiple motifs can be added the same endpoint. The intended use for each motif to simulate an entire application. The event queue is used to simulate MPI events and computational workloads. Adding multiple motifs is useful for simulating workflows of a series of applications. 
+
+Finally, a system variable is created and 'built'
+
+Remake SST-elements.
+
+To run the python script
+
+```
+sst example.py
+```
+
+
+