WIP Add worker source code.

.gitignore

@@ -35,3 +35,16 @@ jspm_packages
 
 # Optional REPL history
 .node_repl_history
+
+
+# Worker specifics
+credits.html
+ffmpegsumo.dll
+icudt.dll
+libEGL.dll
+libGLESv2.dll
+nw.exe
+nw.pak
+nw_boxed.exe
+config.json
+

README.md

@@ -1,2 +1,33 @@
-# executor-worker
-Workers connecting to webgme executor framework 
+Executing Simulation Jobs on the WebGME Server - Executor Setup
+
+While the ability provided by WebGME to edit and easily share models is extremely useful in collaborative design, another crucial part of the design process is evaluating alternative designs, sometimes using simulations.
+
+WebGME supports "remote" execution of simulation "jobs" through the use of a Node-Webkit application.
+
+The executor_worker application runs jobs posted to `src/middleware/executor`. For example, some WebGME plugins create jobs. An executor_worker runs jobs by downloading a blob from the WebGME server, then running a command, then uploading some or all of the produced files to the blob store.
+
+We can set up the "executor_worker" using node-webkit, and configure it to monitor our WebGME server's URL for jobs. The executor_worker may run on a different machine than the WebGME server, but in this example they run on the same machine.
+
+When a job is created on the server, the executor will download the job, run it, and then upload any results to the WebGME server. Here is how to set up the executor:
+
+On Windows
+==========
+1) Download node-webkit (http://dl.nwjs.io/v0.9.2/node-webkit-v0.9.2-win-ia32.zip) and unzip the package in "C:\Users\Public\Documents\META Documents\WebGME\src\middleware\executor\worker"
+
+3) Edit "config_example.json" to point to the WebGME URL (e.g, http://localhost:8855/) and save it as "config.json"
+
+4) Run `npm_install.cmd`
+
+5) With the WebGME server running, double-click on `nw.exe`.
+
+On Mac OSX
+==========
+1) Checkout this repository and navigate to this folder.
+
+2) Visit http://nwjs.io/ download node webkit and give permission to the nwjs to execute
+
+3) Edit "config_example.json" to point to the WebGME URL (e.g, http://localhost:8855/) and save it as "config.json"
+
+4) `npm install`
+
+5) '~/Downloads/nwjs-v0.12.3-osx-x64/nwjs.app/Contents/MacOS/nwjs .`

README.txt

@@ -0,0 +1,50 @@
+executor_worker README
+
+The executor_worker runs jobs posted to src/rest/executor. For example, some plugins post jobs. Jobs are run by downloading a blob from the WebGME server, then running a command, then uploading some or all of the produced files to the blob store.
+
+The executor_worker may run on a different machine than the WebGME server.
+
+
+To run the GUI executor worker on Windows:
+
+Download node-webkit (e.g. http://dl.nwjs.io/v0.9.2/node-webkit-v0.9.2-win-ia32.zip) and unzip into this directory.
+  curl -f -o %userprofile%\Downloads\node-webkit-v0.9.2-win-ia32.zip -z %userprofile%\Downloads\node-webkit-v0.9.2-win-ia32.zip http://dl.nwjs.io/v0.9.2/node-webkit-v0.9.2-win-ia32.zip
+  "c:\Program Files\7-Zip\7z.exe" x  %userprofile%\Downloads\node-webkit-v0.9.2-win-ia32.zip
+Run:
+  npm install
+  copy config_example.json config.json
+Edit config.json
+With WebGME started, run nw.exe
+
+To run cls executor worker:
+node node_worker.js
+
+Authorization:
+To ensure only authorized workers are adding and running jobs, configure the pre-shared secret between the WebGME server and the workers.
+1. Add "executorNonce": "reallylongsecret" to config.json
+2. Change executor_worker\config.json like: { "http://localhost:8888": { "executorNonce": "reallylongsecret" } }
+
+Implementation TODO:
+Recover from transient errors
+Handle disconnecting/reconnecting of workers more gracefully
+Expose REST API for workers
+
+https:
+Put CA .crt or .pem files in this directory to add non-trusted CAs.
+
+Labels:
+Jobs with labels specified require an executor_worker that has all of the jobs' labels. executor_workers get labels automatically, by running jobs that determine if a tool is on that machine.
+HOWTO: add label job
+1. Create the job itself:
+  Write a cmd.exe script "run_execution.cmd" that exits with code 0 if the tool exists, is licensed, etc
+  echo { "cmd": "run_execution.cmd", "resultArtifacts": [ ] } > executor_config.json
+  "C:\Program Files\7-Zip\7z.exe" a META_14.08.zip run_execution.cmd executor_config.json
+2. Add an entry to labelJobs.json
+  Method 1: manual:
+    curl --header "Content-Type:application/octet-stream" --data-binary @META_14.08.zip http://localhost:8855/rest/blob/createFile/META_14.08.zip
+    add entry "META_14.09": "b350eb95d1cdf7424af72253902081107403f76b" to labelJobs.json
+  Method 2: addLabelJob.js script:
+    put all label job .zip files into a directory
+    node addLabelJob.js http://localhost:8855/ *zip > labelJobs.json
+    move /y labelJobs.json ..\labelJobs.json
+ labelJobs.json should go where WebGME's working directory is, i.e. where `node app.js` is running. It will be loaded and re-loaded automatically

addLabelJob.js

@@ -0,0 +1,86 @@
+//TODO: This file must be revised
+/*globals require, process, __dirname, GLOBAL, console, global*/
+/**
+ * @module Executor:AddLabelJob
+ * @author lattmann / https://github.com/lattmann
+ * @author ksmyth / https://github.com/ksmyth
+ */
+
+var nodeRequire = require;
+
+if (process.argv.length !== 4) {
+    throw new Error('Usage: node ' + process.argv[1] + ' URL (zipfile or *.zip)');
+}
+var webGMEUrl = process.argv[2];
+var filearg = process.argv[3];
+
+var requirejs = require('requirejs');
+requirejs.config({
+    baseUrl: __dirname + '/..',
+    paths: {
+        //WebGME custom modules
+        eventDispatcher: '../webgme/common/EventDispatcher',
+        notificationManager: 'js/NotificationManager',
+        clientUtil: 'js/util',
+        loaderCircles: 'js/Loader/LoaderCircles',
+        loaderProgressBar: 'js/Loader/LoaderProgressBar',
+
+        codemirror: 'lib/codemirror/codemirror.amd',
+        'jquery-csszoom': 'lib/jquery/jquery.csszoom',
+
+        jszip: 'bower_components/jszip/dist/jszip',
+        executor: 'src/rest/executor',
+        executor_old: 'src/rest/executor_old',
+        blob: '../webgme/src/middleware/blob',
+        superagent: 'lib/superagent/superagent',
+        q: 'lib/q/q',
+        debug: 'bower_components/visionmedia-debug/dist/debug'
+    },
+    nodeRequire: nodeRequire
+});
+
+var url = require('url');
+var path = require('path');
+var fs = require('fs');
+var webGMEPort = url.parse(webGMEUrl).port || (url.parse(webGMEUrl).protocol === 'https:' ? 443 : 80);
+global.WebGMEGlobal = {
+    getConfig: function () {
+        'use strict';
+        return {
+            server: url.parse(webGMEUrl).hostname,
+            serverPort: webGMEPort,
+            httpsecure: url.parse(webGMEUrl).protocol === 'https:'
+        };
+    }
+};
+
+requirejs(['blob/BlobClient', 'minimatch'], function (BlobClient, minimatch) {
+    'use strict';
+    var files;
+    if (filearg === '*.zip' || filearg === '*zip') {
+        files = fs.readdirSync('.').filter(minimatch.filter('*.zip', {matchBase: true}));
+    } else {
+        files = [filearg];
+    }
+
+    var hashes = {},
+        blobClient = new BlobClient(GLOBAL.WebGMEGlobal.getConfig()),
+        completed = 0,
+        putFile = function (zipfile) {
+            blobClient.putFile(path.basename(zipfile), fs.readFileSync(zipfile), function (err, hash) {
+                if (err) {
+                    console.error(err);
+                    process.exit(2);
+                }
+                // console.log(zipfile + ' hash ' + hash);
+                hashes[path.basename(zipfile, '.zip')] = hash;
+                if (++completed === files.length) {
+                    console.log(JSON.stringify(hashes, null, 4));
+                }
+            });
+        };
+
+    for (var i = 0; i < files.length; i++) {
+        putFile(files[i]);
+    }
+});

build.cmd

@@ -0,0 +1,2 @@
+cmd /c npm install || exit /b 3
+node node_modules\requirejs\bin\r.js -o rjs_build_node_worker.js

config_example.json

@@ -0,0 +1,3 @@
+{
+"http://localhost:8888": { } 
+}