API V3 (nightscout#4250)

* extended .gitignore for Visual Studio 2017 * creating a lib for api3 and exposing it's swagger file * adding pilot test (for /swagger.yaml) * implementing public GET /version * setting api version to 3.0.0-alpha * creating authorization skeleton + fetching some API env variables * reusing authorization library * implementing security * forcing HTTPS and removing x-powered-by from response * moving messages to constants, creating https instance fixture * testing HTTPS requiring * testing Date header * testing permission check * testing allowed operation * refactoring + storage stub * create architecture for generic operations * beginning of READ operation * tidying the code up * basic READ part * going further with READ operation * DELETE operation * handling fields parameter * refactoring to classes * going further with SEARCH operation * refactoring file structure * filtering for SEARCH operation * preparations for fallback deduplication * CREATE operation * UPDATE operation * PATCH operation * HISTORY operation * creating more precise variant of HISTORY operation * autopruning * long for timestamps in swagger * bug fix (when search fields=srvCreated) * creating skeleton for generic collection API test * specific HISTORY skeleton * distinguish between collection logical and storage name * renaming operation to LAST MODIFIED and getting it to work * fallback for LAST MODIFIED operation * tidying a bit * LAST MODIFIED documentation * bugfix + emitting data-received * adding some validations * bugfix - remove 'token' parameter from filtering * testing and debugging generic workflow * test fix for empty db * fixing security test fixture * trying to fix Travis CI testing DB problem * multiple auth callback bugfix + adding user field on authed create/update * messages for Travis CI debugging * messages for Travis CI debugging * messages for Travis CI debugging * test fix (to be prepared for future dates in db) * test fix * adding fallback created_at filling on each create/update * STATUS operation with API permissions * querying srvDate from storage + include storage version info * bugfix of missing apiConst require * getting mongo version with read-only user rights * getting mongo current date with read-only user rights * trying to diagnose travis CI timeout * refactoring storage version caching (due to some environments problems) * making VERSION work on empty database * more fixes * skipping API HTTPS test for node 8 * making code more readable using ES6 (Promises, async + await) * extending treatments collection docs by inspecting the careportal code * tidying existing API3 tests up to allow further grow * tidying the authorization code up to increase readability and performance a bit * more refactoring to ES6 and making APIv3 files structure more extendable * normalizing incoming dates to UTC and storing utcOffset * fixing srvDate to be of node.js server, not the mongo DB * preparing test fixtures for permissions testing + skeleton for CREATE operation test * intensive CREATE operation testing + minor bug fixes * correcting the deduplication test * more deduplication testing of CREATE operation * adding test skeletons for other generic operations * added variability in filtering by date, created_at, srvModified, srvCreated fields * fixing test accordingly to previous commit * adding new collection settings for centralized apps' settings storage * trying to solve travis CI testing problem - adding default collections names * another attempt to travis CI test fix * adding some tests for READ operation * adding custom error handler (overriding bodyparser's errors) * securing settings collection more and updating swagger accordingly * making HISTORY timestamp parameter more flexible + updating swagger documentation * more testing and bug fixing * sending only HTTP status with empty body, when there is no message + minor bug fixing * more refactoring and testing (especially of UPDATE operation) * PATCH testing + adding userModified field for troubleshooting purposes * basic SEARCH operation testing * more SEARCH operation testing * adding alternative 'now' query parameter to 'Date' header to make GET easier * adding 'now' to reserved query parameters for SEARCH operation * more testing * renaming field user to subject (and modifiedBy) * bugfix - fixing RFC 2822 constant for moment parsing * storageSocket: creating skeleton for new Socket.IO namespace * storageSocket: authentication by accessToken * storageSocket: authorizing to subscribe rooms * storageSocket: emitting create, update and delete events * APIv3: adding support for swagger UI at /api/v3/swagger-ui-dist * solving some problems detected by eslint * solving some problems detected by eslint * APIv3: testing and debugging Socket.IO * APIv3: testing and debugging Socket.IO * APIv3: Socket.IO documentation * APIv3: making the sample real * APIv3: starting to create a simple tutorial MD file * APIv3: small corrections * APIv3: minor corrections after dev merge * APIv3: adding CREATE and READ operations to the tutorial.md * APIv3: adding SEARCH, LAST MODIFIED, UPDATE operations to the tutorial.md * APIv3: finishing the tutorial.md * APIv3: minor bugfix (bad location after upsert) * APIv3: refactoring SEARCH complexity * APIv3: refactoring mongoCollection complexity * APIv3: refactoring complexity * APIv3: tidying up a bit * APIv3: refactoring security (start) * APIv3: refactoring lastModified * APIv3: refactoring create (start) * APIv3: refactoring create (finish) * APIv3: refactoring delete * APIv3: refactoring history * APIv3: refactoring update * APIv3: refactoring patch * APIv3: refactoring read * APIv3: refactoring search + removing deprecated authorizationBuilder * APIv3: adding best practise for identifier constructing * APIv3: refactoring and enhancing the validation (immutable fields) * APIv3: adding security.md documentation file * APIv3: refactoring - splitting index.js into multiple files * APIv3: calculating identifier on server side + deduplicating * APIv3: refactoring cosmetics * APIv3: updating the documentation * APIv3: making basic and security tests more readable using async/await * APIv3: making the rest of tests more readable using async/await * APIv3: adapting test of previous API * APIv3: adapting test of previous API
dev4good · Oct 9, 2019 · 2dd576a · 2dd576a
1 parent dfbcf62
commit 2dd576a
Show file tree

Hide file tree

Showing 61 changed files with 7,578 additions and 9 deletions.
diff --git a/.eslintrc.js b/.eslintrc.js
@@ -9,6 +9,15 @@ module.exports = {
       "commonjs": true,
       "es6": true,
       "node": true,
+      "mocha": true,
       "jquery": true
+    },
+    "rules": {
+      "no-unused-vars": [
+        "error",
+        {
+          "varsIgnorePattern": "should|expect"
+        }
+      ]
     }
   };
diff --git a/.gitignore b/.gitignore
@@ -24,3 +24,9 @@ npm-debug.log
 *.heapsnapshot
 
 /tmp
+/.vs
+/cgm-remote-monitor.njsproj
+/cgm-remote-monitor.sln
+/obj/Debug
+/bin
+/*.bat
diff --git a/app.js b/app.js
@@ -114,11 +114,12 @@ function create (env, ctx) {
     });
   }
 
-  ///////////////////////////////////////////////////
-  // api and json object variables
-  ///////////////////////////////////////////////////
-  var api = require('./lib/api/')(env, ctx);
-  var ddata = require('./lib/data/endpoints')(env, ctx);
+    ///////////////////////////////////////////////////
+    // api and json object variables
+    ///////////////////////////////////////////////////
+    var api = require('./lib/api/')(env, ctx);
+    var api3 = require('./lib/api3/')(env, ctx);
+    var ddata = require('./lib/data/endpoints')(env, ctx);
 
   app.use(compression({
     filter: function shouldCompress (req, res) {
@@ -172,6 +173,8 @@ function create (env, ctx) {
   app.use('/api/v2/authorization', ctx.authorization.endpoints);
   app.use('/api/v2/ddata', ddata);
 
+  app.use('/api/v3', api3);
+
   // pebble data
   app.get('/pebble', ctx.pebble);
 

diff --git a/env.js b/env.js
@@ -112,6 +112,7 @@ function setStorage() {
   env.authentication_collections_prefix = readENV('MONGO_AUTHENTICATION_COLLECTIONS_PREFIX', 'auth_');
   env.treatments_collection = readENV('MONGO_TREATMENTS_COLLECTION', 'treatments');
   env.profile_collection = readENV('MONGO_PROFILE_COLLECTION', 'profile');
+  env.settings_collection = readENV('MONGO_SETTINGS_COLLECTION', 'settings');
   env.devicestatus_collection = readENV('MONGO_DEVICESTATUS_COLLECTION', 'devicestatus');
   env.food_collection = readENV('MONGO_FOOD_COLLECTION', 'food');
   env.activity_collection = readENV('MONGO_ACTIVITY_COLLECTION', 'activity');

diff --git a/lib/api3/const.json b/lib/api3/const.json
@@ -0,0 +1,51 @@
+{
+  "API3_VERSION": "3.0.0-alpha",
+  "API3_SECURITY_ENABLE": true,
+  "API3_TIME_SKEW_TOLERANCE": 5,
+  "API3_DEDUP_FALLBACK_ENABLED": true,
+  "API3_CREATED_AT_FALLBACK_ENABLED": true,
+  "API3_MAX_LIMIT": 1000,
+
+  "HTTP": {
+    "OK": 200,
+    "CREATED": 201,
+    "NO_CONTENT": 204,
+    "NOT_MODIFIED": 304,
+    "BAD_REQUEST": 400,
+    "UNAUTHORIZED": 401,
+    "FORBIDDEN": 403,
+    "NOT_FOUND": 404,
+    "GONE": 410,
+    "PRECONDITION_FAILED": 412,
+    "INTERNAL_ERROR": 500
+  },
+
+  "MSG": {
+    "HTTP_400_BAD_LAST_MODIFIED": "Bad or missing Last-Modified header/parameter",
+    "HTTP_400_BAD_LIMIT": "Parameter limit out of tolerance",
+    "HTTP_400_BAD_REQUEST_BODY": "Bad or missing request body",
+    "HTTP_400_BAD_FIELD_IDENTIFIER": "Bad or missing identifier field",
+    "HTTP_400_BAD_FIELD_DATE": "Bad or missing date field",
+    "HTTP_400_BAD_FIELD_UTC": "Bad or missing utcOffset field",
+    "HTTP_400_BAD_FIELD_APP": "Bad or missing app field",
+    "HTTP_400_BAD_SKIP": "Parameter skip out of tolerance",
+    "HTTP_400_SORT_SORT_DESC": "Parameters sort and sort_desc cannot be combined",
+    "HTTP_400_UNSUPPORTED_FILTER_OPERATOR": "Unsupported filter operator {0}",
+    "HTTP_400_IMMUTABLE_FIELD": "Field {0} cannot be modified by the client",
+    "HTTP_401_BAD_DATE": "Bad Date header",
+    "HTTP_401_BAD_TOKEN": "Bad access token or JWT",
+    "HTTP_401_DATE_OUT_OF_TOLERANCE": "Date header out of tolerance",
+    "HTTP_401_MISSING_DATE": "Missing Date header",
+    "HTTP_401_MISSING_OR_BAD_TOKEN": "Missing or bad access token or JWT",
+    "HTTP_403_MISSING_PERMISSION": "Missing permission {0}",
+    "HTTP_403_NOT_USING_HTTPS": "Not using SSL/TLS",
+    "HTTP_500_INTERNAL_ERROR": "Internal Server Error",
+    "STORAGE_ERROR": "Database error",
+    "SOCKET_MISSING_OR_BAD_ACCESS_TOKEN": "Missing or bad accessToken",
+    "SOCKET_UNAUTHORIZED_TO_ANY": "Unauthorized to receive any collection"
+  },
+
+  "MIN_TIMESTAMP": 946684800000,
+  "MIN_UTC_OFFSET": -1440,
+  "MAX_UTC_OFFSET":  1440
+}
diff --git a/lib/api3/doc/security.md b/lib/api3/doc/security.md
@@ -0,0 +1,48 @@
+# APIv3: Security
+
+### Enforcing HTTPS
+APIv3 is ment to run only under SSL version of HTTP protocol, which provides:
+- **message secrecy** - once the secure channel between client and server is closed the communication cannot be eavesdropped by any third party
+- **message consistency** - each request/response is protected against modification by any third party (any forgery would be detected)
+- **authenticity of identities** - once the client and server establish the secured channel, it is guaranteed that the identity of the client or server does not change during the whole session
+
+HTTPS (in use with APIv3) does not address the true identity of the client, but ensures the correct identity of the server. Furthermore, HTTPS does not prevent the resending of previously intercepted encrypted messages by an attacker.
+
+
+---
+###  Authentication and authorization
+In APIv3, *API_SECRET* can no longer be used for authentication or authorization. Instead, a roles/permissions security model is used, which is managed in the *Admin tools* section of the web application.
+
+
+The identity of the client is represented by the *subject* to whom the access level is set by assigning security *roles*. One or more *permissions* can be assigned to each role. Permissions are used in an [Apache Shiro-like style](http://shiro.apache.org/permissions.html "Apache Shiro-like style").
+
+
+For each security *subject*, the system automatically generates an *access token* that is difficult to guess since it is derived from the secret *API_SECRET*. The *access token* must be included in every secured API operation to decode the client's identity and determine its authorization level. In this way, it is then possible to resolve whether the client has the permission required by a particular API operation.
+
+
+There are two ways to authorize API calls:
+- use `token` query parameter to pass the *access token*, eg. `token=testreadab-76eaff2418bfb7e0`
+- use so-called [JSON Web Tokens](https://jwt.io "JSON Web Tokens")
+  - at first let the `/api/v2/authorization/request` generates you a particular JWT, eg. `GET https://nsapiv3.herokuapp.com/api/v2/authorization/request/testreadab-76eaff2418bfb7e0`
+  - then, to each secure API operation attach a JWT token in the HTTP header, eg. `Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6InRlc3RyZWFkYWItNzZlYWZmMjQxOGJmYjdlMCIsImlhdCI6MTU2NTAzOTczMSwiZXhwIjoxNTY1MDQzMzMxfQ.Y-OFtFJ-gZNJcnZfm9r4S7085Z7YKVPiaQxuMMnraVk` (until the JWT expires)
+
+
+
+---
+###  Client timestamps
+As previously mentioned, a potential attacker cannot decrypt the captured messages, but he can send them back to the client/server at any later time. APIv3 is partially preventing this by the temporal validity of each secured API call.
+
+
+The client must include his current timestamp to each call so that the server can compare it against its clock. If the timestamp difference is not within the limit, the request is considered invalid. The tolerance limit is set in minutes in the `API3_TIME_SKEW_TOLERANCE` environment variable.
+
+There are two ways to include the client timestamp to the call:
+-  use `now` query parameter with UNIX epoch millisecond timestamp, eg. `now=1565041446908`
+- add HTTP `Date` header to the request, eg. `Date: Sun, 12 May 2019 07:49:58 GMT`
+
+
+The client can check each server response in the same way, because each response contains a server timestamp in the HTTP *Date* header as well.
+
+
+---
+APIv3 security is enabled by default, but it can be completely disabled for development and debugging purposes by setting the web environment variable `API3_SECURITY_ENABLE=false`.
+This setting is hazardous and it is strongly discouraged to be used for production purposes!
diff --git a/lib/api3/doc/socket.md b/lib/api3/doc/socket.md
@@ -0,0 +1,142 @@
+# APIv3: Socket.IO storage modifications channel
+
+APIv3 has the ability to broadcast events about all created, edited and deleted documents, using Socket.IO library.
+
+This provides a real-time data exchange experience in combination with API REST operations.
+
+### Complete sample client code
+```html
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <meta http-equiv="x-ua-compatible" content="ie=edge" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+
+    <title>APIv3 Socket.IO sample</title>
+
+    <link rel="icon" href="images/favicon.png" />
+  </head>
+
+  <body>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/2.2.0/socket.io.js"></script>
+
+	<script>
+      const socket = io('https://nsapiv3.herokuapp.com/storage');
+
+      socket.on('connect', function () {
+        socket.emit('subscribe', { 
+          accessToken: 'testadmin-ad3b1f9d7b3f59d5',
+          collections: [ 'entries', 'treatments' ]
+        }, function (data) {
+          if (data.success) {
+            console.log('subscribed for collections', data.collections);
+          }
+          else {
+            console.error(data.message);
+          }
+        });
+      });
+
+      socket.on('create', function (data) {
+        console.log(`${data.colName}:created document`, data.doc);
+      });
+
+      socket.on('update', function (data) {
+        console.log(`${data.colName}:updated document`, data.doc);
+      });
+
+      socket.on('delete', function (data) {
+        console.log(`${data.colName}:deleted document with identifier`, data.identifier);
+      });
+	</script>
+  </body>
+</html>
+```
+
+**Important notice: Only changes made via APIv3 are being broadcasted. All direct database or APIv1 modifications are not included by this channel.**
+
+### Subscription (authorization)
+The client must first subscribe to the channel that is exposed at `storage` namespace, ie the `/storage` subadress of the base Nightscout's web address (without `/api/v3` subaddress).
+```javascript
+const socket = io('https://nsapiv3.herokuapp.com/storage');
+```
+
+
+Subscription is requested by emitting `subscribe` event to the server, while including document with parameters:
+* `accessToken`: required valid accessToken of the security subject, which has been prepared in *Admin Tools* of Nightscout. 
+* `collections`: optional array of collections which the client wants to subscribe to, by default all collections are requested)
+
+```javascript
+socket.on('connect', function () {
+  socket.emit('subscribe', { 
+    accessToken: 'testadmin-ad3b1f9d7b3f59d5',
+    collections: [ 'entries', 'treatments' ]
+  },
+```
+
+
+On the server, the subject is first identified and authenticated (by the accessToken) and then a verification takes place, if the subject has read access to each required collection. 
+
+An exception is the `settings` collection for which `api:settings:admin` permission is required, for all other collections `api:<collection>:read` permission is required.
+
+
+If the authentication was successful and the client has read access to at least one collection, `success` = `true` is set in the response object and the field `collections` contains an array of collections which were actually subscribed (granted).
+In other case `success` = `false` is set in the response object and the field `message` contains an error message.
+
+```javascript
+function (data) {
+    if (data.success) {
+      console.log('subscribed for collections', data.collections);
+    }
+    else {
+      console.error(data.message);
+    }
+  });
+});
+```
+
+### Receiving events
+After the successful subscription the client can start listening to `create`, `update` and/or `delete` events of the socket.
+
+
+##### create
+`create` event fires each time a new document is inserted into the storage, regardless of whether it was CREATE or UPDATE operation of APIv3 (both of these operations are upserting/deduplicating, so they are "insert capable"). If the document already existed in the storage, the `update` event would be fired instead.
+
+The received object contains:
+* `colName` field with the name of the affected collection 
+* the inserted document in `doc` field
+
+```javascript
+socket.on('create', function (data) {
+  console.log(`${data.colName}:created document`, data.doc);
+});
+```
+
+
+##### update
+`update` event fires each time an existing document is modified in the storage, regardless of whether it was CREATE, UPDATE or PATCH operation of APIv3 (all of these operations are "update capable"). If the document did not yet exist in the storage, the `create` event would be fired instead.
+
+The received object contains:
+* `colName` field with the name of the affected collection 
+* the new version of the modified document in `doc` field
+
+```javascript
+socket.on('update', function (data) {
+  console.log(`${data.colName}:updated document`, data.doc);
+});
+```
+
+
+##### delete
+`delete` event fires each time an existing document is deleted in the storage, regardless of whether it was "soft" (marking as invalid) or permanent deleting.
+
+The received object contains:
+* `colName` field with the name of the affected collection 
+* the identifier of the deleted document in the `identifier` field
+
+```javascript
+socket.on('delete', function (data) {
+  console.log(`${data.colName}:deleted document with identifier`, data.identifier);
+});
+```