diff --git a/README.md b/README.md index 0bb2da8..3df3e39 100644 --- a/README.md +++ b/README.md @@ -1 +1,64 @@ -This repository is used to track issues that apply cross-services in the full semantic.works stack. +# Semantic.works (mu-semtech) +This repository is used to store information and track issues that apply cross-services in the full semantic.works stack. + +## Getting started +If you do not know where to begin, check out any/all of the following documents: +1. Read about why and how the semantic.works stack works: [discussions - design philosophy](docs/discussions/design-philosophy.md) +2. Recognising our different project types and how they help you: [discussions - project categories](docs/discussions/project-categories.md) +3. Learn how to [create a full-fledged application](docs/how-tos/creating-applications.md) or a [microservice to support it](docs/how-tos/creating-microservices.md) +4. Learn how to ideally [deploy your applications](docs/how-tos/deploying-applications.md) + + +## How-to +### Create... +- [Applications](docs/how-tos/creating-applications.md) +- [Microservices](docs/how-tos/creating-microservices.md) +- [Templates](#create-1) + +### Development +- [Add services to your project](docs/how-tos/adding-services-to-your-project.md) +- [Deploy applications](docs/how-tos/deploying-applications.md) +- [Quickstart writing documentation](docs/how-tos/quickstart-writing-documentation.md) + +### Troubleshooting +- [Troubleshooting - Slow starting containers using 100% CPU](docs/how-tos/troubleshooting---slow-starting-containers.md) + +## Tutorials +### Create... +- [Templates](docs/tutorials/creating-templates.md) + +### Development +- [Develop with your local IDE and tools inside a Docker container](docs/tutorials/developing-inside-containers.md) + +## Reference +For technical information in semantic.works, you can see the following references: +- [Commonly used headers](docs/reference/commonly-used-headers.md) +- [Representing logged in users](docs/reference/representing-logged-in-users.md) + +## Discussions +If you want more information behind the design of semantic.works, you can read the following discussions: +- [Design philosophy](docs/discussions/design-philosophy.md) +- [Documentation structure](docs/discussions/documentation-structure.md) +- [Project categories](docs/discussions/project-categories.md) +- [Sharing authorization](docs/discussions/sharing-authorization.md) + +## Writeups +Perspectives on... +- [Experimentation / all or nothing fails to innovate](writeups/perspectives/all-or-nothing-fails-to-innovate.md) +- [mu.semte.ch primer](writeups/perspectives/mu-semtech-primer.md) +- [Reactive programming](writeups/perspectives/) +- [Why semantic microservices](writeups/perspectives/why-semantic-microservices.md) +- [Why semantic tech](writeups/perspectives/why-semantic-tech.md) + +Retrospectives on... +- [Dockercon EU 2017](writeups/retrospectives/dockercon-eu-2017.md) +- [Developerweek 2018](writeups/retrospectives/developerweek-2018.md) +- [OSLO²](writeups/retrospectives/oslo2.md) + +- [Semantic.works - Implementing Docker multi-stage builds benefits](writeups/retrospectives/sw-implementing-docker-multi-stage-builds.md) +- [Semantic.works - Microservice reuse and authorization](writeups/retrospectives/sw-microservice-reuse-and-authorization.md) +- [Semantic.works - Supporting MacOS](writeups/retrospectives/sw-supporting-mac-os.md) + +Or.. +- Discover who governs semantic.works in [who - mu-semtech](writeups/who---mu-semtech.md) +- Find our external publications in [publictations](writeups/publications.md) \ No newline at end of file diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..5781750 --- /dev/null +++ b/TODO.md @@ -0,0 +1,7 @@ +- [ ] Expand semantic model example as requested by Aad + +We should make and/or link (a) tutorial(s) for the following things: +- [ ] Docker & Docker Compose +- [ ] Linked data & SPARQL +- [ ] Ember.js +- [ ] Accept headers diff --git a/archive/How to build a microservice template.docx b/archive/How to build a microservice template.docx new file mode 100644 index 0000000..409a23b Binary files /dev/null and b/archive/How to build a microservice template.docx differ diff --git a/archive/README.md b/archive/README.md new file mode 100644 index 0000000..cfeba11 --- /dev/null +++ b/archive/README.md @@ -0,0 +1,5 @@ +# mu.semte.ch archive + +The items in this folder have been archived for any (but not limited to) the following reasons: +- They were info about the mu.semte.ch/ blog +- The services they describe have been deprecated diff --git a/archive/auto-expanding-uploaded-semantic-files.md b/archive/auto-expanding-uploaded-semantic-files.md new file mode 100644 index 0000000..7269c52 --- /dev/null +++ b/archive/auto-expanding-uploaded-semantic-files.md @@ -0,0 +1,43 @@ +**Editors note** +The following has been archived due to delta-service being superseded by delta-notifier. More info on the deprecation can be found in the [delta-service repo](https://github.com/mu-semtech/archived-mu-delta-service) + +*This document has been adapted from Jonathan Langens' mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/06/01/auto-expanding-uploaded-semantic-files/)* + +--- + +# Auto-expanding uploaded semantic files + +By adding 2 new microservices to our regular [mu.semte.ch](http://mu.semte.ch/) setup ([https://github.com/mu-semtech/mu-project](https://github.com/mu-semtech/mu-project)) we can create a very nifty workflow that will automatically expand semantic files in our graph database. + +## File uploader service +We need a file uploader service that after accepting a POST with a file saves that![](http://mu.semte.ch/wp-content/uploads/2017/05/fileservice-200x300.png) file and adds information about that file in our triple store. This information can be expressed using the following rather limited vocabulary: +A file is of class [http://mu.semte.ch/vocabularies/file-service/File](http://mu.semte.ch/vocabularies/file-service/File) +and has the following properties + +* [http://mu.semte.ch/vocabularies/file-service/internalFilename](http://mu.semte.ch/vocabularies/file-service/internalFilename) +* [http://mu.semte.ch/vocabularies/file-service/filename](http://mu.semte.ch/vocabularies/file-service/filename) +* [http://mu.semte.ch/vocabularies/file-service/uploadedAt](http://mu.semte.ch/vocabularies/file-service/uploadedAt) +* [http://mu.semte.ch/vocabularies/file-service/status](http://mu.semte.ch/vocabularies/file-service/status) + +## Semantic expander service +Next we need to have a semantic expander service. This service is a little bit more complicated because it handles 2 separate functionalities. + +The first functionality that this service should have is support to consume delta’s as they are generated by the delta service ([https://github.com/mu-semtech/mu-delta-service](https://github.com/mu-semtech/mu-delta-service)). In these reports we will need to filter out the files that contain semantic data and whose status has changed to uploaded. We can achieve this in a rather brute way by first making a set of all URIs of the subjects in the insert reports. After this we can make a simple query that looks somewhat like this: + +``` +SELECT DISTINCT ?internal_filename + WHERE { + ?uri a http://mu.semte.ch/vocabularies/file-service/File . + ?uri http://mu.semte.ch/vocabularies/file-service/internalFilename ?internal_filename . + ?uri http://mu.semte.ch/vocabularies/file-service/filename ?filename . + FILTER(strends(?filename, “.ttl”) || ?filename, “.rdf”)) + FILTER(?uri in ([LIST OF ALL THE URI’s FOUND])) + FILTER NOT EXISTS { + ?uri http://mu.semte.ch/vocabularies/semantic-expander/expanded ?date + } + } +``` + +This query will provide us with a list of filenames. We can now expand each of theses filenames. This can be done either (1) by converting the files to one or more insert queries, (2) by using a graph protocol to load an entire file or (3) by using store specific logic to load the files (i.e. using iSQL on Virtuoso to create a load list and then starting the load endpoint). + +And tada! Whenever we upload a file with semantic content to our backend, the semantic expander service will pick it up automatically and load the contents in the triple store. Almost magic. \ No newline at end of file diff --git a/archive/get-to-know-mu-cl-resources.md b/archive/get-to-know-mu-cl-resources.md new file mode 100644 index 0000000..4597265 --- /dev/null +++ b/archive/get-to-know-mu-cl-resources.md @@ -0,0 +1,12 @@ +**Editors note** +The following has been archived due to it being a blog post that was meant as a status update & linking to different blog posts. +The blog posts that the last two links refer to have been merged & imported into mu-projects' documentation. You can view this [here](https://github.com/mu-semtech/mu-project#creating-a-json-api) + +*This document has been adapted from Erika Pauwels' mu.semte.ch article. You can view it [here](https://mu.semte.ch/2018/03/22/get-to-know-mu-cl-resources/)* + +--- + +# Get to know mu-cl-resources +Long awaited, but it’s finally there: [a README for the mu-cl-resources service](https://github.com/mu-semtech/mu-cl-resources). Get to know all features (there are a lot!) of our microservice producing a [JSONAPI](http://jsonapi.org/) compliant API for your resources based on a simple configuration describing the domain. Find things such as [how to define your domain](https://github.com/mu-semtech/mu-cl-resources#configurationdomainlisp), [how to filter](https://github.com/mu-semtech/mu-cl-resources#basic-filtering), [how to paginate](https://github.com/mu-semtech/mu-cl-resources#pagination) and a lot more. + +You can also have a look at our previous blog posts [Generating a JSONAPI compliant API for your resources, part 1](https://mu.semte.ch/2017/07/27/generating-a-jsonapi-compliant-api-for-your-resources/) and [part 2](https://mu.semte.ch/2017/08/17/generating-a-jsonapi-compliant-api-for-your-resources-part-2/) to get started. diff --git a/archive/the-delta-service-and-its-benefits.md b/archive/the-delta-service-and-its-benefits.md new file mode 100644 index 0000000..046ad28 --- /dev/null +++ b/archive/the-delta-service-and-its-benefits.md @@ -0,0 +1,48 @@ +**Editors note** +The following has been archived due to delta-service being superseded by delta-notifier. More info on the deprecation can be found in the [delta-service repo](https://github.com/mu-semtech/archived-mu-delta-service) + +*This document has been adapted from Jonathan Langens' mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/05/11/the-delta-service-and-its-benefits/)* + +--- + +# The delta service and its benefits +The delta-service has already been mentioned and used in [previous blog posts on reactive programming](https://mu.semte.ch/tag/delta-service/) to make microservices hook into changes introduced by other microservices. In this article we will elaborate in more depth on this service. What is the delta-service and what are the benefits of using it? + +## What is the delta-service? +![](http://mu.semte.ch/wp-content/uploads/2017/05/wild_e_delta-300x200.png)The delta-service is a microservice that offers a SPARQL endpoint. It accepts SPARQL queries, analyses them, calculates the differences a query introduces into the graph store and notifies interested parties about these differences. + +The reports that will be sent to interested parties are formed with triples as the basic building blocks as triples are the resources that those SPARQL stores (mentally) have in common. + +Conceptually such a report would looks somewhat like this: + + + + + + - + - + - + + +All triples with a ‘+’ in front will be inserted and all triples preceded by ‘-’ are triples that will be deleted by the query. + +## What are the benefits? + +### Graph store independence +The delta-service has the obvious benefit that the framework can make abstraction from the graph store you are actually using. For instance, on a Virtuoso graph database both update (INSERT, DELETE) and select queries (SELECT, ASK, DESCRIBE) are sent to the same endpoint: [http://localhost:8890/sparql](http://localhost:8890/sparql) by default. On an OWLIM database on the other hand update queries are sent to [http://localhost:8001/openrdf-workbench/\[repository\]/statements](http://localhost:8001/openrdf-workbench/%5Brepository%5D/statements) while a select query is sent to [http://localhost:8001/openrdf-workbench/\[repository\]/query](http://localhost:8001/openrdf-workbench/%5Brepository%5D/query). The delta-service may even support idiosyncracies of a certain triple store like reforming queries if you know that the graph store cannot properly handle a specific query. + +### Messaging +Additionally a service that can notify interested parties of the changes in the graph store can easily replace messages and message queues. While message queues do scale, they still require a shared mental model between the producing and the consuming microservices. This mental model is enforced by most messaging systems while with the delta-service you are free to subscribe to whatever you want. As long as it is expressible in a triple, a microservice can hook into it. + +The producing microservice does not need to compose a specific message for the message queue. It just writes the triples to the triple store as it normally would (within the [mu.semte.ch](http://mu.semte.ch/) framework the triple store holds the ground truth) and the delta-service takes care of the rest. So here the win is: no mental model change.  Also the consuming microservice on the other hand does not need to know the transformation model the producing microservice uses to transform the message to the message queue format. Again it only needs to know triples. + +And here we win a lot with the added semantic value of the controlled vocabularies used with linked data technology. For example should I install a microservice on my platform that performs actions every time a concept scheme is added to the SPARQL store, then that microservice just needs to listen to all changes on URIs that are a SKOS:ConceptScheme. + +In short, while you can build a message queue with this technology the messages would always handle a view of the model but by being able to consume the delta reports all the reactiveness is being leveraged by nothing but manipulating the model. + +### Scaling +The delta-service may also facilitate the scaling of the mu.semte.ch platform. If we would introduce a master graph database instance to which all updates (but preferably no selects) are proxied then we could put a delta-service in front of this master graph database and send mutation reports of the master graph database to a slave battery updater. This slave battery updater can then update an array of slave stores with every report. The slaves can be used for select queries. This way we have free scaling of the graph database and even more: we can have multiple different types of graph databases all in sync. + +### Reactiveness +The delta-service enables us to use a new approach to developing applications: reactive programming. Make a microservice hook into changes produced by another microservice. Have a look at [the blog posts we’ve already published on this topic](https://mu.semte.ch/tag/reactive-programming/) to learn how to make a microservice reactive. + +## Conclusion +The delta-service is just a microservice, but one that offers a lot of benefits. In the future we will publish more blog posts illustrating how you can use the delta-service in your mu.semte.ch project and easily benefit from the difference reports it generates. \ No newline at end of file diff --git a/archive/thoughts-on-how-a-distributed-SPARQL-endpoint-might-be-implemented.md b/archive/thoughts-on-how-a-distributed-SPARQL-endpoint-might-be-implemented.md new file mode 100644 index 0000000..94b6cb1 --- /dev/null +++ b/archive/thoughts-on-how-a-distributed-SPARQL-endpoint-might-be-implemented.md @@ -0,0 +1,47 @@ +**Editors note** +The following has been archived due to delta-service being superseded by delta-notifier. More info on the deprecation can be found in the [delta-service repo](https://github.com/mu-semtech/archived-mu-delta-service) +Also note that this is a direct import of the article written in 2017, before the master/slave terminology that is used within has come under larger scrutiny. I (the editor/importer) nor the person who originally wrote the article have made recent changes on it, and it has been asked to be left as-is, albeit archived. + +*This document has been adapted from Jonathan Langens' mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/10/26/thoughts-on-how-a-distributed-sparql-endpoint-might-be-implemented/)* + +--- + +# Thoughts on how a distributed SPARQL endpoint might be implemented +![](http://mu.semte.ch/wp-content/uploads/2017/10/replications.png) + +The problem with most triple stores is … they tend to be slow. At least slow compared to their schema-full counterparts. It seems the cost of not defining a fixed schematic structure upfront is a not easy to avoid. We have been thinking about various models and solutions to mitigate this issue for specific applications. With the simplicity of the [mu.semte.ch](http://mu.semte.ch/) stack it is easy to add caches left and right but in the end that does not solve all problems. This post is intended to present an idea on how to add a solution by making the triple store distributed. + +## What kind of distributed? +There are various models of distributed stores and maybe we explore other in subsequent posts.  In this post we present one way of distributing a triplestore across multiple instances. + +The high-level idea is that a delta-service detects changes which would be introduced by update-queries.  These updates are dispatched to a set of replicated triplestores.  Queries and updates are ordered correctly to ensure updates are visible when a combination of read and write queries are executed. + +## Terminology +A quick overview of the terminology used below as it is quite ambiguous. + +- query: a SPARQL query that is not an insert or delete query. +- update: a SPARQL query that is an insert, delete or delete-insert-where query. +- SPARQL query: a query or an update. + +## Architecture +A DSE (Distributed Sparql Endpoint) consists of a distributed SPARQL Endpoint Controller, a delta calculating microservice, a slave controller microservice, optionally a base slave store and then one or more slaves stores. + +![](http://mu.semte.ch/wp-content/uploads/2017/10/distributedSPARQLEndpoint-1.png) + +## Distributed SPARQL Endpoint Controller (DSEC) +The DSEC acts as the entrypoint for the DSE. The most basic form of its functionality would be to split the incoming SPARQL queries into queries and updates. Although updates change the database’s state, the DSEC is itself is almost fully stateless.  A notable exception is an increasing number attached to each incoming SPARQL query. This number allows us to prevent the execution of queries before all updates that were sent earlier have been applied. + +## Delta Calculating Microservice +The delta microservice calculates which triples are changed based on a received query.  It has three distinct responsibilities: + +1. Maintain a queue of pending updates; +2. Calculate the delta that would be introduced by executing the \[lowest\] update on the queue; +3. Execute those delta’s on it’s own \[master\] data store. + +*Note: The Delta Calculating Microservice could maintain it’s state in a private SPARQL store to facilitate rebooting the component without loss of state should it go down. Rolling back should not happen as any update would be translated into an update of the form `WITH INSERT DATA { … triple … }`. In this case we can assume the underlying triple store to handle the cases where roll back is needed.* + +## Slave Master Microservice +The Slave Master Microservice maintains the slaves stores’ lifecycle including creation and termination. This microservice also manages a queue of queries and uses a load balancing algorithm to assign the query to a slave. When the Delta Calculating Microservice sends out a delta report the Slave Master Microservice will inform all slaves of the change in state (in essence nothing more than running the above mentioned ‘simple’ update). The Slave Master knows the latest update number for each slave and can take this into account before sending queries. + +## Conclusion +The ideas in this post build heavily on the reactive programming ideas presented in earlier posts. The main weakness of this approach is still the single point of failure that is the delta calculating component itself. This component could be made distributed by further enhancements.  It is currently the only component that processes update queries in this ecosystem. Maybe for a system with low write and high read access this would be a valuable asset. Keep tuned for the PoC! diff --git a/assets/README.md b/assets/README.md new file mode 100644 index 0000000..bc9d160 --- /dev/null +++ b/assets/README.md @@ -0,0 +1 @@ +Some of these files end with `excalidraw.svg`. This means that they were generated - and can be edited with - excalidraw. \ No newline at end of file diff --git a/assets/logged-in-user.svg b/assets/logged-in-user.svg new file mode 100644 index 0000000..e99c6e1 --- /dev/null +++ b/assets/logged-in-user.svg @@ -0,0 +1,149 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + "Aad Versteden" + + + + app:users/1337 + + + + app:accounts/1337 + + mu:session/account + + + + SID:USSESION:example.mu.semte.ch/ae86-gt86... + + foaf:account + foaf:accountName + mu:password + foaf:name + + + + + "madnificent" + + + + "sekrit" + + + + + + + "not-a-miata.png" + + + + "1c0f-fee4-1337-11..." + + + + "image/png" + + + + + file:///var/www/z4.png + + + + + + + + app:documents/E85 + + nfo:fileUrl + dc:format + nfo:fileName + dc:identifier + + \ No newline at end of file diff --git a/assets/shared-data-models.excalidraw.svg b/assets/shared-data-models.excalidraw.svg new file mode 100644 index 0000000..8cf3fe2 --- /dev/null +++ b/assets/shared-data-models.excalidraw.svg @@ -0,0 +1,17 @@ + + + eyJ2ZXJzaW9uIjoiMSIsImVuY29kaW5nIjoiYnN0cmluZyIsImNvbXByZXNzZWQiOnRydWUsImVuY29kZWQiOiJ4nO2YW0/jOFx1MDAxNMff+Vx1MDAxNFXndenE93jeXHUwMDA2yiwsMMN1uaxWKKRu422ahCS9MeK770mgcdpcdTAwMTRoXHUwMDE4diVcdTAwMDSpVCknvlx1MDAxY1x1MDAxZv9/PrZ/rjVcdTAwMWHNdFx1MDAxYanml0ZTTVxcx9ed2Fx1MDAxOTd/y+wjXHUwMDE1JzpcZuBcdTAwMTPO35NwXHUwMDE4u3lJL02j5Mvnz6ZGy1xyXHUwMDA397WUr1x1MDAwNipIXHUwMDEzKPdcdTAwMTe8N1x1MDAxYT/zf/iiO1nd8cXV1kC4nueNnO5+e7SxI/lFXjUvVDjj+zpKlPkwXHUwMDAxK2O0eJ9mfiFUvI91J/XARpBV2Dyle16aXHUwMDE5LV5cdTAwMTidoOdnXZhiSVx1MDAxYYd9tVx1MDAxOfphnHX9SVlcdTAwMDRcdTAwMTFkur523H4vXHUwMDBlh0GnKJPGTpBETlxmIzXlutr3j9Opf1x1MDAxZiPH9YZxaVx1MDAwMPe9nD24iVx1MDAxNuxFvSSEiJpa0G3PXHUwMDBiVJLM1Vx0I8fVaVx1MDAxNlx1MDAwMmSZcWQ+RjudPPR/XHUwMDFir2JnoHay2Fx1MDAwN0PfLzdcdTAwMWN0XHUwMDFlXHUwMDFhnk2RiT9+sNxcdTAwMTk3lerk/WGJXHUwMDA048JcdTAwMDTU6IQgtmj9XHUwMDFlXHUwMDA2uWaITVx1MDAwNLEthotcdTAwMDI6aYNW0rzVruMnysQ7c22rpCMzmGHUce6rIC65TW3KXHUwMDE4KanC10F/caR+6PZNL2ulYS1I83Zjet2+PWT/XHUwMDA07eveyfn4+1x1MDAwZZl4VWmmapLO69JcdTAwMTb2nC6pVdUlkqxFXHUwMDEw55xxKimiJUVcdTAwMTYyxexDpY+rdO7Dg1x1MDAxY4WgNrPhb4lcdTAwMWGRqFhnakTcJtySXHUwMDEy01x1MDAxN8hxzo+KIFx1MDAxOVx1MDAwMkWiXHUwMDFhgjT6ynRcdTAwMDWxPVI9XHLhdlJwuJGoeKTd0lx1MDAwNHXDID3Wtzmk1pz1mzPQ/nRuXHUwMDA2sia/+rqXhaPpq25JXHUwMDA2XHUwMDEwkVTD8l18TsOoLKZEget5rExcdTAwMTRd6MVcdTAwMDFrXFydqDDWPVx1MDAxZDj+yUqDyJrenqlcdTAwMWW1MHuCStpccslkM944XHUwMDE42INvRyeEk/hcXK9CJVx1MDAxNyZcdTAwMTB5tmBVKiVpXHRCXGKWQmBJJTbLk4HSaOR5KJHKfu9cdTAwMWNKIFx1MDAwMGIvKFmWIyhdNM6gXHUwMDA0JGE5L9d6LSapXHUwMDA04K2y+uoyeVxuXHUwMDEyXHUwMDBlIFx1MDAxOLU4JK/KYamXXHUwMDFhXHUwMDFjVlx1MDAxZF9k71x09Fx1MDAxMteadp3e2eWPtur+sX482e2xs1XQY9KAlEeCmUmdoSesXHUwMDE2seAhXHUwMDE2g1xyhbCNRD/QezF6XHUwMDA0c5sxXFxcbn8pXHUwMDFmWo/nQyFcdTAwMDVcdTAwMTewsXvJ9uxcdTAwMTn2OIVk+yvsba1cdTAwMGZcdTAwMWPtvznyXHUwMDE23a7B3e3u5ni8vz3oXHUwMDFj+0eXe2N5c3PBg1W4XHUwMDEzdD7lwVx1MDAwZfEj5f0vKc8mllx1MDAxNETwSnbLUl4lXHUwMDBmXHUwMDE23FGYXHUwMDA2S6JScnq9nIeptMvqq8vdgZMk4zDuvDnyqo7XYG+ERe8oujw/8ta3+4dcdTAwMDdk/Xe7u9IhkCMzzfkhXHUwMDEwLzlcdTAwMDQy/Pwh8Fx1MDAwM766h0BJYKHDJbGslPQwhz0qUPL6Z0DKXHUwMDA1JrisvrrstaHBRthtXFzrOPXeXHUwMDFjgI94X4PCTW9cdTAwMGbF/b3dP6eH+uD0qnN+dTGhK95cdTAwMTJyMr/5RKVcdTAwMTPsXGZELORcdTAwMTLySsZcdTAwMTXQk1x1MDAwMrn4XHUwMDFkoVfnlpBa2Fx1MDAxNoTRZUjyinFGpIR68HAzX69GpC05qbNcdTAwMGJ9Qptq9HV9n/bl2fbNqXT44CDwtk5WyVx1MDAxMJIsXl+LaobApGVBhlx1MDAxMFx1MDAxNsbCYlbpXHUwMDBl62XXhO9OpssyXHUwMDA0zL3NQY5Lr1x0S3vkxe2ZLZgkXGJb/8U1IbF/7Vi0XHUwMDE3wpr7Zu9cdTAwMDdcdTAwMWbx/pGLQfjPNdB0oug4hUhcdTAwMTbLUXOk1XijqulP3fxprj1QnOlZ5avY3drdv1x1MDAwN97jbCJ9 + + + + + Registration serviceaccountNameE-mailpasswordbirthdayLogin service \ No newline at end of file diff --git a/assets/simple-mental-model-advanced.excalidraw.svg b/assets/simple-mental-model-advanced.excalidraw.svg new file mode 100644 index 0000000..8d01b8b --- /dev/null +++ b/assets/simple-mental-model-advanced.excalidraw.svg @@ -0,0 +1,17 @@ + + + eyJ2ZXJzaW9uIjoiMSIsImVuY29kaW5nIjoiYnN0cmluZyIsImNvbXByZXNzZWQiOnRydWUsImVuY29kZWQiOiJ4nO1dWVPbyFx1MDAxNn6fX0ExVfdp0O3tdPeZN1xiYVx0+5aQ3JmihC1cdTAwMWLFxja22Hxr/vs9bVx1MDAxMmu1bMBcdTAwMDZzg7NcdTAwMTTIstTqPt9yevN/f1tYWIzuO8Hin1x1MDAwYovBXcVvhtWuf7v4hzt+XHUwMDEzdHthu0VvicHvvfZ1tzI481wiijq9P//97/hcdTAwMTNepX358KmgXHUwMDE5XFxcdTAwMDatqEfn/Yd+X1j47+D/xH26QSXyW/VmMPjA4K34Vlx1MDAxMlX26G67NbgtVyClRm3t8Iywt0r3i4IqvV3zm70gfsdcdTAwMWRa3Lizq58jYVb2cOl86zjC2qq9jG9bXHUwMDBim82j6L758FB+5eK6myhUL+q2XHUwMDFiwZewXHUwMDFhXbi7Z45cdTAwMGY/12tTXHUwMDE1xJ/qtq/rXHUwMDE3raDXS32m3fErYXTvjjE2PPpQXHUwMDBify7ER+5cXFx1MDAxNSj0XHUwMDEwpFFKoNVcdTAwMWFEXFwj7lx1MDAwMkJli/Kh3Wx3XVF+XHUwMDBmmOSSx4U59yuNOpWoVVx1MDAxZJ5cdTAwMTN1/Vav43epjeLzbn88pOBieOxcIlxi61x1MDAxN5ErMUePJV+J+1x1MDAwN4Oq51xc0F+tXHUwMDEyLePu2tmsXHUwMDBlwuDvuMK7/mWw6T7Sum42k3XWqv6os5/hXHUwMDEyXHUwMDA3jPxx5J/4sdz5XHUwMDFms4GWXGa2VMBFwV38rInowKhxsH59fNfb//xlr1x1MDAxMuDW5YfgcHF43j9/XHUwMDE0X/bhw2xHhWyrvb6+Ybebn/ZXOpVK0E7f5ef9/W63fZu47o+f4mq57lT9hyDmLr6pxZnmUlx1MDAwZt9vhq1Gts6a7UojjvvfXHUwMDEyXHUwMDA1XHUwMDFlXHUwMDAyruwph9cprKSHODTKXHUwMDAzLZhRzHDNldLpONTgQS6CuFx1MDAxNp6xUiqmQHHGQedcdTAwMDPKlCFgWiH92th+Mlx1MDAwMFJv/Fx1MDAwNFx1MDAxODCwXGZcdTAwMTXGtVx1MDAxOVOmZtmDQ8bUXFxI1GBjXHUwMDA2XHUwMDE5zZhZUKXK8fxcYo1cdTAwMDPOXHUwMDA1XHUwMDFh1e1R6Np+oePXg4W/Wn6nk2i7dis6XG777lx1MDAxMYRNXHUwMDFkXfMvw+Z9qvrd9ZabYd1VxWKFXG5cdTAwMWZ0U8JcdTAwMTKFJFLDXHUwMDEzLsNqNak8534voMJcdTAwMDfpeqzQrXw62t2cREba3bBcdTAwMWW2/OZxwZOlnsvdaGNIq56AQthOpJNKmlGtblx1MDAxNGjgNlx1MDAxMYvjZHJcdTAwMTfXu1x1MDAxMavJjeqx3Kx07f7aTVx1MDAxNedcdTAwMDdKxTKpJfckXHUwMDAwckQ0ysZcdTAwMTdcdTAwMTmQkzSeQMakXHUwMDE2mmulXHUwMDExMuV6dc00Vlx1MDAwYmH0XHUwMDE0XGLjXHUwMDA1XHUwMDE1s1x1MDAxMq7ytb3ubX8l4H63cX7YXHUwMDBmjz69gGKWXvf86mOr/6m9/fX47Duu9zeuXHUwMDBlard386jExbU3gVx1MDAxMmsjPSZRgJLKMm3SoW5wVKhcdTAwMGZFWVhPS3pxcpRW2KKIlVCCt3dRLlx1MDAxMmWrkSnkpkiUS9JcdTAwMTiQgqGh/+ZQlDerdKOwXHUwMDE2JjX0ddRYxPGYUeMxapVV46JHmqZcdTAwMTArXHUwMDE41dKW8IpcXCZaZJxcdTAwMTArPLtdbpyf6F6A31aPri4+X3XM/MCnWIjdXHUwMDEzelxurcNcdTAwMDOBXHUwMDAxZCZPeCBcdTAwMWZKXHUwMDA3XHUwMDA0XHUwMDAzZdFkXG726kosODPGXHUwMDAwiOLsNVZXJT6sNL/511dfP13ss5VDddv7fvNTTV5Lslx1MDAxZlx1MDAwMues3d1oVbvtu2NcdTAwMTVGh/tHX6o3+7UpaOv6zeXNl/vdtU2xebW/wXfPv271p6HZ91fy9Pbw48eNlf1TvrWn71t7zZspXFx35dOHq9bW57PvkamHWsEt9O33efRcdTAwMDJwtH1XNe3Tbu+uwXy+ub19Yy4m8Vx1MDAwMlxcXHUwMDFh8FxiJlpxSu64kjKNNiU8a1x1MDAxOGfWWIWUuVx1MDAxN7lcdTAwMDH0pC13XHUwMDAz4ldzXHUwMDAzs0B5kWmQlKswhlwil7O7sqHIXHUwMDFl/SklgikrmLR6LjP5qN1cclx1MDAxNv61cHTfqlx1MDAxNNtcdTAwMDaWOjrONjSDWlRiXHUwMDFhonan0DHwmL/TjiH1LLlkvbDwk1x1MDAxOYRcdTAwMDco1/tXSyeNw5XvsHZ+3ZD9fuh37GRQVrZMOJGnhDPh+4dQZsZDrjWRgZAggVx1MDAxN/S2PVxuyjxwf96hvDCJ/1dGgaGaL8SyXHUwMDFliWXiZUKzNfFcdPNcdTAwMDPl427YaVx1MDAwNj2HiTeH5MKyP1x1MDAwNsj26KRz+enGXFy2P5zK5YOjm898eXlcIiArKFx1MDAwNbKx44BMms5cYsZcdTAwMWE0iTandszjXHUwMDE4jFx1MDAwN5h8xVLwjutp4tpIXHUwMDAwxrTMXHUwMDAxeFx1MDAwMLTRne1WM665tfOY1++1XCJq5/r9Xy0qUeT/1TpcbirXXVffRShPyEhcbuUydd3J0/xROIc4hFx1MDAxZoHzXHSeJYv6QtBPlt5rXHUwMDFj1eCohUBcdTAwMDCchMdcdTAwMWZcYuZELNf7q/7W953ube3uTJxFvt2cXHUwMDFmsFx1MDAxNaf3yFxc/6LWjtfAXHUwMDEyh6W5TSrPMlx1MDAwNM5cdTAwMTllXHUwMDEyxiiVKdirp/d0XHUwMDA2s6BNop3eQld79eNu69Oaf8ZcdTAwMGXPztnB3t79l9VITdwl3rltbVx1MDAxZcJH2W1e9PdXP1Z2vu1M2CVeel11tbHfOunh8T7u9pubrHZ+fbwxhetGXHUwMDE3X1rnn1x1MDAwZva/7m2cXHUwMDA2rFpbP2tcXK1O4bp3a1xu75e7O9jo93tyaXn/ZHv35E1cZjnEXHUwMDE24YndXGbFbiWhZKNcdTAwMDdcdTAwMTAtouDAXHUwMDFmQWzFwTrvxKa4h4qIzVKFXHUwMDFiKTMjiEa/XHUwMDE0sXGlPe1cdTAwMWOdRsrylShcdTAwMWad+dlrqaXhiUJNZ2gjR17TjNPChGKiXHUwMDExXHUwMDA1V4WrIVVgVLlIOo3njTZkxlx1MDAxNcaoc9aGXHUwMDE0lSeXbVx1MDAxNLme54w2QFwiNcr6T4pjpZk0k1x1MDAwZjdcdTAwMWM09mpfXHUwMDFiXHUwMDA3347vXHUwMDBmb9Z2w5N1JVx1MDAwZbfnXHUwMDFktpxcdTAwMGLwXGKPhEmjJUJ6NJSgZKVlXFxcdTAwMTFydVx1MDAwMklPwOzvtZo11kzPiFxiNEKLZH/BWzBcIvfd7sXJgert6Cg83e2s3KzvLy+9tlx1MDAxMXnGuMRcdTAwMDRcdTAwMDJMlC9VXHUwMDEy1LNcdTAwMTBgwUamklxulYDUbK9xSC5upLlHsrKeRCxGsrVTQ3K5+mrtJUZJSiSXXHUwMDBiSoGUsmxcbtNcXN+K6lx1MDAxZVx1MDAwNvWQatyPXFzQzkZ3x6hQVneLSzR75cWR4/xcXDKFRmo2+cR0XGZ3XHUwMDFh6lhcdTAwMWVcdTAwMWT1Vlx1MDAxYTvdqlxmPsq+P/d4XHUwMDA1zT1mNFh6WmbIcJRJb6JcdTAwMGZzXHUwMDFlxFx1MDAxNyx305hcdTAwMTPzkd+C+Na/Wmhgey1cdTAwMTJb+7dwsLO+02l8f+1s/Vx1MDAxOZNcciZcdTAwMTFfaVx1MDAxMpM5ZyW+Vo5Cs1JgNFx1MDAxYT05mItcdTAwMWJp3sGspfIko1SfMl8rtUrPI8io7/PAXFyqv25cdTAwMTmBm5soKPU1zDCZh3deiq1gRqDFKaD5rSjxdptkcEZcdTAwMTI8Ro6yXHUwMDEynCnK7LVXj0QrV1x1MDAxNFx0zCo1+Wz3XHUwMDFh1ndh+/xT/aCmXHUwMDFhd+xcdTAwMGL3r9aO51x1MDAxZa7ZrDc9xKi49lx1MDAwMI2RrjqUMXa+xFdrZFx1MDAxYdU0ZuK+ZFx1MDAxN3zv8q62flirbW3J61O0IVvV7LW7tJ8xc25cdTAwMTLxNVxuMFx07JmIL4y20ii1RWH4I/qeXHUwMDBiW2nu4aylx8iUKq00cJFILlx1MDAwNnBcdTAwMDY7RTiXp7/SeOCmXHUwMDEwXHS6IdljMVEqTKdZt9zg15Hfw+Bh/XNvRlx1MDAxMjxGlfJZcK44M5dhrUfjXHUwMDE2XHUwMDE0eWZj5eS4ve1947X7PqxXt28/nK7VVm+/1Hpzj1t6Tk9ITcgkIZYsXHUwMDAz27lWYbCaMeQjprnPq1xir5xfqqhfjzrtI99cXPDNzZPTm9Zrj//Odpq5JFVgSVjPQoTl6Fx0ipwhXHUwMDA1JzxiIKm4keZcdTAwMWTLlNd6XHUwMDFjgWlrSIaTqHphXHLWzGNuQr5wXHUwMDAz0dqAzcO7oDfaTWmhlvqFNHgtbM5Mf8fIUVZ/M0V5jvZOMlKWoZN0XHUwMDFjs4HsaGEpXHUwMDFhUFx1MDAxOczsXHUwMDE3XCLI4Ek6R1x1MDAxYWmsXHUwMDEwiaGUeFx1MDAxYbn0SGqQeMdIa5Nd3bG+MOtZXHUwMDAxmlx0RJ6YXHUwMDE0N5XlISR1XHUwMDE1rMzRkvMpKqPIKeOwXHUwMDBmXHUwMDBiODdC5FaKOq1cdTAwMWXptMhcdTAwMTFcdTAwMThcdTAwMDGzWjz6JCx32mFW4uOfXHUwMDE24shcdTAwMTj8Mvz57z9cbs+mdIQhumC1ksI2XHUwMDE2IfdassKzXHUwMDAwXHUwMDE2tFx1MDAxNVJwKvW4y41cdTAwMGXtwfUyUVx1MDAxZF8tJ9NNv1x1MDAxN31oX16GXHUwMDEx1dq+e+Js7fRcIr9cdTAwMWKthK1q2KqnI+HHnkiTTDNcdTAwMTmQYOXa1eZcdTAwMTLzXHUwMDA0XHUwMDA3LZlcdTAwMDS3uJuqXCJRXHUwMDE3i3W/46LBXHUwMDEzXHUwMDA2yYtcbqSzwNVILtqCVnV8ocrH4Fx1MDAxMoVinnQ9OdqSapJHUFx1MDAxMK9IXHUwMDFhXHUwMDE2XG481zXLhdu1hGqV5cs0qKplR2VcdTAwMTeBn8NcdTAwMTOVOPle0kJcdTAwMTVzZ/lAR1x1MDAxOXei5Vx1MDAxZWpEXHUwMDEwhitcdTAwMGJcdTAwMTk/L1x1MDAwNHiMYo1cdTAwMWWTQkiwXHUwMDAy6nRDYkpcdTAwMTGOkVx1MDAxOWohVcCcXHUwMDA0c48pZo2jaEjaqnfufCp3XG5yqGR/XG65s2TpXHKXgprBiolmVjyOPoFxSFxmiL5cdTAwMTZ9XHUwMDEy3DyjgSHQo2opeZrvcrE47noj43twOS6Uh8xt/kX6XHUwMDAxblx1MDAxOGfeXHUwMDE4XHUwMDE0JJXPdWqCUsisSJz1QFboXHUwMDE5XHUwMDEwbt2NQlx1MDAwMUZwnVx1MDAwYrmJXGK0fFx1MDAwNCVdJkk3XHUwMDE0rrLodlZcdTAwMTSUiVx1MDAxYk9rRzqowE0/VWrWXGZa3ls9xn0qj1x1MDAwYoFuPlx1MDAwMFx1MDAwM2PSWZSkLEpzXHUwMDAwhZwsKmlHPJLzk0KRnlx1MDAxNsFQjJHCkeLH3jJhPsnDWovKKrqVfHefU2BQwq7UULh4UY6cZEpOiYhusr2l3qb9ZG5BrSQ6sFpcdTAwMTiAeLtcbvcyzFOWpMeB0lx1MDAxMJuocZdcdTAwMWJcdTAwMTnb7pWN6rniTjKfrkzWuo5Ow4ROKslcdTAwMGaeckt/NJNkXHUwMDA03d6cXHUwMDFj88E2XHUwMDExd5Z3fae5Uzm761beXHUwMDFh7lxme1x1MDAwMXdKXHUwMDBmXHUwMDFk3SjBtFFI/nnW3FneyVjqPlxyelxutVbk3YHCID1cdTAwMDFSatImRbrFpCHDjTa/lVx1MDAwM8HRXHUwMDEzTFx1MDAxMXqIX4nninZyUNKj5IFcdTAwMWMuXHUwMDA3Slx1MDAwZpJLXHUwMDA23snzqeTJhUtcdTAwMWEsVWpcdTAwMDF7UoOM9J9WuoV0elx1MDAwNv5Tka2Fx0xanpH/JIrUYMl4olZuJrVMfjxcdTAwMWaM4/3niFxiXHUwMDFm3Cyb3M9cdTAwMTmDasaIQiWCWz4pWT595+ChXHUwMDA0pGxmMFxurPjTXGK0vO8yVSRF5SFQc7LyXHUwMDAyScZyRdJkmVx1MDAwMVx1MDAwNuVliv7lizRl+iyfTVhqPTVcdCjZTkJcdTAwMDBS3On0dFQ67lk7yFrIbVx1MDAwYp3P3Vx1MDAxMZzGKadtZFutLuh2V9pDJK1XjOSGQixm6HfyfDJ5UqxcdTAwMDMyXZi789EzvTi1s5VsVuvrX997Lo1cZkj3yoXiS5Ddo9Ja4i+k3Fx1MDAwMI0g5k843SGzMCW0m9zDkJEyPDHTLt8jL1MkzshVXHRtKNtcdTAwMDaLXFzmXG7l0n+0rkddkp2npHvWdFe+cqmU7lx1MDAwNFJpkXFDibQ0WqY7K8fznUW3i+hgXHUwMDE5XHUwMDFhU0ZCTFx1MDAwNjHfWc9taiBISy24dcDvfPf8TJtZsvBK59d7LbrNRkbyXHUwMDFk5dmSJFjNaMuv1+e7kfHoXrlIfFx0unvEMIigzMwgMrcxm+bCJvpcdIr5riBcdTAwMTGdLt9cdTAwMTHdke3W4DbrXHUwMDEwlLfbfMJuPXDjZlx1MDAwNpWSVOGI+ZidLtuVz1YtZTvJ0NOWkn1NlSjJ5qXYTnFOXFzo4pn4jrnO5zzdXHUwMDAxSVx1MDAxMp1CXHUwMDAx5Fx1MDAxNt2qgj2VKLlcdTAwMDB3XHUwMDA154qJXHUwMDE22LS3Pfwl+Vx1MDAwZVxcf4ZM7oOU9Hejt0XjYFxm4zDJXGbCt0l3o+LRvZbyofhcdTAwMTKE95iuN7dBmZBumSNYrXRcdTAwMDG9pPtcdTAwMDPlSzg8XCIzyv7oXpZy1oIuSum+woRrN4GLpNTYmfu78qmB5eksZ1x1MDAxZSepcGNtgIZl5qNcdTAwMTFcdTAwMWa6lVqDPea4XHUwMDEwXHUwMDA1jJfJXHUwMDFmXG5cdTAwMDZSgHtujpBb9KVRkfK+XHUwMDBmRk8hoSXguu6pPLc5iVx1MDAxOT1n2lhmXdfu/63Byya0qdGPpXwsvlx1MDAwNOM9oq/MTZhcdTAwMWKMXHUwMDFkc04uXFzlXHUwMDFkXnr2XHKZ9Zk7PFx1MDAwMSQhxo1Wc5Aj8tnkkIGc+dyb8m/1KOM70NIjXGag+95cbpBkW1N0R97fXHUwMDAzpd2KbXJ4omDwwy1cdTAwMWNNXHUwMDE47KKFK9zN1Y9H0lx1MDAwMd7776ZAd0QxVJtcdTAwMDVbYTq9Lem/s0KY1JjJ/1x1MDAxOd2NXGZI91rKx+JL0F35V1dlXGaecns8XCI5POFcdTAwMTZsYt5N0TNcYjfLkNJMbVxy/X3idMPyr/BIMVx1MDAxZTNUa6QhqKTrXG5g+UJcdE9LqnalnSuSVlx1MDAxNlx1MDAwNO50Ka98978yyrNKelx1MDAxNtHN1Fx1MDAwNJbqgXugPJdcdTAwMDMgKCeVViQ6foaUxz16Pve2If/HWcGIxZS/W+/XZDhBTWO4LpxeKEdcdTAwMWE6cuhSaTnRvolvk+BGhV/m07Ojs4mJg+iMM8WU4ypycIJcdJM3S09cdTAwMWNufcRcYjCjjFx1MDAxOdxcdTAwMGVcdTAwMWJkJZ1cdTAwMGJm+dnSLkOlpFx1MDAxZlx1MDAxNFx1MDAxN6DdXHUwMDE2fvms+en89duPVlj0O52jiIJq+FiLN2Fwu1JcYnj3cp9cdTAwMWZcdTAwMTTAXHUwMDAxPlx1MDAxOODnn9/++Vx1MDAxZlx1MDAxNC9cdTAwMDdcIiJ9 + + + + + Single page appIdentifierStore & SyncTriplestoreOntologyDeltaSecurityDispatcherRegistrationLoginResourcesFiles \ No newline at end of file diff --git a/assets/simple-mental-model.excalidraw.svg b/assets/simple-mental-model.excalidraw.svg new file mode 100644 index 0000000..ad36134 --- /dev/null +++ b/assets/simple-mental-model.excalidraw.svg @@ -0,0 +1,17 @@ + + + eyJ2ZXJzaW9uIjoiMSIsImVuY29kaW5nIjoiYnN0cmluZyIsImNvbXByZXNzZWQiOnRydWUsImVuY29kZWQiOiJ4nO2ca1PiSFx1MDAxN4C/z6+g3Kr305jt+2W/oXhcdTAwMWRcdTAwMWNcdTAwMTlBZNzdsmKIXHUwMDEygSQmXHUwMDAxxS3/+9uJklx1MDAwNFximDAyhTMwU1x1MDAxNnS6Oyfd5znn9Okk/30qlbaCkWtu/VXaMlx1MDAxZlxyvWe1Pf1h63NYPjQ933JsdVxiRb99Z+BcdTAwMTlRzU5cdTAwMTC4/l9//pm00Fxmp//SyuyZfdNcdTAwMGV8Ve9v9btU+i/6q45Y7bAtcdyz1nHLPfs+OMNPYu9cbpVPqlHTqNJYmMB8XGaS0kdVRFx1MDAxMIx/j0KhKIl/P1jtoKPKIEVcdTAwMWEhUkomiUBcdTAwMDShuEbHtG47QdQsLtPt2154Mlx1MDAxMJf4ged0zV2n53ihXHUwMDEwf5hcdTAwMDBDXGZcdTAwMTM5rnWje+s5XHUwMDAzu1x1MDAxZNdcdDzd9l3dU9ec1Luxer16MOq9jJZudFx1MDAwNp65NXWWi7HMU+VxO99RY5u0Uqe97dim70+0cVxc3bCCcDwgSK4jlNE9akeT8G9cIpWn982jcFx1MDAxNuxBr5fu2G6/djxxwDfNsDJcdTAwMDacIVx1MDAwMEUy3IlmyJnCr45cdTAwMWQpXHTEUkiEMFx1MDAxMnFccsuvKO1cYqJOb/SebybjXHUwMDFhirCXaM6EXHUwMDE4XHUwMDAzt62/NIJMMkGBlFx1MDAwNDBcdTAwMWNcdTAwMWbvWXZ3uk3PMbpcdTAwMTnnibRKXHJt3Vxup75U02/NUtl1U1x1MDAxM+fYQd16XG4vXHUwMDAwgYnSfb1v9UZcdTAwMTNjXHUwMDFm9lbuWbfhQGz1zJuUXHUwMDAyqMFcYixcdTAwMDVHfDhw3LRcdTAwMWH5ppI6XHUwMDFhpmR4XGZ1XHUwMDE2XZV6s1PkeNatZeu9xlvyh71cdTAwMWWOVVx1MDAxZGqIRlx1MDAwN54/Z5F4QXGnzmrCbF+cn1x1MDAwZeDxY/fMOJgl0TON4IWUXHRcdTAwMWNcdTAwMTlJ4IpwJMnAjHFEMFx1MDAwM0BcYqVcdTAwMDbSn6TdXHUwMDA2yHHHMZDjOUsmXHUwMDA0v5Y8T3MqXHTGXGZcdTAwMTCQVsKxXHUwMDAxp2i6dMwplYRKXGZYXHUwMDAycn5M/46FS8SM1at1OWw2zlx1MDAxZH5zdffQXGYu9kb1WudLPHpcdTAwMTMqpnue87BcdTAwMTVcdTAwMWZ5/ryo3yt+YcHa1cm9u91cdTAwMWVcXJ98r4PWXHUwMDEzy9fv67dkXHUwMDAypk1cdTAwMGJcdTAwMDSEMS7SOL1hWlx1MDAxNvDVZGa9Su/279nll3vGXHUwMDBlnJs9hPN4urRcdTAwMDRzPVx1MDAxZFx1MDAxNlx1MDAxYVx1MDAxMVx1MDAxOGNOXHUwMDAwwlx1MDAwNHM6S9rG1Vx1MDAxNXV1kHAqmFx1MDAwNJBlQFx1MDAwNImYXHUwMDA3keSYUcyWQGjlnm7guo5cdTAwMTeUqvrI9D6en8uWPp+XW+DAkknFXHUwMDE0zptUjCWRmMA80/pC/OGjqDRcdTAwMDPEd07l9vWXRiBvKqK/PqjMXHUwMDFhgdDaYDnfkb+XkVg2IFx1MDAxOGNcdJH6z4hIlGilXHUwMDBl9Wf5uVx1MDAxY/5cYlx1MDAxMFx1MDAwNPiP+qN8JOC55lxyci4xJjJcdTAwMTVFvIVcdTAwMDKRV1x1MDAwZuXu9TnzTXlZqd93mvcuX3dcdTAwMTQgpHDdWUBcdTAwMTBwzilF68hCnepO0G18PfGPTs7w4X21xrujd2OBqlx1MDAwYkc/h1x1MDAwNTpcdTAwMTNFj1lcdTAwMTBcdTAwMDRcdTAwMTBcdTAwMWVcbpRcdTAwMWKFr/LAXHUwMDBiwFxyPmw38JHhidr+sC3XXHUwMDFkXHUwMDA1Idm6k8BcdTAwMDVDiLOftMj61Th4XHUwMDExtL/Tsu/OYTnoiGbryLtET7T8Pc9cdTAwMWFFklx1MDAxY9k4xLTNsuQ9liVSQEShXHUwMDEwWWt7gadcdTAwMGJjt1xymFx1MDAwMJCjVEpgfVx1MDAxNiYnluE5vulcci3D9D/cwmSO9EXSb7RefWxzp+X5j12gw6Nqdcg7edCDXGJMReyZ7Elccof5XHUwMDAx9UVcYiSyTOxcdTAwMDbEoiBiyVx0XHUwMDAwXHUwMDEyZXI4N8WGXHUwMDEwpVx1MDAwMkqYJ3r+6fmBwPHM0v9K9ZFtfDhcbrOFL1x1MDAwMuHdt+OnwYBefT1Eg73q0f3xfbNxnWs3ik3GR1x1MDAxODJtlkLKNCTDXHUwMDBmwlx1MDAxYzBAM7yhXG5zXHUwMDEyvXhcdTAwMWJDaIb/fnNcZlx1MDAxOVx1MDAxN1xcoTib1Fx1MDAwZaNCPl1cdTAwMTi7Q1wiIYeS51xu3Vx1MDAwYnLIlU1maVxyLMrhbs+anK+EwFRcdTAwMDJqgkA80clcdTAwMGZcdTAwMTOYnKVcdTAwMDCB02JPs7dcdTAwMDC9b/buQ9V1q+VcdTAwMWVcdTAwMWLUjreblap9mC89rqiCWFxyPJNcdTAwMDAhXGKTXHRcdTAwMWaDiJEkSkXUokFpyVxmlFx1MDAwMmZFpapcdTAwMTlcdTAwMTOCYiBcYlx1MDAwMlx1MDAxMGKaNNxQOe54IZVcdTAwMTJTolx1MDAwNm82S1x1MDAxZYUj8zeKIcOUUsjePUpV8S+XkKSVsyiWR211XCLrxpqTO4dYk4BDIFx1MDAwMaYqLiCpXbRVopp0V1x1MDAwMNWsSymAK66fVFx1MDAxMGly48i1vlxmr7f9k1x1MDAwM/MkXHUwMDBmrpyhXHKua4grYUhcdTAwMDWyKHNNXHTp3EUlXHUwMDEzmEtB39+JvtBK07pZlNaKpaYsMDq/XHUwMDAwrVmXUoBWXHUwMDA3KL35hs9deXh5vnNZXHUwMDFlom24kyuvXHUwMDAz1CAhXHRcdTAwMTlHKmJlIFx1MDAxOaJopSm4xmTqk1xcW4yrzICVajDdKrW/vYF13PFcdTAwMWJcdTAwMWLTTFx1MDAxMspQ9sb0fOcqoDKrXGauJa3pXHUwMDFjSjavSGPpTyqyWDtesy+mXHUwMDAwsbdP99vn3bOdO7p/PejipydLd0WudFx1MDAxMExlXHUwMDAwouFcdTAwMDBcdTAwMTnpIMA1XHUwMDA1tVx1MDAxYUOIMMU0pUfLpYM2VEZUckK5XHUwMDFh0ixcdTAwMWbK5uaDhEBYMlx1MDAwNFaQlv3hdWjDs9ye6YeJlVx1MDAwZpdcdTAwMGXKlL1INkjUz93+8ZD3nd1cdTAwMTYuf6tcdTAwMGabsFxczpeS5ZNcZipcdTAwMGY6wyCnXHUwMDFhXGLNXHUwMDE4ZURcdTAwMTJIQFx1MDAwNoKUazTtKFO3126QXHUwMDFjd7xcdTAwMTBJrtaSyuPhrNRcdTAwMTCbu6krlWuVdD0zQ6d2oKbvdvSPrSRcbvR/7LppXGa8cFx1MDAxODN9ZrFkkaEuXCJcdTAwMWRi5iY0teVQgNBcdTAwMWPXkstlZpOYXr/IeVNccjGkXHUwMDAwYUVhjrl+MVx1MDAwYpduu3E3ko1cdTAwMDFcdTAwMTnt48puu1aDkq1cdTAwMGY/2Vx1MDAxYviLXHUwMDAzeVx1MDAxY8ZcdTAwMDPvbkmWXFxcdTAwMDOM80eQXHUwMDAxrNaj73Bjy8SBQjv3q727eOmY+Yej4vkx6tL++t2j4+xofElcdTAwMDNAp0vj2zo5J5ySPPe7v6hcdTAwMDQ0rqtccnxcdTAwMDLN0aBROf9mu62Li8ZcdTAwMDfHXHUwMDFmydRu1XrgzyVcdTAwMTZcYvLUXdQrwv8tQJfy1lx1MDAxYkBcdTAwMGJcdTAwMDLK51x1MDAwMlxuXHUwMDAxoTRKTuZGtMxcdTAwMWFcdTAwMGZcdTAwMGaNu/6d22fn9rBb7dbuxbojqmpQjSyTa1tcdTAwMGZgmeBAMMw3wP5cdTAwMTbAsrnAXCLAgYrdUH6X6l1cdTAwMGZcclx1MDAwMfdcdTAwMGYqT6hcdTAwMGYh/mp877SqXHUwMDFmnNd1XGapYfg8IGJpU7phND766zE6P+pV/lx1MDAxNFxigHNtsr8mw1rN7i46deq9yo5+urf7dHpZMT86o2tcdTAwMTj3XHUwMDEyqfCEdONGf1x0RPM8XHUwMDEzNpVBmFDgmShQilx1MDAxOT3DPEPPXHUwMDE2gPHb38o7+3xcdTAwMDd6LZl5iFx1MDAxZUpCXHShPHNLZ7YwTiBcdTAwMGLKOSXvv6VT/Fx0wLjYdazphFfyrZToR/Qj/v7v58zaic6Fn22Y1P801W6rp/vBrtPvW4G6kFooxLTAfqB7wY5lty37dnJ6Xt9cdTAwMWVzlONx2sisXHUwMDE4g/BcdTAwMDKBpsy+oICIcFx1MDAwZVx1MDAxOFx1MDAxMlgknjBUXHUwMDFj3VxyXHUwMDA3dGbOTbv9tlx1MDAxNItfkzEhXHUwMDA1VDGwhIJcdTAwMTOMkEhcdTAwMDfEXHUwMDBiZIjGolx1MDAxY1x1MDAxYYKOqc9osZIwfSydc8y2PIvzl4ssj6BMXHUwMDAznFx1MDAxMDV8Uo1hXHUwMDEySbyYIaxcdTAwMTFcbqVSRqWVqbX7XCKbVOS+5o1NWmCTXHUwMDA0YVx1MDAxNGXvaaG5N2qphTmSkq7itsrwJVx1MDAxOUmss1x1MDAxZSZcdCXVV2eRctuCbWVcZiRnQDBCXHUwMDA1o1x1MDAxY+BkXHUwMDEzMDZcdTAwMDZogrflrNPivamURFBTQjAshORcdTAwMTIhNY2ztil8d5ZcdTAwMDAylFx1MDAxNqJwXHUwMDE3ddXGavFjkouMXHUwMDE1hFx1MDAxMC2yVvRccmuVUFx1MDAxM1urRUuLjbUqYK04YVxcXGKS+VwiXCIy96ZcdTAwMTg1niEnMNc7XHUwMDA2XG7aq6JcdTAwMGbMrshe4TnVV2evXHUwMDE2P3o+aa+AIIKo8JdQXHQwZjAl7auBIO9gr1x1MDAxNr9cdTAwMTdiMqZ7UyCqSUQpXHUwMDBlb+nFfFag5a3Vp9fZ2NJdt1x1MDAxZShtiq9oa2iZXHUwMDBmO7O4/3FcdTAwMTN9wvaRXHUwMDAwIepmRM7zp+f/XHUwMDAz/mph/CJ9 + + + + + Single Page AppSupport LayerMicroservicesStore & SyncClientIdentifierDispatcherMicroserviceTriplestoreOntologyDeltaSecurityMicroserviceMicroserviceMicroserviceMicroserviceMicroservice \ No newline at end of file diff --git a/docs/discussions/design-philosophy.md b/docs/discussions/design-philosophy.md new file mode 100644 index 0000000..58b3e59 --- /dev/null +++ b/docs/discussions/design-philosophy.md @@ -0,0 +1,87 @@ +# Design philosophy +In this document, you will learn the following: +1. The [core values](#core-values) of the semantic.works stack and their benefits. +2. How our [design decisions](#design-decisions) reflect those values. +3. What [implementations](#implementations) this resulted in. + +## Core values +### 1. Productivity through simplicity +- **1a) *Readability***: We wanted to get stuff done, without the requirement of being an expert on every topic, or creating a system that only the computer would understand. +- **1b) *Clear overview***: Efficient development also comes in the form of avoiding overbloated applications, where you can easily lose track of what is handled where. We want to be able to look at a project and know immediately what it does. +- **1c) *Less required reading***: There exists a saying in computer science that developers will end up with spending 90% of their time reading through code and 10% writing code. This reduces our effectiveness. For every line we write we will end up reading 10 lines first. So the question is: can we reduce the amount of lines that have to be written? + +### 2. Functionality through interoperability +- **2a) *Maximize freedom***: There need to be rules: as few and liberating as possible. A lack of rules would come at the cost of interoperability, which would subsequently limit your freedom in the end. +- **2b) *Expandability***: We also wanted orthogonal features: features that would extened eachother at no extra cost. +- **2c) *Longevity***: Something built years ago should still work as expected. +- **2d) *Allow reusing***: Re-writing the same boilerplate and/or copy-pasting functionality for every project is a waste of time. + +
+ +## Design decisions +### Micro +- By isolating functionality by the boundaries of a microservice you get small but focused services: making them reusable & easy to understand +([1a: readability, 1b: clear overview](#1-productivity-through-simplicity), +[2d: allow reusing](#2-functionality-through-interoperability)) +- They are oftenly very easy to read, allowing a basic understanding even for those who do not know the language or framework in question ([1c: less required reading](#1-productivity-through-simplicity)) +- Due to the small size of the services, debugging is easier: there are only a few hundred lines of code per service. Even if those 100 lines are important and take you longer to read, it reduces the area you need to look in ([1a: readability & 1b: clear overview](#1-productivity-through-simplicity)). + +### Standard API's +- Thanks to using technologies like HTTP and JSON:API, different services built years prior can still work as expected, no matter how they're built internally. ([2a: maximize freedom, 2c: longevity](#2-functionality-through-interoperability)) +- Using these standards means more assumptions can be made and less code needs to be read ([1c: less required reading](#1-productivity-through-simplicity)) + +### Semantic models +![A venndiagram of two circles. The registration service circle encapsulates E-mail, Date of birth, Username and Password. The login service circle goes over Username and Password](../../assets/shared-data-models.excalidraw.svg) +Semantic models have a bunch of advantages: +- They can easily be appended upon, meaning there's no limitation to what data you can store ([2a: maximize freedom, 2b: expandability](#2-functionality-through-interoperability)) +- They allow re-using the same models: the same model can be used for username/password, OAuth, or even mock logins ([2a: maximize freedom, 2b: expandability, 2d: allow reusing](#2-functionality-through-interoperability)) + +### Centralised communication +Microservices use a semantic model ([as seen above](#semantic-models)), but this is complemented by the fact that the microservices never talk to eachother directly. Instead they use a **shared linked-data** database. This replaces our need for a message queue. One can of course point out that such a message queue has several advantages, such as speed. But it also comes with a hidden disadvantage: namely every action taken or message sent has to be formatted in some way. This is yet again another dependency in your software system. Any microservice that you want include in your system and that needs to process the messages from certain other components will need to be developed with the knowledge of that message model in mind. ([2a: maximize freedom & 2b: expandability](#2-functionality-through-interoperability)) + + +## Implementations + +### Simple Mental Model +#### Basic +![A flow chart. Single Page App contains "Client" and points to Support Layer. Support layer contains "Identifier" and "Dispatcher", and points to Microservices. Microservices contains many microservices, and points to Store & Sync. This contains a Triplestore header, under which Ontology, Delta and Security are noted](../../assets/simple-mental-model.excalidraw.svg) + +#### In-depth +![A flow chart. Single Page App points to Identifier, Identifier to Dispatcher. Dispatcher points to 4 differently coloured blocks: they say Registration, Login, Resources, and Files. These all point towards a regular coloured block in between them. This block says Store & Sync, and it contains a Triplestore header, under which Ontology, Delta and Security are noted](../../assets/simple-mental-model-advanced.excalidraw.svg) +([micro](#micro), [centralised communication](#centralised-communication)) + +1. The **identifier** will create a session cookie. This doesn't identify you as a person (as this depends on what is in the database), but it gives a general hook ([centralised communication](#centralised-communication)) +2. The **dispatcher** will decide which **microservice** will be called for the made request +3. Then the **triplestore** will change the requested state, yield a response of what has changed in the database, read the information you requested... + +For example: this setup also means that the other services don't care what registration method was used, as they all work from the same [semantic models](#semantic-models) and database, yet care about different aspects of it. + +### Limited base technologies +- **HTTP for communication:** If you build web application, you're probably already familiar with HTTP. ([Standard api's](#standard-apis)) +- **JSON(:API) for sending data:** any programming language or framework that can read the widely known and used JSON will be able to be used. We then selected JSON:API to ensure consistency in our structure ([Standard api's](#standard-apis)) +- **SPARQL for storing data:** SPARQL queries are also just strings: any technology that can manage JSON will be able to use SPARQL in one way or another. This way we can store all our data as linked data, whilst enjoying the compatibility of JSON. ([Semantic models](#semantic-models)) + + +### Docker & Docker-Compose +Docker allows for most technologies to be used on most systems. Docker-Compose are YAML files describing the topology of Docker containers and how they interact whith eachother, which gives us a place to define a central structure of our project. ([Micro](#micro), [Standard API's](#standard-apis)) + +#### Naming conventions: +To ensure quick onboarding, a lot of our docker-compose files following a naming scheme: +| Filename | Description | +| ----------------------------- | ----------- | +| `docker-compose.yml` | General **production** setup, basis for all other docker-compose-*.yml | +| `docker-compose.dev.yml` | Overrides for **development** | +| `docker-compose.override.yml` | Overrides for specific systems/deployments. Do *not* commit | +| `docker-compose.demo.yml` | (Extra) Demo to help people get started | + + +
+ +### Categories +We structure our projects into different categories for easier re-use and discoverability. Please check out [project-categories.md](project-categories.md) + + +*This document has been adapted by Denperidge from...* +- *Aad Versteden's masterclass video series. You can view the original video [here]()* +- *Jonathan Langens' mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/08/31/how-mu-semte-ch-can-help-you-beat-the-10-odds/)* +*another Jonathan Langens' mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/06/15/semantic-micro-services-why-bother/)* diff --git a/docs/discussions/documentation-structure.md b/docs/discussions/documentation-structure.md new file mode 100644 index 0000000..c4b0ecd --- /dev/null +++ b/docs/discussions/documentation-structure.md @@ -0,0 +1,73 @@ +# Semantic.works Documentation + +Documentation is hard, but it is instrumental to helping your code be useful to people that aren't you, and that's a worthwhile pursuit. Even if you're making things just for yourself, the aforementioned "people that aren't you" descriptor also includes you in a few years, months or weeks. To have our documentation be as useful and in-tune as possible with the semantic technologies we craete, the following principles were outlined. + +## Design principles +1. [*Don't Repeat Yourself.*](#1-dont-repeat-yourself) +2. [*All-encompassing.*](#2-all-encompassing) +3. [*Ease of use: writing.*](#3-ease-of-use-writing) +4. [*Ease of use: reading.*](#4-ease-of-use-reading) +5. [*Micro-first.*](#5-micro-first) +6. [*Non-proprietary.*](#6-non-proprietary) + + +### 1. Don't Repeat Yourself +If we're copying documentation/text across places, we're doing something wrong: +- This may cause inconsistencies and out of date versions, breaching [(4) ease of use: readers](#4-ease-of-use-readers), as well as [(2) all-encompassing](#2-all-encompassing) +- This requires extra upkeep/work, breaching [(3) ease of use: writers](#3-ease-of-use-writers) + +### 2. All-encompassing +Many documentations are incredibly lacking, as you've probably experienced over your time as a developer. The divio documentation standard, which you can view on [documentation.divio.com](https://documentation.divio.com/), attempts to clearly outline *what* should be documented, and *where* you can organise it. While I recommend viewing the talk and/or text on their website, a [quick start is included in this repository's how-tos](../how-tos/quickstart-writing-documentation.md). + +### 3. Ease of use: writing +Minimising maintenance for the people who make the code is essential. Lots of people do not have the time, resources, training, or quite simply the desire to write down docs. This is one of the reasons we stay away from external documentation sites like gitbook: having them sign in to a whole different platform to write the documentation for their code would be way too much friction. This would also make us reliant on the external documentation provider, breaching [(6) Non-proprietary](#6-non-proprietary), and would require navigating to an external location for readers and writers alike, thus additionally breaching [(4) Ease of use: reading](#4-ease-of-use-reading). + +### 4. Ease of use: reading +- We have a bunch of projects going on, written in Ruby, Lisp, Elixer, JavaScript, Python... The documentations should nevertheless resemble eachother and be consistent. This is additionally made possible thanks to the the divio documentation standard, as seen in [(2) all-encompassing](#2-all-encompassing). +- End-users should not have to jump through hoops to read the documentation. If you have a local clone of a repository, you should not be required to have an active internet connection to read its documentation. + +### 5. Micro-first +Semantic.works is about microservices, so that means a bunch of documentation will be **small**. This means it is important to adhere to the structures of big projects involuntarily. An entire documentation folder for something that can be explained in one README is a no-go. Start from a single (README) file, and expand only if necessary. + +### 6. Non-proprietary +Our repositores and projects are made to work anywhere, so this should be the case for our documentation as well. Websites, services and hosts shutting down\* or changing their licenses\*\* is not unheard of. No editor lock-in, no host lock-in. + +- \*[Freshcode/freshmeat](https://jeffcovey.net/2014/06/19/freshmeat-net-1997-2014/), [Microsoft CodePlex](https://devblogs.microsoft.com/bharry/shutting-down-codeplex/), [Google Code](https://code.google.com/archive/) +- \*\*[Terraform/Hashicorp products](https://blog.gruntwork.io/the-future-of-terraform-must-be-open-ab0b9ba65bca), [Red Hat Enterprise Linux](https://thenewstack.io/how-red-hats-license-change-is-reinvigorating-enterprise-linux-distros/) + + +## Implementation +To achieve all of the above goals, we are putting the following structure(s) in place: + +### Write the documentation +All documentation ends up in one of the following places +- If it's about the semantic works stack in its entirety, it should go in the `project` repository (see: this document) +- If it's something useful for people getting started with the semantic works stack as a whole, it should go in the `mu-project` repository. +- If it's about a specific microservice or project, it should go inside of that projects repository + +The documentation should ideally be written as presented in [project/how-tos/quickstart-writing-documentation](../how-tos/quickstart-writing-documentation.md). This also covers how to tag new revisions. + +For microservice developers and most people working on semantic.works, this is all you have to do! This will ensure that most design principles are met. However, to have a more readily available & traditionally structured documentation, the following steps get executed. + +### Semantic.works +[View the fork on GitHub here](https://github.com/Denperidge-Redpencil/semantic.works) + +Semantic.works was (re-)developed to contain all necessary links & information about our repositories. This includes links & documentations to current & previous revisions. However, we ensured that *none* of this would have to be hardcoded into the site, thanks to... + +### App-mu-info +[View the fork on GitHub here](https://github.com/Denperidge-Redpencil/app-mu-info-rework) + +Which is a service that holds a linked-data database and an endpoint about repositories and their revisions. This includes links to the repos and images, as well as the contents of the documentation, both parsed and unparsed. And this service fetches its data dynamically thanks to... + +### Repo-harvester +[View the repo on GitHub here](https://github.com/mu-semtech/repo-harvester) + +Which is a microservice that - after some basic configuration - will collect & clone all repositories and images from specified accounts & hosts. It collects all the information app-mu-info's data model needs, including all documentation aggregated & parsed into the divio sections. The documentation stuff is thanks to... + +### Divio-docs-parser +[View the repo on GitHub here](https://github.com/Denperidge-Redpencil/divio-docs-parser) + +Which is a Python package that, when given a directory, will handle everything in terms of finding & parsing documentationo. It returns a class filled with fully parsed & easily useable Python dictionaries. + + +*This document has been loosely adapted from Denperidge's learning writeup. You can view it [here](https://github.com/Denperidge-Redpencil/Learning.md/blob/main/Notes/docs.md)*. diff --git a/docs/discussions/project-categories.md b/docs/discussions/project-categories.md new file mode 100644 index 0000000..9e8e24b --- /dev/null +++ b/docs/discussions/project-categories.md @@ -0,0 +1,84 @@ +# What are the project categories? +These categories structure our re-usable code. The semantic.works framework consists of a number of components. A number that is growing steadily and will keep growing in the future. If you’re new to the stack, it’s not always easy to find your way through it. Therefore, we introduced different categories, findable through naming conventions. By adhering to these naming conventions we try to facilitate the hunt for the component you’re looking for. + +All the rules described below are just some simple and straightforward naming conventions, but adhering to them may drastically speed up the search process in the future. + + + +## Applications (`app-*`) +A full project, combining microservices into a full application. An example structure can be seen below: +![A flow chart. Single Page App points to Identifier, Identifier to Dispatcher. Dispatcher points to 4 differently coloured blocks: they say Registration, Login, Resources, and Files. These all point towards a regular coloured block in between them. This block says Store & Sync, and it contains a Triplestore header, under which Ontology, Delta and Security are noted](../../assets/simple-mental-model-advanced.excalidraw.svg) + +## Core components (`mu-*`) +As the name reveals – core components are the core of the semantic.works framework. Without these components, the framework cannot function correctly. + +Core examples include [mu-identifier](https://github.com/mu-semtech/mu-identifier) and [mu-dispatcher](https://github.com/mu-semtech/mu-dispatcher). + + +## Services (`*-service`) +### Naming conventions: +- Repo: `*-service` (e.g. *acmidm-login-service*) +- Image: `org-name/*-service` (e.g. *lblod/acmidm-login-service*) +- Docker Compose name: `*` (e.g. *login*) + +In case a microservice needs to be configured in a specific programming language the name may also include the name (or an abbreviation) of the programming language name. For example, a service named export-ruby-service requires an export configuration in Ruby while export-js-service requires a configuration in javascript + + +Services are the configurable semantic microservices that compose the backend of an application, which don't require (a lot) more coding to implement a feature, generic or more specific. + +For example, `mu-cl-resources` is an easily configurable solution in case you need to "list" resources (e.g. a shopping cart, people in a database...). Frontends are hard to re-use as your customer will probably want a specific feel or implementation, but universal features can be re-used to fit the needs of multiple projects. + +Services in the semantic.works stack should... +- Contain hareable/re-usable/runnable functionality (see the paragraph above) +- Require minimal configuration (sensible defaults are very okay) +- Persist its state in the database +- Be user facing (usually) + + +Services can be found through... +- Searching for the `mu-service` topic on GitHub +- Finding projects extending the templates (e.g. `mu-javascript-template`) +- Asking around + + + +## Templates (`mu-*-template`) +These are a base for a custom built microservice with minimal overhead. From these you should be able to have a microservice going in your preferred language in minutes. + +Since the programming language of a template is important – you will need to develop the microservice in this language – the name of the language is included in the template name. + +Templates in the semantic.works stack allow... +- Minimal setup time +- Nudging (yet not enforcing) conventions +- Flexpoints: e.g. allowing template updates where new features & conventions get added with minimal effort + +There are also some guidelines to using a template most effectively: +- Ground truth in database + *(Caching is allowed, but keep in sync with the db)* +- Only talks to the database, not other microservices + *(If this happens, it's for a well thought out reason: it is the exception, not the rule)* +- Limit abstractions + *Keep the service easy to understand from a first read* +- Think about reuse + *(Split where others might reuse the service)* + + +To allow for easier development, the templates... +- Often support live reload, so you do not have to manually restart during development +- Don't require (nor recommend) cloning the template repository. Simply using the Dockerfile is enough, and allows for a cleaner codebase +- Some templates have a debugger on one of the ports + + +## Ember add-ons (`ember-mu-*`/`ember-*`) +### Naming conventions: +- We follow the Ember community naming convention by prefixing the package names with `ember-` +- To indicate when they are part of the semantic.works stack, we instead use `ember-mu-` (e.g. *ember-mu-login*) + +Like the configurable services, but for Ember.js frontend components and functionalities. These are the building blocks for the frontend application. Basically, it are just NPM packages you can install in your Ember app. + +## Utilities +Utilities is the collective term of all tools that facilitate the initiation and development of a mu.semte.ch application. Since they are so diverse, they don’t follow a strict naming convention. The name just tries to cover the functionality. Nonetheless, similar utilities have a similar name. For example, all generators are postfixed with `-generator``. + + + +*This document includes content adapted from Erika Pauwels' mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/06/22/find-your-way-through-the-stack/)* diff --git a/docs/discussions/sharing-authorization.md b/docs/discussions/sharing-authorization.md new file mode 100644 index 0000000..155a20d --- /dev/null +++ b/docs/discussions/sharing-authorization.md @@ -0,0 +1,84 @@ +# On sharing authorization +Authorization is a cross-cutting concern\*.  By carving out the pieces each component needs to know, we minimize dependencies, maximize code-reuse, and make the system easy to understand.  In this article, we’ll describe the inner and outer workings of the accessibility model. + +## Who needs to know? +Various entities need to have different information from the authorization system.  The database needs all knowledge, caching systems need a well-structured system for knowing whether a cached result can be used or not. + +The database needs to have full access to the authorization system.  It needs to know what information can be read, and what information can be written.  As all information about the users should be stored in the database, the database itself should be able to define the access rights for a particular user.  Requirement: the database needs to receive the user’s session.  To some effect, the database can be considered the authority on user access information.  The database can derive the access rights and interpret them based on the session identifier. + +Common microservices shouldn’t be bothered with access rights.  A simple microservice would execute operations (read or write) on the database in the name of the user.  If the microservice requests information which it is not allowed to see, the database will reject the operation.  Otherwise, the request passes through, based on the information this particular user is allowed to see.  Responsibility is thereby fully handed over to the database and the microservice needs no information about the access rights. + +Microservices are allowed to cache results in separate databases, as long as they abide the semantic model.  With authorization handled by the database, this model now also consists of the authorization information.  The construction of a cache could take many shapes, and optimizations may be dependent on the specific authorization scheme.  It is our goal to provide a usable interface for common caching scenario’s.  A cache should be able to automatically detect whether it applies to the access rights of a particular user, and whether or not it can be used for other users. + +In an ideal world, these caches can work automatically.  Any system may have unsupported cases.  We aim to support them with escape hatches as they appear. + +## Defining access right tokens +All operations which occur in the stack are based on SPARQL queries.  Hence, the access right tokens should apply to SPARQL queries.  We consider two interesting sets: one set contains all access rights of the current user, the other set contains the access rights used to answer the query. + +### Scoping access right tokens + +In order to intelligently apply and use access right tokens we need to be able to compare them.  The tokens are JSON objects with two keys: *name* and *variables*. + +Examples are: +```js +{ name: "user_basket", variables: ["42"] } +{ name: "public", variables: [] } +{ name: "suborganization_tickets", variables: ["rocket_engineers","nozzle_team"] } +``` + +In each of these examples, the *name* specifies a certain grouping, and the variables present a configuration of that grouping.  The logic constraints of these two properties will be described later. + +### Access rights and executing queries +The SPARQL endpoint has all information available to execute a query.  The endpoint will emit both the access rights of the current user, as well as the used access rights to answer the query.  Let’s explain that in a bit more detail. + +The access rights of the current user is an array of access tokens.  An example could be: +```js +[{ name: "public", variables: [] }, { name: "user_basket", variables: ["42"] }] +``` + +This user has access to all public information as well as to their own shopping basket.  These access rights will not change during normal operation.  Changes can only occur by logging in/out or explicitly receiving new access rights.  Hence the access rights can be cached. + +Now assume the user executes a query for the names of all publicly available products in the webshop.  In such a case, the “public” access token is used, but no content for the “user_basket” token would be used.  In fact, in a realistic scenario, no user_basket would ever contain a product name.  The access right system could share that the user had these two access tokens, but that only the *public* access token was used in this request. + +We specify the data structures of a token by use of the *name* property.  Each token with the same *name*, regardless of its *variables* may contain the same data structures.  The variables define scopings of the actual data.  the user_basket with variable 42 may contain different information than the user_basket with variable 68. + +The access right system shares all groups to which a user had access in the MU_AUTH_ALLOWED_GROUPS response header.  The groups which could have been used to answer the query is returned in the MU_AUTH_USED_GROUPS.  The latter contains all groups which, based on the structure of their data, could have helped to provide an answer. + +## Resulting properties +The authorization system has some interesting implications for practical implementation.  The most prominent ones are described in this section. + +Tokens are communicated as JSON properties in the MU_AUTH_ALLOWED_GROUPS and MU_AUTH_USED_GROUPS headers.  These base technologies are already part of the mu.semte.ch microservice requirements. + +A large set of microservices operate in the name of the current user, and can be implemented transparently.  These microservices can be upgraded solely by upgrading their microservice template.  Resulting in virtually no impact on existing services. + +The tokens for a particular session are near-constant.  The only very common case for changing tokens is logging in/out.  This means that for almost all requests the user makes, we can use a cached set of access tokens.  These access tokens can be stored in the user’s session. + +When executing a SPARQL query, the set of mu_auth_allowed_groups will always be a superset of mu_auth_used_groups. If a group with name foo was in mu_auth_allowed_groups and not in mu_auth_used_groups, then this will be the case for all further executions of this query, regardless of the variables applied to the tokens. Because of this we can... +1. Cache the result of a product listing query for a user which has tokens: + ```js + [{ name: “public”, variables: [] }, { name: “user_basket”, variables: [“42”] }] + ``` +2. Discover only... + ```js + [{ name: “public”, variables: [] }] + ``` + ...was used to answer the query by checking the mu_auth_used_groups +3. And use the response for a user with tokens ... + ```js + [{ name: “public”, variables: [] }] + ``` + ... or ... + ```js + [{ name: “public”, variables: [] }, { name: “user_basket”, variables: [“68”] }] + ``` + + +Results of executing multiple SPARQL queries can be simply combined. Say a user has `tokens [a,b,c,d]`. Assume a first SPARQL query has `used_groups [a,b]` and a second SPARQL query has `used_groups [a,c]`. The resulting required access tokens for the full response would become `[a,b,c]`. As such, it is easy for systems to combine the access rights for SPARQL queries. This property allows for simple `access-rights-aware` caching. + +## Conclusion +The authorization system can work by use of access tokens.  Most microservices can be upgraded merely by bumping their base microservice template.  The access rights tokens are JSON objects which allows us to inspect them.  The system is near-transparent for the end-user. + + +(*) The vast majority of authorization could be handled by the database. Based on who is logged in, we can define what information they are allowed to see or manipulate. Knowing the session, we could derive all information about access rights at the moment the user logs in. However, there’s a catch. Microservices are allowed to have local caches. As these caches can be stored locally, responses may be generated without requests to the database. In order to keep this path open, we need to be able to share access rights to these systems. + +*This document has been adapted from Aad Versteden's mu.semte.ch article. You can view it [here](https://mu.semte.ch/2019/01/02/on-sharing-authorization/)* diff --git a/docs/how-tos/adding-services-to-your-project.md b/docs/how-tos/adding-services-to-your-project.md new file mode 100644 index 0000000..70582a4 --- /dev/null +++ b/docs/how-tos/adding-services-to-your-project.md @@ -0,0 +1,9 @@ +# Using services/adding services to your project + +1. Add it to your docker-compose.yml + (See the service's README) +2. Add the endpoints in the dispatcher + [View the dispatcher docs on GitHub](https://github.com/mu-semtech/mu-dispatcher) +3. Fire up the new stack + - Restart dispatcher + - Run `docker compose up` diff --git a/docs/how-tos/creating-applications.md b/docs/how-tos/creating-applications.md new file mode 100644 index 0000000..5a64a8d --- /dev/null +++ b/docs/how-tos/creating-applications.md @@ -0,0 +1,81 @@ +# Creating applications +This document will help you get a basic understanding of how to spin up your own application. + +*Note: every service with an asterisk (*) is required.* + +## Adding mu-identifier * +This is the entrypoint of your application. It... +- ... identifies the browser agent (mu-session-id header) +- ... caches user's access rights (mu-auth-allowed-groups header) + +```yml +# docker-compose.yml +services: + # ... + identifier: + image: semtech/mu-identifier:1.10.1 + ports: + - "80:80" + links: + - dispatcher:dispatcher +``` + +## Adding mu-dispatcher * +The identifier selects the right service for a request + +```yml +# docker-compose.yml +services: + # ... + dispatcher: + image: semtech/mu-dispatcher:2.0.0 + volumes: + - ./config/dispatcher:/config +``` + +```ex +# config/dispatcher/dispatcher.ex + +defmodule Dispatcher do + use Matcher + + # This can be expanded down the road to allow paths to only accept a certain type (e.g. json, html) + # These are called accept headers + # See the mu-dispatcher docs for more information + define_accept_types [ ] + + # Defines a GET path on /example, including any and all subpaths (/example/test, /example/example) + get "/example/*path", _ do + # The servicename should be replaced with the service name defined in docker-compose + forward conn, path, "http://servicename/examplepath/" + end + + # General catch-all, returning a 404 error message + match "/*_", %{ last_call: true } do + send_resp( conn, 404, "{ \"error\": { \"code\": 404, \"message\": \"Route not found. See config/dispatcher.ex\" } }" ) + end + +end +``` + +## Adding virtuoso * +It stores your data! + +```yml +services: + # ... + dispatcher: + image: redpencil/virtuoso:1.2.0-rc.1 + environment: + SPARQL_UPDATE: "true" + DEFAULT_GRAPH: "http://mu.semte.ch/application" + volumes: + - ./data/db:/data +``` + +## Adding mu-cl-resources +Our frontends speak JSON:API, whilst our backends speak SPARQL. +Mu-cl-resources allows these two to talk to eachother easily, by taking an API description and generating an implementation from it. + +For information on implementing mu-cl-resources, view the documentation on the [semantic.works website](https://semantic.works/docs/mu-cl-resources). + diff --git a/docs/how-tos/creating-microservices.md b/docs/how-tos/creating-microservices.md new file mode 100644 index 0000000..6fa126c --- /dev/null +++ b/docs/how-tos/creating-microservices.md @@ -0,0 +1,27 @@ +# Creating a microservice +Creating a microservice is simply 2 steps! + +You can find templates on [semantic.works/docs](https://semantic.works/docs) or by searching [`template` on the mu-semtech GitHub page](https://github.com/mu-semtech?q=template) + +*Note: the code examples provided use the mu-javascript-template. If you prefer using a different template, adapt accordingly* + +## Step 1: Create a Dockerfile + +```Dockerfile +# Dockerfile +FROM semtech/mu-javascript-template:1.3.5 +LABEL maintainer="yourname@example.com" +``` + +## Step 2: Create app.* + +```js +// app.js +import { app, query } from "mu"; +app.get("/count", async function(req, res) { + const resp = await query("SELECT COUNT(*) WHERE { ?s ?p ?o }"); + res.send(JSON.stringify(resp)); +}); +``` + + diff --git a/docs/how-tos/creating-templates.md b/docs/how-tos/creating-templates.md new file mode 100644 index 0000000..6036492 --- /dev/null +++ b/docs/how-tos/creating-templates.md @@ -0,0 +1,3 @@ +# Creating templates + +Due to the complexity of creating a template (at least relatively to microservices & applications), a more in-depth tutorial is available at [../tutorials/creating-templates.md](../tutorials/creating-templates.md). diff --git a/docs/how-tos/deploying-applications.md b/docs/how-tos/deploying-applications.md new file mode 100644 index 0000000..c93b551 --- /dev/null +++ b/docs/how-tos/deploying-applications.md @@ -0,0 +1,3 @@ +# Deploying applications +Location: `/data/app-*` +Server-specific docker-compose: `/data/app-*/docker-compose.override.yml` \ No newline at end of file diff --git a/docs/how-tos/quickstart-writing-documentation.md b/docs/how-tos/quickstart-writing-documentation.md new file mode 100644 index 0000000..239a695 --- /dev/null +++ b/docs/how-tos/quickstart-writing-documentation.md @@ -0,0 +1,60 @@ +# mu-semtech documentation standards +- Our documentation follows the amazing documentation system used at Divio. [You can view the 30min talk and/or documentation on their website](https://documentation.divio.com/), and/or use the [checklist below](#checklist) to see if the documentation you've written fits. +- The docs... + - ... about semantic microservices/mu-semtech as a whole should be located in this repository ([mu-semtech/project](https://github.com/mu-semtech/project)). + - ... that would be good for a first-timer should be located in [the mu-semtech/mu-project repo](https://github.com/mu-semtech/mu-project) + - ... concerning specific microservices should be located inside their respective repositories. + + +## Checklist +This checklist is an attempt at boiling down the divio documentation system on how to write better documentation into something unobtrusive enough for people familiar with it to use as a checklist, but instructive enough for people unfamiliar with it to somewhat do the thing. + +### Tutorials? +For each tutorial: write as if you're *teaching a child how to cook*. + +Checklist: + +- [ ] Holds the readers hand from start to finish +- [ ] Gives a sense of achievement +- [ ] Works for everyone, with minimal explanation +- [ ] Also teaches what ***you*** take for granted + +### How-To guides? +For each how-to: write down how to achieve **1** specific thing as if it were a *cooking recipe*. + +Checklist: + +- [ ] Only 1 practical goal per how-to +- [ ] Minimal explanation +- [ ] Flexible: works for different but similar uses + + +### A reference? +For each reference: write as if you are writing an *encyclopedia* article. + +Checklist: + +- [ ] Structured like the codebase +- [ ] Doesn't explain common tasks ([how-to guides](#how-to-guides)) +- [ ] Doesn't explain basic concepts ([tutorials](#tutorials)) +- [ ] Fully describes the machinery + +### Explanations: discussions/background material? +For each background material: write as if you're writing about the *history and context* of a subject. + +Checklist: + +- [ ] Explains design decisions +- [ ] Considers alternatives +- [ ] Helps the reader make sense of things + + +## Releasing a new version +Once a new version of a semantic.works repository is made it should... +- Be tagged using `git tag` +- Have a DockerHub image with the same tag (NOTE: this happens automatically if your repository has a `.woodpecker/.tag.yml` configuration) + +*This will ensure that the documentations in app-mu-info get updated. See [project/discussions/how-tos/documentation-structure](../discussions/documentation-structure.md) for more info* + + + diff --git a/docs/how-tos/troubleshooting---slow-starting-containers.md b/docs/how-tos/troubleshooting---slow-starting-containers.md new file mode 100644 index 0000000..1fec9a5 --- /dev/null +++ b/docs/how-tos/troubleshooting---slow-starting-containers.md @@ -0,0 +1,25 @@ +# Troubleshooting - Slow starting containers using 100% CPU + +Some docker images used in Semantic Works stacks, notably those based on sbcl (common lisp) and elixir images, are very slow and CPU intensive to start if the limits of open file descriptors are very high for the container. This leads to a process using 100% of a CPU for some time before that container becomes usable. This can be worked around by setting the defaults for new containers in the docker daemon config (/etc/docker/daemon.json (create it if it doesn't exist)): + +```json +{ + "default-ulimits": { + "nofile": { + "Hard": 104583, + "Name": "nofile", + "Soft": 104583 + } + } +} +``` + +Or, if you want these high defaults for some reason, you can set per-container limits in a docker-compose file for each of the mu-project services: + +```yml + ulimits: + nofile: + soft: 104583 + hard: 104583 +``` + diff --git a/docs/reference/commonly-used-headers.md b/docs/reference/commonly-used-headers.md new file mode 100644 index 0000000..20d5811 --- /dev/null +++ b/docs/reference/commonly-used-headers.md @@ -0,0 +1,5 @@ + +| Header name | Set in | Description | +| ---------------------- | ------------- | ----------- | +| mu-session-id | mu-identifier | | +| mu-auth-allowed-groups | mu-identifier | | diff --git a/docs/reference/representing-logged-in-users.md b/docs/reference/representing-logged-in-users.md new file mode 100644 index 0000000..9ee0b31 --- /dev/null +++ b/docs/reference/representing-logged-in-users.md @@ -0,0 +1,53 @@ +# Representing logged in users + +![](https://mu.semte.ch/wp-content/uploads/2017/08/customumize_for_user-1024x768.png) + +How do microservices know who’s logged in?  [mu.semte.ch](https://mu.semte.ch) makes the assumption that microservices can work together automatically if they use the same semantic model.  Microservices know who is logged in by following the correct semantic model.  The semantic model to follow is explained here. + +The difference in life-span of a session and a user account is very different.  The session is ephermal, if a user agent doesn’t show up for too long we may want to clean its session information.  The user’s account is longer lasting.  If you revisit our app five years from now, we want to remember your profile. + +## Account information +The [FOAF](http://xmlns.com/foaf/spec/) ontology makes a difference between a User and a User Account.  This split allows you to have many accounts through which you can log in. + +The user information is commonly represented by an instance of class [foaf:Person](http://xmlns.com/foaf/spec/#term_Person).  This subclass of [foaf:Agent](http://xmlns.com/foaf/spec/#term_Agent) represents people with their naming info.  If you need access to display the name of the person, find it in [foaf:name](http://xmlns.com/foaf/spec/#term_name).  This contains the preferred name for the user, be it a full name, given name, or nickname.  Unless the application domain requires it, this name may not be unique. + +The account information is represented by an object of class [foaf:OnlineAccount](http://xmlns.com/foaf/spec/#term_OnlineAccount).  We attach the username to the account by connection of [foaf:accountName](http://xmlns.com/foaf/spec/#term_accountName).  An example would be the twitter handle, the email address, or the nickname by which you can login into the account.  If your users can login both through twitter and through username/password then a user can have two instances of foaf:OnlineAccount linked to their foaf:Person.  Find the accounts for a user by following the [foaf:account](http://xmlns.com/foaf/spec/#term_account) predicate from the foaf:Person. + +The distinction between foaf:OnlineAccount and foaf:Person is important when adding information to the user.  Most information will be attached to the foaf:Person.  Information may be specific to a single account, or attached to the user in general.  Each account through which the user may login may have its own icon attached to it.  The provider of the account stores the icon and we may want to display it.  This icon would be attached to the foaf:OnlineAccount.  If the user has an icon to be shown when the application refers to the user, then this icon should be attached to the foaf:Person.  The same goes for interests, age group, past orders, etc. + +## Session information +The session URI for each user is generated by the mu-identifier.  It is available in the MU\_SESSION\_ID header.  The session URI points to the foaf:OnlineAccount after the user logged in by use of the predicate musession:account. + +You may want to attach much more information to the session than just the account as explained [here](https://mu.semte.ch/2017/04/27/attaching-data-to-the-users-session/).  You could let users play with your application before they create an account.  Getting them hooked before they have to invest.  You may want to let them fill in their shopping cart, and only let them login or register when they’re ready to place their order.  A reactive microservice can move the information around as necessary. + +## Overview of used ontologies +A few ontologies, types, and predicates have been mentioned here.  We list them in this overview for easy reference. + +***Used prefixes*** +| Prefix | URL | +| --------- | ----------------------------------------------- | +| foaf | http://xmlns.com/foaf/0.1/ | +| mu | http://mu.semte.ch/vocabularies/core/ | +| musession | http://mu.semte.ch/vocabularies/session/account | + +***Used types & predicates*** +| Prefixed name | Description | +| ------------------ | -------------------------------------- | +| foaf:Person | Type representing a user on your site. +| foaf:OnlineAccount | Type representing an account through which a user can login to your site. | +| foaf:name | Predicate attaching a ‘name’ to a foaf:Person. Can be anything ranging from handle, nickname to official name. +| foaf:accountName | Predicate attaching a ‘name’ to a foaf:Account. Use this when showing multiple accounts of a specific user. | +| foaf:account | Predicate connecting the Person to his OnlineAccount. From Person to OnlineAccount. | +| musession:account | Predicate connecting the current session to the OnlineAccount by which the user logged in. From session URI to OnlineAccount. | + +***Used HTTP headers*** +| header | Description | +| ------------- | ------------------------------ | +| MU_SESSION_ID | HTTP header containing the URI of the current user’s session. Might not be used in the database yet. | + +Documentation of foaf can be found at [http://xmlns.com/foaf/spec/](http://xmlns.com/foaf/spec/). + +## Conclusion +The model mu.semte.ch uses for representing users has three levels: the session, the account, and the user.  Information can be attached to either of these three, microservices using this model automatically interact with the logged in user. + +*This document has been adapted from Aad Versteden's mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/08/24/representing-logged-in-users/)* diff --git a/docs/tutorials/creating-templates.md b/docs/tutorials/creating-templates.md new file mode 100644 index 0000000..1466f1f --- /dev/null +++ b/docs/tutorials/creating-templates.md @@ -0,0 +1,286 @@ +# Creating templates + +## The importance of templates +Templates... +- ... make it easier to implement new microservices +- ... help you get started with the technology you know and love, or help you understand the structure of a service written in a technology/language you barely know +- ... should be predictable. The naming and structure of templates +should therefore match certain standards +- ... make it easy to follow the constraints semantic.works places on services, and should therefore contain a basic set of features which ensures users can easily develop applications in them + + +Templates are our way to familiarize users with semantic.works and to ensure users follow the best practices where possibl + +## The steps of developing a template +A template should make sure that the semantic.works ecosystem keeps +running. The construction of a template happens in a set of stages. + +Building a template without knowing the domain is near impossible. + +### Step 1: Basic service +First one should implement a basic service in the base technology, to get a feeling of which technologies are suited for the template + +To begin, select the technology you would like to support. Look around for web frameworks which match the ideology of microservices. Search for the 'Ruby Sinatra' competitor for your language. + +Your next challenge is to implement a simple microservice for this +purpose. As an example, you can write a service that returns the amount of Tasks in the store, and which allows you to set a Task as done. The service itself is not of importance, you'll learn lessons about the framework when you implement the microservice. + +Don't forget to run your microservice in a toy semantic.works project. +That way you'll know that it actually works. +[mu-project](https://github.com/mu-semtech/mu-project) +offers a good starting point. + +### Step 2: PoC (Proof of Concept) +Next up is the construction of a PoC (Proof of Concept). + +Once your service works, you can start abstracting it. Much of the code you've built would be used again when you implement another service in the same language. Go over the list of the PoC, ensure you've implemented each feature, and put this in a separate project. This project will become the template to use for the Tasks service. If all goes well, you'll end with a PoC template service, and a much shorter Tasks service. + +#### PoC requirements +- Respond to HTTP requests + - Parse JSON & JSONAPI payloads + - Support implementation of GET PUT PATCH POST DELETE + - On port 80 during production + +- Send out SPARQL queries & updates over HTTP + - Perform JSON request to SPARQL HTTP endpoint + - Parse JSON query response + - Support Virtuoso Open Source [[1]](#1) + +- Generate HTTP response + - Build JSON & JSONAPI body + - Set outgoing HTTP headers + +- User-code + - Load user-code from `/app` + - Download & install dependencies on startup + + +### Step 3: Full template +Then comes the standardisation, which requires some extra features to ensure the template can be used in practice. + +This includes going over the [list of features needed for the full template](#full-requirements). +Go over the list of features and try to implement them in a clean +fashion. More complex, and realistic microservices, like those +automatically creating Tasks for all starred books you haven't read yet now become possible. This is the space where your microservice should be in. + +Document the use of the microservice, and get us to publish it on our GitHub space. + +#### Full requirements + +- [**All requirements for the PoC**](#poc-requirements) + +- Configuration + - Access environment variables [[2]](#notes) + - Support live-reload for development [[3]](#notes) + +- Respond to HTTP requests + - Parse incoming HTTP headers + - Destructure request URL + - Parse query parameters [[4]](#notes) + +- Send out SPARQL queries & updates over HTTP + - Set headers of the SPARQL HTTP request [[5]](#notes) + - Provide escaping methods for SPARQL input + +- User-code + - Load user configuration code from /config + - Use ONBUILD to automate template extension [[6]](#notes) + - Support the generation of UUIDs, namely uuid-v1 + +- Enhancing workflow + - Use /template-name/setup.sh for downloading all dependencies + into the image [[7]](#notes) + - Use /template-name/startup.sh for launching the webserver. + - Use the ENVIRONMENT variable to enable development mode & + live-reload [[8]](#notes) + - Debugging console (if possible) + - Enable debugging symbols (if possible) + - Enable error output (if configurable) + + +## Extra considerations +### Levels of abstraction + +Most developers like to abstract things. Make sure that nothing is +repeated. Many of us like to get started quickly. These two ideas often don't go hand-in-hand. We dismiss non-generic abstractions and require consistent naming. + +Getting started should be trivial and has the highest priority. It +should be intuitive to start using a new template. It should be easy to read code that was written in a template, even if it isn't yours. + +The following is a description of various topics on which consistency should be held. The description should be adapted to the use of your language. Ie: if you'd call sparql_query in Ruby, sparqlQuery in Java and sparql-query in Common Lisp. + +#### Defining HTTP endpoints + +Users will define HTTP endpoints to which their service will respond. +Separate methods should be defined for separate types of call. The +method should simply contain the name of HTTP verb, and the URL to which we respond. The system should allow the user to destructure the input and access the destructured contents. Calls may be defined in a block, but if that is the case, then the block should be repeatable. + +**Essentials:** +- Use simple name of HTTP verb +- Allow straight-forward destructuring of URL +- Support finding query parameters + +Example in Common Lisp: + +```lisp +(defcall :get (“say” “hello” “to” name) + ;; responds to GET /say/hello/to/Aad?title=Sir + ;; name is bound to Aad + ;; (get-parameter “title”) yields “Sir” +) + +``` + +Example in Ruby: +```ruby +get “say/hello/to/:name” do |name| + # responds to GET /say/hello/to/Erika?title=Madam + # name is bound to erika + # params[:title] yields “Madam” +end + +``` + +Another example in Ruby: +```ruby +define_calls do |server| + server.get “say/hello/to/:name” do |name| + # responds to GET /say/hello/to/Erika?title=Madam + # name is bound to erika + # params[:title] yields “Madam” + end +end +``` + +#### Executing SPARQL queries + +We should mitigate SPARQL-injection whilst still ensuring the SPARQL +queries are easy to recognise. Allow for injecting variables in a +SPARQL-query string. Support escaping as separate methods. Query results +should be parsed automatically so the user can easily process them. + +A SPARQL endpoint often differs in query and update endpoints. Hence we +advise to use those two names for executing the queries. + +For languages which don't allow injecting variables into strings, magic +is allowed. Keep it to an absolute minimum and try to mimic what is +commonly found in other languages. It should be something which you can +explain in a single sentence. + +Common Lisp example: +```lisp +(sparql-query “SELECT ?s ?p ?o WHERE + GRAPH { + ?s a foaf:Agent; + foaf:name ~A; + ?p ?o. + }”, (sparql-escape-string “Felix”)) + +(sparql-update “INSERT DATA + GRAPH { + ext:Aad a foaf:Agent. + }”) +``` +J + +Ruby example: +```ruby +query “SELECT ?s ?p ?o WHERE + GRAPH { + ?s a foaf:Agent; + foaf:name #{username.sparql_escape}; + ?p ?o. + }” +``` + +### Helper functions +Ideally, your template should expose some helper functions, including +but not limited to the following. Extra parameters can be added as +needed or wanted. + + +| Function name | Parameters | Description | +| ------------------ | --------------------- | ----------- | +| error | title: `string`, http_error_code: `int` | Returns a JSONAPI compliant error response with the given status code (default: 400). | +| query | query: `string` | Executes the given SPARQL select/ask/construct query. [This function should automatically pass the correct headers](#passing-headers). Also see `update`. | +| session_id_header | request: `object` | Get the session id from the HTTP request headers. | +| sparql_escape | value: `any` | Detects the value type and runs the appropiate [sparql escape function](#sparql_escape). | +| update | query: `string` | Executes the given SPARQL update query. [This function should automatically pass the correct headers](#passing-headers). Also see `query`. | +| uuid | None | Generates an uuid. Meant for use in linked data (`mu:uuid`). | + +*Note: camelcase is used in this documentation. However, when writing the helpers in the language of your choice, use the notation that is in line with the language in question.* + +#### sparql_escape +The following helper functions all have the same description/funcionality, but for different types. As such, the description column is omitted. + +**Description:** Converts the given object to a SPARQL-safe RDF object string with the right RDF-datatype. +This functions should be used especially when inserting user-input to avoid SPARQL-injection. + +| Function name | Parameters | +| ---------------------- | ----------------- | +| sparql_escape_string | value: `string` | +| sparql_escape_uri | value: `url` | +| sparql_escape_decimal | value: `decimal` | +| sparql_escape_int | value: `int` | +| sparql_escape_float | value: `float` | +| sparql_escape_date | value: `date` | +| sparql_escape_datetime | value: `datetime` | +| sparql_escape_bool | value: `bool` | + +#### Passing headers +The following headers should be passed when making queries: + +| Header name | Type | Description | +| ---------------------- | -------- | ----------------------- | +| MU-SESSION-ID | | | +| MU-CALL-ID | | | +| MU-AUTH-ALLOWED-GROUPS | | (bidirectional) | +| MU-AUTH-USED-GROUPS | | | + + + + + +## Notes + +1. This constraint should be dropped in the distant future, but it is + our default choice at the time. If you don\'t do anything special, + this will be ok. + +2. Docker makes it easy to set POSIX environment variables. These are + the variables you\'d be able to access in a shell (`$PATH` is a + good example). These should be accessible so services can override + your service. + +3. Live-reload does not exist on all systems, but you can almost always + fake it. Some languages need a restart (like nodejs and ruby), for + some others you should connect to the service and reload the code + through your editor (like Common Lisp with slime). This reloading + does not need to auto-reload the dependencies, as these rarely + change and thus warrant restarting the service. + +4. Query parameters is the content that goes behind the question mark + (?) in the URL. It constitutes of key-value pairs in which the + keys needn\'t be unique. + +5. This requirement comes forth from the work Jonathan Langens is + performing on Sharing Knowledge (link should be added to basic + ground work in the field). + +6. The ONBUILD statement of Docker allows you to execute commands when + someone extends your image. This means you can automatically copy + the relevant contents of the repository in your /app and your + /config. This further reduces the overhead of implementing a new + template. + +7. This construction makes it predictable how extensions of this + template should behave. If someone builds a template on top of + your template (inception-style), then it\'s easily understandable + what needs to be called. + +8. It may be that development mode can not be enabled using an + environment variable as it requires too many extra dependencies. + This is strongly advised, but it is an option. In that case, the + development image should be maintained in the same repository as + the template repository. The name of the development should be + `semtech/-dev` diff --git a/docs/tutorials/developing-inside-containers.md b/docs/tutorials/developing-inside-containers.md new file mode 100644 index 0000000..e2623b0 --- /dev/null +++ b/docs/tutorials/developing-inside-containers.md @@ -0,0 +1,84 @@ +# Develop with your local IDE and tools inside a Docker container + +## docker-compose +[mu.semte.ch](http://mu.semte.ch/) applications consist, in the back end, of microservices. Those microservices are docker containers that run a lightway server framework and offer logic on certain routes. Those routes are described in the dispatcher and the composition of the microservices is described in the docker-compose.yml file. + +Generally it is undesired that a microservice would talk to another microservice. In exceptional circumstances this is however allowed. We will maybe at some other point publish a blog post describing these cases but for this post I will use the example of a cache microservice. Conceptually this is a simple idea that caches the responses that a microservice returns on certain requests. In this case you would need to know the calls that come in from the front end, through the identifier and the dispatcher and the responses that come from the microservice in question. + +It is clear that mocking this is non-trivial. + +## front end complexity +Within the [mu.semte.ch](http://mu.semte.ch/) applications a lot of complexity is sewn together in the front end. The back end is an API that offers logic on the world (SPARQL endpoint) and the front end calls it. There are many examples of times when it is hard to figure out exactly what is going wrong. You could use a library that abstracts the API you are consuming away but how will you then know exactly which calls it makes? + +You could open your developer tools and copy the network calls but this would still fail to capture the mutations that happen in the identifer and the dispatcher. Furhter it can be really handy to just click and see the calls happening while your break points are conditional and thus only fire when needed. Clickig costs less time than fabricating curls. + +## wrapping 3rd party applications +While it is true that it’s not too hard to wrap applications like SOLR, Elastic Search, Spark, Unified Views, … in a docker. It is far more convenient to see how the tool reacts when you can run it locally as opposed to in a docker. While this is certainly not the largest use case it can be frustrating to figure out what is wrong with such an application when something goes wrong. In this case it also helps if you can develop on localhost. + +## so we develop locally +One could exec in the running docker and dev from there. Most major language offer reasonable dev and debugging tools for the command line. But still this never matches the full blown capabilities of an IDE such as IntelliJ, Visual studio, Atom or \[you favorite here\]. + +![](http://mu.semte.ch/wp-content/uploads/2017/07/mu-proxy-1024x768.png) + +But we also want to have the convenience of having the docker composition, the complete application, there. The wiring can be complex, so can the logic, so mocking this entirely is in most cases unfeasable. + +Therefor we use, for these use cases, a proxy that proxies the calls that would be made to your microservice in the docker-composition (dispatcher by the dispatcher to “http://your-microservice:port/route”) to your localhost. + +## how to +Suppose we have a docker-compose.yml file that somewhere between it’s lines has the following: +```yaml + # ... + - ./config/resources:/config +YOUR_MICROSERVICE: + image: your_microservice:1.0 + ports: + - "3000:3000" + volumes: + - ./config/your-microservice:/config +``` + +Then when running in the complete application your microservice would receive it’s real calls on port 3000 within its container. To have this redirect to your localhost we use just another instance of the dispatcher image, changing the configuration to: +```yaml + # ... + - ./config/resources:/config +YOUR_MICROSERVICE: + image: semtech/mu-dispatcher:1.0.1 + volumes: + - ./config/localhost_dispatcher:/config + ports: + - "3000:80" +``` + +Just as the standard [mu.semte.ch](http://mu.semte.ch/) dispatcher the localhost dispatcher has a `dispatcher.ex` file. Here you would find something along the lines of: +```ex +defmodule Dispatcher do + + use Plug.Router + + def start(_argv) do + port = 80 + IO.puts "Starting Plug with Cowboy on port #{port}" + Plug.Adapters.Cowboy.http __MODULE__, [], port: port + :timer.sleep(:infinity) + end + + plug Plug.Logger + plug :match + plug :dispatch + + match "/*path" do + Proxy.forward conn, path, "http://[YOUR LOCALHOST'S IP]:5678/" + end +end +``` + +As you can see this will forward any incoming call on port 80, which is in the `docker-compose.yml` file mapped to port 3000 ensuring that this dispatcher will serve (as an underboundary) the calls that your microservice would need to support. + +A second observation would be the exposed port on your localhost, that can of course not be 3000 in this setup because we already fire a docker on that port. + +## tada +To make this work you only have to fire up whatever you want to test locally on port 5678 and TADA you can dev on your local machine with your local tools as if you were running everything in a docker and in a docker-compose.yml composition. + +Of course sane thing to do here is to use emacs for development. The terminal version can be installed in a docker and you can dev by exec-ing into it :). + +*This document has been adapted from Jonathan Langens' mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/07/20/develop-with-your-local-ide-and-tools-inside-a-docker-container/)* diff --git a/writeups/perspectives/all-or-nothing-fails-to-innovate.md b/writeups/perspectives/all-or-nothing-fails-to-innovate.md new file mode 100644 index 0000000..916f867 --- /dev/null +++ b/writeups/perspectives/all-or-nothing-fails-to-innovate.md @@ -0,0 +1,76 @@ +# Experimentation + +## All or nothing fails to innovate + +Introducing new technologies is hard.  Very hard.  No matter how promising the technology is, the amount of unknowns is daunting.  Who knows what the hidden risks of [Ballerina](http://ballerinalang.org/) will be in practice? The price of technical innovation is too damn high.  mu.semte.ch makes experimentation easy, and cheap. + +[![The price of innovation is too damn high -- made at imgflip.com](https://i.imgflip.com/1k6pqj.jpg)](https://imgflip.com/i/1k6pqj) + +### Monetheism is not your future + +Hind-sight 20/20.  As easy as it is to see what turned out to be the best way, it is hard to figure out what will turn out ok in the future.  Should you guess on building your stack on the über-efficient [Elixir](http://elixir-lang.org/) language?  Will you find the devs?  How about [Haskell](https://www.haskell.org/)?  How about just taking the safe bet that you’ll find enough devs tackling the problem in [Java](https://java.com)?  It is no easy choice. + +It is hard to find good developers.  In any language.  But it seems that some rock-star developers have found the holey grail in one or another strange language.  It happened with [Ruby](https://www.ruby-lang.org) as [Ruby on Rails](http://rubyonrails.org/) became big.  Some great thinkers swear by the simplicity of Haskell, others by the speed of Elixir.  We can spend decades arguing our choice is the best.  There is a catch though, we do tend to conclude that all options have their merits.  Most options have their own reason to exist.  The best solution simply depends on your problem. + +When we look at specific tasks, the problem of choosing a programming language shifts from a religious war of _“the best programming language ever”_ to _“the best language for the problem”_.  A much simpler question, with a much more reasonable answer.  Need to tackle many long-running requests?  Better pick Elixir.  Want to see how purely functional programming would work in practice?  Use Haskell.  Do you think it’d be easier if the backend and the frontend used one language?  Javascript is your friend.  In the holy war of programming languages, monotheism is dead. + +As much as I’d personally like for us to agree that [Common Lisp is sent to us from the gods](https://www.youtube.com/watch?v=5-OjTPj7K54), I’ve learned that most others don’t agree.  It has its place, but betting on it for the sake of your company is an HR suicide mission.  mu.semte.ch uses microservices, all wrapped in their [Docker](https://www.docker.com/) container, meaning they can all choose their programming language.  For an innovation-driven company, the hard choice is figuring out a minimal set of hard constraints on the implementation, so a wide array of unknown choices can be made in the future, without causing extra friction. + +It is not an easy choice to pick the right programming language, so why make it?  Making the choice stifles innovation.  It is not to be made. + +### The lack of priests + +Clearly, there are downsides of having loads of programming languages in your active code-base.  A single language brings benefits.  There must be a balancing act to be made. + +As everyone can introduce new programming languages, how do you keep an oversight over the space?  There is an inherent risk accompanied in using a microservice in a language you don’t know.  Or, god forbid, the lone Haskell developer leaving your company.  Fear not, these are not the risks you are looking for. + +*I don’t understand a microservice in a strange language.* A common worry would be that developers have problems fixing an application in a strange programming language.  This can be true for complex applications, but small microservices are much easier to understand.  mu.semte.ch augments this with templates, offering basic constructs with similar naming in a variety of programming languages.  This makes it easier to dive into a piece of code you’ve never seen, and understand what is going on.  Worst case, it’s a microservice.  You take ownership and rewrite it in your own language of choice. + +*How do I adapt a microservice in a strange language?* Each microservice should be configurable.  It turns out that basic configuration can happen through environment variables.  Meaning it’s exactly the same, regardless of the underlying programming language.  More complex configuration may need to happen in the common method for the specific language.  In Ruby, YAML is common to be used.  In Lisp, S-expressions can be used.  Regardless, the syntax of the configuration is contained in examples in the sources or the README, you just follow that syntax. + +*What if my rock-star Ballerina dev leaves?* It is supposed to be microservice.  The hard part is figuring out what the service should entail exactly.  For the large majority of the services, it holds that they can be rewritten in a matter of days, given knowledge on the requests that will come in.  Fact of the matter is, that a motivated dev is way faster than a bored one.  It’s better to keep using the microservice of your rock-star dev, and replace it when substantial changes need to be made. + +There are other risks on using many programming languages.  There must be.  But they are greatly overshadowed by the benefits.  Namely the motivation and efficiency of the developers turning your company strategies into an automated version of reality. + +### An all-inclusive playground + +All or nothing fails to innovate.  We should start embracing the new kids on the block.  Languages, or frameworks, with a limited track-record could be of great benefit.  I will give Ballerina a swirl, a language released just a few days ago.  I’ll test run it to appreciate its benefits, and discover its shortcomings.  For a virtually non-existent cost, I will have replaced an existent microservice in a real-world application. + +Diversity is what makes all ecosystems thrive.  It is time to apply it to your company. + +*This part has been adapted from Aad Versteden's mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/02/23/all-or-nothing-fails-to-innovate/)* + +## mu.semte.ch as the ideal playground +![](http://mu.semte.ch/wp-content/uploads/2017/01/juggling_with_languages.png-200x300.png)The [mu.semte.ch](http://mu.semte.ch/) framework is the ideal playground for the developer that wants to try a new language. Using the standards set out by the framework you can, in literally no time, get a (tiny) microservice up and running in your next favourite language. We are ever expanding the range of languages covered by the templates. + +### What are these templates? +Within [mu.semte.ch](http://mu.semte.ch/) a template is everything a microservice needs except for your domain specific logic. Take the javascript template ([https://github.com/mu-semtech/mu-javascript-template](https://github.com/mu-semtech/mu-javascript-template)) for instance: to start a new microservice you first create a directory and a Dockerfile. In that Dockerfile you specify the FROM as the template +```Dockerfile + FROM semtech/mu-javascript-template +``` + +and then you add a app.js file where you enter something like +```js + import { app, query } from 'mu'; + + app.get('/', function( req, res ) { + res.send('Hello mu-javascript-template'); + } ); +``` + +With this you have your first javascript microservice that will respond to the ‘/’ route with a simple hello world string (I know not entirely true, you will also need to configure the dispatcher but for that look at our docs on [mu.semte.ch](http://mu.semte.ch/)). After this you can go wild with javascript and you don’t have to worry about the boring stuff that make the webservice tick, allow you to write/read to the database, etc. + +### Other languages +If we compare this with the [ruby template](https://github.com/mu-semtech/mu-ruby-template#building-your-first-microservice-with-sinatra/) then we find that for ruby your FROM statement is slightly different (you guess what changes) and instead of a app.js file you now create a web.rb file and again the simplest form looks almost like the example above. + +Suppose you want to go for python? Well exactly the same. + +Doing this we ensure that you have a similar workflow where you know what to expect with every language that you try through a [mu.semte.ch](http://mu.semte.ch/) template. Allowing you to focus on learning and playing with your newest toy. + +### Just playing? +But wait there is more! You do not have to restrict yourself to just playing. Since the ideas behind a microservice are that they have a very limited set of responsibilities, they can very easily be replaced . If there is a necessity for it you can include your new pet language in a production system with little to no risk at all! This shows one of the biggest strengths of the platform. By standardising on HTTP, JSONAPI and graph databases we ensure freedom to use any tool or language and as such also the tool or language that is the best for the job! + +### Future +The set of templates is, for now, rather limited but check us out regularly because we hope to expand and add many many languages to our portfolio. Of course if you want to implement a template feel free to contact us. We will provide you with a simple set of guidelines that will help you develop that template. + +*This part has been adapted from Jonathan Langens' mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/05/04/mu-semte-ch-as-the-ideal-playground/)* diff --git a/writeups/perspectives/mu-semtech-primer.md b/writeups/perspectives/mu-semtech-primer.md new file mode 100644 index 0000000..879775d --- /dev/null +++ b/writeups/perspectives/mu-semtech-primer.md @@ -0,0 +1,17 @@ +# mu.semte.ch primer (About) +Microservices are a good idea, if only they were easy to understand and integrate.  The answer is here, mu.semte.ch.  Your framework for building microservices that are easy to install, integrate, and reuse.  All microservices are reusable in the backend, the frontend integrates everything into a unified look-and-feel. + +## How do we do it? +We cheat.  Instead of letting microservices depend on each other, we lean on the semantic web for data integration.  The semantic web, as envisioned by the W3C, allows computers from all over the world to form one gigantic database.  All without really knowing about each other.  As that looks like a nice feature for microservices, we used our extensive knowledge on semantic technologies to let microservices interoperate using these models.  Only using the database as a syncing point. + +## Deployment? +We tackle that using Docker and Docker Compose.  It’s simple to understand, and trivial to get running.  We use the docker-compose.yml file to describe the main topology, containing all microservices in your stack.  With a single command, you can download all the microservices of your stack, and start them up.  This makes deployment a fun and easy task. + +## Reuse? +Many backend services are easy to reuse.  We have services available for login, registration, regular resource listing and creation (mu-cl-resources), migrations, exports, …  But the real gem lies in how to roll your own.  We have templates for various languages, like JavaScript and Ruby, making it trivial to build your own microservice. + +## Integration? +The look and feel of an application is important.  It needs to look nice and integrated, but it also needs to be cheap.  Mass-production, but custom.  We integrate all microservices in a Single Page App.  We standardize on EmberJS, the most stable choice for frontend development and reuse.  Its structure makes it easy for us to reuse know-how in disconnected projects.  It also allows us to share frontend logic, and sometimes visualisation, using addons.  In the past three years, it has been the logical step forward. + +## Sustainability? +After open sourcing our hobby project, various entities have contributed to mu.semte.ch.   As such, we are being used at the European Commission, in various research institutes, at the Flemish government, and even by some SMEs.  We are actively working together with each of them, to ensure the platform is used optimally, and in line with our future vision. diff --git a/writeups/perspectives/reactive-programming.md b/writeups/perspectives/reactive-programming.md new file mode 100644 index 0000000..0b2c2db --- /dev/null +++ b/writeups/perspectives/reactive-programming.md @@ -0,0 +1,36 @@ +# Reactive programming +We are experimenting with reactive programming. Why? Orchestration! + +The traditional mu.semte.ch architecture provides user-facing microservices. The frontend orchestrates the microservices as it is best suited to communicate their effects to the end-user. But what about backend microservices? How do we let those communicate? Can we indicate to the user where things were left off? Yes. Yes we can. With reactive programming. + +A drawing of a scientist experimenting with RDF + +With reactive programming, our services respond to a certain state being available in the database. As the state changes, the service is informed, and it can react accordingly. An email service could detect that an email is currently in the Outbox, mail it to the right user, and move it to the Sent box. We can keep the database as the only sync-point and have services start tasks based on other services’s work. Backend services can communicate without direct dependencies, using triples to describe their state in the triplestore. The end-user can be informed on the process by visualizing this state in the frontend. + +## An email example +Backend services write contents to the triplestore, which is discovered by other microservices hooking into this content. + +Let’s assume an email system. As the user creates an email, it is in Draft status. Once the email should be sent, we move that email into the Outbox. As this email gets connected to the Outbox, the email sending service picks it up. It sends the email, and moves it to the Sent mailbox. Each of these states is trivially easy to express in the semantic model. + +## Embracing failure +It may be that a microservice drops out. Perhaps we ran out of emails to send under our current plan, perhaps the server hosting our server decided to go on a holiday. Our approach ensures that, once the service gets back up, it picks up the work from where it left off. + +Assume we prepare 5 emails to be sent at once. We place each of the emails in the Outbox, and the sending service starts mailing. As it has sent the third email, we kill the service. Two emails are left unsent. They are still present in the Outbox. When we restart our mailer-service, it checks what emails are still in the Outbox. The two mails that match are sent. As new emails arrive, the mailer picks them up and sends them. + +Failures can happen. It is important to ensure the failure of a single microservice doesn’t bring down the whole application. A big win for reactive microservices. + +## Rich combinations +Reactive programming can make the construction of an application a lot simpler. As we inform the user about state changes, an understanding of the system as a whole can be supplied. + +Let us consider a complex mailing and tagging system. As we draft an email, we see it in the Draft box. When we send it, our service moves it to the Outbox. Our user interface reflects this change, and shows the email in the Outbox now. The mailer service picks up our email, sends it, and moves it to the Sent box. This too, can be reflected in the user interface. This whole picture makes it simple for the user to understand what is going on in the application. + +We can easily push the boundaries for these rich microservices further with more complex systems. For instance, we may extend our mail client with automatic tagging of emails. As we classify a set of emails, a Neural Network service indicates that it has started training on the new examples. Our user interface shows this state. Once the network has been trained, the updated parameters are written to the store. Our classification interface hooks into this to launch a new auto-classification of emails we haven’t checked yet. The microservices are fully decoupled, and the user can easily get a grip on the fairly complex set of operations going on in the backend. + +Rich applications are easier to construct when clear boundaries exist and are being communicated about. + +## What's next? +We have ran advanced experiments and have PoCs of the necessary tooling for reactive programming on the mu.semte.ch stack. + +The [delta-service](https://github.com/mu-semtech/mu-delta-service) calculates triples that would be changed by the execution of an update query. It can inform other services about these changes so they can update as necessary. We are building some example microservices leveraging this approach, whilst applying the concept to end-user PoC applications. + +*This document has been adapted from Aad Versteden's mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/02/09/reactive-programming/)* diff --git a/writeups/perspectives/why-semantic-microservices.md b/writeups/perspectives/why-semantic-microservices.md new file mode 100644 index 0000000..c83cd46 --- /dev/null +++ b/writeups/perspectives/why-semantic-microservices.md @@ -0,0 +1,31 @@ + +# Why Semantic Micro Services? (Semantic Micro Services, Why Bother?) +![](http://mu.semte.ch/wp-content/uploads/2017/06/semantic_architect-1024x768.png) + +## What is a semantic microservice? +What I mean (in this post) with a semantic microservice is a small self-sustainable piece of software. That software should only handle one specific logic. And it has to have a semantic model, a model that is expressed in RDF with the usage of (preferably) published controlled vocabularies. + +## Decoupling the dependencies + +Each piece of software essentially operates on a model, an abstract representation of the world, and manipulates that model to fulfil the tasks it has. In traditional software approaches this model, for persistence purposes at least, is described in normalised tables. Most of the time the internal software model is designed to reflect this. + +In any case there is a requirement on the logical level of the software to be aware of that model that can be found in the database. This is a dependency that can be found between all microservices that compose a model. True, this is not an obstacle that cannot be overcome. If you look at Drupal you see that all modules that you install happily work together. This is because they share the same database model. Of course the pieces of logic that are not Drupal-specific also need to be aware of this model. + +But it goes further than that. Traditional microservices are bound together by that shared model. In any case if I have a table describing company personnel then that table will contain all data that is available to the software. Of course there is a practice of splitting of information blocks that are shared between different types of entities (addresses being the typical example). + +With RDF there is no such dependency. Since triples are used to express the model there is no need for a combined model. When it comes to the standard user information in the [mu.semte.ch](http://mu.semte.ch/) stack for instance, you will see that a user has the following properties: a login name, a password, a real name, an email address and a date of joining. But the login microservice doesn’t care about the last 3 “fields”. So it restricts itself to the predicates for login and password. After that it adds a session “field” to that user in the database which in turn is not even being considered by the subscription microservice. + +## Real reusage of microservices +This semantic model also allows us to completely reuse microservices without having to consider the model used. Here the controlled vocabularies help a lot. If I have knowledge described in my application and should I have had the sensibility to describe that knowledge with the SKOS vocabulary then I can take just about any taxonomy/concept scheme microservice and install it in my system. The beauty of these semantic vocabularies is that the disambiguate meaning. In a normalised configuration we would have ended up with tables that would for instance be called concept\_scheme, ConceptScheme, conceptScheme, CONCEPT\_SHEMA, taxonomy, Taxonomy, … not to mention the same in dutch or whatever language. The controlled vocabularies allow us to identify the “class” of a concept scheme without ambiguity. When you consider properties this advantage only gets bigger. + +If this interests you just google “semantic web” or “linked open data”. + +## SPARQL, Triples and Reactive software + +SPARQL is the language that is used to query an RDF store. These store usually express statements in triples. Within the [mu.semte.ch](http://mu.semte.ch/) microservice framework we have developed a component that calculates changes introduced by queries in the database. This component then notifies microservices that have subscribed to these changes. This allows us to take a microservice that was developed for a certain task, plug it in our application and without any wiring have it “react” to what other microservices are doing. + +This also replaces our needs for a message queue. One can of course point out that such a message queue has several advantages such as speed. But it also comes with a hidden disadvantage. Namely every action taken or message send has to be formatted in some way. This is yet again another dependency in your software system. Any microservice that you want include in your system and that needs to process the messages from certain other components will need to be developed with the knowledge of that message model in mind. + +Another dependency removed with semantic technologies… + +*This document has been adapted from Jonathan Langens' mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/06/15/semantic-micro-services-why-bother/)* diff --git a/writeups/perspectives/why-semantic-tech.md b/writeups/perspectives/why-semantic-tech.md new file mode 100644 index 0000000..f3dd655 --- /dev/null +++ b/writeups/perspectives/why-semantic-tech.md @@ -0,0 +1,16 @@ +### Why semantic tech? (The future of web apps is mashed up!) +Interactive websites have been the norm for a few years now. Simple contact forms don’t seal the deal anymore. We need logins, [Instagram](https://www.instagram.com/) integration and instant [Twitter](https://twitter.com/) notifications. It’s time to turbocharge our web development to match the new demand. Single page web applications, mashed up from various services, will push developer productivity to a whole new level. + +Take the example of a complaint form with an image upload. You probably want the upload to work smoothly, to start the upload eagerly. You may want to show an image preview of the uploaded file. Allow drag and drop, maybe? All that new shiny stuff. But let’s take a step back here. Is image upload really your core business? No? Then why would you develop it? + +a drawing of someone using a masher + +The future of web applications is mashed up apps. Single page apps that use a multitude of services which are readily available, or which are easy to deploy on your own premises. The app orchestrates the used services so the application works as expected. No project-specific development of components which aren’t core to your business. You can use [Firebase](https://firebase.com/) to store the data of your app, take pictures from [Flickr](https://www.flickr.com/), and stream videos from [YouTube](https://www.youtube.com/). Maybe you’ll still have an in-house service for some custom business logic, but that will be limited. We should take the sharing of open source code to a new level, from software libraries, to microservices. Web apps can use these services, whether they’re hosted at your own premises, or offered as an external service. + +There’s more to web development than the backend though. New JavaScript technologies, like [ES6 modules](http://www.ecma-international.org/ecma-262/6.0/#sec-modules) or even [webcomponents](http://webcomponents.org/), make it easier than ever to share code. Although browsers may not fully support them yet, big web development frameworks [already](https://guides.emberjs.com/v2.3.0/) [support](https://www.polymer-project.org/1.0/) [much](https://github.com/klaemo/react-es6) [of](https://github.com/gocardless/es6-angularjs) [it](https://guides.emberjs.com/v2.3.0/components/defining-a-component/). The problems of the frontend are similar to those of the backend. Why would you develop the code to talk to Flickr? Why would you implement the login form for each of your projects? We can share it, plug it in our app, and extend it to match our needs. By sharing whole components in the backend and in the frontend, we get a clear feel for the final product long before it has been polished. Most of the time spent will be time spent on polishing the app. We’re looking forward to the new paradigms of software development. + +And this isn’t the end stage yet. Standardization efforts mark a new way of development. JSON requests are becoming standardised using [JSONAPI](http://jsonapi.org/), allowing us to automatically build the glue code between the frontend and the backend. CSS3 marks major changes in terms of [what](https://developer.mozilla.org/en-US/docs/Web/CSS/filter) [we](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Flexible_Box_Layout/Using_CSS_flexible_boxes) [can](https://drafts.csswg.org/css-backgrounds-3/#border-radius) [visualise](https://drafts.csswg.org/css-transitions/), [websockets](http://www.html5rocks.com/en/tutorials/websockets/basics/) will allow for more interactive applications, [web storage](https://www.w3.org/TR/webstorage/#the-localstorage-attribute) has been standardised, [web](http://www.html5rocks.com/en/tutorials/workers/basics/) [workers](https://www.w3.org/TR/workers/) allow for multithreaded applications and there’s a whole slew of standards which we refuse to mention here because we are already mentioning too many shiny new things! + +Mashup-like architectures will make developing new applications easier and less time-consuming. It’s a logical architectural change driven by the evolution in standards, frontend frameworks, and backend automation. We’re jumping on the train with [mu.semte.ch](http://mu.semte.ch/), and you’re free to take a ride with us. + +*This document has been adapted from Aad Versteden's mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/01/14/the-future-of-web-apps-is-mashed-up/)* diff --git a/writeups/publications.md b/writeups/publications.md new file mode 100644 index 0000000..f82982a --- /dev/null +++ b/writeups/publications.md @@ -0,0 +1,39 @@ +# Publications + +## State-of-the-art web applications fuelled by Linked Data aware microservices + +*Presentation in the Industry Track of [SEMANTICS](http://2016.semantics.cc/), September 13th, 2016, Leipzig, Germany* + +The presentation introduces and discusses the mu.semte.ch framework. Our experience so far shows that the framework offers a productive way of quickly developing practical applications on top of Linked Data covering various situations and domains. Code can be extensively shared across projects in the frontend as well as in the backend which shortens the application development time drastically. + +[Read](http://www.slideshare.net/AadVersteden/musemtech-a-journey-from-tenforces-perspective-semantics2016) + + + +## State-of-the-art Web Applications using Microservices and Linked Data + +*Paper for the [4th SALAD workshop](http://salad2016.linked.services/) at ESWC, May 29th, 2016, Heraklion, Crete* + +Description of mu.semte.ch, a platform for building state-of-the-art web applications fuelled by Linked Data aware microservices. The platform assumes a mashup-like construction of single page web applications which consume various services. In order to reuse tooling built in the community, Linked Data is not pushed to the frontend. + +[Read](http://ceur-ws.org/Vol-1629/paper4.pdf) + + + +## An Ecosystem of User-facing Microservices supported by Semantic Models + +*Position paper for the [5th International USEWOD Workshop](http://usewod.org/usewod2015.html) at ESWC, May 31st, 2015, Portoroz, Slovenia* + +Microservices promise to vastly simplify application complexity. Tiny applications offering +very specific functionality are easy to build and maintain. But how do you keep your +models in sync? Is there a way to share the microservices between parties? By leveraging +the power of semantic technologies, we can create an ecosystem in which many +microservices complete a unified model. A small semantic web to be used within an +organization. This lets organizations have a taste of semantic technologies cheaply with +minimal risk. Our strategy gives focus on supporting single page web applications, where +javascript and REST API’s are king. In this short position paper we describe an +architecture for building, combining and sharing user-facing microservices for state-of-the- +art web applications. + +[Read](http://usewod.org/files/workshops/2015/papers/USEWOD15_versteden_pauwels_papantaniou.pdf) + diff --git a/writeups/retrospectives/developerweek-2018.md b/writeups/retrospectives/developerweek-2018.md new file mode 100644 index 0000000..ae86886 --- /dev/null +++ b/writeups/retrospectives/developerweek-2018.md @@ -0,0 +1,56 @@ +# mu.semte.ch at DeveloperWeek +![](http://mu.semte.ch/wp-content/uploads/2018/03/developer-week-1024x576.jpg) + +Last February I was lucky enough to be able to give a presentation on the grand ballroom stage at one of the bigger developer conferences in the bay area: [DeveloperWeek](http://www.developerweek.com/). + +I gave a presentation on developing reusable microservices through 10 principles. Principles which we have explored over the years on the mu.semtech blog. What is a reusable microservice? Without giving a precise definition here are some properties to which such a microservice should adhere: + +- a microservice  should be swappable for another microservice that offers the same endpoints (the typical example is an update of the same microservice) +- a microservice can be considered stateless +- a microservice communicates over HTTP + +This article gives an overview and a small description of the principles and why it helps to develop microservices that are resuable. + +## 1 Single source of truth +From the microservice’s perspective there is only one single endpoint which holds the complete truth about the world. This source can be distributed using this functionality on a database or software functionality layer such as the [SANSA stack](http://sansa-stack.net/). But it can also be composed, think for instance of [Ontario](https://www.ontario.ca/search/data-catalogue). + +## 2 Set of actions on data +Each microservice contains only a little consistent piece of functionality on data. Now that is super vague, right? What is consistent? Well, I don’t mean consistent to the other actions offered by this microservice, but rather consistent to its data model. + +Take for instance login and registration services. Both of them are user related so you might think it is a good idea to offer both functionalities in the same microservice. However this is far from optimal since both functionalities require vastly different data models. The login service will need a data model that manages username, password, date of birth, real name, … while the registration service will be interested in the session, username and password. Even though at a first glance this is a single set of functionalities, namely user management, it is not the case from a data perspective. + +## 3 No horizontal communication +Within the application block no microservice talks directly to another. Take the setup shown below for example. You will see that #white# talks to #dark-blue# but they only talk to each other and in this configuration. But from the perspective of the dispatcher and from that of the database they are a single microservice. This is one of the most important principles. Without this we would start introducing hidden dependencies all over the application. + +![](http://mu.semte.ch/wp-content/uploads/2018/03/developer-week-2-1024x603.png) + +Take for instance if #pink# would call an endpoint on #dark-blue# every time a certain hook gets executed. Or to make it more concrete, #pink# creates an invoice and #dark-blue# sends that invoice to the customer. That is innocent enough but if we remove the #dark-blue# microservice some other service will need to pick the ‘send-invoice’ action up. By allowing these horizontal hooks we create a hidden dependency, a contract between the microservice and the maintainer of the application. If you install me in your application then I expect you to install something to handle actions X, Y and Z. + +The second unwanted side effect of this is that #pink#, even though it only creates the invoice, can only be used if you want to create and mail an invoice. But we still want to handle business logic, if not through hooks and messages then how do we go about implementing that? + +## 4 Business logic through UX +We came up with two solutions to the business logic problem. The first one is to solve it through the UX. Note that this should only be done in cases where it makes sense. We don’t want to just shift the complexity of the backend to the frontend, where it is almost always worse! To illustrate when this is OK we use the example of search. If I have a search microservice and a resource microservice then I feed the search microservice a string and it will give me back an ID. I then use this ID to obtain an object from the resource microservice. This a clean decoupling of both functionalities. The search needs not to know how an object is composed and the resource has no idea about the indexing scheme used. Both can operate, be changed and installed independently. + +## 5 Business logic through reactive programming +To go back to the create invoice and mail example, that is clearly a case of complexity that should not be handled in the frontend. To handle this we have come up with what we call ‘reactive programming’. I know it is an overloaded term but we don’t intend that to be what you get if you google the term. Rather it is a framework functionality that allows a microservice to be notified of the pure data changes (think triples here) that are about to be introduced to the database. For more information have a look at the [reactive programming discussion](../docs/discussions/reactive-programming.md). + +## 6 Configuration +Code that gets used for more than one use case and from more than one angle tends to be robuster and more bug free than code that doesn’t get the same usage.  Furthermore, because it has already been used in a variety of ways chances are that the way you want to put it to use may already be largely supported. Within mu.semte.ch the champion when it comes to reuse is [mu-cl-resources](https://github.com/mu-semtech/mu-cl-resources). Which is our default resource offering microservice. It supports a full fledge JSONAPI compliant service on top of a SPARQL backend. + +## 7 Controlled vocabularies +Controlled vocabularies serve not only to facilitate standardization – which they do very well – but because concepts in vocabularies are represented using URIs they have the ability to be materialized so that they also contain, within themselves, unambiguous documentation as to their intent. + +My favorite example of the incredible power of these controlled vocabularies is the work that Aad and I did recently on [an Ember component that supports Oauth2](https://github.com/mu-semtech/ember-oauth-github-generator). We have designed a backend microservice that handles the storing of the Oauth2 credentials in the database. And it all just magically worked together with a component that Erika had written well over 3 years ago! Out of the box, without us going about checking her implementation. + +## 8 Push updates +We want to provide a standardized way to provide the user with push updates. These push updates will allow us to create applications that are from the user’s perspective incredibly responsive. Our standardization builds on either long pulling or web sockets and Ember Data as a frontend datastore. + +## 9 Reasoning +With the advent of powerful database technologies such as the [SANSA stack](http://sansa-stack.net/) comes the ability to apply reasoning to large datasets and to do it in a time that allows application to behave normally. This reasoning can be used for classic purposes such as implicit knowledge definitions and data quality control but it can also help us reuse more services. + +## 10 Access control +The last part to touch upon is a database that has access control built-in. In this way it allows us to separate this important part of application design from both the business logic and from the data it operates on. The separation from business logic means that we can now build microservices without any access control in mind and still deploy them with access control present. Doing so we achieve, yes, even more reusability. Separation from data empowers us to change the access control rules easier and it allows us more freedom when it comes to defining those rules. + +The complete presentation as given at DeveloperWeek can be found [here](https://docs.google.com/presentation/d/1MJWLmAWXi8pFZqUWS2Tp4bM_bTA-C03iH3p50Nh6MZ4/edit#slide=id.g34f8d1cab0_2_50). + +*This document has been adapted from Jonathan Langens' mu.semte.ch article. You can view it [here](https://mu.semte.ch/2018/04/19/mu-semte-ch-at-developerweek/)* diff --git a/writeups/retrospectives/dockercon-eu-2017.md b/writeups/retrospectives/dockercon-eu-2017.md new file mode 100644 index 0000000..bc30f80 --- /dev/null +++ b/writeups/retrospectives/dockercon-eu-2017.md @@ -0,0 +1,31 @@ +# mu.semte.ch at DockerCon EU +The mu.semte.ch stack relies on Docker as it perfectly suites a microservice architecture. We don’t only use it for deployment, but also for development. By using Docker we try to make our workflow as productive as possible. Since the Docker community is moving very fast, we attended DockerCon EU in Copenhagen last week to keep up to date with the latest Docker news. + +![](http://mu.semte.ch/wp-content/uploads/2017/10/dockercon-eu.png) + +The Docker team announced two major topics in the keynotes: + +- [seamless integration of Kubernetes into the Docker platform](https://www.docker.com/kubernetes) (alongside Swarm) +- [expanded partnership for the Modernize Traditional Applications (MTA) program](https://goto.docker.com/rs/929-FJL-178/images/SB_MTA_04.14.2017.pdf)  + +The video recordings of the keynotes are freely available on [Keynote Day 1](https://play.vidyard.com/h5fj14BB2Gkai1WHcbAyTv) and [Keynote Day 2](https://play.vidyard.com/wztT1ekFnTjDYLcYfJWALX). + +Traditonal applications are typically developed as monoliths. With the MTA program Docker tries to support companies to upgrade their traditionally deployed apps to a more modern container infrastructure like Docker. Although the mu.semte.ch architecture builds on the concept of microservices, Docker’s MTA program may teach us useful lessons on the migration from monoliths to microservices. After all, projects don’t always start from scratch but often originate from an existing (monolith) solution. + +Next to the keynote sessions there were also a lot of breakout sessions on a broad range of topics. All slides will soon become available on [Docker’s slideshare](https://www.slideshare.net/Docker). We will just highlight a few of the topics covered: + + +- Abby Fuller from AWS explained how to create effective Docker images covering topics like the recently introduced [multistage builds](https://docs.docker.com/engine/userguide/eng-image/multistage-build/). Some of the mu.semte.ch templates and services will benefit from these tips to reduce the size of their Docker image. + +- Adrian Mouat, one of the Docker captains, gave [a talk with some tips and tricks](https://www.slideshare.net/Docker/tips-and-tricks-of-the-docker-captains). Nothing new, no mind blowing stuff, but just some useful practical tips to make your day-to-day Docker usage more efficient. + +- Dan Finneran, an ex-Docker captain that recently became a Docker employee, prestented [an overview of the Docker networking](https://www.slideshare.net/Docker/practical-design-patterns-in-docker-networking). This included the Swarm overlay networks and some practical design patterns the mu.semte.ch architecture might benefit from when deploying on multiple machines. + +- Bret Fisher, another Docker captain, gave a _playful_ talk on [the deployment to production with Docker](https://d.pr/f/d7diCc/1AEjBc2m). He warned for bad practices like turning servers from cattles into pets and making Dockerfiles environment specific. For sure some lessons we should take into account when deploying a mu.semte.ch app. + +- Riccardo Tommasini, a PhD Student from Milano, gave a short presentation in the Community Theatre on his ongoing research to empower Docker with Linked Data principles. One of his research results, a Docker ontology to represent Dockerfiles in RDF, will be [presented next week at ISWC 2017](https://iswc2017.semanticweb.org/paper-528/). + + +These are just a few highlights of all the topics covered at the conference. The video recordings of all presentations will soon become available on the Docker website. A lot of interesting material to learn from within the context of mu.semte.ch. + +*This writeup has been adapted from Erika Pauwels' mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/10/19/mu-semte-ch-at-dockercon-eu/)* diff --git a/writeups/retrospectives/oslo2.md b/writeups/retrospectives/oslo2.md new file mode 100644 index 0000000..10d2996 --- /dev/null +++ b/writeups/retrospectives/oslo2.md @@ -0,0 +1,52 @@ +# (Lessons learned from) OSLO² + +[Dacota.one](http://dacota.one) used mu.semte.ch to publish [CRAB-LOD as Linked Open Data](http://data.vlaanderen.be/doc/adres/2179183). + +Publishing linked open data is not at the core of the mu.semte.ch stack, yet the choice makes sense. As Linked Data is the core data model in mu.semte.ch to connect the microservices, all necessary information is available. + +In the light of publishing Linked Open Data correctly, we should publish the contents in various ways. One of the ideal methods of publication are the subject pages.  These pages provide a human view on the rendered contents. + +Following is a set of lessons learned on using the mu.semte.ch stack for publishing Linked Open Data. + +## Frontend generators + +[OSLO²](https://overheid.vlaanderen.be/producten-diensten/OSLO2) uses mu-cl-resources to generate the API of the resources.  In this case [addresses, streets and municipalities](https://overheid.vlaanderen.be/producten-diensten/adressenregister-crab).  The content of the subject pages is everything specified in the mu-cl-resources configuration. This means we can derive the contents of the subject pages by interpreting the mu-cl-resources configuration. + +Based on a core idea of generated pages on mu.semte.ch, similar to the [generators in Rails,](http://guides.rubyonrails.org/generators.html) the CRAB-LOD team extended our PoC efforts.  The core benefit of generating the pages is that they are adaptable.  The generators construct views which you can alter as needed, giving you a fast starting point.  The generators are handy when adding new fields, you can overwrite the generated contents.  But you can also override the general feel of the website whilst looking at the real data to publish, thereby making large style changes easier.  By materialising the generated views, customisation becomes easier.  Not all of these features were needed by CRAB-LOD, but it allows the approach to be used in more places. + +The approach of using frontend generators worked fine.  We will further clean up the repositories and document the approach for future use. + +## Content negotiation + +There are multiple ways in which we may share a piece of information.  Content negotiation allows the browser to request the content in the way he wants it shared. + +Based on the type of content the user requests, we should be able to dispatch to a different microservice.  One service may be responsible for hosting content as triples, another as a json-api. The mu-dispatcher does not support this.  In order to make the platform better suited for Linked Data publishing, we should extend the dispatcher with content negotiation capabilities. + +We are strongly considering this extension to the mu-dispatcher. + + +## Read-only query optimisation + +There are no live updates to the data in OSLO².  As such, many database queries can be cached, relieving the triplestore and rendering the pages more swiftly. + +SPARQL queries can be sent as HTTP requests, which is the method we use for our microservices.  These requests are easy to cache.  Slight problem is that the requests are HTTP POST requests, which by HTTP semantics are not supposed to be cached.  This is correct, the contents of the triplestore may change and should thus, in the traditional interpretation, not be cached. + +Bert Van Nuffelen built [a proxy microservice which caches the HTTP POTS requests to the triplestore](https://github.com/bertvannuffelen/SimpleSparqlCache).  Given a static database, like CRAB-LOD, this removes stress from the triplestore.  We have not ran benchmarks yet, but the API feels much more snappy this way. + +## Linked Data Fragments + +In a short wrap-up discussion with Pieter Colpaert on the subject, [Linked Data Fragments](http://linkeddatafragments.org/) (LDF) were pushed forward.  Although not connected to CRAB-LOD yet, it is a method of publishing Linked Data. + +Given the brief discussion with Pieter, it appears to be easy to add this method of publishing Linked Open Data to the outside world.  LDF can be easy on the triplestore and makes federated SPARQL querying the default.  Check their [website](http://linkeddatafragments.org/) for more information. + +We are looking into making Linked Data Fragments a drop-in addition to the framework.  Thereby giving you a free and promising extra way of sharing your contents with the world. + +## Conclusion + +The mu.semte.ch stack has proven to be a nice way of publishing Linked Data. + +Given content available in the triplestore, manipulation is easy.  Performance is swift.  And with the generated subject pages, both adding content, as well as customising views when necessary is feasible.  With the integrated application, it is possible to add a different look and feel to the frontend, whilst still having much reuse. + +It is nice to be used at an organisation like the Flemish Government and expect similar projects to appear in the future. + +*This document has been adapted from Aad Versteden's mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/04/06/lessons-learned-from-oslo%c2%b2/)* diff --git a/writeups/retrospectives/sw-implementing-docker-multi-stage-builds.md b/writeups/retrospectives/sw-implementing-docker-multi-stage-builds.md new file mode 100644 index 0000000..c2f2e51 --- /dev/null +++ b/writeups/retrospectives/sw-implementing-docker-multi-stage-builds.md @@ -0,0 +1,43 @@ +# Docker multi-stage builds benefits (Publishing an EmberJS app using Docker multi-stage builds) +![](http://mu.semte.ch/wp-content/uploads/2017/05/mu_semte_ch_of_oz-1024x683.png) + +In [the ember-proxy-service repo](https://github.com/mu-semtech/ember-proxy-service#tutorial-hosting-an-emberjs-app-with-a-backend-api) we explained how to host an EmberJS application in nginx with a backend API.  First, we had to build the EmberJS app through ember-cli’s build command: +```bash +ember build -prod +``` + +Next, we copied the resulting `dist/` folder into the nginx Docker image before building it. This process is cumbersome since it requires two manual steps to publish an EmberJS Docker image. + +It would be more clean if we could execute the ember build step during the Docker image build. However, this requires that (a) the ember build command is available and (b) all our source files and dependencies (e.g. the node_modules folder) are copied inside the Docker container. As a consequence the resulting image would be much larger since it contains a lot more than just the built javascript assets from the dist/ folder. + +To tackle this problem Docker introduced the concept of [multi-stage builds in Docker 17.05.](https://docs.docker.com/engine/userguide/eng-image/multistage-build/)  Multi-stage builds allow to use multiple FROM statements in your Dockerfile. Each FROM statement begins a new stage of the build starting from the specified base image. The base images may differ per FROM statement. Artifacts can be copied from one stage to another. Everything else will be left behind when moving to a next stage. Therefore the resulting image will be much smaller. It only contains the image built in the final stage and the artifacts copied from previous stages. Have a look at the [Docker documentation](https://docs.docker.com/engine/userguide/eng-image/multistage-build/#before-multi-stage-builds) for a more in-depth explanation. + +Our case would benefit from a 2-stage build: + +1. Building the EmberJS assets +2. Building the nginx image to host the EmberJS app + +This would result in the following Dockerfile: +```Dockerfile +# Stage 1 to build the EmberJS assets +FROM madnificent/ember:2.15.0 as builder + +MAINTAINER Erika Pauwels <[\[email protected\]](/cdn-cgi/l/email-protection)\> + +WORKDIR /app +COPY package.json . # postpone cache invalidation +RUN npm install +COPY . . +RUN ember build -prod +# End stage 1 + +# Stage 2 to build the nginx with our EmberJS app +FROM semtech/ember-proxy-service:1.1.0 + +COPY --from=builder /app/dist /app +# End stage 2 +``` + +Docker multi-stage builds are a simple, yet powerful concept to fully automate application build processes while keeping the resulting Docker image small. In our case we were able to reduce the image size from 1.24GB to 185MB. Note however that multi-stage builds don’t work on Docker Hub, which is deprecated in favor of Docker Cloud. + +*This document has been adapted from Erika Pauwels' mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/11/02/publishing-an-emberjs-app-using-docker-multi-stage-builds/)* diff --git a/writeups/retrospectives/sw-microservice-reuse-and-authorization.md b/writeups/retrospectives/sw-microservice-reuse-and-authorization.md new file mode 100644 index 0000000..742c65b --- /dev/null +++ b/writeups/retrospectives/sw-microservice-reuse-and-authorization.md @@ -0,0 +1,55 @@ +# On microservice reuse and authorization +Microservices make it easy to reuse logic across applications.  You can easily share the logic between applications by sharing the full microservice.  But there’s a catch, you must share the full microservice.  Hence, you must be able to use that exact logic.  *When a microservice embeds authorization logic, it can only be reused in domains with the same authorization logic.* + +## Examples of authorization rules +Authorization rules boil down to the definition of the boundaries of information sharing.  In order to define *who* can see *what* information, the *who*<> and the *what* need to be defined. + +As described in earlier blog-posts, the *who* can be connected to the session of the user.  Based on the information in the session, and a SPARQL query, we should be able to formalize the access groups of a user.  Practical examples of *who* are + +- an individual user, +- a user belonging to an organization, +- a user which has a specific role in an organization. + +The *what* is a less explored domain in the mu.semte.ch stack.  Our goal with *what* is to find a definition which is sufficiently easy to use, but which covers sufficient use-cases.  Cases we’ve discovered are: + +- access to instances of an entity, +- access to properties of an entity, +- access to aggregated results. + +These two topics can be combined.  The *who* and the *what* can be combined.  The visitor of a webshop could be allowed to see only *his own* (who) *shopping baskets* (what).  Without logging into the webshop, *any user* (who) might be allowed to see all *products* (what). + +Aggregated results are a different beast.  A user which hasn’t logged in might be allowed to see the total amount of orders placed on the webshop, to indicate the trustworthiness.  This is an aggregated result.  A user may also be able to see how many orders were placed within his region, given that sufficient orders were placed in the region.  The constraints we could place in this domain seem very broad, and we feel we’ve only scratched the surface.  We’re leaving this to be described and tackled as we gain more experience. + +Access rights could be defined in terms of *who* can access *what*.  The *who* boils down to users, groups, and individuals. The *what* which we aim to tackle are instances of a type, and properties of such instances.  The combination of *who* and *what* define what users will eventually be able to access. + +## The boundaries of a microservice +The mu.semte.ch stack makes it super easy to build and share microservices.  With the semantic model, the intended data is clear, with Docker it’s easy to share and launch the services.  We consider these challenges intrinsic to the offered services.  You need to know what you are talking about, in order to offer operations on that data.  For most services, authorization is not an intrinsic challenge.  The same service could be offered in many authorization setups. + +We have proven that we’re able to share a great set of microservices when the authorization domain is constant.  Application clusters built for a single customer, or solutions which work on open data.  Both of these cases hide the complexity of authorization, but in order to reuse the services with other authorization domains, a solution is necessary once again. + +## Moving around authorization +Authorization is a problem which we should tackle transparently to the microservice . When we take a peek at the examples of authorization rules, we notice that many of them work on the data level.  As such, if we could handle authorization in the triplestore directly, we could unlock the reuse of many more microservices. + +We have run experiments and have shown this could be achieved with a meta-SPARQL-endpoint.  A SPARQL endpoint which rewrites the queries of each microservice so they obey the access rights of an individual user.  Although there are quite a few side-remarks, such an implementation makes it plausible to reuse services which weren’t reusable earlier.  This is a complex challenge which we will need to face head-on. + +With the right primitives, we can transparently rewrite the queries a microservice executes so it only operates on information a user has access to. + +## Thoughts on information sharing and access control +Initial discussions revealed various ways in which information can be shared between multiple entities.  Like sharing a book, or filling in a form together. + +In database terms, the most obvious sharing is bidirectional sharing.  Two people work on the same dataset, as if it were a shared spreadsheet.  All changes one person makes are visible to the other. + +Another easy to understand way of sharing is unidirectional.  You rent a book from the library.  You can read the contents, but you can’t push any changes back to the author. + +If you buy the book, you’re allowed to make annotations.  You can highlight contents, add side-remarks, rip whole chapters out of the book and rewrite them in your own phrasing.  You received the information, manipulated it, but do not share it back. + +There’s also the notion of live versus offline collection of information.  In a live collection of information, we’re seeing the updates pass on immediately.  Whereas with bulk updates, we’d download some new contents and integrate them in our memory. + +Secondly, we discover the act of reading versus writing.  You may be allowed to see certain contents, whilst not being able to write them.  You may be able to write certain contents, but not be allowed to read them (you could send log-files to an admin but never be allowed to read them again).  Or you could do both reading and writing. + +In order to keep the approach conceivable, we’re working towards authorization with the three last mentioned forms of reading and writing.  We will consider bidirectional sharing only, and will require push updates to be solvable through escape hatches of the authorization system. + +## Conclusion +Authorization is not an intrinsic complexity of microservices.  Extracting authorization rules allows sharing microservices across more applications.  Many authorization rules can transparently be handled by rewriting the SPARQL queries. + +*This document has been adapted from Aad Versteden's mu.semte.ch article. You can view it [here](https://mu.semte.ch/2019/04/02/on-microservice-reuse-and-authorization/)* diff --git a/writeups/retrospectives/sw-supporting-mac-os.md b/writeups/retrospectives/sw-supporting-mac-os.md new file mode 100644 index 0000000..d61b1bb --- /dev/null +++ b/writeups/retrospectives/sw-supporting-mac-os.md @@ -0,0 +1,25 @@ +# Supporting MacOS (Hello MacOS) +![](http://mu.semte.ch/wp-content/uploads/2017/11/Mac-semtech-1024x768.png) + +Many developers use MacOS on their daily basis.  With our Docker background, we’ve mainly focused on Linux support.  That is about to change, MacOS is becoming a first class citizen. + +## What does this entail? +Base scripts will receive Docker support, documentation will be verified to run both on Linux as well as on MacOS.  We will supply installation instructions for MacOS. + +For most supporting projects we start, we base ourselves on Linux.  The ember docker is a good example of that.  Although Docker abstracts a lot, there are differences.  The scripts for this Docker have received updates already, ensuring they run correctly on MacOS.  By trying out solutions both on Linux as well as on MacOS, we ensure a smooth operating experience. + +We are looking into ways of supplying code for MacOS so the installation is simplified.  For the backend, installation is trivial with Docker for Mac.  For the frontend, we’re considering to support homebrew installation of edi. + +## What differences are considered? +Bigger differences for us come in the form of filesystem performance, supplied shell scripts, and user namespaces. + +Filesystem performance for mounted volumes is an issue for MacOS.  Both starting of containers, as well as syncing content between the container and the host seems to be an order of magnitude slower.  We are updating scripts to ensure the system works as expected.  Performance improvements are currently only noticeable on MacOS. + +With Linux we target a modern kernel and a modern Bash environment.  The shell scripts Linux ships with are slightly different than those MacOS has in store.  This means some scripts need to be adapted. Some will become cleaner, some will become more complex. + +User Namespaces ensure we can generate files from Docker, with your access rights.  This alleviates the issue where you need to chown generated files.  Docker for Mac doesn’t have that issue.  It does not run as root, hence the containers generate files under your own username. + +## In conclusion +If your organisation is a cross-breed between Linux and MacOS, it will become easier to get everyone on board with mu.semte.ch.  We’ll keep you posted as we hum along. + +*This document has been adapted from Aad Versteden's mu.semte.ch article. You can view it [here](https://mu.semte.ch/2017/11/09/hello-macos/)* diff --git a/writeups/who---mu-semtech.md b/writeups/who---mu-semtech.md new file mode 100644 index 0000000..f0c5e23 --- /dev/null +++ b/writeups/who---mu-semtech.md @@ -0,0 +1,19 @@ +# Who + +## … governs mu.semte.ch? + +The project is disconnected from individual commercial entities ensuring it stays free to use.  The core team meets every two weeks and discusses raised topics whilst prioritising and adding new features. + +![](http://mu.semte.ch/wp-content/uploads/2017/05/erika.jpg) +![](http://mu.semte.ch/wp-content/uploads/2017/05/jonathan.jpg) +![](http://mu.semte.ch/wp-content/uploads/2017/05/aad.jpg) +![](http://mu.semte.ch/wp-content/uploads/2017/05/felix.jpg) + +## … uses mu.semte.ch? +![](http://mu.semte.ch/wp-content/uploads/2017/05/fabbrikka.png)![](http://mu.semte.ch/wp-content/uploads/2017/05/european-commission-e1496161303799.jpg) + +![](http://mu.semte.ch/wp-content/uploads/2017/05/pintafish-1024x1024.jpg)![](http://mu.semte.ch/wp-content/uploads/2017/04/your-data-stories.png)![](http://mu.semte.ch/wp-content/uploads/2017/05/BDE_vertical_noslogan.png) + +![](http://mu.semte.ch/wp-content/uploads/2017/05/vlaamse-overheid.jpg)![](http://mu.semte.ch/wp-content/uploads/2017/05/bravoer.png) + +![](http://mu.semte.ch/wp-content/uploads/2017/05/veeakker.png)![](http://mu.semte.ch/wp-content/uploads/2017/04/moof.jpg)