This is a guide to get you started with Vase - the data-driven microservice container. It will help you write your first descriptor file.
After reading this guide, you will be able to:
- Write a Vase API descriptor
- Exchange data with Vase services
It may help to understand the design of the system and the file formats used before you begin. The most important pieces:
- Vase descriptors are usually stored in Extensible Data Notation.
- Vase descriptors are just Clojure data in memory, so the EDN format is a convenience not a requirement.
- Vase services use JSON as their data exchange format.
It also important to note that the terms service, server, container are all used interchangeably.
First, be sure you have a Datomic transactor running. See
the Datomic docs to get set up. Note that the
template project uses Datomic Pro. (You can change the project.clj
dependency to use Datomic Free. Be sure to make the corresponding
change in the :datomic-uri later.)
Create a new project from the Vase leiningen template:
lein new vase your-first-api
The exact sequence varies a little bit depending on whether you prefer Leiningen or Boot.
Start the service by running the following command at a terminal:
lein repl
Once the REPL is running, start a dev mode server like this:
(def srv (run-dev))That means your server is up and running, listening on port 8080.
Start the service by running the following command at a terminal:
boot repl
Once the REPL is running, start a dev mode server like this:
(require 'your-first-api.server)
(in-ns 'your-first-api.server)
(def srv (your-first-api.server/run-dev))There will be a lot of logging, but it should end with something like this:
INFO o.e.jetty.server.AbstractConnector - Started ServerConnector@5d04c566{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:8080}
INFO org.eclipse.jetty.server.Server - Started @85814ms
Let's build a system to handle basic user accounts and can track items they own.
With this simple goal statement, we can already envision some pieces of the data model and even some URLs that might appear in the API.
Our data model needs to handle:
- Items (with item IDs, names, and descriptions)
- Users (with user IDs, emails, and a list of items they own)
And we might have URLs that:
- Fetch all items in the system
- Fetch all users in the system
- Fetch a user given their user ID
- Fetch a user's owned-item list given their user ID
- etc.
We put the description into a .edn file that our service loads. (EDN
is specified by https://github.com/edn-format/edn.) This description
tells Vase what API a service offers, what schema it defines, and what
specifications the data must follow.
The Vase template created resources/your-first-api_service.edn with
a lot of examples. For the moment, it has too much. Just delete that
file and start a fresh one with these contents:
{:activated-apis []
:datomic-uri "datomic:mem://example"
:descriptor
{:vase/norms {}
:vase/specs {}
:vase/apis {}}}This says we have no schema, no APIs, no specs, and nothing will run. Not very exciting, is it? It gets better in a bit.
Here are the meanings of the top-level keys:
| Key | Type | Meaning |
|---|---|---|
:activated-apis |
Vector of API "names." | Routes for these APIs will be created. Schema used by these APIs will be applied. |
:datomic-uri |
String with a Datomic URI | This is the database that Vase will use. If it doesn't exist, Vase will create it. |
:descriptor |
Map with specific keys expected | Holds the schemas (:vase/norms), specs (:vase/specs), and APIs (:vase/apis |
We often start with a model of the entities a service must manipulate.
Data models are defined with a schema (called "norms" for historical reasons.) You may split your schema up into logical pieces that map directly to your application domain. In our example, we'll define one part of the schema for Items, and another part for Users.
The :vase/norms key is where we put schema definitions. It holds a
map of "schema name" to schema definition. The schema name is a
keyword that you choose. We recommend using a namespaced keyword. It's
best to avoid using vase as part of your namespace. It's not a
reserved name, but you may confuse parts of your application with
parts that Vase requires.
To start with, let's make space for our two schemas:
:vase/norms
{:accounts/item {}
:accounts/user {}
}(This goes inside the map attached to :descriptor.)
This says we're building one schema named :accounts/item and another
named :accounts/user. The first component of a schema is defining
its normalized, master form. That's done using the :vase/norms
entry. In this map, we'll define attributes for Items and Users.
You'll notice that the Users schema requires the Items schema, so that
we can describe a user's owned items. You'll also notice that
attributes of a given entity are defined in terms of database
transactions (txes) that describe them.
Vase offers a reader literal shorthand called #vase/schema-tx for
defining Datomic schema transactions within :vase.norms/txes. Its
body is a vector of vectors. Each subvector defines an attribute
name, its cardinality (:one or :many), its type, some optional
flags, and a doc string.
The allowed flags are translated into parts of the attribute definition for Datomic:
| Flag | Meaning | Equates to |
|---|---|---|
:unique |
Only one entity in the DB can have this value for this attribute | :db/unique :db.unique/value |
:identity |
An entity is uniquely identified by this value. Upsert is enabled | :db/unique :db.unique/identity |
:index |
This attribute should be indexed | :db/index true |
:fulltext |
This attribute should be searchable as text. (Mildly deprecated.) | :db/fulltext true |
:component |
The entity referenced by this attribute should be retracted when this one is. (Cascading delete) | :db/isComponent true |
:no-history |
Do not preserve old values of this attribute | :db/noHistory true |
Even though the short-schema version covers most of the use cases you'll need, you're always free to use full Datomic schema definitions like:
:vase.norm/txes [[{:db/id #db/id[:db.part/db]
:db/ident :item/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "The name of an item"}]]A common convention for attribute names is that the namespace identifies the "entity type" that the attribute belongs to. This is strictly for human consumption, though. Datomic doesn't have any notion of entity type and doesn't care how you mix and match attributes.
Fill in the schema parts like this:
:vase/norms
{:accounts/item
{:vase.norm/txes [#vase/schema-tx
[[:item/itemId :one :long :identity "The unique identifier for an item"]
[:item/name :one :string "The name of an item"]
[:item/description :one :string "A short description of the item"]]]}
:accounts/user
{:vase.norm/requires [:accounts/item]
:vase.norm/txes [#vase/schema-tx
[[:user/userId :one :long :identity "The unique identifier for a user"]
[:user/email :one :string :unique "The email address for a given user"]
[:user/items :many :ref "The collection of items a user owns"]]]}}This says the :accounts/item schema has three attributes:
:item/itemId, :item/name, and :item/description. The
:accounts/user schema has three more attributes: :user/userId,
:user/email, and :user/items. Furthermore, the :accounts/user
schema requires that the :accounts/item schema must be in place
first. (It doesn't really... there are no definitions in
:accounts/user that depend on :accounts/item. This is just for
illustration in this tutorial.)
With the data model in place, we can build our API.
Vase defines routes defined with URL strings, HTTP verbs (get, post, etc), and action literals. The foundation of Vase's routing is based on Pedestal's capabilities, but with care given to represent routes in an external data file.
Fill in :vase/apis with this route that gives some information about our API:
:vase/apis
{:accounts/v1
{:vase.api/routes
{"/about" {:get #vase/respond {:name :accounts.v1/about-response
:body "General User and Item Information"}}}}}Routes are defined as nested maps. Each map defines a single route,
and then a map of HTTP verbs to action literals---in this case
#vase/respond. Every action literal requires a unique :name. Other
keys and their interpretations are defined per literal. Here we see
that GET /api/accounts/v1/about will respond with a string.
These are the action literals:
| Literal | Purpose |
|---|---|
#vase/respond |
Respond with static data, optionally formatted as JSON |
#vase/redirect |
Redirect an incoming request to a different URL |
#vase/query |
Respond with the results of a database query |
#vase/transact |
Add or update data in the database |
#vase/validate |
Validate data against specs |
#vase/intercept |
Apply a hand-crafted, artisinal interceptor |
An API usually depends on some amount of schema. In this example,
we've added a dependency from the :accounts/v1 API to the User
schema. Since the User schema depends on the Item schema, both parts
of schema will be applied to our database.
{:vase/apis
{:accounts/v1
{:vase.api/routes ,,, ;; skipping the routes for space
:vase.api/schemas [:accounts/user]}}}An API can depend on any number of schemas. You should feel free to grow and evolve your schema by adding new "norms."
One note: Vase tracks which schemas it has already applied to a
database. Think of each schema like a migration in other database
frameworks: once it's applied to the database you don't change
it. Just add new schemas under :vase/schemas and add them to your
APIs' :vase.api/schemas dependencies.
You can use curl to test the new URL, but it won't work yet.
curl http://localhost:8080/api/accounts/v1/about
If you've been following along in this guide, you got a 404 response just now. That's because Vase has one more concept about APIs: activation.
A single EDN file can define many APIs and many schema fragments. It
is up to a service instance to determine which of these to activate
using the top-level key :activated-apis. This is in the EDN file for
ease, but could be supplied separately as part of your service
config. (For example, as EC2 instance data.)
To activate the accounts API, modify the top of your EDN file like this:
{:activated-apis [:accounts/v1]
,,,Now re-run curl and you'll get back a 200 status code with the body
string from our #vase/respond literal.
Some Javascript and Clojurescript clients provide request IDs that they use to correlate requests and responses. We can tell Vase to forward headers from the request through to the client. This is at the level of the whole API:
{:vase/apis
{:accounts/v1
{:vase.api/routes ,,, ;; skipping the routes for space
:vase.api/schemas [:accounts/user]
:vase.api/forward-headers ["vaserequest-id"]}}}This says to forward the vaserequest-id header from every
request to every response. This HTTP header is used to trace and
debug requests to the service (and is automatically added if it's not
sent in).
Vase does some code generation to make HTTP parameters available as
Clojure bindings for a route's action literal(s). These are conveyed
from the route to Vase by Pedestal, using the :params keyword in the
request map.
Params arrive in various forms, extracted from:
- EDN payload for POSTs
- JSON payload for POSTs
- Form data for POSTs
- Query string arguments
- URL (i.e., path) parameters.
Parameters are resolved with an order of precedence, as listed below:
- EDN POST payloads override
- JSON POST payloads override
- POST form data payloads override
- Query string args override
- URL parameters
See Pedestal's docs for a complete reference on parameters.
Let's look at them from the bottom up.
A path parameter binds a single value from a URL to a symbol name in your action literal.
Add a route with a path parameter:
:vase/apis
{:accounts/v1
{:vase/routes
{"/about" {:get #vase/respond {:name :accounts.v1/about-response
:body "General User and Item Information"}}
"/about/:your-name" {:get #vase/respond {:name :accounts.v1/about-yourname
:params [your-name]
:body your-name}}}
:vase.api/schemas [:accounts/user]
:vase.api/forward-headers ["vaserequest-id"]}}In this trivial example we bind part of the URL path to the symbol
your-name and return it as the body of our response. A path
parameter value is always a string.
Try it out with curl:
curl http://127.0.0.1:8080/api/accounts/v1/about/paul
Let's see how we might take multiple parameters for a given route.
Query string args are typically used for filtering the result of returned data. Here's an example were we'll return a JSON response for all of our expected query args
:vase/apis
{:accounts/v1
{:vase/routes
{"/about" {:get #vase/respond {:name :accounts.v1/about-response
:body "General User and Item Information"}}
"/about/:your-name" {:get #vase/respond {:name :accounts.v1/about-yourname
:params [your-name]
:body your-name}}
"/aboutquery" {:get #vase/respond {:name :accounts.v1/about-query
:params [one-thing another-thing]
:body {:first-param one-thing
:second-param another-thing}}}}
:vase.api/schemas [:accounts/user]
:vase.api/forward-headers ["vaserequest-id"]}}With curl:
curl 'http://127.0.0.1:8080/api/accounts/v1/aboutquery?one-thing=hello&another-thing=world'
(Make sure you quote the whole string properly... the '?' and '&' mean something entirely different to the shell.)
Notice that the response that comes back is JSON, not text like the
other #vase/respond action we specified. That's because the new action
returned a Clojure data structure instead of a string. When the body
of a response is not a string, Vase converts it to JSON.
You can provide a default value for any :params binding.
For example, :params [your-name [age 42]]. Of course, if a path
parameter is nil, then the route didn't even match. But query and body
parameters can be defaulted this way.
So far, we've lead you to believe that action literals are purely data. That's not entirely true.
Many parts of the action literals are evaluated as code, in an
environment where the parameter names are bound. Everything from
clojure.core is available. That is, unless you shadow something from
clojure.core with a parameter name!
This means we can add a function call directly to our #vase/respond
actions. Let's update our url-param route to print a more interesting
string using clojure.core/str.
Modify the action for "/about/:your-name" like this:
:vase/apis
{:accounts/v1
{:vase.api/routes
{"/about" {:get #vase/respond {:name :accounts.v1/about-response
:body "General User and Item Information"}}
"/about/:your-name" {:get #vase/respond {:name :accounts.v1/about-yourname
:params [your-name]
:body (str "You said your name was: " your-name)}}
"/aboutquery" {:get #vase/respond {:name :accounts.v1/about-query
:params [one-thing another-thing]
:body {:first-param one-thing
:second-param another-thing}}}}
:vase.api/schemas [:accounts/user]
:vase.api/forward-headers ["vaserequest-id"]}}With great power comes great responsibility - bending this ability is dangerous and can cause the container to crash, but in a pinch it can allow you to shape an API to meet your needs.
Obviously, this means anyone who can alter your descriptor can run arbitrary code in your server. Don't accept user inputs as descriptors!
In addition to rendering content, the Vase system also provides a
#vase/transact action allowing the storage of incomming POST data.
Vase expects transaction data to arrive as a JSON entity body. The top
level of the body is an object with the single key payload. The
payload should be a collection of entity bodies (i.e., maps) to transact.
Add the "/user" route shown here to your descriptor:
:vase/apis
{:accounts/v1
{:vase.api/routes
{"/about" {:get #vase/respond {:name :accounts.v1/about-response
:body "General User and Item Information"}}
"/about/:your-name" {:get #vase/respond {:name :accounts.v1/about-yourname
:params [your-name]
:body (str "You said your name was: " your-name)}}
"/aboutquery" {:get #vase/respond {:name :accounts.v1/about-query
:params [one-thing another-thing]
:body {:first-param one-thing
:second-param another-thing}}}
"/user" {:post #vase/transact {:name :accounts.v1/user-create
:properties [:db/id
:user/userId
:user/email]}}}
:vase.api/schemas [:accounts/user]
:vase.api/forward-headers ["vaserequest-id"]}}The new route has a single #vase/transact literal with a name and
properties. The :properties key holds a whitelist of attribute names
that Vase will accept in the incoming POST data. In this case, the
payload should have a sequence of maps that each have :user/userId
and :user/email keys and values. These will be asserted in Datomic.
We want to POST the following JSON
{"payload":
[{
"user/userId" : 42,
"user/email" : "[email protected]"
}]
}This is how you would use cURL to POST such a payload:
curl -H "Content-Type: application/json" -X POST -d '{"payload": [{"user/userId": 42, "user/email": "[email protected]"}]}' http://localhost:8080/api/accounts/v1/user
The quoting is a little different on Windows:
curl -H "Content-Type: application/json" -X POST -d "{\"payload\":[{\"user/userId\":42,\"user/email\":\"[email protected]\"}]}" http://localhost:8080/api/accounts/v1/user
Give it a try!
The properties :user/userId and :user/userEmail are fairly
self-explanatory, but the :db/id property is handled specially.
That is, the :db/id key signifies if the incoming data refers to
existing entities in the Vase database, or to new entities.
In the JSON packet above, the entity did not contain a :db/id
field. Vase treats it as a new entity and attaches a tempid before
asserting it.
At this point, one of three things will happen:
- The entity has an
:identityattribute whose value matches an existing entity in the database. This becomes an "upsert" on that entity: the new attribute values are merged with the existing entity. - The entity has a
:uniquevalue that already exists in the database. Because it has a tempid, Datomic will reject the transaction and Vase will return an error to the client. - Neither of the above are true, and Datomic will create a new entity.
It is possible to supply a db/id directly, like this:
{"payload":
[{
"db/id" : 100,
"user/userId" : 9,
"user/email" : "[email protected]"
}]
}Because the :db/id field is set to a value, Vase will try to
resolve the entity in the database before transacting the
data. Obviously, if no such entity exists then a failure will occur,
thus notifying the calling client.
One final way to refer to existing entities is to set the value at the
:db/id field to correspond to a unique value for the entity in
question. For example:
{"payload":
[{
"db/id" : ["user/userId", 9],
"user/email" : "[email protected]"
}]
}We declared :user/userId as an :identity attribute. That means we
can use it in lookup refs as well as doing upsert with it.
See Datomic's lookup refs for more details.
The #vase/query action provides a way to define service routes that
return data based on Datalog queries.
Go ahead and add a :get action to the "/user" route like this:
:vase/apis
{:accounts/v1
{:vase.api/routes
{"/about" {:get #vase/respond {:name :accounts.v1/about-response
:body "General User and Item Information"}}
"/about/:your-name" {:get #vase/respond {:name :accounts.v1/about-yourname
:params [your-name]
:body (str "You said your name was: " your-name)}}
"/aboutquery" {:get #vase/respond {:name :accounts.v1/about-query
:params [one-thing another-thing]
:body {:first-param one-thing
:second-param another-thing}}}
"/user" {:post #vase/transact {:name :accounts.v1/user-create
:properties [:db/id
:user/userId
:user/email]}
:get #vase/query {:name :accounts.v1/user-page
:params [email]
:query [:find ?e
:in $ ?email
:where [?e :user/email ?email]]}}}
:vase.api/schemas [:accounts/user]
:vase.api/forward-headers ["vaserequest-id"]}}The "/user" route now supports both POST and GET requests. The POST
request hits the #vase/transact the same as before. Now a GET runs
the #vase/query action. The query looks up an entity based on a
query string parameter. The two main keys of interest in the
#vase/query action are :params and :query. (We'll discuss an
optional third property a bit later.)
The :params property defines
the accepted keyed data names that are used as external arguments to
the query to resolve those listed parameters to the incoming
values. The #vase/query action passes these as extra arguments to
datomic.api/q, following the database value itself.
If the :params field is empty or missing, they query doesn't accept
any arguments. In that case all parameters in the URL, query string,
form, etc. will be ignored.
The :query property contains a Datomic
datalog query.
One limitation of providing query parameters as URL arguments or path parameters is that only string types are shuttled across to the server. Often it's useful to refer to arguments that are other useful types, numbers are a common case. Here's a new route to illustrate that conversion:
"/user/:id" {:get #vase/query {:name :accounts-v1/user-id-page
:params [id]
:edn-coerce [id]
:query
[:find ?e
:in $ ?id
:where [?e :user/userId ?id]]}}(We're going to stop repeating all the routes in the interest of saving space and helping focus on the new stuff. It should be clear by now where this goes in the EDN file.)
The new route /user/:id allows the a similar information lookup, but
with the argument as a path parameter
(e.g. http://example.com/user/id). By default the :id parameter
would be a string value, but by using the :edn-coerce property of
the #query action we tell Vase to attempt to parse the string as a
valid EDN data type. Therefore, when clients hit the URL bound to
that query the proper types will match (i.e. the DB expects integer
IDs, not string IDs).
It's often useful to model a query that match against anything within
a given data set, for example, "Give me all users whose email is in
["[email protected]", "[email protected]"]"
In Datomic this is called a "parameterized in/or query," and it's achieved by
binding the names with the :in clause. To do this, we supply additional
constant data to the Vase query with the :constants option.
This can also be used with parameter binding (covered above), in which case parameters
are passed in to the query before the constants.
An example of simple constants follows - observe our last change to the schema below:
"/special-users" {:get #vase/query {:name :accounts.v1/special-users
:params []
:constants [["[email protected]" "[email protected]"]]
:query
[:find ?e
:in $ [?email ...]
:where
[?e :user/email ?email]]}}So far, we've used an in-memory URI for Datomic. That means just what
it sounds like: values are only stored in memory. To make it
persistent, you need to pick a storage engine and update the
:datomic-uri value.
We've covered most of the action literals. The examples in this guide created a real, if somewhat quirky, API for an accounts system.
The next step is to read the Action Literals reference.