Skip to content

Magic Metadata

Jump to bottom
josephjclark edited this page Mar 2, 2023 · 7 revisions

This page is a one-stop guide to creating magic metadata for adaptor functions

OpenFn magic works by first building a model of a datasource - basically a hierarchical list of schema - and then querying that model with json path.

If you want to add magic to an adaptor, you need to:

  • Ensure there's a metadata model for that datasource (and that it has the right information)
  • Write json path queries as jsdoc annotations in Adaptor.js
  • Write unit tests against your annotations

The Model

The metadata model is a very lightweight heirarchical model.

The basic structure is a tree of Entities which look like this:

type Entity = {
  name: string; // the value when inserted
  type: string; // domain-specific type string (eg OrgUnit, sObject)

  label?: string; // human readable label
  datatype?: string; // the javascript type (shown in monaco)
  desc?: string; // a longer description

  children?: Entity[] | Record<string, Entity>;

  // arbitrary extra stuff  goes in the meta object
  meta?: Record<string, any>;
}

An entity can either have an array of arbitrary children, like Salesforce:

{
  name: "my-salesforce-instance",
  children: [{
    name: "a"
    type "sobject"
  }]
}

Or can have typed children, like DHIS2:

{
  name: "my-dhis2-instance",
  children: {
    "orgUnits": [...],
    "trackedEntityTypes": [...]
  }
}

This can make writing json path queries a little easier.

CLI Utils

The adaptors repo has metadata command which can generate metadata and mock metadata for a given adaptor.

To generate metadata based on some config, run this from the repo root:

pnpm metadata generate <adaptor> <path/to/config>

Where the config path is relative to the adaptor.

For example, to generate salesforce metadata:

pnpm metadata generate salesforce src/metadata/config.js

Note: you will need to ensure the OPENFN_SF_USER, OPENFN_SF_PASS, and OPENFN_SF_TOKEN env vars are set (see the config.js file).

You should commit a sandbox model into this repo so that you can write unit tests against it later.

Creating metadata a function

If it doesn't already exist (or doesn't generate the right data for you), the first job is to create or modify a metadata function.

In the src/meta directory in the adaptor, create a metadata.js file with the following signature:

export default function(configuration = {}, mock = false): Model

Your metadata functions has two jobs:

  1. Fetch some kind of raw schema data from the datasource
  2. Convert that schema into an openfn model

You should write a set of helper functions to help you with (1), where each function basically runs a query against the datasource (ie, get all sobjects or get all orgunits). This helper can be auto-mocked to cache the results for unit testing later, which is super useful.

The helper function is a factory which takes config as an argument and returns an object with functions.

export default function createHelper(configuration = {}): { [fnName: string]: (...any) => object

See Salesforce for an example.

To test your metadata function, use the metadata command from the root: pnpm metadata generate <adaptor> <config>. The results will be saved to src/metada/data/metadata.json for you to inspect.

You can cache data from helper functions by providing a src/meta/populate-mock-data.js file, which should exercise the helper functions and write the results to disk. The auto-mock will:

  • Wrap each function defined in your helper
  • Save the results to a file in src/meta/data of the form fnName-__arg1__argN (ie, the function name followed by its arguments)
  • When called, will return the locally saved data rather than calling out to a live endpoint.

Using a mock metadata function (or mock helpers) you can then write unit tests against this cached data.

Testing and mocks

It's a good idea to create unit tests early for your metadata function. You may want separate tests on your metadata function and on its helpers.

The metadata takes as second parameter a mock flag. If true, the metadata should use a mock helper (which can be autogenerated). This is useful when writing unit tests because it'll test against saved data.

Use the createMock function from @openfn/metadata (found in tools):

import { createMock, createEntity } from '@openfn/metadata';
import helper from './helper';

const metadata = async (configuration = {}, mock = false) => {
  let helper = helper(configuration);
  if (mock) {
    helper = createMock(helper);
  }
}

Create a populate-mock-data.js file which calls to a live backend, calls helper functions, and saves the results to disk:

export default async () => {
  const helper = await createHelper(state.configuration);
  const mock = createMock(helper);
  await mock.getGlobals();
}

Use the populate-mock command to run this file and update your cached data:

pnpm metadata populate-mock salesforce src/metadata/config.js

Writing JSONPath queries

Given a metadata model, you can start writing queries to drive magic functions.

Magic is simply a JSONPath query which pulls appropriate metadata out of the model in order to populate a picklist. Depending on what type of data you're looking up, you'll either want to return a list of strings or a list of entities.

Write your query as a @lookup (name TBD) JSDoc annotation with the parameter name and a query

* @param {string} resourceType - Type of resource to create. E.g. `trackedEntityInstances`, `programs`, `events`, ...
* @paramlookup resourceType $.children.resourceTypes[*]

Your json query will run against the model, with an entity at its root ($). It'll always query the children, so start with $.children and then some query/path.

You can insert templated varibles into your query. At the time of writing, the only substitutable value is the args object, which for each argument name passed to the function, will provide a value (if known). This works well for string values passed to other arguments. Variablesare inserted via {{x}} syntax

* @paramlookup externalId - $.children[?(@.name=="{{args.sObject}}")].children[?(@.meta.externalId)].name

Writing queries is hard. One day we'll have a tool to help, but for now the recommended process is:

  • Paste some or all of your cached metadata into https://jsonpath.com
  • Write a query in the search bar
  • Paste the query into JSDoc

Testing Queries

Unit tests can automatically parse your adaptor's jsdoc and extract the query strings, so you can test against them directly.

First, load the query strings from your source file:

import extractLookups from '@openfn/parse-jsdoc';

let queries;
before(async () => {
  queries = await extractLookups(path.resolve('src/Adaptor.js'));
});

The queries object will have a key for every magic function (ie, upsert), and a nested key for every parameter (ie, sObject). Using this, you can execute queries against some test data:

import data from 'src/metadata/data/metadata.json' assert { type: 'json' };

it('upsert.sObject: should list non-system sObject names', () => {
  const results = jp.query(data, queries.upsert.sObject);
  expect(results[0]).to.equal('vera__Beneficiary__c');
});
Clone this wiki locally