Magic Metadata

This page is a one-stop guide to creating magic metadata for adaptor functions

OpenFn magic works by first building a model of a datasource - basically a hierarchical list of schema - and then querying that model with json path.

If you want to add magic to an adaptor, you need to:

• Ensure there's a metadata model for that datasource (and that it has the right information)
• Write json path queries as jsdoc annotations in Adaptor.js
• Write unit tests against your annotations

The Model

The metadata model is a very lightweight heirarchical model. It still need a little work.

The basic structure is a tree of Entities which look like this:

Note: the current implementation isn't quite the same, I'm just about to update it and bring it all into line

type Entity = {
  label?: string; // human readable label
  type: string; // domain-specific type string (eg OrgUnit, sObject)
  datatype?: string; // the javascript type (shown in monaco)
  desc?: string;
  value?: string; // the value when inserted (TODO: support templating?)

  // Is this system/admin entity?
  system?: boolean;
  
  // children can be an array or a named map
  children?: Entity[] | Record<string, Entity>;
}

CLI Utils

The adaptors repo has metadata command which can generate metadata and mock metadata for a given adaptor.

To generate metadata based on some config, run this from the repo root:

pnpm metadata generate <adaptor> <path/to/config>

Where the config path is relative to the adaptor.

So, to generate salesforce metadata, do

pnpm metadata generate salesforce src/metadata/config.js

You will need to ensure the OPENFN_SF_USER, OPENFN_SF_PASS, and OPENFN_SF_TOKEN env vars are set (see the config.js file).

Unit tests for metadata functions run against cached data objects (more on this later). To (re)create these, run the populate-mock command:

pnpm metadata populate-mock salesforce src/metadata/config.js

Creating metadata a function

If it doesn't already exist (or doesn't generate the right data for you), the first job is to create or modify a metadata function.

In the src/meta directory in the adaptor, create a metadata.js file. This should export a single default function which takes state (with config) and returns a model.

Your metadata functions has two jobs:

1. Fetch some kind of raw schema data from the datasource
2. Convert that schema into an openfn model

You should write a set of helper functions to help you with 1, where each function basically runs a query against the datasource (ie, get all sobjects or get all orgunits). This helper can be automocked to cache the results for unit testing later, which is super useful.

The helper function is a factory which takes config as an argument and returns an object with functions.

To test your metadata function, use the metadata command from the root: pnpm metadata generate <adaptor> <config>. The results will be saved to src/metada/data/metadata.json for you to inspect.

You can also test the helpers by providing a src/meta/populate-mock-data.js file, which should exercise the helper function and write the results to disk.

Testing a metadata function

It's a good idea to create unit test early for your metadata function. You may want separate tests on your metadata function and on its helpers.

The metadata takes as second parameter a mock flag. If true, the metadata should use a mock helper (which can be autogenerated). This is useful when writing unit tests because it'll test against saved data.

Use the createMock function from @openfn/metadata (found in tools):

import { createMock, createEntity } from '@openfn/metadata';
import helper from './helper';

const metadata = async (configuration = {}, mock = false) => {
  let helper = helper(configuration);
  if (mock) {
    helper = createMock(helper);
  }
}

Writing Queries

Magic can be applied to parameters and object values used by the adaptor functions.

Magic is simply a JSONPath query which pulls appropriate metadata out of the model in order to populate a picklist.

Writing queries is hard. One day we'll have a tool to help, but for now the recommended process is:

• Paste some or all of your cached metadata into https://jsonpath.com
• Write a query in the search bar
• Paste the query into JSDoc

TODO - examples of jsdoc attributes for parameters and properties

Testing Queries

Unit tests can automatically parse your adaptor's jsdoc and extract the query strings, so you can test against them directly.

First, load the query strings from your source file:

import extractLookups from '@openfn/parse-jsdoc';

let queries;
before(async () => {
  queries = await extractLookups(path.resolve('src/Adaptor.js'));
});

The queries object will have a key for every magic function (ie, upsert), and a nested key for every parameter (ie, sObject). Using this, you can execute queries against some test data:

import data from 'src/metadata/data/metadata.json' assert { type: 'json' };

it('upsert.sObject: should list non-system sObject names', () => {
  const results = jp.query(data, queries.upsert.sObject);
  expect(results[0]).to.equal('vera__Beneficiary__c');
});