feat: preserve flows ids, add hooks and handle translations collection (

#23) * ci: run PR CI on next branch * feat(cli): move options to commands * feat(cli): manage config with a service * chore: optimize imports * feat: add config loader from file * test: add tests for ConfigFileLoader * feat: format zod errors * feat: merge options from default to CLI options * chore: move tests * chore: ignore directus-config in prettier config * feat: auth with email and password (#13) * feat: allow authentication with email/password * docs: add config example directus-sync.config.js * chore: merge version 3.1 from main * feat: preserve flow ids on pull and push (#20) * feat: configuration improvement (#14) * ci: run PR CI on next branch * feat(cli): move options to commands * feat(cli): manage config with a service * chore: optimize imports * feat: add config loader from file * test: add tests for ConfigFileLoader * feat: format zod errors * feat: merge options from default to CLI options * chore: move tests * chore: ignore directus-config in prettier config * feat: auth with email and password (#13) * feat: allow authentication with email/password * docs: add config example directus-sync.config.js * chore(release): bump version [skip ci] - [email protected] - [email protected] * feature: treat non-existent directories as empty (#18) * chore(release): bump version [skip ci] - [email protected] - [email protected] * feat: preserve id of flows --------- Co-authored-by: tractr-bot <[email protected]> Co-authored-by: Evgeniy <[email protected]> * feat: add hooks (#21) * feat(cli): add version command * feat: add hooks schema in zod * feat: add transform data hooks for onLoad and onSave events * feat: add onDump ,onSave and onLoad hooks * feat: provides Directus client in hooks * docs: add hooks examples * docs: add warning about onDump * feat: add onQuery hook * feat: import pull process * docs: add onQuery and mermaid * docs: add onQuery and mermaid * docs: improve mermaid * feat: handle translations collection (#22) --------- Co-authored-by: tractr-bot <[email protected]> Co-authored-by: Evgeniy <[email protected]>
tractr · Jan 23, 2024 · 0b3faf9 · 0b3faf9
1 parent 89a3931
commit 0b3faf9
Show file tree

Hide file tree

Showing 36 changed files with 568 additions and 38 deletions.
diff --git a/README.md b/README.md
@@ -16,7 +16,8 @@ for targeted updates and clearer oversight of your Directus configurations.
 # Requirements
 
 - Node.js 18 or higher
-- `directus-extension-sync` installed on your Directus instance. See the [installation instructions](#dependency-directus-extension-sync).
+- `directus-extension-sync` installed on your Directus instance. See
+  the [installation instructions](#dependency-directus-extension-sync).
 
 # Usage
 
@@ -146,6 +147,181 @@ module.exports = {
 };
 ```
 
+### Hooks
+
+In addition to the CLI commands, `directus-sync` also supports hooks. Hooks are JavaScript functions that are executed
+at specific points during the synchronization process. They can be used to transform the data coming from Directus or
+going to Directus.
+
+Hooks are defined in the configuration file using the `hooks` property. Under this property, you can define the
+collection
+name and the hook function to be executed.
+Available collection names are: `dashboards`, `flows`, `operations`, `panels`, `permissions`, `roles`, `settings`, `translations`,
+and `webhooks`.
+
+For each collection, available hook functions are: `onQuery`, `onLoad`, `onSave`, and `onDump`.
+These can be asynchronous functions.
+
+During the `pull` command:
+
+- `onQuery` is executed just before the query is sent to Directus for get elements. It receives the query object as parameter and must
+  return the query object. The second parameter is the Directus client.
+- `onDump` is executed just after the data is retrieved from Directus and before it is saved to the dump files. The data
+  is the raw data received from Directus. The second parameter is the Directus client. It must return the data to be
+  saved to the dump files.
+- `onSave` is executed just before the cleaned data is saved to the dump files. The "cleaned" data is the data without
+  the columns that are ignored by `directus-sync` (such as `user_updated`) and with the relations replaced by the
+  SyncIDs. The first parameter is the cleaned data and the second parameter is the Directus client. It must return the
+  data to be saved to the dump files.
+
+During the `push` command:
+
+- `onLoad` is executed just after the data is loaded from the dump files. The data is the cleaned data, as described
+  above. The first parameter is the data coming from the JSON file and the second parameter is the Directus client.
+  It must return the data.
+
+#### Simple example
+
+Here is an example of a configuration file with hooks:
+
+```javascript
+// ./directus-sync.config.js
+module.exports = {
+  hooks: {
+    flows: {
+      onDump: (flows) => {
+        return flows.map((flow) => {
+          flow.name = `🧊 ${flow.name}`;
+          return flow;
+        });
+      },
+      onSave: (flows) => {
+        return flows.map((flow) => {
+          flow.name = `🔥 ${flow.name}`;
+          return flow;
+        });
+      },
+      onLoad: (flows) => {
+        return flows.map((flow) => {
+          flow.name = flow.name.replace('🔥 ', '');
+          return flow;
+        });
+      },
+    },
+  },
+};
+```
+
+> [!WARNING]  
+> The dump hook is called after the mapping of the SyncIDs. This means that the data received by the hook is already
+> tracked. If you filter out some elements, they will be deleted during the `push` command.
+
+#### Filtering out elements
+
+You can use `onQuery` hook to filter out elements. This hook is executed just before the query is sent to Directus, during the `pull` command.
+
+In the example below, the flows and operations whose name starts with `Test:` are filtered out and will not be tracked.
+
+```javascript
+// ./directus-sync.config.js
+const testPrefix = 'Test:';
+
+module.exports = {
+  hooks: {
+    flows: {
+      onQuery: (query, client) => {
+        query.filter = {
+          ...query.filter,
+          name: { _nstarts_with: testPrefix },
+        };
+        return query;
+      },
+    },
+    operations: {
+      onQuery: (query, client) => {
+        query.filter = {
+          ...query.filter,
+          flow: { name: { _nstarts_with: testPrefix } },
+        };
+        return query;
+      },
+    },
+  },
+};
+```
+
+> [!WARNING]
+> Directus-Sync may alter the query after this hook. For example, for `roles`, the query excludes the `admin` role.
+
+#### Using the Directus client
+
+The example below shows how to disable the flows whose name starts with `Test:` and add the flow name to the operation.
+
+```javascript
+const { readFlow } = require('@directus/sdk');
+
+const testPrefix = 'Test:';
+
+module.exports = {
+  hooks: {
+    flows: {
+      onDump: (flows) => {
+        return flows.map((flow) => {
+          flow.status = flow.name.startsWith(testPrefix)
+            ? 'inactive'
+            : 'active';
+        });
+      },
+    },
+    operations: {
+      onDump: async (operations, client) => {
+        for (const operation of operations) {
+          const flow = await client.request(readFlow(operation.flow));
+          if (flow) {
+            operation.name = `${flow.name}: ${operation.name}`;
+          }
+        }
+        return operations;
+      },
+    },
+  },
+};
+```
+
+### Lifecycle & hooks
+
+#### `Pull` command
+
+```mermaid
+flowchart
+  subgraph Pull[Get elements - for each collection]
+    direction TB
+		B[Create query for all elements]
+    -->|onQuery hook|C[Add collection-specific filters]
+    -->D[Get elements from Directus]
+    -->E[Get or create SyncId for each element. Start tracking]
+    -->F[Remove original Id of each element]
+    -->|onDump hook|G[Keep elements in memory]
+  end
+	subgraph Post[Link elements - for each collection]
+    direction TB
+		H[Get all elements from memory]
+    --> I[Replace relations ids by SyncIds]
+    --> J[Remove ignore fields]
+    --> K[Sort elements]
+    -->|onSave hook|L[Save to JSON file]
+  end
+ A[Pull command] --> Pull --> Post --> Z[End]
+```
+
+#### `Diff` command
+
+**Coming soon**
+
+#### `Push` command
+
+**Coming soon**
+
 ### Tracked Elements
 
 `directus-sync` tracks the following Directus collections:
@@ -157,6 +333,7 @@ module.exports = {
 - permissions
 - roles
 - settings
+- translations
 - webhooks
 
 For these collections, data changes are committed to the code, allowing for replication on other Directus instances. A
@@ -186,7 +363,7 @@ configurations and schema within Directus. Here is a step-by-step explanation of
 Upon execution of the `pull` command, `directus-sync` will:
 
 1. Scan the specified Directus collections, which include dashboards, flows, operations, panels, permissions, roles,
-   settings, and webhooks.
+   settings, translations and webhooks.
 2. Assign a SyncID to each element within these collections if it doesn't already have one.
 3. Commit the data of these collections into code, allowing for versioning and tracking of configuration changes.
 

diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts
@@ -62,6 +62,7 @@ const forceOption = new Option(
 );
 
 program
+  .version(process.env.npm_package_version ?? 'unknown')
   .addOption(debugOption)
   .addOption(directusUrlOption)
   .addOption(directusTokenOption)

diff --git a/packages/cli/src/lib/loader.ts b/packages/cli/src/lib/loader.ts
@@ -7,6 +7,7 @@ import {
   PermissionsCollection,
   RolesCollection,
   SettingsCollection,
+  TranslationsCollection,
   WebhooksCollection,
 } from './services';
 import { createDumpFolders } from './helpers';
@@ -62,6 +63,7 @@ export function loadCollections() {
   // The collections are populated in the same order
   return [
     Container.get(SettingsCollection),
+    Container.get(TranslationsCollection),
     Container.get(WebhooksCollection),
     Container.get(FlowsCollection),
     Container.get(OperationsCollection),

diff --git a/packages/cli/src/lib/services/collections/base/data-client.ts b/packages/cli/src/lib/services/collections/base/data-client.ts
@@ -4,6 +4,7 @@ import {
   DirectusId,
   Query,
   WithoutIdAndSyncId,
+  WithoutSyncId,
 } from './interfaces';
 import { MigrationClient } from '../../migration-client';
 
@@ -38,7 +39,7 @@ export abstract class DataClient<DirectusType extends DirectusBaseType> {
    * Inserts data into the target collection using the rest API.
    * Remove the id and the syncId from the item before inserting it.
    */
-  async create(item: WithoutIdAndSyncId<DirectusType>): Promise<DirectusType> {
+  async create(item: WithoutSyncId<DirectusType>): Promise<DirectusType> {
     const directus = await this.migrationClient.get();
     return await directus.request(await this.getInsertCommand(item));
   }
@@ -73,7 +74,7 @@ export abstract class DataClient<DirectusType extends DirectusBaseType> {
     | Promise<RestCommand<DirectusType[], object>>;
 
   protected abstract getInsertCommand(
-    item: WithoutIdAndSyncId<DirectusType>,
+    item: WithoutSyncId<DirectusType>,
   ):
     | RestCommand<DirectusType, object>
     | Promise<RestCommand<DirectusType, object>>;

diff --git a/packages/cli/src/lib/services/collections/base/data-differ.ts b/packages/cli/src/lib/services/collections/base/data-differ.ts
@@ -36,7 +36,7 @@ export abstract class DataDiffer<DirectusType extends DirectusBaseType> {
    * Returns the diff between the dump and the target table.
    */
   async getDiff() {
-    const sourceData = this.dataLoader.getSourceData();
+    const sourceData = await this.dataLoader.getSourceData();
 
     const toCreate: WithSyncIdAndWithoutId<DirectusType>[] = [];
     const toUpdate: UpdateItem<DirectusType>[] = [];
@@ -81,7 +81,6 @@ export abstract class DataDiffer<DirectusType extends DirectusBaseType> {
     const idMap = await this.idMapper.getBySyncId(sourceItem._syncId);
     if (idMap) {
       const targetItem = await this.dataClient
-
         .query({ filter: { id: idMap.local_id } } as Query<DirectusType>)
         .then((items) => items[0])
         .catch(() => {

diff --git a/packages/cli/src/lib/services/collections/base/data-loader.ts b/packages/cli/src/lib/services/collections/base/data-loader.ts
@@ -1,26 +1,40 @@
 import { DirectusBaseType, WithSyncIdAndWithoutId } from './interfaces';
 import { readJsonSync, writeJsonSync } from 'fs-extra';
+import { Hooks } from '../../config';
+import { MigrationClient } from '../../migration-client';
 
 export abstract class DataLoader<DirectusType extends DirectusBaseType> {
-  constructor(protected readonly filePath: string) {}
+  constructor(
+    protected readonly filePath: string,
+    protected readonly migrationClient: MigrationClient,
+    protected readonly hooks: Hooks,
+  ) {}
 
   /**
    * Returns the source data from the dump file, using readFileSync
    * and passes it through the data transformer.
    */
-  getSourceData(): WithSyncIdAndWithoutId<DirectusType>[] {
-    return readJsonSync(
+  async getSourceData(): Promise<WithSyncIdAndWithoutId<DirectusType>[]> {
+    const { onLoad } = this.hooks;
+    const loadedData = readJsonSync(
       this.filePath,
     ) as WithSyncIdAndWithoutId<DirectusType>[];
+    return onLoad
+      ? await onLoad(loadedData, await this.migrationClient.get())
+      : loadedData;
   }
 
   /**
    * Save the data to the dump file. The data is passed through the data transformer.
    */
-  saveData(data: WithSyncIdAndWithoutId<DirectusType>[]) {
+  async saveData(data: WithSyncIdAndWithoutId<DirectusType>[]) {
     // Sort data by _syncId to avoid git changes
     data.sort(this.getSortFunction());
-    writeJsonSync(this.filePath, data, { spaces: 2 });
+    const { onSave } = this.hooks;
+    const transformedData = onSave
+      ? await onSave(data, await this.migrationClient.get())
+      : data;
+    writeJsonSync(this.filePath, transformedData, { spaces: 2 });
   }
 
   /**