Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Devrel 1074 import taxonomies #9

Merged
merged 2 commits into from
Dec 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/commands/export.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import { languageVariantsExportEntity } from "./importExportEntities/entities/la
import { previewUrlsExportEntity } from "./importExportEntities/entities/previewUrls.js";
import { rolesExportEntity } from "./importExportEntities/entities/roles.js";
import { spacesExportEntity } from "./importExportEntities/entities/spaces.js";
import { taxonomiesExportEntity } from "./importExportEntities/entities/taxonomies.js";
import { taxonomiesEntity } from "./importExportEntities/entities/taxonomies.js";
import { workflowsExportEntity } from "./importExportEntities/entities/workflows.js";
import { EntityDefinition } from "./importExportEntities/entityDefinition.js";

Expand Down Expand Up @@ -50,7 +50,7 @@ export const register: RegisterCommand = yargs => yargs.command({
const entityDefinitions: ReadonlyArray<EntityDefinition<any>> = [
collectionsEntity,
spacesExportEntity,
taxonomiesExportEntity,
taxonomiesEntity,
languagesEntity,
previewUrlsExportEntity,
rolesExportEntity,
Expand Down
12 changes: 8 additions & 4 deletions src/commands/import.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { RegisterCommand } from "../types/yargs.js";
import { serially } from "../utils/requests.js";
import { collectionsEntity } from "./importExportEntities/entities/collections.js";
import { languagesEntity } from "./importExportEntities/entities/languages.js";
import { taxonomiesEntity } from "./importExportEntities/entities/taxonomies.js";
import { EntityDefinition, ImportContext } from "./importExportEntities/entityDefinition.js";

export const register: RegisterCommand = yargs => yargs.command({
Expand Down Expand Up @@ -34,7 +35,8 @@ export const register: RegisterCommand = yargs => yargs.command({
// Keep in mind that there are dependencies between entities so the order is important.
const entityDefinitions: ReadonlyArray<EntityDefinition<any>> = [
collectionsEntity,
languagesEntity
languagesEntity,
taxonomiesEntity,
];

type ImportEntitiesParams = Readonly<{
Expand All @@ -55,7 +57,9 @@ const importEntities = async (params: ImportEntitiesParams) => {

let context: ImportContext = {
collectionIdsByOldIds: new Map(),
languageIdsByOldIds: new Map()
languageIdsByOldIds: new Map(),
taxonomyGroupIdsByOldIds: new Map(),
taxonomyTermIdsByOldIds: new Map(),
};

await serially(entityDefinitions.map(def => async () => {
Expand All @@ -69,12 +73,12 @@ const importEntities = async (params: ImportEntitiesParams) => {
?? context;

console.log(`${def.name} imported`);

console.log(`All entities were successfully imported into environment ${params.environmentId}.`);
}
catch (err) {
console.error(`Failed to import entity ${def.name} due to error ${JSON.stringify(err)}. Stopping import...`);
process.exit(1);
}
}));

console.log(`All entities were successfully imported into environment ${params.environmentId}.`);
};
34 changes: 31 additions & 3 deletions src/commands/importExportEntities/entities/taxonomies.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,39 @@
import { TaxonomyContracts } from "@kontent-ai/management-sdk";

import { zip } from "../../../utils/array.js";
import { serially } from "../../../utils/requests.js";
import { EntityDefinition } from "../entityDefinition.js";

export const taxonomiesExportEntity: EntityDefinition<ReadonlyArray<TaxonomyContracts.ITaxonomyContract>> = {
export const taxonomiesEntity: EntityDefinition<ReadonlyArray<TaxonomyContracts.ITaxonomyContract>> = {
name: "taxonomies",
fetchEntities: client => client.listTaxonomies().toAllPromise().then(res => res.data.items.map(t => t._raw)),
serializeEntities: taxonomies => JSON.stringify(taxonomies),
importEntities: () => { throw new Error("Not supported yet.")},
deserializeEntities: () => { throw new Error("Not supported yet.")},
importEntities: async (client, fileTaxonomies, context) => {
const projectTaxonomies = await serially<ReadonlyArray<() => Promise<TaxonomyContracts.ITaxonomyContract>>>(
fileTaxonomies.map(taxonomy => () =>
client
.addTaxonomy()
.withData(addExternalIds(taxonomy))
.toPromise()
.then(res => res.data._raw))
);

return {
...context,
taxonomyGroupIdsByOldIds: new Map(zip(fileTaxonomies.map(t => t.id), projectTaxonomies.map(t => t.id))),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so we can assume that the order of taxonomies in the project are the same that are in the file? Even though that's probably true and makes sense, is there no possibility that the MAPI response can't mess this up and change the order?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The order of taxonomies is intentional, it is not a dictionary, but a list. You can even reorder them in the UI (not sure about MAPI). So I would be really surprised if it reordered them from what I sent it.

taxonomyTermIdsByOldIds: new Map(zip(fileTaxonomies.flatMap(t => t.terms), projectTaxonomies.flatMap(t => t.terms)).flatMap(extractTermIdsEntries)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need two Maps for taxonomies? we can have the same internal ID for taxonomy group and term? I think that it is not probable.. if not, what is the advantage having map for groups and terms?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought it would be a better separation. You never have an id that can be a group or a term. In content types you reference groups in variants you reference terms so why not have them in two maps.

};
},
deserializeEntities: JSON.parse,
};

const addExternalIds = (taxonomy: TaxonomyContracts.ITaxonomyContract): TaxonomyContracts.ITaxonomyContract => ({
...taxonomy,
external_id: taxonomy.external_id ?? taxonomy.codename,
terms: taxonomy.terms.map(addExternalIds),
});

const extractTermIdsEntries = ([fileTaxonomy, projectTaxonomy]: readonly [TaxonomyContracts.ITaxonomyContract, TaxonomyContracts.ITaxonomyContract]): ReadonlyArray<readonly [string, string]> => [
[fileTaxonomy.id, projectTaxonomy.id] as const,
...zip(fileTaxonomy.terms, projectTaxonomy.terms).flatMap(extractTermIdsEntries),
];
4 changes: 4 additions & 0 deletions src/commands/importExportEntities/entityDefinition.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,8 @@ export type EntityDefinition<T> = Readonly<{
export type ImportContext = Readonly<{
collectionIdsByOldIds: ReadonlyMap<string, string>;
languageIdsByOldIds: ReadonlyMap<string, string>;
taxonomyGroupIdsByOldIds: IdsMap;
taxonomyTermIdsByOldIds: IdsMap;
}>;

type IdsMap = ReadonlyMap<string, string>;
29 changes: 29 additions & 0 deletions src/utils/array.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
export const zip = <T1 extends ReadonlyArray<any>, T2 extends ReadonlyArray<any>>(arr1: T1, arr2: T2): Zip<T1, T2> =>
arr1
.slice(0, Math.min(arr1.length, arr2.length))
.map((el1, i) => [el1, arr2[i]] as const) as unknown as Zip<T1, T2>;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is really hard to follow.. maybe adding some comments, why is the array different from Tuple,

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean on the type-level or where?

type Zip<T1 extends ReadonlyArray<any>, T2 extends ReadonlyArray<any>> =
true extends IsEmptyTuple<T1> | IsEmptyTuple<T2>
? readonly []
: [IsNonEmptyTuple<T1>, IsNonEmptyTuple<T2>] extends [true, true]
? ZipTuples<T1, T2>
: ZipArrays<T1, T2>;

type IsEmptyTuple<T extends ReadonlyArray<any>> = T extends readonly [] ? true : false;

type IsNonEmptyTuple<T extends ReadonlyArray<any>> = T extends readonly [any, ...ReadonlyArray<any>] ? true : false;

/**
* Handles zip of two types where at least one is an array of unknown length.
*/
type ZipArrays<T1 extends ReadonlyArray<any>, T2 extends ReadonlyArray<any>> = ReadonlyArray<readonly [T1[number], T2[number]]>;

/**
* Handles zip of two tuples (arrays of known length and exact types for different positions).
* This type expects two tuples and doesn't work well with arrays of unknown length.
*/
type ZipTuples<T1 extends ReadonlyArray<any>, T2 extends ReadonlyArray<any>, Accum extends ReadonlyArray<readonly [any, any]> = readonly []> =
[T1, T2] extends [readonly [infer First1, ...infer Rest1 extends ReadonlyArray<any>], readonly [infer First2, ...infer Rest2 extends ReadonlyArray<any>]]
? ZipTuples<Rest1, Rest2, readonly [...Accum, readonly [First1, First2]]>
: Accum;