Merge pull request #15 from commercetools/documentation

Documentation
commercetools · Oct 10, 2023 · 88340e8 · 88340e8
2 parents c4f1a3e + c83ffa9
commit 88340e8
Show file tree

Hide file tree

Showing 11 changed files with 270 additions and 4 deletions.
diff --git a/README.md b/README.md
@@ -1 +1,119 @@
-# connect-search-template
+# connect-search-template
+This repository provides a template of search connector used in [connect service](https://github.com/commercetools/connect-services), which helps users to customize their own connector application to synchronize product changes between specific store in the commercetools project and external search index.
+
+## Features
+- NodeJS supported.
+- Uses Express as web server framework.
+- Uses [commercetools SDK](https://docs.commercetools.com/sdk/js-sdk-getting-started) for the commercetools-specific communication.
+- Includes local development utilities in npm commands to build, start, test, lint & prettify code.
+- Uses JSON formatted logger with log levels
+- Setup sample integration tests with [Jest](https://jestjs.io/) and [supertest](https://github.com/ladjs/supertest#readme)
+
+##Prerequisite
+#### Develop your search-specific SDK 
+To import the [commercetools Product Projections](https://docs.commercetools.com/api/projects/productProjections) to external search index, users need to develop their own SDK which is responsible mainly for the following tasks
+- Data Mapping: The custom SDK needs to transform the product projection objects from commercetools structure to users-desired structure for the search index.
+- Data Persistence: The custom SDK is capable to save/remove product data to/from the specific search index. Please remember that the product data might not be saved into the external search index in a single attempt. It is because of performance concern in case of huge amount of product projections exported from commercetools platform.
+
+To install the custom SDK, please publish your developed SDK as a package to the npm registry, and run following npm command to install the package.
+
+#### Create external search index
+Users are expected to create search index in external platform themselves. The search connector application does not create search index during the application running. Therefore, please provides details of the search index including its identity after it is created. Details of search index can be provided as environment variables `SEARCH_PLATFORM_CONFIG` when starting the deployment.
+For details about `SEARCH_PLATFORM_CONFIG`, please also read [Deployment Configuration](./README.md#Deployment Configuration).
+
+##Getting started
+The template contains two separated modules :
+- Full Ingestion : Provides a REST-API to users to export all products from specific store of a commercetools project to external search index. 
+- Incremental Updater : Receives message from GCP Pub/Sub once there are product changes in commercetools store. The modified products are then synchronized to the existing external search index.
+
+Regarding the development of both modules, please refer to the following documetations:
+- Development of Full Ingestion
+- Development of Incremental Updater
+
+##Deployment Configuration
+In order to deploy your customized search connector application on commercetools-provided infrastructure, it needs to reviewed by certification team. For details, please refer to [documentation about commercetools Connect](https://docs.commercetools.com/connect/concepts)
+In addition, in order to support connect service, the search connector template has a folder structure as listed below
+```
+├── full-ingestion
+│   ├── src
+│   ├── test
+│   └── package.json
+├── incremental-updater
+│   ├── src
+│   ├── test
+│   └── package.json
+└── connect.yaml
+```
+
+Connect deployment configuration is specified in `connect.yaml` which is required information needed for certification of the application. Following is the deployment configuration used by full ingestion and incremental updater modules
+```
+deployAs:
+  - name: full-ingestion
+    applicationType: service
+    endpoint: /fullSync
+    scripts:
+      postDeploy: npm install
+    configuration:
+      securedConfiguration:
+        - key: CTP_PROJECT_KEY
+          description: commercetools project key
+        - key: CTP_CLIENT_ID
+          description: commercetools client ID
+        - key: CTP_CLIENT_SECRET
+          description: commercetools client secreet
+        - key: CTP_SCOPE
+          description: commercetools client scope
+        - key: CTP_REGION
+          description: Region of commercetools project
+        - key: SEARCH_PLATFORM_CONFIG
+          description: Escaped JSON object including credentails to search platform and other settings
+  - name: incremental-updater
+    applicationType: event
+    endpoint: /deltaSync
+    scripts:
+      postDeploy: npm install && npm run connector:post-deploy
+      preUndeploy: npm install && npm run connector:pre-undeploy
+    configuration:
+      securedConfiguration:
+        - key: CTP_STORE_KEY
+          description: Unique key of commercetools Store
+        - key: CTP_PROJECT_KEY
+          description: commercetools project key
+        - key: CTP_CLIENT_ID
+          description: commercetools client ID
+        - key: CTP_CLIENT_SECRET
+          description: commercetools client secreet
+        - key: CTP_SCOPE
+          description: commercetools client scope
+        - key: CTP_REGION
+          description: Region of commercetools project
+        - key: SEARCH_PLATFORM_CONFIG
+          description: Escaped JSON object including credentails to search platform and other settings
+```
+
+Here you can see the details about various variables in configuration
+- CTP_PROJECT_KEY: The key of commercetools project.
+- CTP_CLIENT_ID: The client ID of your commercetools user account. It is used in commercetools client to communicate with commercetools platform via SDK.
+- CTP_CLIENT_SECRET: The client secret of commercetools user account. It is used in commercetools client to communicate with commercetools platform via SDK.
+- CTP_SCOPE: The scope constrains the endpoints to which the commercetools client has access, as well as the read/write access right to an endpoint.
+- CTP_REGION: As the commercetools APIs are provided in six different region, it defines the region which your commercetools user account belongs to.
+- SEARCH_PLATFORM_CONFIG: It defines the configurations required by the external search index, such as credentials, search index unique identifier, etc.
+  Following is a sample JSON object of this variable.
+
+    ```
+    {
+        SEARCH_INDEX_CREDENTIAL_ID: xxx,
+        SEARCH_INDEX_CREDENTIAL_SECRET: yyy,
+        SEARCH_INDEX_ID: zzz
+    }
+
+    ```
+  The value of this configuration variable needs to be in escaped JSON format. Hence, based on the sample above, the expected value of this variable becomes
+  ```
+  '{ "SEARCH_INDEX_CREDENTIAL_ID": "xxx", "SEARCH_INDEX_CREDENTIAL_SECRET": "yyy", "SEARCH_INDEX_ID": "zzz" }'
+  ```
+- CTP_STORE_KEY : Only used in incremental updater. It specifies the key of commercetools store so that connector can look up the modified product under the specific store in commercetools platform.
+
+##Recommendations
+#### Implement your own test cases
+We have provided simple integration test cases with [Jest](https://jestjs.io/) and [supertest](https://github.com/ladjs/supertest#readme). The implementation is under `test` folder in both `full-ingestion` and `incremental-updater` modules. It is recommended to implement further test cases based on your own needs to test your development. 
diff --git a/full-ingestion/.env.example b/full-ingestion/.env.example
@@ -0,0 +1,10 @@
+SEARCH_PLATFORM_CONFIG=
+CTP_CLIENT_ID=
+CTP_CLIENT_SECRET=
+CTP_PROJECT_KEY=
+CTP_SCOPE=
+CTP_REGION=
+CTP_STORE_KEY=
+
+CONNECT_GCP_TOPIC_NAME=
+CONNECT_GCP_PROJECT_ID=
diff --git a/full-ingestion/README.md b/full-ingestion/README.md
@@ -0,0 +1,39 @@
+# Full Ingestion
+This module provides a REST-API hosted on commercetools-provided infrastructure. Once the API is triggered, it exports all products under the specific store of the commercetools project, and then import to the external search index.
+
+## Get started
+#### Install your search-specific SDK 
+Assuming you have already published your developed SDK as a package to the npm registry, please run following npm command under full-ingestion folder to install the package.
+```
+$ npm install <your-search-index-sdk>
+```
+#### Install dependencies
+```
+$ npm install
+```
+#### Run integration test
+```
+$ npm run test:integration
+```
+#### Run the application in local environment
+```
+$ npm run start
+```
+## Development
+
+#### Set the required environment variables
+
+Before starting the development, we advise users to create a .env file in order to help them in local development.
+
+For that, we also have a template file .env.example with the required environement variables for the project to run successfuly. To make it work, rename the file from `.env.example` to `.env`. Remember to fill the variables with your values.
+
+#### Execute full synchronization
+After deployment by connect service or starting up the application in local environment, you can trigger the full synchronization by sending HTTP POST request to REST-API as below
+```
+curl 
+--location 
+--request POST 'https://<your-search-connector-host>/fullSync/<your-commercetools-store-key>' \
+--data ''
+
+```
+Remind that you need to change the host name and append the key of commercetools Store as query parameter in the URL.
diff --git a/full-ingestion/package.json b/full-ingestion/package.json
@@ -10,7 +10,8 @@
     "lint": "node_modules/.bin/eslint src --ext .js",
     "prettier": "node_modules/.bin/prettier --write '**/*.{js,ts}'",
     "test": "echo Comment: It is npm command dedicated for running test in connect service",
-    "test:ci": "node_modules/.bin/jest --config jest.config.cjs"
+    "test:integration": "node_modules/.bin/jest --config jest.config.cjs",
+    "test:ci": "npm run test:integration"
   },
   "author": "",
   "license": "MIT",

diff --git a/full-ingestion/src/clients/create.client.js b/full-ingestion/src/clients/create.client.js
@@ -4,7 +4,7 @@ import { readConfiguration } from '../utils/config.utils.js';
 
 /**
  * Create client with apiRoot
- * apiRoot can now be used to build requests to de Composable Commerce API
+ * apiRoot can now be used to build requests to Composable Commerce API
  */
 export const createApiRoot = ((root) => () => {
   if (root) {

diff --git a/incremental-updater/.env.example b/incremental-updater/.env.example
@@ -0,0 +1,10 @@
+SEARCH_PLATFORM_CONFIG=
+CTP_CLIENT_ID=
+CTP_CLIENT_SECRET=
+CTP_PROJECT_KEY=
+CTP_SCOPE=
+CTP_REGION=
+CTP_STORE_KEY=
+
+CONNECT_GCP_TOPIC_NAME=
+CONNECT_GCP_PROJECT_ID=
diff --git a/incremental-updater/README.md b/incremental-updater/README.md
@@ -0,0 +1,68 @@
+# Incremental Updater
+This module provides an application hosted on commercetools-provided infrastructure, which receives messages from Google Cloud Pub/Sub when product changes under specific [commercetools Store](https://docs.commercetools.com/api/projects/stores) occur. 
+
+The module also provides scripts for post-deployment and pre-undeployment action. After deployment via connect service completed, [commercetools Subscription](https://docs.commercetools.com/api/projects/subscriptions) is created by post-deployment script which listen to any product changes under specific store.
+Once products in the store have been changed, the commercetools Subscription sends message to Google Cloud Pub/Sub topic and then notify the incremental updater to handle the corresponding product changes.
+
+The commercetools Subscription would be cleared once the search connector is undeployed.
+
+## Assumption
+#### Support single store
+Since search index in external platform is supposed to be store-specific, each deployed search connector handles product changes from single commercetools Store. The key of commercetools Store has to be defined as environment variable before deployment.
+
+For details how to set environment variables for search connector in non-local environment, please refer to [Deployment Configuration](../README.md#Deployment Configuration).  
+## Get started
+#### Change the key of commercetools Subscription
+Please specify your desired key for creation of commercetools Subscription [here](https://github.com/commercetools/connect-search-ingestion-template/blob/c4f1a3e04988a4a44842d3e1607638c96983ef29/incremental-updater/src/connectors/actions.js#L1).
+#### Install your search-specific SDK 
+Assuming you have already published your developed SDK as a package to the npm registry, please run following npm command under incremental-updater folder to install the package.
+```
+$ npm install <your-search-index-sdk>
+```
+#### Install dependencies
+```
+$ npm install
+```
+#### Run integration test
+```
+$ npm run test:integration
+```
+#### Run the application in local environment
+```
+$ npm run start
+```
+#### Run post-deploy script in local environment
+```
+$ npm run connector:post-deploy
+```
+#### Run pre-undeploy script in local environment
+```
+$ npm run connector:pre-undeploy
+```
+
+## Development in local environment
+Different from staging and production environments, in which the out-of-the-box settings and variables have been set by connect service during deployment, the search connector requires additional operations in local environment for development.
+#### Create Google Cloud pub/sub topic and subscription
+When an event-type connector application is deployed via connect service, a GCP pub/sub topic and subscription are created automatically. However it does not apply on local environment. To develop the search connector in local environment, you need to follow the steps below:
+1. Create a Pub/Sub topic and subscription in Google Cloud platform.
+2. Use HTTP tunnel tools like [ngrok](https://ngrok.com/docs/getting-started) to expose your local development server to internet.
+3. Set the URL provided by the tunnel tool as the destination of GCP subscription, so that message can be forwarded to the incremental updater in your local environment.  
+
+For details, please refer to the [Overview of the GCP Pub/Sub service](https://cloud.google.com/pubsub/docs/pubsub-basics).
+
+#### Set the required environment variables
+
+Before starting the development, we advise users to create a .env file in order to help them in local development.
+
+For that, we also have a template file .env.example with the required environement variables for the project to run successfuly. To make it work, rename the file from `.env.example` to `.env`. Remember to fill the variables with your values.
+
+In addition, following two environment variables in `.env.example` are not needed to be provided by users during staging or production deployment. 
+```
+CONNECT_GCP_TOPIC_NAME=<your-gcp-topic-name>
+CONNECT_GCP_PROJECT_ID=<your-gcp-project-id>
+```
+It is because they are only required in local development server. For staging or production environment, connect service sets the Pub/Sub topic name and GCP project ID into these environment variables automatically after the Pub/Sub service has been created in Google Cloud platform.
+
+
+
+      
diff --git a/incremental-updater/package-lock.json b/incremental-updater/package-lock.json
diff --git a/incremental-updater/package.json b/incremental-updater/package.json
@@ -29,6 +29,7 @@
     "@commercetools/platform-sdk": "^4.1.0",
     "@commercetools/sdk-client-v2": "^2.0.1",
     "body-parser": "^1.20.1",
+    "dotenv": "^16.3.1",
     "express": "^4.18.2",
     "supertest": "^6.3.3",
     "validator": "^13.7.0"

diff --git a/incremental-updater/src/clients/create.client.js b/incremental-updater/src/clients/create.client.js
@@ -4,7 +4,7 @@ import { readConfiguration } from '../utils/config.utils.js';
 
 /**
  * Create client with apiRoot
- * apiRoot can now be used to build requests to de Composable Commerce API
+ * apiRoot can now be used to build requests to Composable Commerce API
  */
 export const createApiRoot = ((root) => () => {
   if (root) {

diff --git a/incremental-updater/src/index.js b/incremental-updater/src/index.js
@@ -1,3 +1,5 @@
+import 'dotenv/config';
+
 import express from 'express';
 import bodyParser from 'body-parser';