diff --git a/README.md b/README.md index b8cd3d7..560011e 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,9 @@ -# connect-search-template -This repository provides a template of search connector used in [connect service](https://github.com/commercetools/connect-services), which helps users to customize their own connector application to synchronize product changes between specific store in the commercetools project and external search index. +# connect-search-ingestion-template +This repository provides a [connect](https://docs.commercetools.com/connect) template for a search ingestion connector for populating a Store specific exnternal search engine. This boilerplate code act as a starting point for such integration. -## Features +This template uses the [Store](https://docs.commercetools.com/api/projects/stores) and [Product Selection](https://docs.commercetools.com/api/projects/product-selections) data models from commercetools composable commerce which can be used for querying Store-specific product data to power Store-specific frontends. Template is based on asynchronous [Subscriptions](https://docs.commercetools.com/api/projects/subscriptions) to keep the search indices up to date. + +## Template Features - NodeJS supported. - Uses Express as web server framework. - Uses [commercetools SDK](https://docs.commercetools.com/sdk/js-sdk-getting-started) for the commercetools-specific communication. @@ -9,30 +11,38 @@ This repository provides a template of search connector used in [connect service - Uses JSON formatted logger with log levels - Setup sample integration tests with [Jest](https://jestjs.io/) and [supertest](https://github.com/ladjs/supertest#readme) -##Prerequisite -#### Develop your search-specific SDK -To import the [commercetools Product Projections](https://docs.commercetools.com/api/projects/productProjections) to external search index, users need to develop their own SDK which is responsible mainly for the following tasks -- Data Mapping: The custom SDK needs to transform the product projection objects from commercetools structure to users-desired structure for the search index. -- Data Persistence: The custom SDK is capable to save/remove product data to/from the specific search index. Please remember that the product data might not be saved into the external search index in a single attempt. It is because of performance concern in case of huge amount of product projections exported from commercetools platform. +## Prerequisite +#### commercetools composable commerce API client +Users are expected to create API client responsible for fetching product, store and product selection details from composable commerce project, API client should have enought scope to be able to do so. These API client details are taken as input as an environment variable/ configuration for connect. Details of compsable commerce project can be provided as environment variables (configuration for connect) `CTP_PROJECT_KEY` , `CTP_CLIENT_ID`, `CTP_CLIENT_SECRET`, `CTP_SCOPE`, `CTP_REGION`. For details, please read [Deployment Configuration](./README.md#Deployment Configuration). + +#### commercetools composable commerce Data setup +Users are expected to create store and link product selection, accordingly linking products to corresponding product selection. The Store detail is taken as input as an environment variable / configuration for connect. Details of store can be provided as environment variables (configuration for connect) `CTP_STORE_KEY`. For details, please read [Deployment Configuration](./README.md#Deployment Configuration). -To install the custom SDK, please publish your developed SDK as a package to the npm registry, and run following npm command to install the package. +#### external search index creation +Users are expected to create search index in external search engine . The index details are taken as input as an environment variable / configuration for connect. Details of search index can be provided as environment variables (configuration for connect) `SEARCH_PLATFORM_CONFIG`.For details, please read [Deployment Configuration](./README.md#Deployment Configuration). -#### Create external search index -Users are expected to create search index in external platform themselves. The search connector application does not create search index during the application running. Therefore, please provides details of the search index including its identity after it is created. Details of search index can be provided as environment variables `SEARCH_PLATFORM_CONFIG` when starting the deployment. -For details about `SEARCH_PLATFORM_CONFIG`, please also read [Deployment Configuration](./README.md#Deployment Configuration). -##Getting started +## Getting started The template contains two separated modules : -- Full Ingestion : Provides a REST-API to users to export all products from specific store of a commercetools project to external search index. -- Incremental Updater : Receives message from GCP Pub/Sub once there are product changes in commercetools store. The modified products are then synchronized to the existing external search index. +- Full Ingestion : Provides a REST-API to users to export all products from specific store of a commercetools project to external search index as initial load or for full reindexing whenever needed. +- Incremental Updater : Receives message from commercetools project once there are product changes in commercetools store. The modified products are then synchronized to the existing external search index. Regarding the development of both modules, please refer to the following documetations: - Development of Full Ingestion - Development of Incremental Updater +#### Develop your search-specific needs +To import the [commercetools composable commerce Product Projections](https://docs.commercetools.com/api/projects/productProjections) to external search index, users need to extend this connector with the following tasks +- Data Mapping: Implementaion to transform the product projection objects from commercetools structure to users-desired structure for the search index. +- Data Persistence: Implementation to save/remove product data to the specific search index using libraries provided by external search product. Please remember that the product data might not be saved into the external search index in a single attempt, it should have needed retry and recovery mechanism. + +#### Register as connector in commercetools Connect +Follow guidelines [here](https://docs.commercetools.com/connect/getting-started) to register the connector for public/private use. + + ##Deployment Configuration -In order to deploy your customized search connector application on commercetools-provided infrastructure, it needs to reviewed by certification team. For details, please refer to [documentation about commercetools Connect](https://docs.commercetools.com/connect/concepts) -In addition, in order to support connect service, the search connector template has a folder structure as listed below +In order to deploy your customized search connector application on commercetools Connect, it needs be published. For details, please refer to [documentation about commercetools Connect](https://docs.commercetools.com/connect/concepts) +In addition, in order to support connect, the search connector template has a folder structure as listed below ``` ├── full-ingestion │ ├── src @@ -45,7 +55,7 @@ In addition, in order to support connect service, the search connector template └── connect.yaml ``` -Connect deployment configuration is specified in `connect.yaml` which is required information needed for certification of the application. Following is the deployment configuration used by full ingestion and incremental updater modules +Connect deployment configuration is specified in `connect.yaml` which is required information needed for publishing of the application. Following is the deployment configuration used by full ingestion and incremental updater modules ``` deployAs: - name: full-ingestion