-
Notifications
You must be signed in to change notification settings - Fork 45
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Doc]Upload EventMesh application scenario case (#238)
* update application scenario * update 05-generative-AI.md * fix bulid errors * upload docs of Chinese version
- Loading branch information
Showing
27 changed files
with
573 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
# Data Steam Processing | ||
|
||
* **Application Scenario Name** | ||
|
||
  **Data Steam Processing** | ||
|
||
|
||
* **Application Scenario Description** | ||
|
||
<div align="center"> | ||
<img src="/images/application-scenario/s0_img1.png" width="80%" /> | ||
</div> | ||
|
||
<!-- ![data-steam-processing-detail](../../static/images/application-scenario/s0_img1.png) --> | ||
|
||
  Based on the figure above, EventMesh is between customer data and target data sources, and can support streaming data processing. Through the filter\transformer capabilities of the EventMesh function, the data format and content desired by the customer can be processed and spit out to the target data source to achieve data cleaning, rule filtering, character replacement, data enrichment, data routing and other capabilities. | ||
|
||
* **Application Scenario Implementation Proposal** | ||
|
||
  To be continued... |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
# Data Synchronization | ||
|
||
* **Application Scenario Name** | ||
|
||
  **Data Synchronization** | ||
|
||
|
||
* **Application Scenario Description** | ||
|
||
  Large enterprises typically have central data management platforms that connect multiple business systems.Upstream business systems often need to distribute data to multiple downstream systems, while each of these systems may also perform asynchronous data changes, additions, and deletions. This requires efficient data synchronization.Traditional data synchronization methods typically rely on message queues or request/response models, but both approaches have limitations: the message queue approach makes it difficult to segment data for different business units, leading to data management challenges; the request/response model, on the other hand, puts a significant performance burden on the central data management platform, resulting in high coupling that hampers system upgrades and maintenance. | ||
|
||
  Therefore, EventMesh is introduced to handle user data synchronization based on the Event-Driven Architecture (EDA). The data management center only needs to publish data change information to different event sources. Each business system can then configure event targets and subscribe to the relevant event sources to achieve data synchronization. During the subscription process, business systems can bind transformation rules (for data splitting and desensitization) and filtering rules to decouple the system architecture. | ||
|
||
* **Application Scenario Implementation Proposal** | ||
|
||
  In the data synchronization scenario, Geely Auto implemented an event integration platform based on EventMesh. For example, in Geely's user center system, most internal business systems cache user data locally after integrating with the user center. As these business systems frequently modify user data asynchronously, there is a strong demand for data synchronization. | ||
|
||
<div align="center"> | ||
<img src="/images/application-scenario/s1_img1.png" width="80%" /> | ||
</div> | ||
|
||
  Geely Automobile's event integration platform, built on EventMesh, enables the user center to publish new and changed user data as "events" within an event-driven architecture. Business systems subscribe to event sources to retrieve necessary user data updates. For instance, the finance system may only be interested in payment information updates, while the customer service system cares about contact information changes. | ||
|
||
  To address the unique data needs of different business systems, the platform utilizes EventMesh's filter/transformer capabilities. Event sources are configured with transformation and filtering rules, enabling business systems to subscribe and process user data splitting and desensitization. This approach decouples direct interactions between the business systems and the user center while reducing the performance load on the user center. | ||
|
||
  To ensure high performance and availability, Geely Automobile's event integration platform is deployed using a multi-region, multi-cluster, cross-cloud architecture, catering to data from various business areas. Different business systems can consume events across regions, clusters, and clouds, enabling data access and synchronization while minimizing data synchronization latency. This architecture ensures stable system operation,even under high concurrency conditions. | ||
|
||
* **Reference** | ||
|
||
  ["Apache EventMesh the event grid helps build an event platform" in CommunityOverCode Asia 2024](https://asia.communityovercode.org/) | ||
|
||
  ["How Geely Automobile implements event integration platform based on EventMesh to promote efficient business integration"](https://mp.weixin.qq.com/s/BLHbIq5KyjnhoVlmw9_DBw) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,122 @@ | ||
# The SaaS composite application integration standardization | ||
|
||
* Application scenario name | ||
|
||
  **The SaaS composite application integration standardization** | ||
|
||
|
||
* Application scenario description | ||
|
||
  A SaaS composite application consists of multiple PBCs (Package Business Capabilities). A PBC can be defined as a SaaS application module with clearly defined business capabilities. Each module is business-driven and capable of independently fulfilling business requirements without external dependencies. Enterprise solutions are typically composed of multiple SaaS application modules, integrating various complex functionalities to present a unified and comprehensive user experience. | ||
|
||
  The diagram below illustrates a single PBC, a combination of PBCs, and multiple composite PBCs: | ||
|
||
<div align="center"> | ||
<img src="/images/application-scenario/s2_img1.png" width="80%" /> | ||
</div> | ||
|
||
<!-- ![PBC-structure](../../static/images/application-scenario/s2_img1.png) --> | ||
|
||
|
||
  From the above architecture, it is clear that each SaaS application module (PBC) has a low degree of coupling. Modifying or adjusting a specific module does not affect the operation of other existing modules.This facilitates agile development and efficient iterative updates. However, one challenge of SaaS composite applications is the lack of standardized integration between different applications, as the absence of a unified communication protocol can hinder the implementation of this architecture. | ||
|
||
  This issue can be solved by EventMesh. EventMesh integrates TCP and HTTP protocols, and supports bi-directional asynchronous communication between Client and Server through gRPC (Google’s open-source high-performance RPC framework based on HTTP/2) with SDKs available for multiple languages such as Java, Python, C, and Go. Users do not need to worry about which communication protocol is used in different scenarios when using the SDK; the event-driven asynchronous communication can be achieved through the SDK APIs integrated by EventMesh, enabling the seamless event flow between different SaaS application modules. | ||
|
||
<div align="center"> | ||
<img src="/images/application-scenario/s2_img2.png" width="60%" /> | ||
</div> | ||
|
||
* Application scenario implementation proposal | ||
|
||
  Regarding the specific implementation of the scenario, EventMesh officially introduced the `gRPC` framework starting from version v1.4.0. gRPC defines API interface data models using Protobuf. In the gRPC Protobuf event model, each event is represented by the `SimpleMessage` data model, with the event content stored in the `content` field. | ||
|
||
<div align="center"> | ||
<img src="/images/application-scenario/s2_img3.png" width="60%" /> | ||
</div> | ||
|
||
  The gRPC event scenarios supported by EventMesh include: event sending and batch event sending, event broadcasting, event request and response, event subscription, and event pushing (for more details, see: [eventmesh-client.proto](https://github.com/apache/eventmesh/blob/master/eventmesh-protocol-plugin/eventmesh-protocol-grpc/src/main/proto/eventmesh-client.proto)). | ||
|
||
  1. The event publishing service provides the following interfaces: | ||
|
||
|
||
```java | ||
service PublisherService { | ||
|
||
rpc publish(SimpleMessage) returns (Response); | ||
|
||
rpc requestReply(SimpleMessage) returns (SimpleMessage); | ||
|
||
rpc batchPublish(BatchMessage) returns (Response); | ||
} | ||
``` | ||
|
||
  Events are presented in the SimpleMessage data model. Event publishing supports three modes: `synchronous publishing`, `asynchronous publishing`, and `batch publishing`. | ||
|
||
   `Synchronous publishing` means the event producer sends the event to EventMesh, waits for the event to be successfully delivered to the event consumer, and receives a response from the event consumer. The end-to-end event publishing process is only considered complete once this is done. | ||
|
||
   `Asynchronous publishing` means the event producer sends the event to EventMesh without waiting for the event to be successfully delivered to the event consumer. | ||
|
||
   `Batch publishing` refers to asynchronously sending a batch of events to EventMesh. | ||
|
||
  2. The event subscription service provides the following interfaces: | ||
|
||
```java | ||
service ConsumerService { | ||
|
||
rpc subscribe(Subscription) returns (Response); | ||
|
||
rpc subscribeStream(stream Subscription) returns (stream SimpleMessage); | ||
|
||
rpc unsubscribe(Subscription) returns (Response); | ||
} | ||
``` | ||
|
||
  Event subscription supports two modes: `cluster` and `broadcast`. In cluster mode, only one instance within the event consumer cluster will consume the event. In broadcast mode, every instance in the cluster will consume the event. | ||
|
||
  These subscription modes are defined in the subscription data model. Additionally, the subscription service provides two subscription interfaces: `the subscribe API` and `the subscribeStream API`. `The subscribe API` pushes events to consumers via a URL, also known as a webhook. This scenario is suitable for cloud-native microservices, custom applications, and functions. `The subscribeStream API` pushes events to consumers using gRPC bidirectional streaming, allowing the event consumer to return an acknowledgment (Ack) to the event producer. This supports the producer's need for synchronous event publishing using RequestReply. | ||
|
||
  3. Event subscription service provides the following interfaces: | ||
|
||
  To improve the performance of event production and consumption, the EventMesh server (`EventMesh Runtime`) defines thread pools within the gRPC service. Moreover, independent parameters are configured based on different performance requirements for event production and consumption. These parameters can be found in the EventMesh configuration file (`eventmesh.properties`). | ||
|
||
  For example, the following are the number of threads for event production, subscription, and pushing: | ||
|
||
```java | ||
eventMesh.server.sendmsg.threads.num=50 | ||
eventMesh.server.clientmanage.threads.num=30 | ||
eventMesh.server.pushmsg.threads.num=50 | ||
``` | ||
|
||
  When the gRPC service starts, it begins listening for client requests. Once a new request arrives, it is dispatched to the thread pool of the corresponding service, and the appropriate processor (`Processor`) handles it. This prevents blocking the handling of subsequent requests, thereby improving concurrency. | ||
|
||
```java | ||
public void publish(SimpleMessage request, StreamObserver<Response> responseObserver){ | ||
cmdLogger.info("cmd={}|{}|client2eventMesh|from={}|to={}", "AsyncPublish", | ||
EventMeshConstants.PROTOCOL_GRPC, request.getHeader().getIp(), | ||
eventMeshGrpcServer.getEventMeshGrpcConfiguration().eventMeshIp); | ||
|
||
EventEmitter<Response> emitter = new EventEmitter<>(responseObserver); | ||
|
||
threadPoolExecutor.submit(() -> { | ||
SendAsyncMessageProcessor sendAsyncMessageProcessor = new SendAsyncMessageProcessor(eventMeshGrpcServer); | ||
try { | ||
sendAsyncMessageProcessor.process(request, emitter); | ||
} catch (Exception e) { | ||
logger.error("Error code {}, error message {}", StatusCode.EVENTMESH_SEND_ASYNC_MSG_ERR.getRetCode(), | ||
StatusCode.EVENTMESH_SEND_ASYNC_MSG_ERR.getErrMsg(), e); | ||
ServiceUtils.sendRespAndDone(StatusCode.EVENTMESH_SEND_ASYNC_MSG_ERR, e.getMessage(), emitter); | ||
} | ||
}); | ||
} | ||
``` | ||
|
||
  For instance, the above code is the implementation of the event publishing service. It uses a `threadPoolExecutor` to send events to the thread pool for downstream `SendAsyncMessageProcessor` handling. | ||
|
||
|
||
* Reference | ||
|
||
  ["How Does Apache Eventmesh Solve The Saas Composite Application Integration Standardization Problem" in ApacheCon Asia 2022](https://www.youtube.com/watch?v=0v-tjIYkDIw) | ||
|
||
  [A Blog by Huawei Cloud Middleware Team](https://mp.weixin.qq.com/s/Z3DAw_YtKJWXftdFjAm2rA) | ||
|
||
  [The Code of EventMesh in Github](https://github.com/apache/eventmesh/tree/master/eventmesh-protocol-plugin/eventmesh-protocol-grpc/src/main/proto) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
# Serverless Event Bus for Cloud Services | ||
|
||
* Application scenario name | ||
|
||
  **Serverless Event Bus for Cloud Services** | ||
|
||
|
||
* Application scenario description | ||
|
||
  Cloud service providers can build a Serverless Event Bus using `EventMesh` to facilitate and manage events across various sources, including traditional applications, cloud-native services, and SaaS partner applications. In this scenario, the Serverless Event Bus powered by EventMesh helps developers create loosely coupled and distributed event-driven serverless microservice applications without the need to manage underlying infrastructure. | ||
|
||
  This Serverless Event Bus supports asynchronous event communication across different cloud environments. It simplifies protocol management by routing events between services, making it suitable for multi-cloud, hybrid cloud architectures, and third-party SaaS systems. Additionally, the bus can be deployed in distributed environments, ensuring reliable event delivery even under heavy load, thanks to `EventMesh`’s cross-region failover and load-balancing capabilities. | ||
|
||
|
||
* Application scenario implementation proposal | ||
|
||
  Cloud service provider Huawei Cloud offers a Serverless Event Bus service called `EventGrid`, which is based on `EventMesh`. `EventGrid` uses `EventMesh` as its runtime engine and supports the integration of Huawei Cloud services, custom applications, and SaaS applications in a standardized, centralized manner. By utilizing the standardized CloudEvents protocol, it flexibly routes events between applications, building a loosely coupled, distributed event-driven architecture. This not only creates more application scenarios for Huawei Cloud but also further enriches the developer ecosystem. | ||
|
||
  As a crucial part of the event-driven architecture in a Serverless environment, it provides elastic, asynchronous, and decentralized event governance services. Key functions include event aggregation, schema validation, filtering, routing, transformation, and pushing. Additionally, it offers enhanced features like fault-tolerant retries, dead-letter storage, event query tracking, and event workflows. | ||
|
||
<div align="center"> | ||
<img src="/images/application-scenario/s3_img1_en.png" width="70%"/> | ||
</div> | ||
|
||
  The diagram above represents the overall architecture of Huawei Cloud `EventGrid`. `EventGrid` connects to various cloud services as event sources, including Distributed Message Service, Object Storage Service (OBS), and Distributed Cache Service. The various events generated by these event sources are pushed to the event channels of the `EventGrid` Event Bus (Bus Runtime). The Event Bus configures event channels for EventGrid users on a per-tenant basis, allowing multiple event channels under a single tenant to carry events from different sources. | ||
|
||
  `EventGrid`, as the standard event hub for Huawei Cloud, enables interconnection between various cloud services. Cloud services, acting as event sources or event targets, are deployed on Huawei Cloud's Serverless application platform. Applications push real-time business events to the event grid, where the event grid filters, routes, and transforms the events to trigger cloud services subscribed to those events. `EventGrid` currently supports over 100+ built-in Huawei Cloud event sources and allows for custom and partner event extensions. Additionally, it is supported by a vast array of official data sources, covering databases, messaging, serverless computing, big data, DevOps platforms, IoT, and more, with custom event integration capabilities. | ||
|
||
* Reference | ||
|
||
  ["How Does Apache Eventmesh Solve The Saas Composite Application Integration Standardization Problem"in ApacheCon Asia 2022](https://www.youtube.com/watch?v=0v-tjIYkDIw) | ||
|
||
  [A Blog by Huawei Cloud Middleware Team](https://mp.weixin.qq.com/s/Z3DAw_YtKJWXftdFjAm2rA) | ||
|
||
  [The Instruction Document of EventGrid by Huawei Cloud](https://www.huaweicloud.com/product/eventgrid.html) | ||
|
||
|
Oops, something went wrong.