Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

全文检索数据同步改造 #7561

Draft
wants to merge 1 commit into
base: v3.12.x
Choose a base branch
from
+6,252 −2,507
Draft

全文检索数据同步改造 #7561

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .code.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ source:
# 此处备注的第三方代码在后续统计代码量环节会被排除，若代码库中不存在需要排除的第三方代码，该项配置标识可为空
third_party_source:
#第三方代码文件的正则表达式，若无统一标识格式，可以指定具体目录，样例可参考test_source举例
filepath_regex: ["/vendor/.*", "/src/tools/monstache/vendor/.*"]
filepath_regex: ["/vendor/.*"]
68 changes: 29 additions & 39 deletions docs/overview/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
* Redis >= 3.2.11
* MongoDB >= 4.2
* Elasticsearch >= 7.0.0 (用于全文检索功能)
* Monstache >= 6.0.0 (用于全文检索功能)

## CMDB 微服务进程清单

Expand All @@ -30,9 +29,11 @@
* cmdb_operationserver
* cmdb_synchronizeserver
* cmdb_taskserver
* cmdb_syncserver

### 4. 资源管理服务进程

* cmdb_cacheservice
* cmdb_coreservice

---
Expand Down Expand Up @@ -182,13 +183,6 @@ MongoDB官方资料：
> db.createUser({user: "cc",pwd: "cc",roles: [ { role: "readWrite", db: "cmdb" } ]})
```

- 开启ES情况(用于全文检索, 可选, 控制开关见第9步的full_text_search)

``` json
> use cmdb
> db.createUser({user: "cc",pwd: "cc",roles: [ { role: "readWrite", db: "cmdb" },{ role: "readWrite", db: "monstache" } ]})
```

**注：以上用户名、密码、数据库名仅作示例展示，用户使用中可以根据实际情况自行配置。如果安装的MongoDB的版本大于等于3.6，需要手动修改init.py自动生成的配置文件，详细步骤参看init.py相关小节。**

详细手册请参考官方资料 [MongoDB](https://docs.mongodb.com/manual/reference/method/db.createUser/)
Expand All @@ -207,13 +201,7 @@ MongoDB官方资料：

如果想部署高可用可扩展的ES，可参考官方文档[ES-Guide](https://www.elastic.co/guide/index.html)

### 7. 部署Monstache (用于全文检索, 可选, 控制开关见第9步的full_text_search)

蓝鲸CMDB针对需求场景采用定制化的Monstache组件，组件以及其插件SO请从指定的Release Package中获取。

插件基于Monstache v6.0.0+, 需要依赖Elasticsearch v7+和MongoDB v4.2+。

阅读[蓝鲸CMDB全文检索插件文档](../../src/tools/monstache/README.md), 按照指引进行安装部署。
CMDB提供场景层服务`cmdb_syncserver`用于将MongoDB中的数据同步到Elasticsearch中。阅读[同步服务文档](../../src/scene_server/sync_server/readme.md)，按照指引进行全文检索同步配置。

### 8. 部署CMDB

Expand All @@ -230,6 +218,7 @@ drwxr-xr-x 4 root root 4096 Jun 18 15:24 cmdb_eventserver
drwxr-xr-x 5 root root 4096 Jun 18 15:24 cmdb_hostserver
drwxr-xr-x 4 root root 4096 Jun 18 15:24 cmdb_operationserver
drwxr-xr-x 5 root root 4096 Jun 18 15:24 cmdb_procserver
drwxr-xr-x 3 root root 4096 Jun 18 10:33 cmdb_syncserver
drwxr-xr-x 3 root root 4096 Jun 18 10:33 cmdb_synchronizeserver
drwxr-xr-x 5 root root 4096 Jun 18 15:24 cmdb_taskserver
drwxr-xr-x 4 root root 4096 Jun 18 15:24 cmdb_toposerver
Expand All @@ -249,31 +238,32 @@ drwxr-xr-x 7 root root 4096 Jun 18 10:33 web

各目录代表的服务及职责：

| 目标 | 类型 | 用途描述 |
| ---------------------- | ---------- | ------------------------------------------------------------ |
| cmdb_adminserver | server | 负责系统数据的初始化以及配置管理工作 |
| cmdb_apiserver | server | 场景层服务，api 服务 |
| cmdb_coreservice | server | 资源管理层，提供原子接口服务 |
| cmdb_datacollection | server | 场景层服务，数据采集服务 |
| cmdb_eventserver | server | 场景层服务，事件推送服务 |
| cmdb_hostserver | server | 场景层服务，主机数据维护 |
| cmdb_operationserver | server | 场景层服务，提供与运营统计相关功能服务 |
| cmdb_procserver | server | 场景层服务，负责进程数据的维护 |
| cmdb_synchronizeserver | server | 场景层服务，数据同步服务 |
| cmdb_taskserver | server | 场景层服务，异步任务管理服务 |
| cmdb_toposerver | server | 场景层服务，负责模型的定义以及主机、业务、模块及进程等实例数据的维护 |
| cmdb_webserver | server | web server 服务子目录 |
| docker | Dockerfile | 各服务的Dockerfile模板 |
| image.sh | script | 用于制作Docker镜像 |
| 目标 | 类型 | 用途描述 |
|------------------------|------------|---------------------------------------------|
| cmdb_adminserver | server | 负责系统数据的初始化以及配置管理工作 |
| cmdb_apiserver | server | 场景层服务，api 服务 |
| cmdb_coreservice | server | 资源管理层，提供原子接口服务 |
| cmdb_datacollection | server | 场景层服务，数据采集服务 |
| cmdb_eventserver | server | 场景层服务，事件推送服务 |
| cmdb_hostserver | server | 场景层服务，主机数据维护 |
| cmdb_operationserver | server | 场景层服务，提供与运营统计相关功能服务 |
| cmdb_procserver | server | 场景层服务，负责进程数据的维护 |
| cmdb_syncserver | server | 场景层服务，负责cmdb与第三方组件的数据同步 |
| cmdb_synchronizeserver | server | 场景层服务，多个cmdb之间的数据同步服务 |
| cmdb_taskserver | server | 场景层服务，异步任务管理服务 |
| cmdb_toposerver | server | 场景层服务，负责模型的定义以及主机、业务、模块及进程等实例数据的维护 |
| cmdb_webserver | server | web server 服务子目录 |
| docker | Dockerfile | 各服务的Dockerfile模板 |
| image.sh | script | 用于制作Docker镜像 |
| init.py | script | 用于初始化服务及配置项，在需要重置服务配置的时候也可以运行此脚本，按照提示输入配置参数 |
| init_db.sh | script | 初始化数据库的数据 |
| ip.py | script | 查询主机真实的IP脚本 |
| restart.sh | script | 用于重启所有服务 |
| start.sh | script | 用于启动所有服务 |
| stop.sh | script | 用于停止所有服务 |
| tool_ctl | ctl | 管理小工具 |
| upgrade.sh | script | 用于全量升级服务进程 |
| web | ui | CMDB UI 页面 |
| init_db.sh | script | 初始化数据库的数据 |
| ip.py | script | 查询主机真实的IP脚本 |
| restart.sh | script | 用于重启所有服务 |
| start.sh | script | 用于启动所有服务 |
| stop.sh | script | 用于停止所有服务 |
| tool_ctl | ctl | 管理小工具 |
| upgrade.sh | script | 用于全量升级服务进程 |
| web | ui | CMDB UI 页面 |

### 9. 初始化

Expand Down
Binary file added docs/resource/img/sync-server/simple_sync.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 0 additions & 4 deletions docs/support-file/dockerfile/monstache/dockerfile
View file
Open in desktop

This file was deleted.

9 changes: 9 additions & 0 deletions docs/support-file/dockerfile/syncserver/dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
FROM ccr.ccs.tencentyun.com/bk.io/centos7-cmdb:base
ENV container docker
COPY cmdb_syncserver /data/cmdb/cmdb_syncserver
RUN mkdir /data/cmdb/cmdb_syncserver/logs
RUN chmod +x /data/cmdb/cmdb_syncserver/cmdb_syncserver
#time zone setting
ENV TimeZone=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TimeZone /etc/localtime && echo $TimeZone > /etc/timezone

Loading