diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d73a45b..3fe03a6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,4 +1,6 @@ -# Code Style and Lint +# Development Guide + +## Code Style and Lint We use [black](https://github.com/psf/black) as the code formatter, the best way to use it is to install the pre-commit hook, it will automatically format the code before each commit @@ -15,13 +17,13 @@ Pre-commit will automatically format the code before each commit, It can also be pre-commit run --all-files ``` -# Install Locally with Test Dependencies +## Install Locally with Test Dependencies ```bash pip install -e .[test] ``` -# Unittest +## Unittest We use pytest to write unit tests, and use pytest-cov to generate coverage reports @@ -32,7 +34,7 @@ pytest --cov=duetector # Generate coverage reports Run unit-test before PR, **ensure that new features are covered by unit tests** -# Typing +## Typing (Optional, python<=3.10) Use [pytype](https://github.com/google/pytype) to check typed diff --git a/README.md b/README.md index 851f4a1..4900a3f 100644 --- a/README.md +++ b/README.md @@ -15,11 +15,15 @@ ## 简介 -> duetector是DataUCON项目中的组件之一，DataUCON项目旨在为数据使用控制提供支持。[了解DataUCON](https://dataucon.idslab.io/)。 +> duetector是DataUCON项目中的组件之一，DataUCON项目旨在为数据使用控制提供支持。 +> +> [查看DataUCON的网页](https://dataucon.idslab.io/) +> +> [深入了解并部署DataUCON](https://github.com/hitsz-ids/dataucon) duetector🔍是一个基于eBPF的数据使用探测器，它可以在Linux内核中对数据使用行为进行探测，从而为数据使用控制提供支持。 - +**🐛🐞🧪 项目正在大力开发中，期待任何Bug报告、功能请求、合并请求** 在[ABAUC控制模型](https://github.com/hitsz-ids/dataucon)当中，duetector可作为PIP（Policy Information Point）来获取数据使用行为，从而为PDP（Policy Decision Point）提供数据使用行为的信息。 @@ -28,16 +32,30 @@ duetector🔍是一个基于eBPF的数据使用探测器，它可以在Linux内 - [主要特性](#主要特性) - [安装](#安装) - [快速开始](#快速开始) -- [API文档](#API文档) +- [API文档与配置文档](#API文档与配置文档) - [维护者](#维护者) - [如何贡献](#如何贡献) - [许可证](#许可证) ## 主要特性 -TBD - -TODO: 特性和[对应的内核配置](https://github.com/iovisor/bcc/blob/master/docs/kernel_config.md) +- [X] 插件化系统 + - [X] 支持自定义tracer + - [X] 支持自定义filter + - [X] 支持自定义collector + - [X] [自定义插件示例](./examples/) +- [ ] 配置管理 + - [X] 使用单一配置文件配置 + - [X] 支持生成插件配置 + - [ ] 支持动态加载配置 +- [ ] 基于eBPF的数据使用探测器 + - [X] 文件打开操作 + - [ ] …… +- [X] 支持SQL数据库的数据收集器 +- [X] CLI工具 +- [ ] PIP服务 + +eBPF探测器需要内核支持，详见[内核支持](./docs/kernel_config.md) ## 安装 @@ -55,21 +73,51 @@ pip install duetector docker pull dataucon/duetector:latest ``` -预发布版本将不会更新到 `latest`上，您可以指定tag进行拉取，如 `v0.1.0` +预发布版本将不会更新到 `latest`上，您可以指定tag进行拉取，如 `v0.0.1a` ```bash -docker pull dataucon/duetector:v0.1.0 +docker pull dataucon/duetector:v0.0.1a ``` +使用docker镜像运行的更多细节请参考[这里](./docs/how-to/run-with-docker.md) + ## 快速开始 -TBD +使用命令行启动monitor，由于bcc需要root权限，所以我们使用 `sudo` 命令，这将启动所有的探测器，并将探测内容收集到当前目录下的 `duetector-dbcollector.sqlite3`文件中 + +```bash +sudo duectl start +``` + +按下 `CRTL+C`可以退出监测，你将看到屏幕上输出了一段总结 + +``` +{'DBCollector': {'OpenTracer': {'count': 31, 'first at': 249920233249912, 'last': Tracking(tracer='OpenTracer', pid=641616, uid=1000, gid=1000, comm='node', cwd=None, fname='SOME-FILE', timestamp=249923762308577, extended={})}}} +``` + +启动时，配置文件将自动生成，对应路径为 `~/.config/duetector` ，可以使用 `--config`指定使用的配置文件 + +```bash +sudo duectl start --config +``` + +当使用插件时，默认的配置文件不会包含插件的配置内容，使用动态生成配置的指令生成带有插件配置的配置文件，这个指令也支持合并当前已有的配置文件和环境变量 + +```bash +duectl generate-dynamic-config --help +``` + +当配置文件出错时，可以使用 `generate-config` 恢复默认状态 + +```bash +duectl generate-config +``` 更多文档和例子可以在[这里](./docs/)找到。 -## API文档 +## API文档与配置文档 -TBD +WIP 这一部分内容是PIP相关的，目前还没有完成，完成后将包括可配置的类的内容，以及如何使用duetector作为PIP的内容。 ## 维护者 @@ -79,7 +127,9 @@ TBD 非常欢迎你的加入！[提一个 Issue](https://github.com/hitsz-ids/duetector/issues/new) 或者提交一个 Pull Request。 -开发环境配置和其他注意事项请参考[开发者文档](./DEVELOP.md)。 +开发环境配置和其他注意事项请参考[开发者文档](./CONTRIBUTING.md)。 + +在这里了解本项目的设计思路和架构：[设计文档](./docs/design/README.md) ## 许可证 diff --git a/README_en.md b/README_en.md index a1d464b..141844f 100644 --- a/README_en.md +++ b/README_en.md @@ -17,6 +17,8 @@ duetector🔍 is an eBPF-based data usage control probe that provides support for data usage control by probing for data usage behavior in the Linux kernel. +**🐛🐞🧪 The project is under heavy development, looking forward to any bug reports, feature requests, pull requests!** + In the [ABAUC control model](https://github.com/hitsz-ids/dataucon), duetector can be used as a PIP (Policy Information Point) to obtain data usage behavior, so as to provide information about data usage behavior for PDP (Policy Decision Point). Provide information on data usage behavior to PDP (Policy Decision Point). ## Table of Contents @@ -31,9 +33,23 @@ In the [ABAUC control model](https://github.com/hitsz-ids/dataucon), duetector c ## Feature -TBD - -TODO: Features and corresponding [kernel config](https://github.com/iovisor/bcc/blob/master/docs/kernel_config.md) +- [X] Plug-in system + - [X] Customized tracer support + - [X] Support for custom filters + - [X] Custom collector support + - [X] [Custom Plugin Examples](./examples/) +- [ ] Configuration Management + - [X] Configuration using a single configuration file + - [X] Generate Plugin Configuration + - [ ] Support for dynamically loading configurations +- [ ] eBPF-based data usage probes + - [X] File Open Operation + - [ ] ...... +- [X] Data collector with SQL database support +- [X] CLI Tools +- [ ] PIP Service + +The eBPF probe requires kernel support, see [Kernel Support](./docs/kernel_config.md) ## Installation @@ -51,19 +67,51 @@ Or use the Docker image that we provide docker pull dataucon/duetector:latest ``` -Pre-releases will not be updated on `latest`, you can specify the tag to pull, e.g. `v0.1.0` +Pre-releases will not be updated to `latest`, you can specify the tag to pull, e.g. `v0.0.1a` ```bash -docker pull dataucon/duetector:v0.1.0 +docker pull dataucon/duetector:v0.0.1a ``` +For more details on running with docker images see [here](./docs/how-to/run-with-docker.md) + ## Quick start -TBD +Start monitor using the command line, since bcc requires root privileges, we use the `sudo` command, which will start all probes and collect the probes into the `duetector-dbcollector.sqlite3` file in the current directory + +```bash +sudo duectl start +``` + +Press `CRTL+C` to exit monitoring and you will see a summary output on the screen + +``` +{'DBCollector': {'OpenTracer': {'count': 31, 'first at': 249920233249912, 'last': Tracking(tracer='OpenTracer', pid=641616, uid=1000, gid= 1000, comm='node', cwd=None, fname='SOME-FILE', timestamp=249923762308577, extended={})}}} +``` + +At startup, the configuration file will be automatically generated at `~/.config/duetector`, and you can specify the configuration file to use with `--config`. + +```bash +sudo duectl start --config + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/design/src/dataflow.drawio b/docs/design/src/dataflow.drawio new file mode 100644 index 0000000..5ac4266 --- /dev/null +++ b/docs/design/src/dataflow.drawio @@ -0,0 +1,181 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/how-to/README.md b/docs/how-to/README.md deleted file mode 100644 index e69de29..0000000 diff --git a/docs/how-to/run-with-docker.md b/docs/how-to/run-with-docker.md new file mode 100644 index 0000000..0b8e9a5 --- /dev/null +++ b/docs/how-to/run-with-docker.md @@ -0,0 +1,31 @@ +# Run with docker + +> https://hub.docker.com/r/dataucon/duetector/ + +BCC relies on kernel headers, either by turning on the kernel compilation parameter `CONFIG_IKHEADERS=m` or by installing the `kernel-development-package` provided by the distribution. + +Docker (`runC`) 's containers are using the same kernel as the host machine, so it is not possible to have kernel headers built-in in Docker images, since the host machines may not be the same. So The host need to meet one of the above conditions. + +There are two options: + +1. check to see if there is a `/sys/kernel/kheaders.tar.xz` file, and if there is, mount it directly into the same location as the container + +```Bash +docker run -it --rm --privileged \ +-v /sys/kernel/kheaders.tar.xz:/sys/kernel/kheaders.tar.xz \ +-v /sys/kernel/debug:/sys/kernel/debug \ +dataucon/duetector +``` + +2. If there is no such file, you need to install headers and mount `/usr/lib/modules` into the container, `/usr/lib/modules` is the directory where kernel modules are installed, and `/lib` -> `/usr/lib` is the directory where kernel modules are installed, and usually `/lib` -> `/usr/lib` is the directory where kernel modules are installed. The steps are as follows: + 1. In WSL2, [you need to compile and install headers by yourself](https://github.com/iovisor/bcc/blob/master/INSTALL.md#install-packages), or [directly replace your own compiled kernel](https:// zhuanlan.zhihu.com/p/324530180) + 2. Debian/Ubuntu `sudo apt-get install linux-headers-$(uname -r)` + 3. arch `sudo pacman -S linux-headers` + 4. For other distributions, see https://github.com/iovisor/bcc/blob/master/INSTALL.md + +```Bash +docker run -it --rm --privileged \ +-v /usr/lib/modules:/usr/lib/modules \ +-v /sys/kernel/debug:/sys/kernel/debug \ +dataucon/duetector +``` diff --git a/docs/kernel_config.md b/docs/kernel_config.md new file mode 100644 index 0000000..2e1409e --- /dev/null +++ b/docs/kernel_config.md @@ -0,0 +1,3 @@ +Currently we use kprobes, uprobes, Tracepoints, Network releatd BPF features. Please refre to [Kernel Configuration for BPF Features](https://github.com/iovisor/bcc/blob/master/docs/kernel_config.md). + +If there is any trouble, [raise an Issue](https://github.com/hitsz-ids/duetector/issues/new).