Skip to content

Commit

Permalink
docs/design (#13)
Browse files Browse the repository at this point in the history 
Docs Updating!
  • Loading branch information
@Wh1isper
Wh1isper authored Aug 23, 2023
1 parent ca6c02c commit 5c691f1
Show file tree
Hide file tree
Showing 11 changed files with 462 additions and 24 deletions.
10 changes: 6 additions & 4 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# Code Style and Lint
# Development Guide

## Code Style and Lint

We use [black](https://github.com/psf/black) as the code formatter, the best way to use it is to install the pre-commit hook, it will automatically format the code before each commit

Expand All @@ -15,13 +17,13 @@ Pre-commit will automatically format the code before each commit, It can also be
pre-commit run --all-files
```

# Install Locally with Test Dependencies
## Install Locally with Test Dependencies

```bash
pip install -e .[test]
```

# Unittest
## Unittest

We use pytest to write unit tests, and use pytest-cov to generate coverage reports

Expand All @@ -32,7 +34,7 @@ pytest --cov=duetector # Generate coverage reports

Run unit-test before PR, **ensure that new features are covered by unit tests**

# Typing
## Typing

(Optional, python<=3.10) Use [pytype](https://github.com/google/pytype) to check typed

Expand Down
74 changes: 62 additions & 12 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,15 @@

## 简介

> duetector是DataUCON项目中的组件之一，DataUCON项目旨在为数据使用控制提供支持。[了解DataUCON](https://dataucon.idslab.io/)。
> duetector是DataUCON项目中的组件之一，DataUCON项目旨在为数据使用控制提供支持。
>
> [查看DataUCON的网页](https://dataucon.idslab.io/)
>
> [深入了解并部署DataUCON](https://github.com/hitsz-ids/dataucon)
duetector🔍是一个基于eBPF的数据使用探测器，它可以在Linux内核中对数据使用行为进行探测，从而为数据使用控制提供支持。

<!-- 这里需要补充ABAUC相关文档，然后替换链接 -->
**🐛🐞🧪 项目正在大力开发中，期待任何Bug报告、功能请求、合并请求**

在[ABAUC控制模型](https://github.com/hitsz-ids/dataucon)当中，duetector可作为PIP（Policy Information Point）来获取数据使用行为，从而为PDP（Policy Decision Point）提供数据使用行为的信息。

Expand All @@ -28,16 +32,30 @@ duetector🔍是一个基于eBPF的数据使用探测器，它可以在Linux内
- [主要特性](#主要特性)
- [安装](#安装)
- [快速开始](#快速开始)
- [API文档](#API文档)
- [API文档与配置文档](#API文档与配置文档)
- [维护者](#维护者)
- [如何贡献](#如何贡献)
- [许可证](#许可证)

## 主要特性

TBD

TODO: 特性和[对应的内核配置](https://github.com/iovisor/bcc/blob/master/docs/kernel_config.md)
- [X] 插件化系统
- [X] 支持自定义tracer
- [X] 支持自定义filter
- [X] 支持自定义collector
- [X] [自定义插件示例](./examples/)
- [ ] 配置管理
- [X] 使用单一配置文件配置
- [X] 支持生成插件配置
- [ ] 支持动态加载配置
- [ ] 基于eBPF的数据使用探测器
- [X] 文件打开操作
- [ ] ……
- [X] 支持SQL数据库的数据收集器
- [X] CLI工具
- [ ] PIP服务

eBPF探测器需要内核支持，详见[内核支持](./docs/kernel_config.md)

## 安装

Expand All @@ -55,21 +73,51 @@ pip install duetector
docker pull dataucon/duetector:latest
```

预发布版本将不会更新到 `latest`上，您可以指定tag进行拉取，如 `v0.1.0`
预发布版本将不会更新到 `latest`上，您可以指定tag进行拉取，如 `v0.0.1a`

```bash
docker pull dataucon/duetector:v0.1.0
docker pull dataucon/duetector:v0.0.1a
```

使用docker镜像运行的更多细节请参考[这里](./docs/how-to/run-with-docker.md)

## 快速开始

TBD
使用命令行启动monitor，由于bcc需要root权限，所以我们使用 `sudo` 命令，这将启动所有的探测器，并将探测内容收集到当前目录下的 `duetector-dbcollector.sqlite3`文件中

```bash
sudo duectl start
```

按下 `CRTL+C`可以退出监测，你将看到屏幕上输出了一段总结

```
{'DBCollector': {'OpenTracer': {'count': 31, 'first at': 249920233249912, 'last': Tracking(tracer='OpenTracer', pid=641616, uid=1000, gid=1000, comm='node', cwd=None, fname='SOME-FILE', timestamp=249923762308577, extended={})}}}
```

启动时，配置文件将自动生成，对应路径为 `~/.config/duetector` ，可以使用 `--config`指定使用的配置文件

```bash
sudo duectl start --config <config-file-path>
```

当使用插件时，默认的配置文件不会包含插件的配置内容，使用动态生成配置的指令生成带有插件配置的配置文件，这个指令也支持合并当前已有的配置文件和环境变量

```bash
duectl generate-dynamic-config --help
```

当配置文件出错时，可以使用 `generate-config` 恢复默认状态

```bash
duectl generate-config
```

更多文档和例子可以在[这里](./docs/)找到。

## API文档
## API文档与配置文档

TBD
WIP 这一部分内容是PIP相关的，目前还没有完成，完成后将包括可配置的类的内容，以及如何使用duetector作为PIP的内容。

## 维护者

Expand All @@ -79,7 +127,9 @@ TBD

非常欢迎你的加入！[提一个 Issue](https://github.com/hitsz-ids/duetector/issues/new) 或者提交一个 Pull Request。

开发环境配置和其他注意事项请参考[开发者文档](./DEVELOP.md)。
开发环境配置和其他注意事项请参考[开发者文档](./CONTRIBUTING.md)。

在这里了解本项目的设计思路和架构：[设计文档](./docs/design/README.md)

## 许可证

Expand Down
66 changes: 58 additions & 8 deletions README_en.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
duetector🔍 is an eBPF-based data usage control probe that provides support for data usage control by probing for data usage behavior in the Linux kernel.

**🐛🐞🧪 The project is under heavy development, looking forward to any bug reports, feature requests, pull requests!**

In the [ABAUC control model](https://github.com/hitsz-ids/dataucon), duetector can be used as a PIP (Policy Information Point) to obtain data usage behavior, so as to provide information about data usage behavior for PDP (Policy Decision Point). Provide information on data usage behavior to PDP (Policy Decision Point).

## Table of Contents
Expand All @@ -31,9 +33,23 @@ In the [ABAUC control model](https://github.com/hitsz-ids/dataucon), duetector c

## Feature

TBD

TODO: Features and corresponding [kernel config](https://github.com/iovisor/bcc/blob/master/docs/kernel_config.md)
- [X] Plug-in system
- [X] Customized tracer support
- [X] Support for custom filters
- [X] Custom collector support
- [X] [Custom Plugin Examples](./examples/)
- [ ] Configuration Management
- [X] Configuration using a single configuration file
- [X] Generate Plugin Configuration
- [ ] Support for dynamically loading configurations
- [ ] eBPF-based data usage probes
- [X] File Open Operation
- [ ] ......
- [X] Data collector with SQL database support
- [X] CLI Tools
- [ ] PIP Service

The eBPF probe requires kernel support, see [Kernel Support](./docs/kernel_config.md)

## Installation

Expand All @@ -51,19 +67,51 @@ Or use the Docker image that we provide
docker pull dataucon/duetector:latest
```

Pre-releases will not be updated on `latest`, you can specify the tag to pull, e.g. `v0.1.0`
Pre-releases will not be updated to `latest`, you can specify the tag to pull, e.g. `v0.0.1a`

```bash
docker pull dataucon/duetector:v0.1.0
docker pull dataucon/duetector:v0.0.1a
```

For more details on running with docker images see [here](./docs/how-to/run-with-docker.md)

## Quick start

TBD
Start monitor using the command line, since bcc requires root privileges, we use the `sudo` command, which will start all probes and collect the probes into the `duetector-dbcollector.sqlite3` file in the current directory

```bash
sudo duectl start
```

Press `CRTL+C` to exit monitoring and you will see a summary output on the screen

```
{'DBCollector': {'OpenTracer': {'count': 31, 'first at': 249920233249912, 'last': Tracking(tracer='OpenTracer', pid=641616, uid=1000, gid= 1000, comm='node', cwd=None, fname='SOME-FILE', timestamp=249923762308577, extended={})}}}
```

At startup, the configuration file will be automatically generated at `~/.config/duetector`, and you can specify the configuration file to use with `--config`.

```bash
sudo duectl start --config <config-file-path
```

When using a plugin, the default configuration file will not contain the plugin's configuration, use the dynamically-generated configuration directive to generate a configuration file with the plugin's configuration, this directive also supports merging existing configuration files and environment variables.

```bash
duectl generate-dynamic-config --help
```

Use `generate-config` to restore the default state in case of configuration file errors.

```bash
duectl generate-config
```

More documentation and examples can be found [here](. /docs/).

## API documentation

TBD
WIP

## Maintainers

Expand All @@ -73,7 +121,9 @@ This project is initiated by **Institute of Data Security, Harbin Institute of T

You are very welcome to join! [Raise an Issue](https://github.com/hitsz-ids/duetector/issues/new) or submit a Pull Request.

Please refer to [Developer Documentation](./DEVELOP.md).
Please refer to the [Developer Documentation](./CONTRIBUTING.md).

Learn about the design ideas and architecture of this project here: [DESIGN DOCUMENTS](./docs/design/README.md).

## License

Expand Down
36 changes: 36 additions & 0 deletions docs/design/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Architecture

![img](./image/architecture.png)

Key Components and Features:

- [ ] **HTTP / RPC Server**: PIP Server, providing API for PDP to get data usage information.
- [ ] **Analyzer**: Analyze data usage information and generate data usage behavior.
- [x] **CLI**: CLI for administrator to manage duetector.
- [x] **BccMonitor**: Monitor data usage behavior in kernel space. Use BCC to implement.
- [ ] **PollingMonitor**: A general monitor for custom command. Polling the output of command.
- [x] **TracerManager**: Manage tracers, support plugin.
- [x] **OpenTracer**: A `bcc` tracer, trace `open` syscall.
- [ ] ...
- [x] **FilterManager**: Manage filters, support plugin.
- [x] **DefaultFilter**: Filtering some meaningless information
- [ ] ...
- [x] **CollectorManager**: Manage collectors, support plugin.
- [x] **DBCollector**: Collect filted trackings and store them into database.
- [ ] ...

# Data Flow

![img](./image/dataflow.png)

Current data flow implementation:

1. Register **Tracer's** `callback` to host, and start **Monitor**.
2. Once **Monitor's** `poll` is called, it will trigger **Tracer's** `callback`
3. **Tracer's** `callback` will call **Filter** to filter the data.
4. **Filter's** `filter` will call **Collector's** `emit` to collect the data.

The following are not yet realized and may be subject to change.

- [ ] **Collector** will get data from **Collector**.
- [ ] **HTTP / RPC Server** will get data from **Collector**.
Binary file modified docs/design/image/architecture.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/design/image/dataflow.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
85 changes: 85 additions & 0 deletions docs/design/src/architecture.drawio
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
<mxfile host="65bd71144e">
<diagram id="71-AI3Yj6Bn5oummquvE" name="第 1 页">
<mxGraphModel dx="658" dy="1562" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
<root>
<mxCell id="0"/>
<mxCell id="1" parent="0"/>
<mxCell id="54" value="" style="edgeStyle=none;html=1;" parent="1" source="52" target="14" edge="1">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="31" value="" style="rounded=0;whiteSpace=wrap;html=1;dashed=1;" parent="1" vertex="1">
<mxGeometry x="285" y="470" width="100" height="310" as="geometry"/>
</mxCell>
<mxCell id="14" value="HTTP / RPC Service" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;dashed=1;" parent="1" vertex="1">
<mxGeometry x="280" y="230" width="420" height="60" as="geometry"/>
</mxCell>
<mxCell id="18" value="BccMonitor" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;" parent="1" vertex="1">
<mxGeometry x="280" y="310" width="170" height="60" as="geometry"/>
</mxCell>
<mxCell id="20" value="TracerManager" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;" parent="1" vertex="1">
<mxGeometry x="280" y="390" width="110" height="60" as="geometry"/>
</mxCell>
<mxCell id="25" value="PollingMonitor" style="rounded=0;whiteSpace=wrap;html=1;dashed=1;fillColor=#fff2cc;strokeColor=#d6b656;" parent="1" vertex="1">
<mxGeometry x="460" y="310" width="170" height="60" as="geometry"/>
</mxCell>
<mxCell id="26" value="FilterManager" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;" parent="1" vertex="1">
<mxGeometry x="400" y="390" width="110" height="60" as="geometry"/>
</mxCell>
<mxCell id="27" value="CollectorManager" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;" parent="1" vertex="1">
<mxGeometry x="520" y="390" width="110" height="60" as="geometry"/>
</mxCell>
<mxCell id="28" value="OpenTracer" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;" parent="1" vertex="1">
<mxGeometry x="290" y="480" width="90" height="60" as="geometry"/>
</mxCell>
<mxCell id="37" value="" style="rounded=0;whiteSpace=wrap;html=1;dashed=1;" parent="1" vertex="1">
<mxGeometry x="405" y="470" width="100" height="310" as="geometry"/>
</mxCell>
<mxCell id="38" value="DefaultFilter" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;" parent="1" vertex="1">
<mxGeometry x="410" y="480" width="90" height="60" as="geometry"/>
</mxCell>
<mxCell id="39" value="GlobFilter" style="rounded=0;whiteSpace=wrap;html=1;dashed=1;fillColor=#e1d5e7;strokeColor=#9673a6;" parent="1" vertex="1">
<mxGeometry x="410" y="560" width="90" height="60" as="geometry"/>
</mxCell>
<mxCell id="40" value="RegFilter" style="rounded=0;whiteSpace=wrap;html=1;dashed=1;fillColor=#e1d5e7;strokeColor=#9673a6;" parent="1" vertex="1">
<mxGeometry x="410" y="640" width="90" height="60" as="geometry"/>
</mxCell>
<mxCell id="41" value="ExecTracer" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;" parent="1" vertex="1">
<mxGeometry x="290" y="560" width="90" height="60" as="geometry"/>
</mxCell>
<mxCell id="43" value="" style="endArrow=none;dashed=1;html=1;dashPattern=1 3;strokeWidth=2;entryX=0.45;entryY=0.613;entryDx=0;entryDy=0;entryPerimeter=0;exitX=0.45;exitY=0.871;exitDx=0;exitDy=0;exitPerimeter=0;" parent="1" source="31" target="31" edge="1">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="380" y="510" as="sourcePoint"/>
<mxPoint x="430" y="460" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="44" value="" style="rounded=0;whiteSpace=wrap;html=1;dashed=1;" parent="1" vertex="1">
<mxGeometry x="524" y="470" width="100" height="310" as="geometry"/>
</mxCell>
<mxCell id="45" value="DBCollector" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;" parent="1" vertex="1">
<mxGeometry x="529" y="480" width="90" height="60" as="geometry"/>
</mxCell>
<mxCell id="48" value="MongoCollector" style="rounded=0;whiteSpace=wrap;html=1;dashed=1;fillColor=#e1d5e7;strokeColor=#9673a6;" parent="1" vertex="1">
<mxGeometry x="529" y="550" width="90" height="60" as="geometry"/>
</mxCell>
<mxCell id="50" value="CLI" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;" parent="1" vertex="1">
<mxGeometry x="140" y="310" width="120" height="60" as="geometry"/>
</mxCell>
<mxCell id="53" value="" style="edgeStyle=none;html=1;" parent="1" source="51" target="50" edge="1">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="51" value="Admin" style="shape=umlActor;verticalLabelPosition=bottom;verticalAlign=top;html=1;outlineConnect=0;" parent="1" vertex="1">
<mxGeometry x="40" y="310" width="30" height="60" as="geometry"/>
</mxCell>
<mxCell id="52" value="PEP" style="shape=umlActor;verticalLabelPosition=bottom;verticalAlign=top;html=1;outlineConnect=0;" parent="1" vertex="1">
<mxGeometry x="475" y="90" width="30" height="60" as="geometry"/>
</mxCell>
<mxCell id="56" value="bcc" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#f5f5f5;strokeColor=#666666;fontColor=#333333;" parent="1" vertex="1">
<mxGeometry x="285" y="340" width="45" height="25" as="geometry"/>
</mxCell>
<mxCell id="58" value="Analyzer" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;dashed=1;" vertex="1" parent="1">
<mxGeometry x="640" y="310" width="60" height="140" as="geometry"/>
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>
Loading

0 comments on commit 5c691f1

Please sign in to comment.