Skip to content

Commit

Permalink
Merge pull request #229 from huawei-noah/zjj_1.8.2_release
Browse files Browse the repository at this point in the history
release 1.8.2
  • Loading branch information
zhangjiajin authored Mar 31, 2022
2 parents 140b2c3 + d7b7241 commit f02bd90
Show file tree
Hide file tree
Showing 57 changed files with 210 additions and 1,430 deletions.
12 changes: 6 additions & 6 deletions README.cn.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@

---

**Vega ver1.8.0 发布**
**Vega ver1.8.2 发布**

- 特性增强
- 错误修正

- 安全增强,组件间通信支持安全协议
- 提供独立的评估服务安装
- 更新Auto-lane模型,提供auto-lane推理代码
- 修正文档中链接错误
- 评估服务支持多输入
- 修正在NPU下使用Apex的错误

---

Expand Down Expand Up @@ -84,7 +84,7 @@ vega ./examples/nas/cars/cars.yml -s

| 对象 | 参考 |
| :--: | :-- |
| **用户** | [安装指导](./docs/cn/user/install.md)[部署指导](./docs/cn/user/deployment.md)[安全配置](./docs/cn/user/security_configure.md)[配置指导](./docs/cn/user/config_reference.md)[示例参考](./docs/cn/user/examples.md)[评估服务](./evaluate_service/docs/cn/evaluate_service.md) |
| **用户** | [安装指导](./docs/cn/user/install.md)[部署指导](./docs/cn/user/deployment.md)[安全配置](./docs/cn/user/security_configure.md)[配置指导](./docs/cn/user/config_reference.md)[示例参考](./docs/cn/user/examples.md)[评估服务](./evaluate_service/README.cn.md) |
| **开发者** | [开发者指导](./docs/cn/developer/developer_guide.md)[快速入门指导](./docs/cn/developer/quick_start.md)[数据集指导](./docs/cn/developer/datasets.md)[算法开发指导](./docs/cn/developer/new_algorithm.md) |

## FAQ
Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@

---

**Vega ver1.8.0 released**
**Vega ver1.8.2 released**

- Feature enhancement:
- Bug Fixed:

- Security enhancement: Security protocols communication.
- Provide evaluation service release package.
- Update the auto-lane model and provide auto-lane inference sample code.
- Fixed bad document links.
- The model to be evaluated supports multiple imputs.
- Fixed using Apex on the NPU.

---

Expand Down Expand Up @@ -85,7 +85,7 @@ vega ./examples/nas/cars/cars.yml -s

| Reader | Refrence |
| :--: | :-- |
| **User** | [Install Guide](./docs/en/user/install.md), [Deployment Guide](./docs/en/user/deployment.md), [Configuration Guide](./docs/en/user/config_reference.md), [Security Configuration](./docs/en/user/security_configure.md), [Examples](./docs/en/user/examples.md), [Evaluate Service](./evaluate_service/docs/en/evaluate_service.md) |
| **User** | [Install Guide](./docs/en/user/install.md), [Deployment Guide](./docs/en/user/deployment.md), [Configuration Guide](./docs/en/user/config_reference.md), [Security Configuration](./docs/en/user/security_configure.md), [Examples](./docs/en/user/examples.md), [Evaluate Service](./evaluate_service/README.md) |
| **Developer** | [Development Reference](./docs/en/developer/developer_guide.md), [Quick Start Guide](./docs/en/developer/quick_start.md), [Dataset Guide](./docs/en/developer/datasets.md), [Algorithm Development Guide](./docs/en/developer/new_algorithm.md) |

## FAQ
Expand Down
2 changes: 1 addition & 1 deletion RELEASE.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
**Vega ver1.8.0 released:**
**Vega ver1.8.2 released:**

**Introduction**

Expand Down
5 changes: 2 additions & 3 deletions docs/cn/algorithms/nago.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,15 +66,14 @@ search_space:
### 4.2 搜索策略设置

NAGO的搜索空间适用于任何贝叶斯优化算法。在VEGA中我们采用了BOHB, 所以我们需要在 `nago.yml` 文件中也设置BOHB算法的基本参数。
例如,下面的设置会跑50个搜索循环的BOHB,并用最少30 epochs和最多120 epochs来训练和评估所生成的神经网络结构。

```yaml
search_algorithm:
type: BohbHpo
policy:
total_epochs: -1
repeat_times: 50
num_samples: 350
repeat_times: 1
num_samples: 7
max_epochs: 120
min_epochs: 30
eta: 2
Expand Down
1 change: 0 additions & 1 deletion docs/cn/developer/quick_start.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,6 @@ nas:
lr_scheduler:
type: MultiStepLR
params:
warmup: False
milestones: [30]
gamma: 0.5
loss:
Expand Down
45 changes: 24 additions & 21 deletions docs/cn/user/security_configure.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,10 @@ openssl genrsa -out ca.key 4096
openssl req -new -x509 -key ca.key -out ca.crt -subj "/C=<country>/ST=<province>/L=<city>/O=<organization>/OU=<group>/CN=<cn>"
```

注意:以上`<country>``<province>``<city>``<organization>``<group>``<cn>`根据实际情况填写,本文后面的配置也是同样的。并且CA的配置需要和其他的不同。
注意:

1. 以上`<country>``<province>``<city>``<organization>``<group>``<cn>`根据实际情况填写,去掉符号`<>`,本文后面的配置也是同样的。并且CA的配置需要和其他的不同。
2. RSA密钥长度建议在3072位及以上,如本例中使用4096长度。

## 3. 生成评估服务使用的证书

Expand Down Expand Up @@ -139,12 +142,12 @@ vega-encrypt_key --cert=client.crt --key=client.key --key_component_1=ksmaster_c
```shell
mkdir ~/.vega
mv * ~/.vega/
chmod -R 600 ~/.vega
chmod 600 ~/.vega/*
```

说明:

1. 如上的秘钥、证书、加密材料也可以放到其他目录位置,注意访问权限要设置为`600`,并在后继的配置文件中同步修改该文件的位置。
1. 如上的秘钥、证书、加密材料也可以放到其他目录位置,注意访问权限要设置为`600`,并在后继的配置文件中同步修改该文件的位置,需要使用绝对路径
2. 在训练集群上,需要保留`ca.crt``client.key``client.crt``ksmaster_client.dat``ksstandby_client.dat``server_dask.key``server_dask.crt``client_dask.key``client_dask.crt`,并删除其他文件。
3. 评估服务上,需要保留`ca.crt``server.key``server.crt``ksmaster_server.dat``ksstandby_server.dat`,并删除其他文件。

Expand All @@ -155,36 +158,36 @@ chmod -R 600 ~/.vega
server.ini:

```ini
[security]
ca_cert=<~/.vega/car.crt>
[security] # 以下文件路径需要修改为绝对路径
ca_cert=<~/.vega/ca.crt>
server_cert_dask=<~/.vega/server_dask.crt>
server_secret_key_dask=<~/.vega/server_dask.key>
client_cert_dask=<~/.vega/client_dask.crt>
client_secret_key_dask=<~/.vega/ client_dask.key>
client_secret_key_dask=<~/.vega/client_dask.key>
```

client.ini:

```ini
[security]
ca_cert=<~/.vega/car.crt>
[security] # 以下文件路径需要修改为绝对路径
ca_cert=<~/.vega/ca.crt>
client_cert=<~/.vega/client.crt>
client_secret_key=<~/.vega/client.key>
encrypted_password=<加密后的client端的口令> #如果使用普通证书, 此项配置为空
key_component_1=<~/.vega/ksmaster_client.dat> #如果使用普通证书, 此项配置为空
key_component_2=<~/.vega/ksstandby_client.dat> #如果使用普通证书, 此项配置为空
encrypted_password=<加密后的client端的口令> # 如果使用普通证书, 此项配置为空
key_component_1=<~/.vega/ksmaster_client.dat> # 如果使用普通证书, 此项配置为空
key_component_2=<~/.vega/ksstandby_client.dat> # 如果使用普通证书, 此项配置为空
```

在评估服务器上,需要配置`~/.vega/vega.ini`

```ini
[security]
ca_cert=<~/.vega/car.crt>
server_cert=<~/.vega/server.crt>
server_secret_key=<~/.vega/server.key>
encrypted_password=<加密后的server端的口令> #如果使用普通证书, 此项配置为空
key_component_1=<~/.vega/ksmaster_server.dat> #如果使用普通证书, 此项配置为空
key_component_2=<~/.vega/ksstandby_server.dat> #如果使用普通证书, 此项配置为空
[security] # 以下文件路径需要修改为绝对路径
ca_cert=<~/.vega/ca.crt>
server_cert=<~/.vega/server.crt>
server_secret_key=<~/.vega/server.key>
encrypted_password=<加密后的server端的口令> # 如果使用普通证书, 此项配置为空
key_component_1=<~/.vega/ksmaster_server.dat> # 如果使用普通证书, 此项配置为空
key_component_2=<~/.vega/ksstandby_server.dat> # 如果使用普通证书, 此项配置为空
```

## 7. 配置评估服务守护服务
Expand All @@ -197,7 +200,7 @@ key_component_2=<~/.vega/ksstandby_server.dat> #如果使用普通证书, 此
vega-evaluate_service-service -i <ip> -w <path>
```

然后再创建一个守护服务的文件`evaluate-service`,脚本内容如下,注意替换为真实的脚本位置:
然后再创建一个守护服务的文件`evaluate-service.service`,脚本内容如下,注意替换为真实的脚本位置:

```ini
[Unit]
Expand All @@ -211,10 +214,10 @@ vega-evaluate_service-service -i <ip> -w <path>
WantedBy=multi-user.target
```

然后将`evaluate-service`拷贝到目录`/usr/lib/systemd/system`中,并启动该服务:
然后将`evaluate-service.service`拷贝到目录`/usr/lib/systemd/system`中,并启动该服务:

```shell
sudo cp evaluate-service /usr/lib/systemd/system/
sudo cp evaluate-service.service /usr/lib/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start evaluate-service
```
Expand Down
6 changes: 3 additions & 3 deletions docs/en/algorithms/nago.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,15 +66,15 @@ The exact code for the architecture generator (return a trainable PyTorch networ

### 4.2 Search Strategy

Our NAGO search space is amenable to any Bayesian optimisation search strategies. In this code package, we use BOHB to perform the optimisation and the configuration of BOHB needs to be specified in `nago.yml`. The example below defines a BOHB run with `eta=2` and `t=50` search iterations. The minimum and maxmimum training epochs used for evaluating a recommended configuration is 30 and 120 respectively.
Our NAGO search space is amenable to any Bayesian optimisation search strategies. In this code package, we use BOHB to perform the optimisation and the configuration of BOHB needs to be specified in `nago.yml`.

```yaml
search_algorithm:
type: BohbHpo
policy:
total_epochs: -1
repeat_times: 50
num_samples: 350
repeat_times: 1
num_samples: 7
max_epochs: 120
min_epochs: 30
eta: 2
Expand Down
1 change: 0 additions & 1 deletion docs/en/developer/quick_start.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,6 @@ nas:
lr_scheduler:
type: MultiStepLR
params:
warmup: False
milestones: [30]
gamma: 0.5
loss:
Expand Down
63 changes: 33 additions & 30 deletions docs/en/user/security_configure.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,17 @@ pip3 install --user pyOpenSSL==19.0.0
## 2. Generate the CA Certificate

Run the following command to generate a CA certificate:
Note: The length of the RSA key must be 3072 bits or more. The following RSA key length configuration also requires the same.

```shell
openssl genrsa -out ca.key 4096
openssl req -new -x509 -key ca.key -out ca.crt -subj "/C=<country>/ST=<province>/L=<city>/O=<organization>/OU=<group>/CN=<cn>"
```

Note: `<country>`, `<province>`, `<city>`, `<organization>`, `<group>`, and `<cn>` should be set based on the situation. The configuration in this document is the same.
In addition, the CA configuration must be different from other configurations.
Note:

1. `<country>`, `<province>`, `<city>`, `<organization>`, `<group>`, and `<cn>` should be set based on the situation. The values do not contain `< >'. In addition, the CA configuration must be different from other configurations.
2. It is recommended that the length of the RSA key be 3072 bits or more.

## 3. Generate the Certificate for Evaluate_service

Expand Down Expand Up @@ -138,15 +141,15 @@ vega-encrypt_key --cert=client.crt --key=client.key --key_component_1=ksmaster_c
Create the `.vega` directory in the home directory of the current user, copy the generated keys, certificates, and encryption materials to this directory, and change the permission.

```shell
mkdir -/.vega
mv * -/.vega/
chmod -R 600 -/.vega
mkdir ~/.vega
mv * ~/.vega/
chmod 600 ~/.vega/*
```

Description:

1. The preceding keys, certificates, and encryption materials can also be stored in other directories. The access permission must be set to 600 and the file location must be changed in subsequent configuration files.
2. In the train cluster, reserve `ca.crt`, `client.key`, `client.crt`, `ksmaster_client.dat`, `ksstandby_client.dat`, and `server_dask.key. `, `server_dask.crt`, `client_dask.key`, `client_dask.crt`, and delete other files.
1. The preceding keys, certificates, and encryption materials can also be stored in other directories. The access permission must be set to 600, and the file location must be changed to an absolute path in subsequent configuration files.
2. In the train cluster, reserve `ca.crt`, `client.key`, `client.crt`, `ksmaster_client.dat`, `ksstandby_client.dat`, and `server_dask.key`, `server_dask.crt`, `client_dask.key`, `client_dask.crt`, and delete other files.
3. In the evaluate service, reserve `ca.crt`, `server.key`, `server.crt`, `ksmaster_server.dat`, and `ksstandby_server.dat` files, and delete other files.

Create `server.ini` and `client.ini` in the `~/.vega` directory.
Expand All @@ -156,36 +159,36 @@ In the train cluster, configure `~/.vega/server.ini` and `~/.vega/client.ini`.
server.ini:

```ini
[security]
ca_cert=<-/.vega/car.crt>
server_cert_dask=<-/.vega/server_dask.crt>
server_secret_key_dask=<-/.vega/server_dask.key>
client_cert_dask=<-/.vega/client_dask.crt>
client_secret_key_dask=<-/.vega/ client_dask.key>
[security] # The following file paths need to be changed to absolute paths.
ca_cert=<~/.vega/ca.crt>
server_cert_dask=<~/.vega/server_dask.crt>
server_secret_key_dask=<~/.vega/server_dask.key>
client_cert_dask=<~/.vega/client_dask.crt>
client_secret_key_dask=<~/.vega/ client_dask.key>
```

client.ini:

```ini
[security]
ca_cert=<-/.vega/car.crt>
client_cert=<-/.vega/client.crt>
client_secret_key=<-/.vega/client.key>
encrypted_password=<Encrypted client password> #If a common certificate is used, leave this parameter blank.
If the key_component_1=<~/.vega/ksmaster_client.dat> #If a common certificate is used, leave this parameter blank.
If the key_component_2=<~/.vega/ksstandby_client.dat> #If a common certificate is used, leave this parameter blank.
[security] # The following file paths need to be changed to absolute paths.
ca_cert=<~/.vega/ca.crt>
client_cert=<~/.vega/client.crt>
client_secret_key=<~/.vega/client.key>
encrypted_password=<Encrypted client password> # If a common certificate is used, leave this parameter blank.
key_component_1=<~/.vega/ksmaster_client.dat> # If a common certificate is used, leave this parameter blank.
key_component_2=<~/.vega/ksstandby_client.dat> # If a common certificate is used, leave this parameter blank.
```

On the evaluation server, configure `~/.vega/vega.ini`.

```ini
[security]
ca_cert=<-/.vega/car.crt>
server_cert=<-/.vega/server.crt>
server_secret_key=<-/.vega/server.key>
encrypted_password=<Encrypted server password> #If a common certificate is used, leave this parameter blank.
If the key_component_1=<~/.vega/ksmaster_server.dat> # uses a common certificate, leave this parameter blank.
If the key_component_2=<~/.vega/ksstandby_server.dat> # uses a common certificate, leave this parameter blank.
[security] # The following file paths need to be changed to absolute paths.
ca_cert=<~/.vega/ca.crt>
server_cert=<~/.vega/server.crt>
server_secret_key=<~/.vega/server.key>
encrypted_password=<Encrypted server password> # If a common certificate is used, leave this parameter blank.
key_component_1=<~/.vega/ksmaster_server.dat> # uses a common certificate, leave this parameter blank.
key_component_2=<~/.vega/ksstandby_server.dat> # uses a common certificate, leave this parameter blank.
```

## 7. Configuring the Evaluation Service Daemon Service
Expand All @@ -198,7 +201,7 @@ Create a script `run_evaluate_service.sh` for starting the evaluation service. R
vega-evaluate_service-service -i <ip> -w <path>
```

Create a daemon service file `evaluate-service`. The script content is as follows. Replace it with the actual script location.
Create a daemon service file `evaluate-service.service`. The script content is as follows. Replace it with the actual script location.

```ini
[Unit]
Expand All @@ -212,10 +215,10 @@ RestartSec=60
WantedBy=multi-user.target
```

Copy `evaluate-service` to the `/usr/lib/systemd/system` directory and start the service.
Copy `evaluate-service.service` to the `/usr/lib/systemd/system` directory and start the service.

```shell
sudo cp evaluate-service /usr/lib/systemd/system/
sudo cp evaluate-service.service /usr/lib/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start evaluate-service
```
Expand Down
8 changes: 6 additions & 2 deletions evaluate_service/README.cn.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@

### 3.1 安装配置Atlas300环境

首先需要配置Ascend 300环境,请参考[配置文档](./ascend_310.md)
首先需要配置Ascend 300环境,请参考[配置文档](./docs/cn/ascend_310.md)

然后请安装评估服务,请执行如下命令安装:

Expand All @@ -57,8 +57,12 @@ cd build/intermediates/host
cmake ../../src -DCMAKE_CXX_COMPILER=g++ -DCMAKE_SKIP_RPATH=TRUE
make && echo "[INFO] check the env sucess!"
```
### 3.2 编译推理程序
参考 [https://gitee.com/ascend/tools/tree/master/msame](https://gitee.com/ascend/tools/tree/master/msame), 下载代码并完成编译。
并把编译后的可执行文件拷贝到`~/.local/lib/python3.7/site-packages/evaluate_service/hardwares/davinci/`目录下。

### 3.2 启动评估服务

### 3.3 启动评估服务

使用如下命令启动评估服务:

Expand Down
7 changes: 5 additions & 2 deletions evaluate_service/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Configure the hardware (Atlas 200 DK, Atlas 300, or mobile phone) by following t
Please contact us.

### 3.1.2 Install and configure the Atlas 300 Environment (Optional)

Please refer to [configuration documentation](./docs/en/ascend_310.md).
For details, see the Huawei official tutorial at <https://support.huawei.com/enterprise/zh/ai-computing-platform/a300-3000-pid-250702915>.

Note: The preceding documents may be updated. Please follow the released updates or obtain the corresponding guide documents. After the environment is installed, you need to set environment variables. For details, see the preceding guide. To facilitate environment configuration, we provide the environment variable configuration template [env_atlas300.sh](https://github.com/huawei-noah/vega/blob/master/evaluate_service/hardwares/davinci/env/env_atlas300.sh) for your reference. The actual environment prevails.
Expand All @@ -55,8 +55,11 @@ Please contact us.

Please contact us.

### 3.2 Compile the inference program
Please refer to [https://gitee.com/ascend/tools/tree/master/msame](https://gitee.com/ascend/tools/tree/master/msame).
Download the code and finish compiling, then copy the compiled executable file to the `~/.local/lib/python3.7/site-packages/evaluate_service/hardwares/davinci/` directory.

### 3.2 Start the evaluation service
### 3.3 Start the evaluation service

Run the following command to start the evaluate service:
```shell
Expand Down
2 changes: 1 addition & 1 deletion evaluate_service/RELEASE.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
**Evaluate Service ver1.8.0 released:**
**Evaluate Service ver1.8.2 released:**

**Introduction**

Expand Down
2 changes: 1 addition & 1 deletion evaluate_service/evaluate_service/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,4 @@

"""Evaluate service."""

__version__ = "1.8.0"
__version__ = "1.8.2"
Loading

0 comments on commit f02bd90

Please sign in to comment.