Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

知识库创建接口对齐\upload_file接口隐藏 #726

Merged
merged 4 commits into from
Jan 23, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 12 additions & 59 deletions docs/BasisModule/Platform/KnowledgeBase/knowledgebase.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,10 @@
| name | string | 是 | 希望创建的知识库名称 | "我的知识库" |
| description | string | 否 | 知识库描述 | "我的知识库" |
| type | string | 是 | 知识库索引存储配置 (public、bes、vdb) | "public" |
| esUrl | string | 否 | bes 访问地址,type填bes时填写 | "http://test/test" |
| clusterId | string | | 集群/实例 ID | "clusterId" |
| esUserName | string | 否 | bes 用户名,type填bes时填写 | "username" |
| esPassword | string | 否 | bes密码,type填bes时填写 | "password" |
|location|string|否|托管资源的区域,type填vdb时填写<br>可选值:<br>- bj:北京<br>- bd:保定<br>- sz:苏州<br>- gz:广州|"bj"|

#### 方法返回值

Expand Down Expand Up @@ -441,41 +442,7 @@ knowledge.upload_documents(
)
```

### 9、上传通用文档 `KnowledgeBase().upload_file(file_path: str)->KnowledgeBaseUploadFileResponse`

#### 方法参数
| 参数名称 | 参数类型 | 是否必传 | 描述 | 示例值 |
| --------- | -------- | -------- | -------------- | -------------- |
| file_path | string | 是 | 本地的文件路径 | "/home/my_doc" |

#### 方法返回值
`KnowledgeBaseUploadFileResponse` 类定义如下

```python
class KnowledgeBaseUploadFileResponse(BaseModel):
request_id: str = Field(..., description="请求ID")
id: str = Field(..., description="文件ID")
name: str = Field(..., description="文件名称")
```


#### 方法示例
```python
import os
import appbuilder
os.environ["APPBUILDER_TOKEN"] = "your_appbuilder_token"

my_knowledge_base_id = "your_knowledge_base_id"
my_knowledge = appbuilder.KnowledgeBase(my_knowledge_base_id)
print("知识库ID: ", my_knowledge.knowledge_id)

upload_res = my_knowledge.upload_file("./test.txt")
print(upload_res)

# 知识库ID: da51a988-cbe7-4b24-aa5b-768985e8xxxx
# request_id='255eec22-ec87-4564-bdeb-3e5623eaxxxx' id='ef12119b-d5be-492a-997c-77f8e84axxxx' name='test.txt'
```
### 10、向知识库添加文档 `KnowledgeBase().add_document()->KnowledgeBaseAddDocumentResponse`
### 9、向知识库添加文档 `KnowledgeBase().add_document()->KnowledgeBaseAddDocumentResponse`

#### 方法参数
| 参数名称 | 参数类型 | 是否必传 | 描述 | 示例值 |
Expand Down Expand Up @@ -525,7 +492,7 @@ print("添加文档结果: ",add_res)
# 添加文档结果: request_id='412e1630-b570-47c9-a042-caf3cd9dxxxx' knowledge_base_id='da51a988-cbe7-4b24-aa5b-768985e8xxxx' document_ids=['5e0eb279-7688-4100-95d1-241f3d19xxxx']
```

### 11、从知识库删除文档 `KnowledgeBase().delete_document()->KnowledgeBaseDeleteDocumentResponse`
### 10、从知识库删除文档 `KnowledgeBase().delete_document()->KnowledgeBaseDeleteDocumentResponse`

#### 方法参数
| 参数名称 | 参数类型 | 是否必传 | 描述 | 示例值 |
Expand Down Expand Up @@ -566,7 +533,7 @@ print("删除文档结果: ",delete_res)
# 删除文档结果: request_id='ba0e8bc0-b799-45b5-bdac-0d4c50e2xxxx'
```

### 12、获取知识库的文档列表`KnowledgeBase().get_documents_list()->KnowledgeBaseGetDocumentsListResponse`
### 11、获取知识库的文档列表`KnowledgeBase().get_documents_list()->KnowledgeBaseGetDocumentsListResponse`

#### 方法参数
| 参数名称 | 参数类型 | 是否必传 | 描述 | 示例值 |
Expand Down Expand Up @@ -617,7 +584,7 @@ print("文档列表: ", list_res)
# 文档列表: request_id='f66c2193-6035-4022-811b-c4cd7743xxxx' data=[{'id': '8f388b10-5e6a-423f-8acc-dd5fdc2fxxxx', 'name': 'test.txt', 'created_at': 1719988868, 'word_count': 16886, 'enabled': True, 'meta': {'source': 'upload_file', 'file_id': '0ebb03fb-ea48-4c49-b494-cf0cec11xxxx'}}, {'id': '5e0eb279-7688-4100-95d1-241f3d19xxxx', 'name': 'test.txt', 'created_at': 1719987921, 'word_count': 16886, 'enabled': True, 'meta': {'source': 'upload_file', 'file_id': '059e2ae2-1e3c-43ea-8b42-5d988f93xxxx'}}]
```

### 13、获取知识库全部文档`KnowledgeBase().get_all_documents()->list`
### 12、获取知识库全部文档`KnowledgeBase().get_all_documents()->list`

#### 方法参数
| 参数名称 | 参数类型 | 是否必传 | 描述 | 示例值 |
Expand Down Expand Up @@ -658,7 +625,7 @@ for message in doc_list:
print(message)
```

### 14. 创建切片`create_chunk(documentId: str, content: str) -> CreateChunkResponse`
### 13. 创建切片`create_chunk(documentId: str, content: str) -> CreateChunkResponse`

#### 方法参数

Expand Down Expand Up @@ -692,7 +659,7 @@ print("切片ID: ", resp.id)
chunk_id = resp.id
```

### 15. 修改切片信息`modify_chunk(chunkId: str, content: str, enable: bool)`
### 14. 修改切片信息`modify_chunk(chunkId: str, content: str, enable: bool)`

#### 方法参数

Expand All @@ -716,7 +683,7 @@ print("知识库ID: ", my_knowledge.knowledge_id)
my_knowledge.modify_chunk("your_chunk_id", "content", True, knowledgebase_id=my_knowledge_base_id)
```

### 16. 删除切片`delete_chunk(chunkId: str)`
### 15. 删除切片`delete_chunk(chunkId: str)`

#### 方法参数

Expand All @@ -738,7 +705,7 @@ print("知识库ID: ", my_knowledge.knowledge_id)
my_knowledge.delete_chunk("your_chunk_id", knowledgebase_id=my_knowledge_base_id)
```

### 17. 获取切片信息`describe_chunk(chunkId: str)`
### 16. 获取切片信息`describe_chunk(chunkId: str)`

#### 方法参数

Expand Down Expand Up @@ -783,7 +750,7 @@ print("切片详情:")
print(resp)
```

### 18. 获取切片列表`describe_chunks(documentId: str, marker: str = None, maxKeys: int = None, type: str = None) -> DescribeChunksResponse`
### 17. 获取切片列表`describe_chunks(documentId: str, marker: str = None, maxKeys: int = None, type: str = None) -> DescribeChunksResponse`

#### 方法参数

Expand Down Expand Up @@ -871,10 +838,6 @@ public class KnowledgebaseTest {
listRequest.setKonwledgeBaseId(knowledgeBaseId);
listRequest.setLimit(10);
Document[] documents = knowledgebase.getDocumentList(listRequest);

// 向知识库上传通用文档
String fileId = knowledgebase.uploadFile("src/test/java/com/baidubce/appbuilder/files/test.pdf");
System.out.println(fileId);

// 向知识库添加文档
DocumentAddRequest request = new DocumentAddRequest();
Expand Down Expand Up @@ -905,7 +868,7 @@ public class KnowledgebaseTest {

// 创建知识库
KnowledgeBaseConfig.Index index = new KnowledgeBaseConfig.Index("public",
"http://localhost:9200", "elastic", "changeme");
"", "", "", "");
KnowledgeBaseConfig config = new KnowledgeBaseConfig(index);
request.setConfig(config);
KnowledgeBaseDetail response = knowledgebase.createKnowledgeBase(request);
Expand Down Expand Up @@ -1031,13 +994,6 @@ func TestKnowledgeBase(t *testing.T) {
}
fmt.Println(documentsRes)

// 向知识库中上传通用文件
fileID, err := client.UploadFile("./files/test.pdf")
if err != nil {
t.Fatalf("upload file failed: %v", err)
}
fmt.Println(fileID)

// 向知识库中添加文档
createDocumentRes, err := client.CreateDocument(CreateDocumentRequest{
KnowledgeBaseID: knowledgeBaseID,
Expand Down Expand Up @@ -1083,9 +1039,6 @@ func TestCreateKnowledgeBase(t *testing.T) {
Config: &KnowlegeBaseConfig{
Index: KnowledgeBaseConfigIndex{
Type: "public",
EsUrl: "http://localhost:9200",
Password: "elastic",
Username: "elastic",
},
},
})
Expand Down
1 change: 1 addition & 0 deletions go/appbuilder/knowledge_base.go
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,7 @@ func (t *KnowledgeBase) GetDocumentList(req GetDocumentListRequest) (*GetDocumen
return &rsp, nil
}

// Deprecated: 此功能即将废弃
func (t *KnowledgeBase) UploadFile(localFilePath string) (string, error) {
var data bytes.Buffer
w := multipart.NewWriter(&data)
Expand Down
9 changes: 5 additions & 4 deletions go/appbuilder/knowledge_base_data.go
Original file line number Diff line number Diff line change
Expand Up @@ -89,10 +89,11 @@ type KnowlegeBaseConfig struct {
}

type KnowledgeBaseConfigIndex struct {
Type string `json:"type"`
EsUrl string `json:"esUrl"`
Username string `json:"username"`
Password string `json:"password"`
Type string `json:"type"`
ClusterId string `json:"clusterId"`
Username string `json:"username"`
Password string `json:"password"`
Location string `json:"location"`
}

type KnowledgeBaseDetail struct {
Expand Down
7 changes: 0 additions & 7 deletions go/appbuilder/knowledge_base_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,6 @@ func TestCreateKnowledgeBaseError(t *testing.T) {
Config: &KnowlegeBaseConfig{
Index: KnowledgeBaseConfigIndex{
Type: "public",
EsUrl: "http://localhost:9200",
Password: "elastic",
Username: "elastic",
},
Expand All @@ -285,7 +284,6 @@ func TestCreateKnowledgeBaseError(t *testing.T) {
Config: &KnowlegeBaseConfig{
Index: KnowledgeBaseConfigIndex{
Type: "public",
EsUrl: "http://localhost:9200",
Password: "elastic",
Username: "elastic",
},
Expand All @@ -303,7 +301,6 @@ func TestCreateKnowledgeBaseError(t *testing.T) {
Config: &KnowlegeBaseConfig{
Index: KnowledgeBaseConfigIndex{
Type: "public",
EsUrl: "http://localhost:9200",
Password: "elastic",
Username: "elastic",
},
Expand All @@ -321,7 +318,6 @@ func TestCreateKnowledgeBaseError(t *testing.T) {
Config: &KnowlegeBaseConfig{
Index: KnowledgeBaseConfigIndex{
Type: "public",
EsUrl: "http://localhost:9200",
Password: "elastic",
Username: "elastic",
},
Expand All @@ -339,7 +335,6 @@ func TestCreateKnowledgeBaseError(t *testing.T) {
Config: &KnowlegeBaseConfig{
Index: KnowledgeBaseConfigIndex{
Type: "public",
EsUrl: "http://localhost:9200",
Password: "elastic",
Username: "elastic",
},
Expand All @@ -358,7 +353,6 @@ func TestCreateKnowledgeBaseError(t *testing.T) {
Config: &KnowlegeBaseConfig{
Index: KnowledgeBaseConfigIndex{
Type: "public",
EsUrl: "http://localhost:9200",
Password: "elastic",
Username: "elastic",
},
Expand Down Expand Up @@ -1210,7 +1204,6 @@ func TestCreateKnowledgeBase(t *testing.T) {
Config: &KnowlegeBaseConfig{
Index: KnowledgeBaseConfigIndex{
Type: "public",
EsUrl: "http://localhost:9200",
Password: "elastic",
Username: "elastic",
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ public Knowledgebase(String knowledgeBaseId, String SecretKey) {
* @throws IOException 当文件上传失败时抛出IOException
* @throws AppBuilderServerException 当服务器返回错误码时抛出AppBuilderServerException
*/
@Deprecated
public String uploadFile(String filePath) throws IOException, AppBuilderServerException {
return innerUploadFile(filePath, java.util.UUID.randomUUID().toString());
}
Expand All @@ -53,6 +54,7 @@ public String uploadFile(String filePath) throws IOException, AppBuilderServerEx
* @throws IOException 如果发生I/O错误
* @throws AppBuilderServerException 如果应用构建服务器发生错误
*/
@Deprecated
public String uploadFile(String filePath, String clientToken) throws IOException, AppBuilderServerException {
return innerUploadFile(filePath, clientToken);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,23 +13,25 @@ public Index getIndex() {

public static class Index {
private String type;
private String esUrl;
private String clusterId;
private String username;
private String password;
private String location;

public Index(String type, String esUrl, String username, String password) {
public Index(String type, String clusterId, String username, String password, String location) {
this.type = type;
this.esUrl = esUrl;
this.clusterId = clusterId;
this.username = username;
this.password = password;
this.location = location;
}

public String getType() {
return type;
}

public String getEsUrl() {
return esUrl;
public String getClusterId() {
return clusterId;
}

public String getUsername() {
Expand All @@ -39,5 +41,9 @@ public String getUsername() {
public String getPassword() {
return password;
}

public String getLocation() {
return location;
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ public void testCreateKnowledgebase() throws IOException, AppBuilderServerExcept

// 创建知识库
KnowledgeBaseConfig.Index index = new KnowledgeBaseConfig.Index("public",
"http://localhost:9200", "elastic", "changeme");
"", "", "", "");
KnowledgeBaseConfig config = new KnowledgeBaseConfig(index);
request.setConfig(config);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,16 @@

import static org.junit.Assert.assertEquals;

import java.beans.Transient;

public class KnowledgeBaseConfigTest {

private KnowledgeBaseConfig knowledgeBaseConfig;
private KnowledgeBaseConfig.Index index;

@Before
public void setUp() {
index = new KnowledgeBaseConfig.Index("es", "http://localhost:9200", "user", "password");
index = new KnowledgeBaseConfig.Index("es", "clusterId", "user", "password", "bj");
knowledgeBaseConfig = new KnowledgeBaseConfig(index);
}

Expand All @@ -27,8 +29,8 @@ public void testIndexType() {
}

@Test
public void testIndexEsUrl() {
assertEquals("http://localhost:9200", index.getEsUrl());
public void testIndexClusterId() {
assertEquals("clusterId", index.getClusterId());
}

@Test
Expand All @@ -40,4 +42,9 @@ public void testIndexUsername() {
public void testIndexPassword() {
assertEquals("password", index.getPassword());
}

@Test
public void testIndexLocation() {
assertEquals("bj", index.getLocation());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ public class KnowledgeBaseDetailTest {
@Before
public void setUp() {
knowledgeBaseDetail = new KnowledgeBaseDetail();
knowledgeBaseConfig = new KnowledgeBaseConfig(new KnowledgeBaseConfig.Index("type1", "esUrl", "user", "pass"));
knowledgeBaseConfig = new KnowledgeBaseConfig(new KnowledgeBaseConfig.Index("type1", "clusterId", "user", "pass", "location"));
}

@Test
Expand Down
4 changes: 3 additions & 1 deletion python/core/console/knowledge_base/data_class.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,10 @@ class KnowledgeBaseGetDocumentsListResponse(BaseModel):

class KnowledgeBaseConfigIndex(BaseModel):
type: str = Field(..., description="索引类型", enum=["public", "bes", "vdb"])
esUrl: Optional[str] = Field(None, description="bes地址")
clusterId: Optional[str] = Field(None, description="集群/实例 ID")
username: Optional[str] = Field(None, description="bes用户名")
password: Optional[str] = Field(None, description="bes密码")
location: Optional[str] = Field(None, description="托管资源的区域", enum=["bj", "bd", "sz", "gz"])


class KnowledgeBaseConfig(BaseModel):
Expand All @@ -117,6 +118,7 @@ class KnowledgeBaseGetDetailRequest(BaseModel):


class KnowledgeBaseDetailResponse(BaseModel):
requestId: str = Field(..., description="请求ID")
id: str = Field(..., description="知识库ID")
name: str = Field(..., description="知识库名称")
description: Optional[str] = Field(None, description="知识库描述")
Expand Down
1 change: 1 addition & 0 deletions python/core/console/knowledge_base/knowledge_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ def create_knowledge(cls, knowledge_name: str) -> "KnowledgeBase":
knowledge_id=response["id"], knowledge_name=response["name"]
)

@deprecated()
def upload_file(
self, file_path: str, client_token: str = None
) -> data_class.KnowledgeBaseUploadFileResponse:
Expand Down
Loading