-
Notifications
You must be signed in to change notification settings - Fork 187
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: add e2e test #11941
base: dev
Are you sure you want to change the base?
test: add e2e test #11941
Changes from all commits
71dc1ab
12d5895
38ca36c
9f4ef2e
1e56315
3862b4a
f8e8268
6948a96
951e0a4
175e560
1f98670
f73ce31
10381b0
2374bf4
fd0de8a
ec73efc
3f2aee2
b7cb7ae
6c75a3e
fe5cf7a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -51,8 +51,10 @@ export abstract class CaseFactory { | |
skipDeploy?: boolean; | ||
skipValidate?: boolean; | ||
skipPackage?: boolean; | ||
skipErrorMessage?: string; | ||
}; | ||
public custimized?: Record<string, string>; | ||
public customized?: Record<string, string>; | ||
public processEnv?: NodeJS.ProcessEnv; | ||
|
||
public constructor( | ||
capability: Capability, | ||
|
@@ -74,16 +76,19 @@ export abstract class CaseFactory { | |
skipDeploy?: boolean; | ||
skipValidate?: boolean; | ||
skipPackage?: boolean; | ||
skipErrorMessage?: string; | ||
} = {}, | ||
custimized?: Record<string, string> | ||
customized?: Record<string, string>, | ||
processEnv?: NodeJS.ProcessEnv | ||
) { | ||
this.capability = capability; | ||
this.testPlanCaseId = testPlanCaseId; | ||
this.author = author; | ||
this.validate = validate; | ||
this.programmingLanguage = programmingLanguage; | ||
this.options = options; | ||
this.custimized = custimized; | ||
this.customized = customized; | ||
this.processEnv = processEnv; | ||
} | ||
|
||
public onBefore(): Promise<void> { | ||
|
@@ -103,14 +108,16 @@ export abstract class CaseFactory { | |
testFolder: string, | ||
capability: Capability, | ||
programmingLanguage?: ProgrammingLanguage, | ||
custimized?: Record<string, string> | ||
customized?: Record<string, string>, | ||
processEnv?: NodeJS.ProcessEnv | ||
): Promise<void> { | ||
await Executor.createProject( | ||
testFolder, | ||
appName, | ||
capability, | ||
programmingLanguage ? programmingLanguage : ProgrammingLanguage.TS, | ||
custimized | ||
customized, | ||
processEnv | ||
); | ||
} | ||
|
||
|
@@ -126,14 +133,15 @@ export abstract class CaseFactory { | |
validate, | ||
programmingLanguage, | ||
options, | ||
custimized, | ||
customized, | ||
processEnv, | ||
onBefore, | ||
onAfter, | ||
onAfterCreate, | ||
onBeforeProvision, | ||
onCreate, | ||
} = this; | ||
describe(`template Test: ${capability}`, function () { | ||
describe(`template Test: ${capability} - ${programmingLanguage}`, function () { | ||
const testFolder = getTestFolder(); | ||
const appName = getUniqueAppName(); | ||
const projectPath = path.resolve(testFolder, appName); | ||
|
@@ -153,7 +161,8 @@ export abstract class CaseFactory { | |
testFolder, | ||
capability, | ||
programmingLanguage, | ||
custimized | ||
customized, | ||
processEnv | ||
); | ||
expect(fs.pathExistsSync(projectPath)).to.be.true; | ||
|
||
|
@@ -173,7 +182,12 @@ export abstract class CaseFactory { | |
expect(result).to.be.true; | ||
process.env["AZURE_RESOURCE_GROUP_NAME"] = appName + "-rg"; | ||
|
||
const { success } = await Executor.provision(projectPath); | ||
const { success } = await Executor.provision( | ||
projectPath, | ||
"dev", | ||
true, | ||
options?.skipErrorMessage | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. if skip this error message, the pipeline may skip some action There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, there are some features that are in private preview. The test account cannot run some templates successfully (lacking some permissions), so we are going to skip this error message for now There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. if so, maybe set skipErrorMessage for specific templates instead of set all rules? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Only some templates have passed the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If we don't add this parameter, the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It seems like everyone has some concerns about this, so let's hold off on adding any test cases related to the Copilot plugin until they've fully rolled it out. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If this error is skipped, is it possible to simulate the actual e2e process? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can simulate most, but not all. We'll remove the skip logic once the feature is GA. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Sorry, I didn't explain clearly enough. There is only one action, "extendToM365," will fail during provisioning, the arm deploy is work. Also, both There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually, there is a common function |
||
); | ||
expect(success).to.be.true; | ||
|
||
// Validate Provision | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,51 @@ | ||
// Copyright (c) Microsoft Corporation. | ||
// Licensed under the MIT license. | ||
|
||
/** | ||
* @author Yimin Jin <[email protected]> | ||
*/ | ||
|
||
import { ProgrammingLanguage } from "@microsoft/teamsfx-core"; | ||
import { CopilotPluginCommonTest } from "./copilotPluginCommonTest"; | ||
|
||
class CopilotPluginWithApiKeyAuthCase extends CopilotPluginCommonTest {} | ||
const validateFiles = { | ||
[ProgrammingLanguage.JS]: [ | ||
"appPackage/ai-plugin.json", | ||
"appPackage/manifest.json", | ||
"src/keyGen.js", | ||
], | ||
[ProgrammingLanguage.TS]: [ | ||
"appPackage/ai-plugin.json", | ||
"appPackage/manifest.json", | ||
"src/keyGen.ts", | ||
], | ||
[ProgrammingLanguage.CSharp]: [ | ||
"appPackage/ai-plugin.json", | ||
"appPackage/manifest.json", | ||
"GenerateApiKey.ps1", | ||
], | ||
}; | ||
new CopilotPluginWithApiKeyAuthCase( | ||
28640069, | ||
"[email protected]", | ||
"api-key", | ||
ProgrammingLanguage.JS, | ||
validateFiles[ProgrammingLanguage.JS] | ||
).test(); | ||
|
||
new CopilotPluginWithApiKeyAuthCase( | ||
28640069, | ||
"[email protected]", | ||
"api-key", | ||
ProgrammingLanguage.TS, | ||
validateFiles[ProgrammingLanguage.TS] | ||
).test(); | ||
|
||
new CopilotPluginWithApiKeyAuthCase( | ||
28640069, | ||
"[email protected]", | ||
"api-key", | ||
ProgrammingLanguage.CSharp, | ||
validateFiles[ProgrammingLanguage.CSharp] | ||
).test(); |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
// Copyright (c) Microsoft Corporation. | ||
// Licensed under the MIT license. | ||
|
||
/** | ||
* @author Yimin Jin <[email protected]> | ||
*/ | ||
|
||
import { ProgrammingLanguage } from "@microsoft/teamsfx-core"; | ||
import { CopilotPluginCommonTest } from "./copilotPluginCommonTest"; | ||
|
||
class CopilotPluginWithNoneAuthCase extends CopilotPluginCommonTest {} | ||
const validateFiles = ["appPackage/ai-plugin.json", "appPackage/manifest.json"]; | ||
|
||
new CopilotPluginWithNoneAuthCase( | ||
27569734, | ||
"[email protected]", | ||
"none", | ||
ProgrammingLanguage.JS, | ||
validateFiles | ||
).test(); | ||
|
||
new CopilotPluginWithNoneAuthCase( | ||
27569734, | ||
"[email protected]", | ||
"none", | ||
ProgrammingLanguage.TS, | ||
validateFiles | ||
).test(); | ||
|
||
new CopilotPluginWithNoneAuthCase( | ||
27569734, | ||
"[email protected]", | ||
"none", | ||
ProgrammingLanguage.CSharp, | ||
validateFiles | ||
).test(); |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
// Copyright (c) Microsoft Corporation. | ||
// Licensed under the MIT license. | ||
|
||
/** | ||
* @author Hui Miao <[email protected]> | ||
*/ | ||
|
||
import { ProgrammingLanguage } from "@microsoft/teamsfx-core"; | ||
import { CopilotPluginCommonTest } from "./copilotPluginCommonTest"; | ||
|
||
class CopilotPluginWithOAuthCase extends CopilotPluginCommonTest {} | ||
const validateFiles = [ | ||
"appPackage/ai-plugin.dev.json", | ||
"appPackage/manifest.json", | ||
]; | ||
|
||
new CopilotPluginWithOAuthCase( | ||
28641204, | ||
"[email protected]", | ||
"oauth", | ||
ProgrammingLanguage.JS, | ||
validateFiles | ||
).test(); | ||
|
||
new CopilotPluginWithOAuthCase( | ||
28641204, | ||
"[email protected]", | ||
"oauth", | ||
ProgrammingLanguage.TS, | ||
validateFiles | ||
).test(); | ||
|
||
new CopilotPluginWithOAuthCase( | ||
28641204, | ||
"[email protected]", | ||
"oauth", | ||
ProgrammingLanguage.CSharp, | ||
validateFiles | ||
).test(); |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
// Copyright (c) Microsoft Corporation. | ||
// Licensed under the MIT license. | ||
|
||
/** | ||
* @author Hui Miao <[email protected]> | ||
*/ | ||
|
||
import { Capability } from "../../utils/constants"; | ||
import { CaseFactory } from "../caseFactory"; | ||
import { ProgrammingLanguage } from "@microsoft/teamsfx-core"; | ||
import { replaceSecretKey, validateFiles } from "./helper"; | ||
import * as path from "path"; | ||
export class CopilotPluginCommonTest extends CaseFactory { | ||
validateFileList?: string[]; | ||
authOption?: string; | ||
|
||
public constructor( | ||
testPlanCaseId: number, | ||
author: string, | ||
authOption: "none" | "api-key" | "oauth", | ||
programmingLanguage?: ProgrammingLanguage, | ||
validateFileList?: string[] | ||
) { | ||
const env = Object.assign({}, process.env); | ||
env["DEVELOP_COPILOT_PLUGIN"] = "true"; | ||
if (programmingLanguage === ProgrammingLanguage.CSharp) { | ||
env["TEAMSFX_CLI_DOTNET"] = "true"; | ||
} | ||
|
||
const skipOptions = { | ||
skipValidate: true, | ||
skipErrorMessage: "No elements found in the manifest", | ||
}; | ||
|
||
const authOptions: Record<string, string> = {}; | ||
authOptions["api-auth"] = authOption; | ||
|
||
super( | ||
Capability.CopilotPluginFromScratch, | ||
testPlanCaseId, | ||
author, | ||
["function"], | ||
programmingLanguage, | ||
skipOptions, | ||
authOptions, | ||
env | ||
); | ||
this.validateFileList = validateFileList; | ||
this.authOption = authOption; | ||
this.onAfterCreate = this.onAfterCreate.bind(this); | ||
} | ||
|
||
public override async onAfterCreate(projectPath: string): Promise<void> { | ||
await validateFiles(projectPath, this.validateFileList || []); | ||
|
||
if (this.authOption === "api-key") { | ||
const userFile = path.resolve(projectPath, "env", `.env.dev.user`); | ||
await replaceSecretKey(userFile); | ||
} | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
// Copyright (c) Microsoft Corporation. | ||
// Licensed under the MIT license. | ||
import * as path from "path"; | ||
import * as fs from "fs-extra"; | ||
import { expect } from "chai"; | ||
|
||
export async function validateFiles( | ||
projectPath: string, | ||
files: string[] | ||
): Promise<void> { | ||
for (const file of files) { | ||
const filePath = path.join(projectPath, file); | ||
expect(fs.existsSync(filePath), `${filePath} must exist.`).to.eq(true); | ||
} | ||
console.log("Files validation successful"); | ||
} | ||
|
||
export async function replaceSecretKey(userFile: string): Promise<void> { | ||
const newSecretKey = 'SECRET_API_KEY="test-secret-api-key"'; | ||
let fileContent = fs.readFileSync(userFile, "utf8"); | ||
fileContent = fileContent.replace(/(SECRET_API_KEY=).*/, newSecretKey); | ||
fs.writeFileSync(userFile, fileContent, "utf8"); | ||
console.log(`Updated ${newSecretKey} in .env.dev.user file`); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it possible not to suppress this check?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As some historical reasons, many files in the 'tests/' folder currently have nested import issues. If there are any PR modify any of these files, the pre-commit check will fail. The pre-commit check seems to be added later on.
We need to take a look at all those affected files and discuss with related owners on how we can fix them. Let me follow up with them.