-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[37, 119, 191, 207, 105] Fix flaky test and remove retries #5142
Changes from all commits
3adf449
580c0fa
5edc129
098120a
87241c9
269b979
6c7a6ef
9c5f5ad
8d4f729
f76f2e5
92a8865
36d8a4d
d457a96
755878c
e04304c
fbdac5c
79e04a3
0e8c9aa
e246fe5
a2b2ab4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
--- | ||
"saleor-dashboard": patch | ||
--- | ||
|
||
Some end-to-end Playwright tests now have extended timeouts for UI elements to load. This means that automation tests should fail less. Playwright retires value has been set to 0. |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -18,45 +18,39 @@ test.beforeEach(({ page }) => { | |
|
||
const PRE_INSTALLATION_TIMEOUT = 20 * 1000; | ||
const INSTALLATION_PENDING_TIMEOUT = 50 * 1000; | ||
const APP_LISTED_TIMEOUT = 15 * 1000; | ||
const APP_INSTALLED_TIMEOUT = 50 * 1000; | ||
const APP_EXPECT_UI_TIMEOUT = 15 * 1000; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we need such long timeouts? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
|
||
test("TC: SALEOR_119 User should be able to install and configure app from manifest @e2e", async ({ | ||
page, | ||
}) => { | ||
await appsPage.gotoAppsList(); | ||
await appsPage.waitForDOMToFullyLoad(); | ||
await expect(appsPage.installExternalAppButton).toBeVisible(); | ||
await appsPage.installExternalAppButton.click(); | ||
await appsPage.typeManifestUrl("https://klaviyo.saleor.app/api/manifest"); | ||
await appsPage.installAppFromManifestButton.click(); | ||
// Klaviyo app can take a while to respond with manifest if it's | ||
// cold-starting | ||
await expect(installationPage.appInstallationPageHeader).toHaveText( | ||
"You are about to install Klaviyo", | ||
{ | ||
// Klaviyo app can take a while to respond with manifest if it's | ||
// cold-starting | ||
timeout: PRE_INSTALLATION_TIMEOUT, | ||
}, | ||
); | ||
await installationPage.installAppButton.click(); | ||
await appsPage.expectSuccessBanner(); | ||
|
||
await expect(appsPage.successBanner).toBeVisible({ timeout: INSTALLATION_PENDING_TIMEOUT }); | ||
await expect(appsPage.installedAppRow.first()).toBeVisible(); | ||
await appsPage.installationPendingLabel.waitFor({ | ||
state: "hidden", | ||
timeout: INSTALLATION_PENDING_TIMEOUT, | ||
}); | ||
await expect(appsPage.appKlaviyo).toContainText("Klaviyo", { | ||
timeout: APP_LISTED_TIMEOUT, | ||
}); | ||
await appsPage.installedAppRow | ||
.filter({ hasText: "Klaviyo" }) | ||
.first() | ||
.waitFor({ state: "visible", timeout: APP_INSTALLED_TIMEOUT }); | ||
await expect(appsPage.installationPendingLabel).not.toBeVisible(); | ||
|
||
await expect(appsPage.appKlaviyo).toContainText("Klaviyo"); | ||
await expect(appsPage.installedAppRow.filter({ hasText: "Klaviyo" }).first()).toBeVisible(); | ||
await appsPage.appKlaviyo.click(); | ||
|
||
const iframeLocator = page.frameLocator("iframe"); | ||
|
||
await expect(iframeLocator.getByLabel("PUBLIC_TOKEN")).toBeVisible(); | ||
await expect(iframeLocator.getByLabel("PUBLIC_TOKEN")).toBeVisible({ | ||
// Klavyio's UI can take a while to load initially | ||
timeout: APP_EXPECT_UI_TIMEOUT, | ||
}); | ||
await iframeLocator.getByLabel("PUBLIC_TOKEN").fill("test_token"); | ||
await iframeLocator.getByText("Save").click(); | ||
await appsPage.expectSuccessBanner(); | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -257,6 +257,10 @@ test("TC: SALEOR_84 Create draft order @e2e @draft", async () => { | |
}); | ||
|
||
test("TC: SALEOR_191 Refund products from the fully paid order @e2e @refunds", async () => { | ||
// All steps of this test pass (including after hooks), but Playwright | ||
// marks it as failed because of exceeding 30s timeout | ||
test.slow(); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this marks test as slow, and triples the timeout, meaning something similar as if you were added 3x base timeout here. I does not look like fix for the flaky test. Since you are fixing couple of test in single PR, please mark this as TODO, and leave ticket number for further fixing. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Last time it failed the test threw the following error:
here's the recording: |
||
|
||
const order = ORDERS.fullyPaidOrderWithSingleTransaction; | ||
|
||
await ordersPage.goToExistingOrderPage(order.id); | ||
|
@@ -268,7 +272,7 @@ test("TC: SALEOR_191 Refund products from the fully paid order @e2e @refunds", a | |
|
||
const productRow = await refundPage.getProductRow(order.lineItems[0].name); | ||
|
||
expect(productRow.locator(refundPage.productQuantityInput)).toHaveValue( | ||
await expect(productRow.locator(refundPage.productQuantityInput)).toHaveValue( | ||
order.lineItems[0].quantity, | ||
); | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do you need that comment?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I want to see how disabling retries behaves and if needed revert this change.