Merge main to 1.x (#121)

* Fixes editing rollup delay (#82) * Fixes editing the delay amount and displaying the schedule for rollups Signed-off-by: Clay Downs <[email protected]> * Bumps version to 1.2 (#103) * Bumps version to 1.2 * Changes test workflows to follow Dashboards 1.x Signed-off-by: Clay Downs <[email protected]> * Fixes flakey tests (#105) * Fixes flakey cypress test 'rollup enable and disable test' * Fixes flakey unit test 'can delete a rollup job' Signed-off-by: Robert Downs <[email protected]> * Fixed Transforms geo_point bug and boolean render (#93) * Fixed Transforms geo_point bug and boolean render * Set default DefineTransforms table render behavior to JSON.stringify * Updated DefineTransforms null cell value check Signed-off-by: Eric Lobdell <[email protected]> * Transform bug fixes (#109) * Rename plugin helper file * Update TransformIndices.tsx * Backport commits from main (#90) * Release 1.1.0.0 (#83) * Release 1.1.0.0 Signed-off-by: bowenlan-amzn <[email protected]> * Correct copyright notices * UI fixes for new ISM UI (#84) * Removes X icon from action/transition flyout footer next to cancel Signed-off-by: Drew Baugher <[email protected]> * Adds tooltips to edit/delete icon buttons on the draggable action/transition components Signed-off-by: Drew Baugher <[email protected]> * Removes overlay for flyout so clicking outside doesn't close flyout and removes X close button on top right Signed-off-by: Drew Baugher <[email protected]> * Adds edit/delete tooltips for state component and removes underline from hovering states Signed-off-by: Drew Baugher <[email protected]> * Changes state flyout from X Close to Cancel Signed-off-by: Drew Baugher <[email protected]> * Adds JSON editor for allocation action Signed-off-by: Drew Baugher <[email protected]> * Adds isValid to action interface and implements in actions Signed-off-by: Drew Baugher <[email protected]> * Correctly removes unused rollover keys from rollover object, handles uncontrolled inputs and NaN minDocs Signed-off-by: Drew Baugher <[email protected]> * Adds error for state name already existing in policy Signed-off-by: Drew Baugher <[email protected]> * Fixes duplicated actions when editing as previously we had a new ID and matching logic didn't find an action Signed-off-by: Drew Baugher <[email protected]> * Fixes other action inputs that can return NaN and creates a no conditions option that is the default Signed-off-by: Drew Baugher <[email protected]> * Updates snapshot Signed-off-by: Drew Baugher <[email protected]> * Correctly show danger toast on update/create failures Signed-off-by: Drew Baugher <[email protected]> * Fixes rollup action nesting multiple ism_template keys Signed-off-by: Drew Baugher <[email protected]> * Fixes timeout/retry settings turning into uncontrolled inputs and NaN value Signed-off-by: Drew Baugher <[email protected]> * Removes text transformation on inputs and updates isValid methods and passes to form labels Signed-off-by: Drew Baugher <[email protected]> * Fixes allocation and rollup adding UIAction properties to policy JSON on updates Signed-off-by: Drew Baugher <[email protected]> * Updates release notes w/ new PR changes Signed-off-by: Drew Baugher <[email protected]> * Wraps actions/transitions in states component and updates default rollup to include ism_rollup key Signed-off-by: Drew Baugher <[email protected]> * Fixes some small UI touchups/issues for new ISM UI (#85) * Moves cancel secondary button next to primary on action/transition flyout Signed-off-by: Drew Baugher <[email protected]> * Removes default ISM template and increases width of empty prompt Signed-off-by: Drew Baugher <[email protected]> * Adds punctuation Signed-off-by: Drew Baugher <[email protected]> * Adds back X icon next to cancel for state flyout Signed-off-by: Drew Baugher <[email protected]> * Makes inputs in flyout full width, updates help text, and adds some spacing Signed-off-by: Drew Baugher <[email protected]> * Updates transition default to not include conditions empty object, and on change transition to delete conditions when selecting none Signed-off-by: Drew Baugher <[email protected]> * Moves edit button out of policy settings content panel and on to global page for view policy Signed-off-by: Drew Baugher <[email protected]> * Updates release note Signed-off-by: Drew Baugher <[email protected]> * Fixes broken link Signed-off-by: Drew Baugher <[email protected]> * Fixes small issues on new ISM UI (#88) * Fixes small issues on new ISM UI Signed-off-by: Drew Baugher <[email protected]> * Updates snapshots Signed-off-by: Drew Baugher <[email protected]> * Updates release notes Signed-off-by: Drew Baugher <[email protected]> * Updates workflows to trigger on 1.* branches Signed-off-by: Drew Baugher <[email protected]> Co-authored-by: Bowen Lan <[email protected]> * Support data filter when viewing sample data * Update TransformOptions.tsx * Add TransformService.test.ts * Draft of date histogram panel * Add helper method and move import * Split calendar and fixed interval to 2 panels * Add rendering test * Add close popover test * Update cypress-workflow.yml * Update links.yml * Update unit-tests-workflow.yml * Update TransformOptions.tsx * Update tests * Update DateHistogramPanel.test.tsx * Make scripted metrics editor larger * Expand code editor related panels * Undo changes to workflow Signed-off-by: Annie Lee <[email protected]> * Update TimeAggregation.tsx * Update DefineTransforms.tsx undo changes to style * Clean up code * Update snapshot Signed-off-by: Annie Lee <[email protected]> * Refactor the order of checking if name is defined before checking for duplicate name * Update CreateTransformForm.tsx * Update DefineTransforms.tsx Signed-off-by: Annie Lee <[email protected]> * Update DefineTransforms.tsx Signed-off-by: Annie Lee <[email protected]> * Update DefineTransforms.tsx * Update DefineTransforms.tsx * Update DefineTransforms.tsx Co-authored-by: Drew Baugher <[email protected]> Co-authored-by: Bowen Lan <[email protected]> * Fix the bugs in visual UI for legacy notification (#111) * Transforms unit and cypress tests (#92) * Transform cypress tests (#21) * Added Cypress tests for Transforms UI * License update * Unit tests initial commit * Added jest tests for Transforms, details, edit and create * Sample transform fix * Fixed cypress test transform and sample data load order Signed-off-by: Eric Lobdell <[email protected]> * Test fix debug * Simplified unit test * Updated transforms cypress test comments and success criteria Signed-off-by: Eric Lobdell <[email protected]> * Fixed quotation mark issue in test Signed-off-by: Eric Lobdell <[email protected]> * Removed debug mapping and index objects from unit test Signed-off-by: Eric Lobdell <[email protected]> * Cypress test improvements and completed edit transform unit test Signed-off-by: Eric Lobdell <[email protected]> * Integ test delete fix and EditTransforms jest test fix Signed-off-by: Eric Lobdell <[email protected]> * Restoring enable disable page refresh delay to fix failing integ test Signed-off-by: Eric Lobdell <[email protected]> * Resolve test constants conflict Signed-off-by: Eric Lobdell <[email protected]> * Fix pagination of managed indices page (#113) * Disables jitter for managed_indices_spec Cypress tests (#122) Signed-off-by: Robert Downs <[email protected]> Co-authored-by: Clay Downs <[email protected]> Co-authored-by: Annie Lee <[email protected]> Co-authored-by: Drew Baugher <[email protected]> Co-authored-by: Bowen Lan <[email protected]> Co-authored-by: Ravi <[email protected]>
opensearch-project · Nov 5, 2021 · 5fb620e · 5fb620e
1 parent 10ba1af
commit 5fb620e
Show file tree

Hide file tree

Showing 52 changed files with 4,364 additions and 297 deletions.
diff --git a/.github/workflows/cypress-workflow.yml b/.github/workflows/cypress-workflow.yml
@@ -10,7 +10,7 @@ on:
       - 1.*
 env:
   OPENSEARCH_DASHBOARDS_VERSION: '1.x'
-  OPENSEARCH_VERSION: '1.1.0-SNAPSHOT'
+  OPENSEARCH_VERSION: '1.2.0-SNAPSHOT'
 jobs:
   tests:
     name: Run Cypress E2E tests

diff --git a/.github/workflows/unit-tests-workflow.yml b/.github/workflows/unit-tests-workflow.yml
@@ -7,8 +7,9 @@ on:
   push:
     branches:
       - main
-      - 1.*
-
+      - development-*
+env:
+  OPENSEARCH_DASHBOARDS_VERSION: '1.x'
 jobs:
   tests:
     name: Run unit tests
@@ -22,7 +23,7 @@ jobs:
         uses: actions/checkout@v2
         with:
           repository: opensearch-project/OpenSearch-Dashboards
-          ref: '1.0'
+          ref: ${{ env.OPENSEARCH_DASHBOARDS_VERSION }}
           path: OpenSearch-Dashboards
       - name: Get node and yarn versions
         id: versions

diff --git a/cypress/fixtures/sample_transform.json b/cypress/fixtures/sample_transform.json
@@ -0,0 +1,40 @@
+{
+  "transform": {
+    "enabled": true,
+    "schedule": {
+      "interval": {
+        "period": 1,
+        "unit": "Minutes",
+        "start_time": 1602100553
+      }
+    },
+    "description": "Test transform",
+    "source_index": "opensearch_dashboards_sample_data_ecommerce",
+    "target_index": "test_transform",
+    "data_selection_query": {
+      "match_all": {}
+    },
+    "page_size": 1000,
+    "groups": [
+      {
+        "terms": {
+          "source_field": "customer_gender",
+          "target_field": "gender"
+        }
+      },
+      {
+        "terms": {
+          "source_field": "day_of_week",
+          "target_field": "day"
+        }
+      }
+    ],
+    "aggregations": {
+      "quantity": {
+        "sum": {
+          "field": "total_quantity"
+        }
+      }
+    }
+  }
+}
diff --git a/cypress/integration/managed_indices_spec.js b/cypress/integration/managed_indices_spec.js
@@ -36,6 +36,8 @@ describe("Managed indices", () => {
   beforeEach(() => {
     // Set welcome screen tracking to false
     localStorage.setItem("home:welcome:show", "false");
+    // Disable jitter so ISM jobs run without an additional delay
+    cy.disableJitter();
 
     cy.wait(3000);
 

diff --git a/cypress/integration/rollups_spec.js b/cypress/integration/rollups_spec.js
@@ -236,6 +236,9 @@ describe("Rollups", () => {
 
       cy.contains(`${ROLLUP_ID}`);
 
+      // Disable button is enabled
+      cy.get(`[data-test-subj="disableButton"]`).should("not.be.disabled");
+
       // Click Disable button
       cy.get(`[data-test-subj="disableButton"]`).trigger("click", { force: true });
 

diff --git a/cypress/integration/transforms_spec.js b/cypress/integration/transforms_spec.js
@@ -0,0 +1,257 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ *
+ * Modifications Copyright OpenSearch Contributors. See
+ * GitHub history for details.
+ */
+
+import { PLUGIN_NAME } from "../support/constants";
+import sampleTransform from "../fixtures/sample_transform";
+
+const TRANSFORM_ID = "test_transform_id";
+
+describe("Transforms", () => {
+    before(() => {
+      // Delete all indices
+      cy.deleteAllIndices();
+
+      // Load ecommerce data
+      cy.request({
+        method: 'POST',
+        url:`${Cypress.env("opensearch_dashboards")}/api/sample_data/ecommerce`,
+        headers: {
+          'osd-xsrf': true
+        }
+      }).then((response) => {
+        expect(response.status).equal(200);
+      });
+    });
+
+    beforeEach(() => {
+      // delete test transform and index
+      cy.request("DELETE", `${Cypress.env("opensearch")}/test_transform*`);
+      cy.request({
+        method: 'POST',
+        url: `${Cypress.env("opensearch")}/_plugins/_transform/${TRANSFORM_ID}/_stop`,
+        failOnStatusCode: false
+      });
+      cy.request({
+        method: 'DELETE',
+        url: `${Cypress.env("opensearch")}/_plugins/_transform/${TRANSFORM_ID}  `,
+        failOnStatusCode: false
+      });
+
+      // Set welcome screen tracking to test_transform_target
+      localStorage.setItem("home:welcome:show", true);
+
+      // Visit ISM Transforms Dashboard
+      cy.visit(`${Cypress.env("opensearch_dashboards")}/app/${PLUGIN_NAME}#/transforms`);
+
+      // Common text to wait for to confirm page loaded, give up to 60 seconds for initial load
+      cy.contains("Create transform", { timeout: 60000 });
+    });
+
+    describe("can be created", () => {
+      it("successfully", () => {
+        // Confirm we loaded empty state
+        cy.contains(
+          "Transform jobs help you create a materialized view on top of existing data."
+        );
+
+        // Route to create transform page
+        cy.contains("Create transform").click({ force: true });
+
+        // Type in transform ID
+        cy.get(`input[placeholder="my-transformjob1"]`).type(TRANSFORM_ID, { force: true });
+
+        // Get description input box
+        cy.get(`textarea[data-test-subj="description"]`).focus().type("some description");
+
+        // Enter source index
+        cy.get(`div[data-test-subj="sourceIndexCombobox"]`)
+          .find(`input[data-test-subj="comboBoxSearchInput"]`)
+          .focus()
+          .type("opensearch_dashboards_sample_data_ecommerce{enter}");
+
+        // Enter target index
+        cy.get(`div[data-test-subj="targetIndexCombobox"]`)
+          .find(`input[data-test-subj="comboBoxSearchInput"]`)
+          .focus()
+          .type("test_transform{enter}");
+
+        // Click the next button
+        cy.get("button").contains("Next").click({ force: true });
+
+        // Confirm that we got to step 2 of creation page
+        cy.contains("Select fields to transform");
+
+        cy.get(`button[data-test-subj="category.keywordOptionsPopover"]`)
+          .click({ force: true });
+
+        cy.contains("Group by terms").click({ force: true });
+
+        // Confirm group was added
+        cy.contains("category.keyword_terms");
+
+        // Add aggregable field
+        cy.contains("50 columns hidden").click({ force: true });
+        cy.contains("taxless_total_price").click({ force: true });
+        // Click out of the window
+        cy.contains("Select fields to transform").click({ force: true });
+
+        cy.get(`button[data-test-subj="taxless_total_priceOptionsPopover"]`)
+          .click({ force: true });
+
+        cy.contains("Aggregate by avg").click({ force: true });
+
+        // Confirm agg was added
+        cy.contains("avg_taxless_total_price");
+
+        // Click the next button
+        cy.get("button").contains("Next").click({ force: true });
+
+        // Confirm that we got to step 3 of creation page
+        cy.contains("Job enabled by default");
+
+        // Click the next button
+        cy.get("button").contains("Next").click({ force: true });
+
+        // Confirm that we got to step 4 of creation page
+        cy.contains("Review and create");
+
+        // Click the create button
+        cy.get("button").contains("Create").click({ force: true });
+
+        // Verify that sample data is add by checking toast notification
+        cy.contains(`Transform job "${TRANSFORM_ID}" successfully created.`);
+        cy.location('hash').should('contain', 'transforms');
+        cy.get(`button[data-test-subj="transformLink_${TRANSFORM_ID}"]`);
+      });
+    });
+
+    describe("can be edited", () => {
+      beforeEach(() => {
+        cy.createTransform(TRANSFORM_ID, sampleTransform);
+        cy.reload();
+      });
+
+      it("successfully", () => {
+        // Confirm we have our initial transform
+        cy.contains(TRANSFORM_ID);
+
+        // Select checkbox for our transform
+        cy.get(`#_selection_column_${TRANSFORM_ID}-checkbox`)
+          .check({ force: true });
+
+        // Click on Actions popover menu
+        cy.get(`[data-test-subj="actionButton"]`).click({ force: true });
+
+        // Click Edit button
+        cy.get(`[data-test-subj="editButton"]`).click({ force: true });
+
+        // Wait for initial transform job to load
+        cy.contains("Test transform");
+
+        cy.get(`textArea[data-test-subj="description"]`).focus().clear().type("A new description");
+
+        // Click Save changes button
+        cy.get(`[data-test-subj="editTransformSaveButton"]`).click({ force: true });
+
+        // Confirm we get toaster saying changes saved
+        cy.contains(`Changes to transform saved`);
+
+        // Click into transform job details page
+        cy.get(`[data-test-subj="transformLink_${TRANSFORM_ID}"]`).click({ force: true });
+
+        // Confirm new description shows in details page
+        cy.contains("A new description");
+      });
+    });
+
+    describe("can be deleted", () => {
+      beforeEach(() => {
+        cy.createTransform(TRANSFORM_ID, sampleTransform);
+        cy.reload();
+      });
+
+      it("successfully", () => {
+        // Confirm we have our initial transform
+        cy.contains(TRANSFORM_ID);
+
+        // Disable transform
+        cy.get(`#_selection_column_${TRANSFORM_ID}-checkbox`).check({ force: true });
+        cy.get(`[data-test-subj="disableButton"]`).click({ force: true });
+        cy.contains(`"${TRANSFORM_ID}" is disabled`);
+
+        // Select checkbox for our transform job
+        cy.get(`#_selection_column_${TRANSFORM_ID}-checkbox`).check({ force: true });
+
+        // Click on Actions popover menu
+        cy.get(`[data-test-subj="actionButton"]`).click({ force: true });
+
+        // Click Delete button
+        cy.get(`[data-test-subj="deleteButton"]`).click({ force: true });
+
+        // Type "delete" to confirm deletion
+        cy.get(`input[placeholder="delete"]`).type("delete", { force: true });
+
+        // Click the delete confirmation button in modal
+        cy.get(`[data-test-subj="confirmModalConfirmButton"]`).click();
+
+        // Confirm we got deleted toaster
+        cy.contains(`"${TRANSFORM_ID}" successfully deleted`);
+
+        // Confirm showing empty loading state
+        cy.contains(
+          "Transform jobs help you create a materialized view on top of existing data."
+        );
+      });
+    });
+
+    describe("can be enabled and disabled", () => {
+      beforeEach(() => {
+        cy.createTransform(TRANSFORM_ID, sampleTransform);
+        cy.reload();
+      });
+
+      it("successfully", () => {
+        // Confirm we have our initial transform
+        cy.contains(TRANSFORM_ID);
+
+        // Click into transform job details page
+        cy.get(`[data-test-subj="transformLink_${TRANSFORM_ID}"]`).click({ force: true });
+
+        cy.contains(`${TRANSFORM_ID}`);
+
+        /* Wait required for page data to load, otherwise "Disable" button will
+         * appear greyed out and unavailable. Cypress automatically retries,
+         * but only after menu is open, doesn't re-render.
+         */
+        cy.wait(1000);
+
+        // Click into Actions menu
+        cy.get(`[data-test-subj="actionButton"]`).click({ force: true });
+
+        // Click Disable button
+        cy.get(`[data-test-subj="disableButton"]`).click();
+
+        // Confirm we get toaster saying transform job is disabled
+        cy.contains(`"${TRANSFORM_ID}" is disabled`);
+
+        cy.wait(1000);
+
+        // Click into Actions menu
+        cy.get(`[data-test-subj="actionButton"]`).click({ force: true });
+
+        // Click Enable button
+        cy.get(`[data-test-subj="enableButton"]`).click({ force: true });
+
+        // Confirm we get toaster saying transform job is enabled
+        cy.contains(`"${TRANSFORM_ID}" is enabled`);
+      });
+    })
+});
diff --git a/cypress/support/commands.js b/cypress/support/commands.js
@@ -157,3 +157,21 @@ Cypress.Commands.add("deleteDataStreams", (names) => {
 Cypress.Commands.add("rollover", (target) => {
   cy.request("POST", `${Cypress.env("opensearch")}/${target}/_rollover`);
 });
+
+Cypress.Commands.add("createTransform", (transformId, transformJSON) => {
+  cy.request("PUT", `${Cypress.env("opensearch")}${API.TRANSFORM_JOBS_BASE}/${transformId}`, transformJSON);
+});
+
+Cypress.Commands.add("disableJitter", () => {
+  // Sets the jitter to 0 in the ISM plugin cluster settings
+  const jitterJson = {
+    persistent: {
+      plugins: {
+        index_state_management: {
+          jitter: "0.0",
+        },
+      },
+    },
+  };
+  cy.request("PUT", `${Cypress.env("opensearch")}/_cluster/settings`, jitterJson);
+});
diff --git a/cypress/support/constants.js b/cypress/support/constants.js
@@ -26,6 +26,7 @@
 
 export const API_ROUTE_PREFIX = "/_plugins/_ism";
 export const API_ROUTE_PREFIX_ROLLUP = "/_plugins/_rollup";
+export const API_ROUTE_PREFIX_TRANSFORM = "/_plugins/_transform";
 
 export const INDEX = {
   OPENDISTRO_ISM_CONFIG: ".opendistro-ism-config",
@@ -41,6 +42,7 @@ export const API = {
   REMOVE_POLICY_BASE: `${API_ROUTE_PREFIX}/remove`,
   CHANGE_POLICY_BASE: `${API_ROUTE_PREFIX}/change_policy`,
   ROLLUP_JOBS_BASE: `${API_ROUTE_PREFIX_ROLLUP}/jobs`,
+  TRANSFORM_JOBS_BASE: `${API_ROUTE_PREFIX_TRANSFORM}`,
 };
 
 export const PLUGIN_NAME = "opensearch_index_management_dashboards";