feat(mrf-admin-view): hawkeye v1.0 #7965

kevin9foong · 2024-12-02T09:35:28Z

Problem

Currently, admins want a way to track the status of their multi-respondent submissions/workflows to see:

is the workflow completed / pending?
what is the current step of the workflow?

Closes FRM-1919

Solution

Add the above-mentioned information to the responses dashboard, csv download and individual responses page so admins can track the status.

Breaking Changes

No - this PR is backwards compatible

Before & After Screenshots

BEFORE:

AFTER:

New columns added and new name for timestamp column.

Tests

TC1: Hawk eye view dashboard shows the new status, current step and submission time of first step

Create 2 step MRF form without approval step
Submit step 1
Check responses dashboard, individual submission and download CSV. It should show pending status and correct step which is 1 of 2.
Submit step 2.
Check responses dashboard, individual submission and download CSV. It should show completed status and correct step which is 2 of 2.

TC2: Reject and approve show correctly.

Create a 3 step form with step 2 being approval step. Also, add a compulsory attachment field at step 1. When submitting the form in subsequent steps of the test case, always upload an attachment.
Submit step 1 and 2 of form, rejecting at step 2. (ie click No)
Check responses dashboard, individual submission and download CSV. It should show completed status and correct step which is 2 of 3.
Try download CSV with attachment. Ensure the csv file is correctly downloaded and the data in the csv is as expected and attachment files are downloaded.
Submit step 1, 2 and 3 of form, ensuring that step 2 is approve (ie click Yes)
Check responses dashboard, individual submission and download CSV. It should show completed status and correct step which is 3 of 3.
Try download CSV with attachment. Ensure the csv file is correctly downloaded and the data in the csv is as expected and attachment files are downloaded.

TC3: Storage mode and storage mode with with payments unaffected.

Create a storage mode form.
Submit and check the dashboard. Ensure status, current step and submission time of first step do not exist in the dashboard and the other expected columns exist (ie, same as before the changes introduced in this PR / refer to prod env)
Download the CSV, ensure that status, current step and submission time of first step do not exist in the csv.
Add payments to the storage mode form.
Submit and check the dashboard. Ensure status, current step and submission time of first step do not exist and the dashboard is same as previous (ie, same as before the changes introduced in this PR / refer to prod env).

datadog-opengovsg · 2024-12-09T15:57:02Z

Datadog Report

Branch report: feat/hawkeye-v1.0
Commit report: 66b411c
Test service: formsg

✅ 0 Failed, 2150 Passed, 1 Skipped, 5m 6.71s Total duration (1m 39.78s time saved)

kevin9foong · 2024-12-10T06:24:37Z

To reduce duplicate for the Step X of Y string into a common util function
To explore reusing types mrfMetadata

kevin9foong · 2024-12-13T03:37:51Z

shared/types/submission.ts

+
+const SubmittedNonApprovalStep = z.object({
+  isApproval: z.literal(false),
+  submittedAt: z.string(),


does zod have a timestamp type for validation? if so, maybe we can use it

yes - fixed referring to zod's datetime documentation with submittedAt: z.string().datetime({ precision: 3 })

kevin9foong · 2024-12-13T12:59:19Z

frontend/src/features/admin-form/responses/common/utils/mrfSubmissionView.ts

@@ -0,0 +1,26 @@
+import { WorkflowStatus } from '~shared/types'
+
+/** Gets the business friendly string for current step of workflow. */


these are helper functions to generate the view for mrf submissions responses

kevin9foong · 2024-12-13T13:00:13Z

frontend/vite.config.ts

+    define: {
+      // On local dev, global is undefined, causing decryption workers to fail.
+      // This is a workaround based on https://github.com/vitejs/vite/discussions/5912.
+      global: 'globalThis',


used the bypass an error faced as described by the comment.

I experimented with removing the whole global access chunk in decryption.worker.ts and it seemed to have resolved the csv generation issue on local.

Might want to look into that solution since I would prefer deleting code than to have more config.

removed the global reference + vite config, tested and it works well!

kevin9foong · 2024-12-13T13:03:06Z

src/app/modules/submission/submission.utils.ts

+/**
+ * Builds the metadata for a multirespondent form submission.
+ */
+export const buildMrfMetadata = ({


mrfMetadata refers to data which is not stored but derived from stored data when buildMrfMetadata is executed. This metadata is used in the submission and submissionMetadata requests to display metadata info about the submission in the responses table, downloaded csv file and individual submission response detail page.

tested to generate the correct metadata given the submission data

kevin9foong · 2024-12-13T13:41:22Z

...s/ResponsesPage/storage/utils/EncryptedResponseCsvGenerator/EncryptedResponseCsvGenerator.ts

@@ -112,7 +112,7 @@ export class EncryptedResponseCsvGenerator extends CsvGenerator {
    if (this.hasBeenProcessed) return

    // Create a header row in CSV using the fieldIdToQuestion map.
-    const headers = this.isMrf ? MRF_CSV_HEADERS : NON_MRF_CSV_HEADERS
+    const headers = this.isMrf ? [...MRF_CSV_HEADERS] : [...NON_MRF_CSV_HEADERS]


Note: The de-structuring is necessary as we are mutating the headers array below. De-structuring creates a new array which prevents mutations from affecting the referenced const array (ie, MRF_CSV_HEADERS or NON_MRF_CSV_HEADERS).

Otherwise, headers points to the MRF_CSV_HEADERS or NON_MRF_CSV_HEADERS const array and mutates the same memory address which is shared across multiple runs of process(). This causes issues in cases where the memory is shared/reused,specifically test cases, which causes the headers to contain the headers from the previous runs of process() and for the test cases to fail.

Good explanation to highlight that we need to create new array as we are mutating it at a later stage. Would recommend to put a short note in the code.

left a NOTE and TODO

kevin9foong · 2024-12-13T14:36:50Z

...min-form/responses/ResponsesPage/storage/UnlockedResponses/ResponsesTable/ResponsesTable.tsx

@@ -193,7 +194,8 @@ const MRF_RESPONSE_TABLE_COLUMNS: Column<ResponseColumnData>[] = [
    Header: MRF_FIRST_STEP_TIMESTAMP_LABEL,
    accessor: 'submissionTime',
    width: 250,
-    disableResizing: true
+    minWidth: 250,


Note: re-using the same fix as existing storage mode responses table columns to include minWidth which allows the bar to expand for the rightmost column when the screen is small and horizontal scroll is enabled.

linear · 2024-12-13T17:23:22Z

FRM-1919 MRF | Hawkeye v1.0

...min-form/responses/ResponsesPage/storage/UnlockedResponses/ResponsesTable/ResponsesTable.tsx

KenLSM · 2024-12-20T05:47:02Z

frontend/src/features/admin-form/responses/common/utils/mrfSubmissionView.ts

+      return MRF_STATUS.COMPLETED
+    case WorkflowStatus.PENDING:
+      return MRF_STATUS.PENDING
+    default:


/nit since it is likely that all the WorkflowStatus is fully defined, we would want the typesystem to let us know that we need to update the cases here rather than it falling through to default.

made it more explicit for undefined to become '' and explicitly specified return literals of the function.

...s/ResponsesPage/storage/utils/EncryptedResponseCsvGenerator/EncryptedResponseCsvGenerator.ts

src/app/models/submission.server.model.ts

src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts

KenLSM · 2025-02-05T06:32:07Z

frontend/src/features/admin-form/responses/common/utils/mrfSubmissionView.ts

  switch (workflowStatus) {
    case WorkflowStatus.COMPLETED:
    case WorkflowStatus.APPROVED:
    case WorkflowStatus.REJECTED:
      return MRF_STATUS.COMPLETED
    case WorkflowStatus.PENDING:
      return MRF_STATUS.PENDING
-    default:
+    case undefined:


Should still have a default case for fall-through, recommend to utilise exhaustive switch

Check for undefined should be outside of switch case. Could be something before line 21.

KenLSM

See comments on exhaustive switch in frontend/src/features/admin-form/responses/common/utils/mrfSubmissionView.ts. That should be the last item before we're good to go!

kevin9foong requested a review from KenLSM December 2, 2024 09:35

kevin9foong self-assigned this Dec 2, 2024

kevin9foong marked this pull request as draft December 2, 2024 09:35

kevin9foong requested a review from scottheng96 December 2, 2024 13:52

kevin9foong force-pushed the feat/hawkeye-v1.0 branch from 342a9e8 to 73eaece Compare December 11, 2024 09:36

kevin9foong commented Dec 13, 2024

View reviewed changes

kevin9foong temporarily deployed to staging-alt2 December 13, 2024 03:43 — with GitHub Actions Inactive

kevin9foong force-pushed the feat/hawkeye-v1.0 branch from db66d56 to b492151 Compare December 13, 2024 12:52

kevin9foong commented Dec 13, 2024

View reviewed changes

kevin9foong added 16 commits December 13, 2024 22:32

feat: add columns for mrf responses page

1c304c4

feat: fix default paddings for table

9f89d28

chore: update comment

42ae99a

feat: pipe step number to FE response table

c5c5cdc

feat: log step meta and pipe to fe response table

b817a59

fix: checking is form rejected error

f213752

feat: add status badges

6983771

feat: add csv download col

3614690

fix: add define global

936c973

feat: support cols in csv

009f840

fix: workflowStep being 0 is defined as falsy

c276fa4

feat: add approved rejected status in individualresponse

d5f044e

feat: map workflow status to business friendly status

6a28c16

fix: change imports to relative

8a8ddac

fix: failing unit tests

a8bdd88

feat: add datetime validation for submittedAt

609ccec

kevin9foong force-pushed the feat/hawkeye-v1.0 branch from fae67dc to c60e95a Compare December 13, 2024 14:34

fix: bug where last col does not expand in scrollable

75ac070

kevin9foong force-pushed the feat/hawkeye-v1.0 branch from c60e95a to 75ac070 Compare December 13, 2024 14:35

kevin9foong commented Dec 13, 2024

View reviewed changes

kevin9foong added 2 commits December 13, 2024 23:08

feat: add tc for testing the metadata builder

90eb8e5

feat: add tc for appending mrf meta to mrf submission stream

2384693

kevin9foong force-pushed the feat/hawkeye-v1.0 branch from 448f2d0 to 2384693 Compare December 13, 2024 15:08

kevin9foong added 7 commits December 13, 2024 23:32

feat: add tc for submission.server.model to fetch mrf meta

cd734b9

feat: add tc for csv generator for mrf

3d80ac3

feat: fix tc missing required fields

2139b9a

fix: bug where approved is marked as completed instead

c6eb27e

fix: submission.server.model tc

90d375b

fix: import ordering for lint

15f6d2f

feat: add chromatic story for mrf dashboard

2b9ca5d

kevin9foong marked this pull request as ready for review December 13, 2024 17:23

chore: remove redundant comment

0ef1e49