Add paging to f1-j1-getPrimeroCasesV2.js

[aleksa-krolls]

Request

Primero API requests are now limited to a maximum of 1,000 records. While it's rare that we an API query returns more than 1,000 records, we should proactively update our OpenFn jobs to consider this limit and to handle paging if more than 1k records do exist.

In the job V2/f1-j1-getPrimeroCasesV2.js, we need to (1) change the parameter per to one thousand per: 1000 and then (2) handle scenarios where multiple pages of data may be returned

expression.js

Please branch off main and update the job V2/f1-j1-getPrimeroCasesV2.js . MAKE SURE TO ONLY UPDATE JOB IN V2 FOLDER:
https://github.com/OpenFn/unicef-cambodia/blob/master/jobs/V2/f1-j1-getPrimeroCasesV2.js

Query to update:

unicef-cambodia/jobs/V2/f1-j1-getPrimeroCasesV2.js

Lines 19 to 29 in 57237db

	getCases(
	{
	remote: true,
	last_updated_at: state => `${state.cursor}..`,
	//last_updated_at: '2023-08-24T00:00:00.862Z..', //for troubleshooting
	//last_updated_at: state => `${state.cursor}..2023-08-14T15:00:00.862Z`, //for troubleshooting
	page: 1,
	per: 10000,
	// These cases have been recently updated and MIGHT have a new referral to send to Oscar; we check & filter below
	// workflow: 'referral_to_oscar', //REMOVED July '23 bc we should rely on services, not case-level statuses
	},

state

Primero (to generate state):

{
"configuration": "Primero V2 Cambodia Demo API user",
"data": {}
}

To test

Feel free to run locally. When ready to test, pls merge to the staging branch so that we can test on the Staging platform project: https://www.openfn.org/projects/primero-oscar-cambodia-io-staging/jobs/jywaga

Toggl

UNICEF Cambodia Primero IO Support [2024]

[mtuchi]

@aleksa-krolls should i branch off from main or staging ?, Because the project link on v1 uses staging branch and not main

[aleksa-krolls]

hey @mtuchi pls branch off main... and we can overwrite staging with your changes. (I don't trust what's currently on staging... so we should "refresh" that branch.)

[mtuchi]

Hiya @aleksa Krolls there is a potential performance implication when using getCases with the { withReferrals: true } What this does is fetch referrals for each retrieved case which can take a while if there is a lot of cases fetched.

I am currently experimenting pagination with a small batch, I will share more tomorrow

[aleksa-krolls]

@mtuchi feel free to query in smaller batches (less than 1000)... we have long-running jobs enabled on platform

[mtuchi]

@josephjclark This is ready for code review now, See #131

[mtuchi]

@aleksa-krolls i have tested the changes locally with the lastRunDateTIme: 2024-06-20T07:12:25.383Z. Although i am getting cases from primero but when we filter to get oscarRefs i am getting empty results. Let me know if you need to sync to test this.

FYI. The changes in #131 uses @openfn/[email protected]

[aleksa-krolls]

hey @mtuchi fyi I've asked @daissatou2 for help testing here - and then I will do a final review/merge to prod when I get back online

[daissatou2]

@mtuchi I want to test this on the staging project but it looks like the PR was created against main. Can you please create the PR against staging?

[mtuchi]

@daissatou2 i think we need to overwrite what is on staging right now because i did ask @aleksa-krolls which branch to use and she suggested we use main

[daissatou2]

@aleksa-krolls we tested this together locally and it passed. I created a case & referral on Primero and we ran the workflow. We did not merge the PR per Mtuchi's last comment.

[aleksa-krolls]

@mtuchi as discussed, pls refresh staging branch based on the main branch... and then we can merge your PR to staging

@aleksa-krolls i have refreshed staging branch and change the branch base from master to stagin in #131