Regularly query interfaces and push data

[ourlang]

To achieve the following functionality in Mirth Connect:

1、Periodically call the URL http://192.168.10.201:8888/queryUserList every hour to retrieve all user information from a third-party source in a paginated format (about 10,000 records).
2、Push the retrieved data to the http://192.168.10.111:19001/pushUser endpoint.

Do you have any reference example code? I don't have any clue yet。thanks

[jonbartels]

I do not have reference code but there are a few ways to do this. I assume that since queryUserList returns rows that each row is sent to pushUser one by one and not as a batch.

If pushUser tolerates a batch, reply and we can find a much simpler solution.

Option 1 - JS Reader and JS HTTP calls

Have a JS reader poll, and call the queryUserList endpoint using JS/Java HTTP calls.
The reader can generate a batch message by returning an array. Check this in the manual. I think you can return an array of messages from a JS Reader and that array is treated as a batch

Then your destination is just an HTTP sender to the pushUser endpoint

Option 2 - Two channels

Channel 1 is a JS Reader that just polls.
Destination 1 is an HTTP Sender that calls queryUserList
Destination 2 takes the response from D1, transforming as needed to csv or json or whatever, and is a channel writer sending to Channel 2

Channel 2 is a Channel Reader with a batch split set for its input data format (CSV or whatever)
Destination 1 calls pushUser

[ourlang]

Is there a simpler way

[jonbartels]

if pushUser accepts batches but only smaller batches than queryUserList maybe:

Channel 1 is a JS Reader that just polls.
Destination 1 is an HTTP Sender that calls queryUserList
Destination 2 takes the response from D1 and splits it every 2000 lines. Then D1 uses JS to call router.routeMessage to send to Channel 2

Channel 2 is a Channel Reader with NO batch split set for its input data format (CSV or whatever)
Destination 1 calls pushUser with the data

[pacmano1]

Simplest way is a JS step (anywhere) that calls the https endpoint via java in a while loop, setting the next page as needed. This can all be done in a single channel.

Start with a simple GET of one paged result and make sure in that code you check the response code and JSON.parse the received result. Make sure that works - then add the paging part and the loop. You can ask chatGPT to generate this code for you, just be specific that you are building this in MIrth and Rhino is used.

It's actually only about 40 lines of code excluding the insert of the returned rows.

[pacmano1]

I should note you can also recurse into your own channel if you don't want to use java.

So on your response transformer you would router.route a message back to the current channel with the next page to get in the message itself, or via message meta data. They key there is the message format needs to match the type of message you are current generating in your source. You need to set the source queue on for that technique.

[ourlang]

Thank you all for providing ideas 。 We have found a solution that may not be very effective

[ourlang]

channel one GetDataChannel

channel two PushDataChannel

The above method can achieve the function, but it doesn't feel very good

[ourlang]

I originally intended to use router.routeMaessage in Source JavaScript
Then select Channel Writer for data forwarding in Destination, but it was unsuccessful, similar to the following

[ourlang]

Thank you all for providing ideas