Skip to content

Commit

Permalink
Smart bulk fhir (openemr#4204)
Browse files Browse the repository at this point in the history
* migrate patientrole to main fhir api route

* Centralized scope checks for rest api. Perm update

Updated the scope permissions so that only the ones we currently support
for patient/<resource>.* is supported.

Centralized the scope checks in the HttpRestRouteHandler to remove
redundant code from the routes and to make sure all possible routes are
checked against the access code.  This also prevents developers who
extend the api from accidently forgetting to check against the
AccessToken.

For now I've only enabled it on the FHIR api, but if it tests well we
can open it up to the rest of the APIs.

* Standalone SMART response handler

Fixed some scope permission checks.  Refactored the route parsing
algorithm into its own class that can be unit tested against.  The
parsing logic could then be leveraged in the scope auth check which made
matching against the REST FHIR resource a lot easier.

Added the additional SMART capabilities we now support with patient
standalone and launch standalone.

Fixed the refresh token issues.  We don't send patient context
parameters as part of the refresh_grant oauth2 flow so we only send the
parameters now in the authorization_grant flow inside our SMARTResponse
object.  There may be a better way to make this work, but for now this
is functioning.

* Fix unit tests and style problems.

* Fix patient context missing for standalone.

Fixing the refresh token issues broke the patient context missing due to
the way the ResponseType object was cloned for the League
AuthorizationController.

* Initial Implementation of JWK Client Credentials Grant

* Implemented Bulk FHIR Client Credentials Grant

Extended the SMART app registration to be able to send a JSON Web Key
Set (JWKS) URI or embed your jwks directly to be able to use system
permissions.

Implemented a JWT signer using the RS384 algorithm per Bulk FHIR spec
incorporating code from the jumbojett/OpenID-Connect-PHP project to
handle the signing.

Implemented a JsonWebKeySet that will retrieve JWKS from a JWK URI or
hold the JWKS to be used for signature verification.

Wrote Test class to verify the JWK signature verification as well as
illustrate how to use class mock's to improve unit testing of our
services.  This will improve testing time in the future as we can proxy
network requests without actually having to hit the network.

We still have outstanding the following requirements for OAUTH2
authorization.
- spec requirement to combine and validate keys against both JWK URI &
client provided keys at registration.  Currently does one or the other.
- Prevent replay attacks by checking against previous JWT jti values for the
current issuer (client id) for the 1 minute expires period.
- Check optional jku value is the same as the JWK URI provided at registration.

That said the current implementation meets all the authorization
testing checks of the ONC inferno tool for the Bulk FHIR requirements.

* Initial support for FHIR Group $export operation.

Refactored the Capability REST operation to use the FHIR object classes
so we could handle the FHIR operation definition types for group-export,
patient-export and the system export.

Fixed the ScopeRepo to handle the updated capability rest object.

* Bulk FHIR Export authentication,api requests

Initial Implementation of the Export Controller to handle Bulk FHIR Export.

Put in the initial $export routes for the Group/Patient/System exports of
data for the BULK FHIR implementation.

Refactored the Capability statement to now support FHIR operations on each
FHIR resource.  Moved the Capability statement to use the FHIR class
resources for the REST response.  This fixed a number of issues in the FHIR
validator for inferno.

Pulled out the oauth2_trusted_user api calls into its own controller so we
can grant a trusted user for Client Credentials grant.

Refactored dispatch.php in order to allow a trusted client access token that
is not tied to a user to still have api access for system calls.

Refactored the hard dependancy on RestConfig in the ScopeRepository so I could
write a unit test against ScopeRepository to fix some scope bugs that were
occurring.

* Fix php 7.4+ error

If the kid is missing it throws an error on php 7.4+

* Export scopes, scope escalation attack fix.

Made it so a the finalizeScope's check removes any scopes from a
permission grant that were not there at the time of client registration.
Before this a client could request and get granted whatever scopes they
wanted as long as the system supported it.

Also added the operation $export scopes needed for Bulk FHIR export.

* Add System User for Bulk FHIR Export.

Based on Brady & Jerry Padgett's recommendation's I've gone head and
added a system user to tie the bulk fhir export to.  I added them to the
Admin ACL which allows ACL's to be adjusted in the future.

* FHIR Export Service architecture

Implemented a service locator to retrieve all of the fhir resource
services that support exporting data.  Created an interface that
services can implement that support exporting data.  All of the US Core
resources will need to implement the interface.

* Improved documentation for library sql command.

* SQL definitions for export jobs.

Export Job represents the Bulk FHIR export requests we need to process.

* Support FHIR operation routes

Made it so the route parser will handle operations just fine.  The http
rest route handler will also now check against operation permissions.
SMART scopes permissions are represented as
system/<resource>.$<operation-name> for example a bulk group export is
system/Group.$export.  A root level operation permission is represented
by a star.  For example a root export is system/*.$export.

Enabled the ScopeRepository to sue system/*.$export as well as
system/*.$bulkdata-status (this idea came from the IBM FHIR server
implementation where they are using $bulkdata-status as their reference
URL.

Also put in place a unit test for the route parser as the route parsing
has gotten more complicated and errors were cropping up, so I put that
in place to try and avoid future problems.

* SMART FHIR Group Compartment Export

Got the SMART FHIR Group Compartment Implementation Guide works.

Added the SystemUser to the UserService and got it so the Client
Credentials grant uses the SystemUser and is checked properly in the api
dispatch file.

Wrote a wrapper class around our embedded Psr17Factory so we can replace
it as needed.

Created a StatusCode class that we can start putting HTTP status in
which makes unit testing and readability much better.

Wrote an ExportJob class to match with the database export_job and that
the FhirExportRestController creates and manages the life cycle of.

Wrote the controller class to properly handle the bulk $export and the
$export status checks.

Right now the file creation of the exports and the deletion of the
exports is mocked out and need to be implemented in a subsequent
iteration.

Minor style guide fixes included as well.

* ONC Client Credentials Grant fixes

Fix JWK validation and use the OpenEMR API SystemUser for the grants

* Prevent registration of insecure clients.

Clients without any JWKs are prohibited.  Client's that are public
cannot have user or system scopes.

* Style fix.

* Fix reviews, remove unused code.

* Make the additional user install after gacl

* Fix namespace use statement.

* Ref openemr#4203 Bulk FHIR Resource Export

Implemented FHIR Document Binary downloads
- Built it so that documents can now be downloaded via FHIR Made it so the document service checks against the category ACLs to allow more fine grained access control on documents. This made it so FHIR export documents require the super user ACL in order to be retrieved from the system.

- Wrote a BaseDocumentDownloader class that is generic and should handle every mime type.  If special mimetype handling is needed someone can implement the IDocumentDownloader interface to handle a specific mime type.

Added an expiration date to Documents
- Documents cannot be downloaded via the FHIR api if they have been marked as deleted or past their expiration date.

- Made additional minor adjustments to the Document class. Added a helper method to Document so you can retrieve the document data using the class method get_data().  It will retrieve it from CouchDB (if enabled) or from the file system. Data is decrypted if you pass the decrypt flag into the function.

- Added a can_access, has_expired, and get_categories helper methods to Document

Built all of the Export FHIR functionality.
- In order for the export to work you have to have the 'Enable OpenEMR Standard FHIR Export API' flag turned on.  By default it is off for security purposes.

- Any FHIR service that wishes to export FHIR resources just needs to implement the IFhirExportableResourceService and reside in the OpenEMR\Services\FHIR namespace.

- Services can state whether they support the patient, group, or system level exports.  Any resource service that is inside the US Core Patient Compartment (https://www.hl7.org/fhir/compartmentdefinition-patient.html) should implement the patient/group export functionality.  Services for FHIR resources outside patient/group should return false for the patient / group methods.

- The FhirPatientService and FhirEncounterService are good reference examples for how to handle an export.  You can send any FHIR resource that extends the FhirResource base class to the ExportWriter and it will handle the data just fine.

- Right now FHIR exports are handled syncronously as the export is all done at the time of the initial export creation.  We assume the OpenEMR server can process this very fast. However, all the code is setup so that this can be moved asynchronously.  Each export has a shutdown time that tracks the last resource exported.  We would need to create a table to store the in progress data but the code should be fairly straightforward to be able to handle this for long running batch jobs.

- The document exports are all handled in memory right now.  If this becomes too much memory usage the ExportStreamWriter is built so it can take a stream.  I intended to add a streaming capability for the Document class but I ran out of time. The export to be more memory efficient can take a stream (such as one from fopen for local file system) and data can be written out that way.  I'm not too sure on how that would work for CouchDB.

Did a bunch of rework on the FhirPatientService for searching.  There's a lot of rework that needs to happen here as we aren't following much
of the specification for searching.  Added in the search parameter types so we can handle our search capability statement correctly.
Also any FHIR resource that is one of the 13 US Core Implementation Guide resources should implement the IResourceUSCIGProfileService so that
we can include the USCIG profiles in the capability statement.

* Implement DELETE for Bulk FHIR

* Style fixes

* Fixed an issue where patient export wasn't working

* Added some documentation around export

* Fix string version of JWKs

* Fix unit tests to handle export global flag

* Fix styles

* Fix test cases with missing root user.

Not sure why the systemuser is missing on the test runner.  However,
mocked the interface so the test doesn't fail, maybe in this use case
there is no database access?  Need to check with @bradymiller about it.

* USCIG core fixes to Patient FHIR resource

Added required address period field.  Partially implemented ethnicity and race
extensions. Added required communication field.

The FHIR specification is inconsistent in the SystemURIs for the
language communication, ethnicity and race.  Different documents state
different URIs for this and I can't seem to find what the correct one
is.  For now its just throwing warnings in inferno and not failing the
test.  We'll need to figure out the correct one, though it seems like
its a moving target as some of the checker's changed as of mid January.

* OpenEMR CLI runner, Client Grant Testing Tool.

Implemented a basic CLI command runner that can be used to execute CLI
commands in the OpenEMR source code ecosystem.  Developers can implement
the IOpenEMRCommand interface and put their Command inside the
src/Common/Command directory and it will be picked up by the command
runner.  The command runner only runs in the cli space so it can't be
picked up by apache.  To list the commands you can execute just run
./bin/command-runner -l

Created the CreateClientCredentialsAssertion command to help users test
and use Client Credential Grants.  Users can use the command to grab
a test OpenEMR JWKS to register at /interface/smart/register-app.php in the JWKS
field.  To get the public keys you just execute
./bin/command-runner -c CreateClientCredentialsAssertion -k

To create a Client Credentials Grant Assertion for using in CURL or with
something like postman you can execute the following (assuming you are
running the docker containers) ./bin/command-runner -c CreateClientCredentialsAssertion -i <ClientID> -a https://localhost:9300/oauth2/default/token

The command will spit out the assertion grant and a CURL request you can execute to get back an access token.

* Setup update now installs additional users.

* Fix Download URL and decryption.

Made the download URL work even if the system has been migrated per
bradymiller's suggestions.

Also defaulted the content to return decrypted data and handled
decrypting the couch db content.

* System scope restrictions, export verifies scopes.

Made the export actually verify the user agent has access to the
resources that are exported using the scopes requested.  Disabled the
system scopes unless a global flag is turned on (bigger restriction than
originally just restricting the export).

Fixed a bug in the AuthorizationController where our checks against
public/private apps and the scopes they were allowed were NOT actually
being used.  Had to fix some of the api tests to handle that correctly.

* Make uuid unique key per code review request

* Fix styles, Fix FHIR Categories.

* Fix database categories

* Make sure SystemUser is setup for Credentials Grant.

* Implement sqlThrowsException method

Made it so the export controller doesn't die if there is a sql error.
Instead an exception is thrown and the API returns properly.

* More minor pr fixes.

* Fix styles.

* Add foreign db key to Documents. Fix document bugs.

Added a foreign reference id & table name to the Documents class so we
can tie a Document to another table record.  Apparently the 'foreign_id'
only is used for Patients per @bradymiller.

I also fixed a bug where the deleted flag was not actually picked up by
the document ORM since it was missing the getters and setters.

Finally fixed a bug in the FhirDocumentRestController that was allowing
access to deleted documents that had not yet expired.
  • Loading branch information
adunsulag authored Feb 10, 2021
1 parent 1d9c3b8 commit 726526a
Show file tree
Hide file tree
Showing 75 changed files with 6,287 additions and 387 deletions.
11 changes: 11 additions & 0 deletions API_README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ Finally, APIs which are integrated with the new `handleProcessingResult` method
- [Authorization Code Grant](API_README.md#authorization-code-grant)
- [Refresh Token Grant](API_README.md#refresh-token-grant)
- [Password Grant](API_README.md#password-grant)
- [Client Credentials Grant](API_README#client-credentials-grant)
- [Logout](API_README.md#logout)
- [More Details](API_README.md#more-details)
- [Standard API Endpoints](API_README.md#api-endpoints)
Expand Down Expand Up @@ -311,6 +312,14 @@ Response:
}
```

### Client Credentials Grant

This is an advanced grant that uses JSON Web Key Sets(JWKS) to authenticate and identify the client. This credential grant is
required to be used for access to any **system/\*.$export** scopes. API clients must register either web accessible JWKS URI that hosts
a RSA384 compatible key, or provide their JWKS as part of the registration. Client Credentials Grant access tokens are short
lived and valid for only 1 minute and no refresh token is issued. Tokens are requested at `/oauth2/default/token`
To walk you through how to do this process you can follow [this guide created by HL7](https://hl7.org/fhir/uv/bulkdata/authorization/index.html).

#### Logout

A grant (both Authorization Code and Password grants) can be logged out (ie. removed) by url of `oauth2/<site>/logout?id_token_hint=<id_token>`; an example full path would be `https://localhost:9300/oauth2/default/logout?id_token_hint=<id_token>`. Optional: `post_logout_redirect_uri` and `state` parameters can also be sent; note that `post_logout_redirect_uris` also needs to be set during registration for it to work.
Expand Down Expand Up @@ -1722,3 +1731,5 @@ Response:
- For business logic, make or use the services [here](src/Services)
- For controller logic, make or use the classes [here](src/RestControllers)
- For routing declarations, use the class [here](_rest_routes.inc.php).

-
55 changes: 55 additions & 0 deletions FHIR_README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ Database Result -> Service Component -> FHIR Service Component -> Parse OpenEMR
- [Authorization Code Grant](API_README.md#authorization-code-grant)
- [Refresh Token Grant](API_README.md#refresh-token-grant)
- [Password Grant](API_README.md#password-grant)
- [Client Credentials Grant](API_README.md#client-credentials-grant)
- [Logout](API_README.md#logout)
- [More Details](API_README.md#more-details)
- [FHIR API Endpoints](FHIR_README.md#fhir-endpoints)
Expand All @@ -58,6 +59,9 @@ Database Result -> Service Component -> FHIR Service Component -> Parse OpenEMR
- [Location](FHIR_README.md#location-resource)
- [CareTeam](FHIR_README.md#careTeam-resource)
- [Provenance](FHIR_README.md#Provenance-resources)
- [System Export](FHIR_README.md#SystemExport-resource)
- [Patient Export](FHIR_README.md#PatientExport-resource)
- [Group Export](FHIR_README.md#GroupExport-resource)

### Prerequisite

Expand Down Expand Up @@ -602,3 +606,54 @@ Provenance resources are requested by including `_revinclude=Provenance:target`
```sh
curl -X GET 'http://localhost:8300/apis/default/fhir/AllergyIntolerance?_revinclude=Provenance:target'
```
### BULK FHIR Exports
An export operation that implements the [BULK FHIR Export ONC requirements](https://hl7.org/fhir/uv/bulkdata/export/index.html) can be requested by issuing a GET request to the following endpoints:
- System Export, requires the **system/\*.$export** scope. Exports All supported FHIR resources
```sh
curl -X GET 'https://localhost:9300/apis/default/fhir/$export'
```
- Group Export, requires the **system/Group.$export** scope. Exports all data in the [Patient Compartment](https://www.hl7.org/fhir/compartmentdefinition-patient.html) for the group.
There is only one group defined in the system currently. If OpenEMR defines additional patient population groups you would change the Group ID in the API call.
```sh
curl -X GET 'https://localhost:9300/apis/default/fhir/Group/1/$export'
```
- Patient Export, requires the **system/Group.$export** scope. Exports all data for all patients in the [Patient Compartment](https://www.hl7.org/fhir/compartmentdefinition-patient.html).
```sh
curl -X GET 'https://localhost:9300/apis/default/fhir/Patient/$export'
```
You will get an empty body response with a **Content-Location** header with the URL you can query for status updates on the export.

To query the status update operation you need the **system/\*.$bulkdata-status** scope. An example query:
- Status Query
```sh
curl -X GET 'https://localhost:9300/apis/default/fhir/$bulkdata-status?job=92a94c00-77d6-4dfc-ae3b-73550742536d'
```

A status Query will return a result like the following:
```
{
"transactionTime": {
"date": "2021-02-05 20:48:38.000000",
"timezone_type": 3,
"timezone": "UTC"
},
"request": "\/apis\/default\/fhir\/Group\/1\/%24export",
"requiresAccessToken": true,
"output": [
{
"url": "https:\/\/localhost:9300\/apis\/default\/fhir\/Document\/97552\/Binary",
"type": "Patient"
},
{
"url": "https:\/\/localhost:9300\/apis\/default\/fhir\/Document\/105232\/Binary",
"type": "Encounter"
}
],
"error": []
}
```

You can download the exported documents which are formatted in Newline Delimited JSON (NDJSON) by making a call to:
```sh
curl -X GET 'https://10.0.0.9:9300/apis/default/fhir/Document/105232/Binary'
```
81 changes: 63 additions & 18 deletions _rest_config.php
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,14 @@
use Nyholm\Psr7Server\ServerRequestCreator;
use OpenEMR\Common\Acl\AclMain;
use OpenEMR\Common\Auth\OpenIDConnect\Repositories\AccessTokenRepository;
use OpenEMR\Common\Http\HttpRestRequest;
use OpenEMR\Common\Logging\EventAuditLogger;
use OpenEMR\Common\Logging\SystemLogger;
use OpenEMR\Common\Session\SessionUtil;
use OpenEMR\Common\Uuid\UuidRegistry;
use OpenEMR\Services\TrustedUserService;
use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\ServerRequestInterface;


// also a handy place to add utility methods
// TODO before v6 release: refactor http_response_code(); for psr responses.
//
Expand Down Expand Up @@ -216,20 +216,23 @@ public static function verifyAccessToken()

public static function isTrustedUser($clientId, $userId)
{
$trustedUserService = new TrustedUserService();
$response = self::createServerResponse();
try {
$trusted = sqlQueryNoLog("SELECT * FROM `oauth_trusted_user` WHERE `client_id`= ? AND `user_id`= ?", array($clientId, $userId));
if (empty($trusted['session_cache'])) {
if (!$trustedUserService->isTrustedUser($clientId, $userId)) {
(new SystemLogger())->debug(
"invalid Trusted User. Refresh Token revoked or logged out",
['clientId' => $clientId, 'userId' => $userId]
);
throw new OAuthServerException('Refresh Token revoked or logged out', 0, 'invalid _request', 400);
}
return $trustedUserService->getTrustedUser($clientId, $userId);
} catch (OAuthServerException $exception) {
return $exception->generateHttpResponse($response);
} catch (\Exception $exception) {
return (new OAuthServerException($exception->getMessage(), 0, 'unknown_error', 500))
->generateHttpResponse($response);
}

return $trusted;
}

public static function createServerResponse(): ResponseInterface
Expand Down Expand Up @@ -276,9 +279,9 @@ public static function getPostData($data)
return null;
}

public static function authorization_check($section, $value): void
public static function authorization_check($section, $value, $user = ''): void
{
$result = AclMain::aclCheckCore($section, $value);
$result = AclMain::aclCheckCore($section, $value, $user);
if (!$result) {
if (!self::$notRestCall) {
http_response_code(401);
Expand Down Expand Up @@ -366,18 +369,30 @@ public static function skipApiAuth($resource): bool

public static function apiLog($response = '', $requestBody = ''): void
{
$logResponse = $response;

// only log when using standard api calls (skip when using local api calls from within OpenEMR)
// and when api log option is set
if (!$GLOBALS['is_local_api'] && !self::$notRestCall && $GLOBALS['api_log_option']) {
if ($GLOBALS['api_log_option'] == 1) {
// Do not log the response and requestBody
$response = '';
$logResponse = '';
$requestBody = '';
}
if ($response instanceof ResponseInterface) {
if (self::shouldLogResponse($response)) {
$body = $response->getBody();
$logResponse = $body->getContents();
$body->rewind();
} else {
$logResponse = 'Content not application/json - Skip binary data';
}
} else {
$logResponse = (!empty($logResponse)) ? json_encode($response) : '';
}

// convert pertinent elements to json
$requestBody = (!empty($requestBody)) ? json_encode($requestBody) : '';
$response = (!empty($response)) ? json_encode($response) : '';

// prepare values and call the log function
$event = 'api';
Expand All @@ -393,7 +408,7 @@ public static function apiLog($response = '', $requestBody = ''): void
'request' => $GLOBALS['resource'],
'request_url' => $url,
'request_body' => $requestBody,
'response' => $response
'response' => $logResponse
];
if ($patientId === 0) {
$patientId = null; //entries in log table are blank for no patient_id, whereas in api_log are 0, which is why above $api value uses 0 when empty
Expand Down Expand Up @@ -421,30 +436,60 @@ public static function emitResponse($response, $build = false): void
echo $response->getBody();
}

public function authenticateUserToken($tokenId, $userId): bool
/**
* If the FHIR System scopes enabled or not. True if its turned on, false otherwise.
* @return bool
*/
public static function areSystemScopesEnabled()
{
return $GLOBALS['rest_system_scopes_api'] === '1';
}

public function authenticateUserToken($tokenId, $clientId, $userId): bool
{
$ip = collectIpAddresses();

// check for token
$authToken = sqlQueryNoLog("SELECT `expiry` FROM `api_token` WHERE `token` = ? AND `user_id` = ?", [$tokenId, $userId]);
if (empty($authToken) || empty($authToken['expiry'])) {
EventAuditLogger::instance()->newEvent('api', '', '', 0, "API failure: " . $ip['ip_string'] . ". Token not found for " . $userId . ".");
$accessTokenRepo = new AccessTokenRepository();
$authTokenExpiration = $accessTokenRepo->getTokenExpiration($tokenId, $clientId, $userId);

if (empty($authTokenExpiration)) {
EventAuditLogger::instance()->newEvent('api', '', '', 0, "API failure: " . $ip['ip_string'] . ". Token not found for client[" . $clientId . "] and user " . $userId . ".");
return false;
}

// Ensure token not expired (note an expired token should have already been caught by oauth2, however will also check here)
$currentDateTime = date("Y-m-d H:i:s");
$expiryDateTime = date("Y-m-d H:i:s", strtotime($authToken['expiry']));
$expiryDateTime = date("Y-m-d H:i:s", strtotime($authTokenExpiration));
if ($expiryDateTime <= $currentDateTime) {
EventAuditLogger::instance()->newEvent('api', '', '', 0, "API failure: " . $ip['ip_string'] . ". Token expired for " . $userId . ".");
EventAuditLogger::instance()->newEvent('api', '', '', 0, "API failure: " . $ip['ip_string'] . ". Token expired for client[" . $clientId . "] and user " . $userId . ".");
return false;
}

// Token authentication passed
EventAuditLogger::instance()->newEvent('api', '', '', 1, "API success: " . $ip['ip_string'] . ". Token successfully used for " . $userId . ".");
EventAuditLogger::instance()->newEvent('api', '', '', 1, "API success: " . $ip['ip_string'] . ". Token successfully used for client[" . $clientId . "] and user " . $userId . ".");
return true;
}

/**
* Checks if we should log the response interface (we don't want to log binary documents or anything like that)
* We only log requests with a content-type of any form of json fhir+application/json or application/json
* @param ResponseInterface $response
* @return bool If the request should be logged, false otherwise
*/
private static function shouldLogResponse(ResponseInterface $response)
{
if ($response->hasHeader("Content-Type")) {
$contentType = $response->getHeaderLine("Content-Type");
if ($contentType === 'application/json') {
return true;
}
}

return false;
}


/** prevents external cloning */
private function __clone()
{
Expand Down
Loading

0 comments on commit 726526a

Please sign in to comment.