Add DICOM SR Support for additional/common annotations

[dannyrb]

I think on first pass we should support:

• Length (done)
• Bidirectional (WIP)
• arrow annotation (…is this doable, could be a single point + a descriptive label?)
• Probe (control point)
• Ellipse
• Rectangle

@pieper has kindly provided some guidance on how to approach these mappings in a discussion with @galelis, @JamesAPetts, and @dannyrb

@galelis, if you transition off this issue, please provide insights/next-steps in a follow-up comment.

[dclunie]

There are 2D (image) relative coordinates in DICOM SR, and 3D (frame of reference e.g., patient, volume, slide).

All of Graphic Types specified for those that are co-planar with a referenced image orientation should be supported (3D types that are not in the same plane as the image would require volume "awareness", which may be out of scope for now):

2D: POINT, MULTIPOINT, POLYLINE, CIRCLE, ELLIPSE
3D: POINT, MULTIPOINT, POLYLINE, POLYGON, ELLIPSE

See:

http://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_C.18.6.html
http://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_C.18.9.html

In DICOM SR TID 1500 (which is the template to focus on), see examples of use in these templates:

http://dicom.nema.org/medical/dicom/current/output/chtml/part16/chapter_A.html#sect_TID_1410
http://dicom.nema.org/medical/dicom/current/output/chtml/part16/chapter_A.html#sect_TID_1411
http://dicom.nema.org/medical/dicom/current/output/chtml/part16/chapter_A.html#sect_TID_320

Examples at:

http://dicom.nema.org/medical/dicom/current/output/chtml/part17/sect_RRR.5.html

David

PS. Note that "generic" SR annotation rendering (as opposed to authoring) code may theoretically "ignore" what template is defined to be or appears to be used, and just search the tree for any SCOORD or SCOORD3D content item, and then explore the local neighborhood for children that are images, and parents that describe its context (i.e., any associated description, identification, measurements, etc.). That approach is robust and can support all sorts of SR templates and applications from the generic TID 1500 to Mammography CAD, as well as private templates.

[dclunie]

Agree "arrows" should be single point.

[JamesAPetts]

Thanks the input David, I thought as much for the arrow.

Yeah the inital scope for this issue would be the in plane annotations that cornerstone supports by default.

Out of plane annotations as well as TID 1411 (?, off the top of my head as I'm on a train, the volumetric measurements for ROIs) will come down the line as separate issues.

[dannyrb]

... "generic" SR annotation rendering (as opposed to authoring) code may theoretically "ignore" what template is defined to be or appears to be used, and just search the tree for any SCOORD or SCOORD3D content item, and then explore the local neighborhood for children that are images, and parents that describe its context (i.e., any associated description, identification, measurements, etc.). - @dclunie

This is a good note. When our application is not the author of the report, and without additional assurances/constraints, this may be the best we can do without introducing hazards.

Overview

Current approach

We parse and render the contents of a DICOM SR by doing the following:

import * as dcmjs from 'dcmjs';

const { DicomMetaDictionary, DicomMessage } = dcmjs.data;

  setContentFromByteArray(byteArray) {
    const arrayBuffer = byteArray.buffer;
    const dicomData = DicomMessage.readFile(arrayBuffer);
    const dataset = DicomMetaDictionary.naturalizeDataset(dicomData.dict);
    dataset._meta = DicomMetaDictionary.namifyDataset(dicomData.meta);

    const mainData = getMainData(dataset);
    const contentSequence = getContentSequence(dataset);
    const content = (
      <>
        {mainData}
        {contentSequence}
      </>
    );

    this.setState({
      content,
    });
  }

A sopClassHandler is registered that creates displaySets for DICOM SR SOP Class UIDs. The OHIFDicomHtmlViewport uses the DicomLoaderService to get the byteArray for a given DisplaySet. It then sets that byteArray as a prop for the Viewport. The Viewport uses DCMJS, as outlined above, to parse and display the content in a more or less raw format. When mapping a measurement to tool state, we use the parsed dataset with a DCMJS adapter.

Issues with current approach

• Extensions are forced to have a hard dependency on DCMJS and the DicomLoaderService
• Extensions are required to have an intimate knowledge of DICOM SR to extract/persist data
• Extensions that need the same information would dispatch multiple (duplicate) network requests
• Mapping SR measurements to tools can be done in a more interoperable way
• Mapping SR measurements to CSTools' State bypasses the platform and makes CSTools the source of truth
• We are unable to display measurements created by other sources

Goals

• Centralize DICOM SR Parsing / loading
• Broad support for rendering DICOM SR measurements
• Ability to create interoperable DICOM SRs that have edit support for primitives
• Ease burden of DICOM SR knowledge for extension authors
• Ease ability to create new extensions that can leverage information in DICOM SRs
• (Showcase) Best practices around consuming enhanced data services in a practical example (DICOM SR Extension)

Blockers

• MVP completion of the MeasurementService
• We should complete the decoupling of the "MeasurementsPanel" extension, so we can remove it for this feature showcase. This extension could reuse the components, but should provide them itself, and marry them with DICOM SR functionality in @ohif/core
• Current studyMetadataManager usage is convoluted

Phases

This task is too large to tackle in one go. Here are some reasonable milestones we can use to iteratively add value:

Shift where/how we query and store DICOM SR information

The MeasurementService works well for our critical path, but it doesn't encapsulate all of the information contained in an SR and it's not immutable. We likely want to extend the existing studyMetadataManager that contains instances of OHIFStudyMetadata. It can be used to hydrate the MeasurementService.

1. Add a sopClassHandler for DICOM SR
   • No longer show SR thumbnails
2. Register/add a copy of the SRs information to a "data service" like the studyMetadataManager
3. Replace the @ohif/extension-dicom-html logic with a query to the data service that fetches and displays content similar to what is already shown

Display DICOM SR measurements in a table and on a 2D canvas

The simplest approach here is likely what @dclunie outlined above. We may want to limit the (initial) supported measurements to SCOORD as they'll have a reference image and are easier to map to Cornerstone's imageIdSpecificToolState. While we can support SCOORD3D, how that impacts rendering and tabular data is a UX concern that requires more thought.

1. Display a list of SRs in a DICOM SR side panel
   • Order from most recent to oldest
2. Similar to DICOM Seg functionality, hydrate the measurements service with the most recent report. Selecting a previous report should swap out Measurements specific to one report with the newly selected reports and trigger a re-render. Some meta summary for each report should exist.
3. It is problematic/hazardous to assume, without further guarantees, that the parsed measurements can be safely mapped to "edit" tools. Until we have more information, we should use one of the following strategies for render-only behavior:
   • A single tool that, given some schema, can render POINT, MULTIPOINT, POLYLINE, CIRCLE, and ELLIPSE value types for a given imageId
     • If we're feeling ambitious, this could be the beginning of a tool state SVG renderer that allows interop between VTK and Cornerstone
   • OR we duplicate existing tools (length, freehand, circle, ellipse) with alternative tool names in the "enabled" (read-only) mode that we designate as renderers for mapped measurements

This may sound more straightforward than it is. It's worth looking over the various templates and how they group measurements. We'll need to account for related measurements that are linked by UID (and span measurement groups), as well as a few other scenarios. This may impact how we display information in our table. Some measurements will explicitly map to a POINT or POLYLINE, while others are implied or derived from neighboring information.

Notes:

• Do timepoints play a part here?
• Would we ever want to show multiple SRs measurements at the same time?

When a TID 1500 Report originated from a supported OHIF version/schema, map to edit tools

We currently accomplish this by keying off of a Measurement's "Code Meaning". For example, we encode a POLYLINE that has a CodeMeaning of Length, then assume if we filter/match on that string key, the measurement works with the cornerstone-tools Length Tool. (This currently takes place in a DCMJS "adapter", not within this monorepo).

The TID 1500 includes Author Observer Sequence. This would allow us to encode information about the application that generated the report and to make further assumptions about the encoded data. For example, if we see that there is an Author Observer (there may be a better field for this) that has:

• (0018,1020) Software Version(s) X
• (0008,0080) Institution Name Y
• etc.

We may be able to reasonably assert that because our application supports Application Y and Software Version X that the encoded measurements are no longer generic, but rather, map 1:1 to our application's tools.

The biggest change here is identifying what information we key off of, and how we might go about supporting schemas as we continue to version/grow what we do support. Ideally, this would be extensible as we grow this to support multi-step and composite tools, but I'm likely getting ahead of myself.

1. Spell out the information we set when persisting information and restoring it
2. Create new fields in the Data Service that help indicate the OHIF version/schema associated, if any, with the SR
3. If the application supports the defined version/schema, map tool state to edit tools instead of the generic render tools
4. Support round trip persistence

This may sound more straightforward than it is. There are several "gotchas" that may not be immediately obvious. For example:

There is no "rectangle" value type. A rectangle ROI would be encoded as a POLYLINE. Rendering a POLYLINE is simple enough, but if we need to know this should map to a "rectangle tool", we would need to look at the points more closely to identify that. Knowing when a square POLYLINE maps to a square tool, a rectangle tool, or a freehand roi tool is an interesting problem.

POLYLINEs are only "closed" if the first point is the same as the end point.

POLYLINEs do not guarantee a "winding" order. You cannot assume that the points are read from left to right, or right to left. This makes encoding an "arrow" with an end point an interesting problem. We may need to use a single point (end point), and a magnitude/direction

Support for persisting tools outside the scope of primitive value types (far future)

We will likely need to create a custom "Code Scheme Designator", "Code Meaning"s, and "Code Value"s when persisting tool state for tools that can't be encoded using a single value type. A GitHub repository (and npm package) can describe these concepts, and create an avenue for other systems that would like to consume the data beyond simply rendering/listing the values.

I am currently unsure of how we might provide a path to add these dynamically or ad-hoc through extensions. Interoperability beyond our application may only be possible with published definitions and a concerted effort.

Not Discussed

SCOORD3D Support

While we have a frame of reference tool state manager in cornerstone tools, it's not commonly used. For most measurements (created in OHIF), we'll want to tie them to a specific instance. As we broaden VTK annotation support, SCOORD3D support will becomes more pressing.

Pathway for rendering w/ VTK Viewports

...

Automatic mapping

While we could make optimistic guesses to map SCOORD and SCOORD3D to edit tools, the outcome is likely too hazardous to pursue. A future enhancement could be made to allow manually transferring the state to a tool that supports that same general schema, but with a very blatant warning.

If there is an obvious pattern where MANUFACTURER maps to these tools, we could build this in as a more robust feature where users can setup the automatic mappings.

Akin to using ImagerPixelspacing or calibrating an image's pixel spacing. "You can do this, but at your own risk."

I am doubtful we will have the bandwidth to tackle this anytime soon, but I believe it's worth listing as it may catch the right financier's eye <3

Parting Words

Please feel free to push on any/all of this. This is my best guess at our path forward with a few sparse implementation details peppered in. I'm fairly new to DICOM SR, and curious to hear how other implementers are tackling these same problems.

Note: I've created a Structured Reports :memo: label to tie together related issues.

[JamesAPetts]

No longer show SR thumbnails

Some meta summary for each report should exist.

Just so we are on the same page, the HTML render will still be available, just accessed from elsewhere?

A single tool that, given some schema, can render POINT, MULTIPOINT, POLYLINE, CIRCLE, and ELLIPSE value types for a given imageId

OR we duplicate existing tools (length, freehand, circle, ellipse) with alternative tool names in the “enabled” (read-only) mode that we designate as renderers for mapped measurements

Duplicating the tools is overkill I reckon? We could add a locked state for all toolState that makes them immutable. This may be too intrusive to CST (unless this a feature others would value?). I think the “SR measurement render only tool” you suggested is better.

If we’re feeling ambitious, this could be the beginning of a tool state SVG renderer that allows interop between VTK and Cornerstone

I feel like this is a mix of concerns that will only confuse intergation, and is better left as a separate ticket/project later.

Would we ever want to show multiple SRs measurements at the same time?

Ideally yes, if they contain different data and not just a “corrected” version, but how could we possibly work this out? :thinking_face: . I don’t think we can. Maybe we have to limit to only one SR per series (even that is a little complicated as SRs can reference lots of images from a study I believe). Of course other than a PWA, a more complicated backend could triage time versioned SR.

We will likely need to create a custom “Code Scheme Designator” …. I am currently unsure of how we might provide a path to add these dynamically or ad-hoc through extensions.

I think the OHIF standard we make must be consumed and added to a Code Scheme library. An extension could add its on CSD + definitions to this library (e.g. an extension adds a tool and how to store it?).

[dannyrb]

Just so we are on the same page, the HTML render will still be available, just accessed from elsewhere?

The HTML Viewport Extension would use the studyMetadataManager or similar service to query for the data it needs to render similar output. Setting it as a viewport, if we remove SR thumbnails, would require an action to be dispatched to Redux.

Improved programmtic control of the Viewport Grid and Viewports is something we have slated.

Duplicating the tools is overkill I reckon?

csTools.addTool(csTools.LengthTool, { name: 'DicomSrLength' });
csTools.setToolEnabled('DicomSrLength');

But I agree. I think a render only tool would be cleaner.

I feel like this is a mix of concerns that will only confuse integration, and is better left as a separate ticket/project later.

I agree. A pipe dream of mine.

I think the OHIF standard we make must be consumed and added to a Code Scheme library.

👍

[galelis]

Regarding the way we display the SRs and how to get old SRs

Just so we are on the same page, the HTML render will still be available, just accessed from elsewhere?

As SR is also used for more things other than just measurements, we could filter the DICOM SR with measurements from thumbnail, maybe using (0018,1020) Software Version(s) X or (0008,0080) Institution Name or other tag. And display the non-measurement ones as we use to do now.

Would we ever want to show multiple SRs measurements at the same time?

We could have a runtime load of a new SR, similar to what we have on series-list <-> viewports, maybe drag and drop into the table or a simple drop down with old SRs (this was suggested by Steve when I first add measurement table on Meteor version)

Maybe we have to limit to only one SR per series (even that is a little complicated as SRs can reference lots of images from a study I believe)

Now a days, we use dcmjs and store one SR per Study, doing one by series, I don't personally don't see any gain on having it by series.

Regarding the way we display the tools, I like the idea of render only tool.

Have you guys thought about tool versioning and synchronization with what we have on a DICOM SR? I mean, what if the data we stored in the past, new app does not know how to render anymore, like the structure has changed..

[fedorov]

Wow, that's a lot of stuff guys! I have to say I could not follow all of the points in the discussion due to lacking understanding of the OHIF internals perhaps. My overall suggestion would be to consider focusing on specific use cases, find a very specific customer that will be invested in using this functionality and provide feedback to guide development. Just one TID 1500 template has so many capabilities and usage flavors that even exhaustively covering those is a huge project!

On a positive side, I guess (considering the "I am doubtful we will have the bandwidth to tackle this anytime soon" comment!), planar annotations are not an immediate priority for the IDC project. Our current target is development of the MVP (see our plans in the IDC MVP document linked from here), which will provide static view of a subset of TCIA collections. None of those collections (with perhaps the exception of the annotations produced by CCC for some of them) have planar annotations. It is much more important for IDC to be able to handle measurements and SR objects derived from volumetric segmentations - plenty of datasets of that kind, and more on the way.

I promised to @pieper and @swederik that I will submit all issues that I encountered and also create issues for functionality we need for IDC MVP. We can discuss those after that in a call to hopefully bring everyone to the same page.

[pieper]

Also just for context: the effort to serialize and deserialize OHIF annotations is not just for IDC, although there's great synergy with the efforts to use SR for this purpose especially given the guidance from @dclunie 's white paper.

Currently we have IDC:candidate and IDC:priority labels, but maybe this would fall into something like an IDC:collaboration category (don't want to invent too many labels, but if needed we can).

[fedorov]

@pieper I agree, and adding "IDC:collaboration" makes sense to me.

[rodrigobasilio2022]

Its not a bug itself in the implementation, but in how the data is presented in Measurement Panel. @fedorov , correct me if i am wrong, but the want to filter the SRs in the panel to the ones related to the active viewport, right?

[igoroctaviano]

Its not a bug itself in the implementation, but in how the data is presented in Measurement Panel. @fedorov , correct me if i am wrong, but the want to filter the SRs in the panel to the ones related to the active viewport, right?

@rodrigobasilio2022

Scenarios:
Given I'm a user
When I open a study with SCOORD3D points
Then The measurement panel displays the exact number of measurements present in the SR

Given I'm a user
When I select a series in the gallery (activating it)
And I select a measurement in the right panel (measurement panel)
Then The measurement should be rendered on the active series, jumping to the slice where the measurement was rendered (If the active series is not the referenced series in the SR, the measurement should be displayed on that series if it's close enough given a tolerance value)

So the idea here is to parse the SR, display the measurements in the measurement panel (the exact number of measurements present in the SR, no duplication allowed) and when the user clicks on a given measurement, that measurement is displayed on the active viewport (the viewer jumps to that slice). This means that the measurement could be rendered on any series if it shares a frame of reference and is close enough to a particular slice given a tolerance value.

[fedorov]

Thank you @igoroctaviano - this is exactly correct.

[rodrigobasilio2022]

This scenario has a problem, if there is no slice near to the SCOORD3D points in the active viewport, nothing will show. What if we change this scenario a little bit, by filtering the Measurements Table based on the current active viewport. If there is annotations they will be displayed, if not the measurements panel will be empty.

[igoroctaviano]

This scenario has a problem, if there is no slice near to the SCOORD3D points in the active viewport, nothing will show. What if we change this scenario a little bit, by filtering the Measurements Table based on the current active viewport. If there is annotations they will be displayed, if not the measurements panel will be empty.

For now, we can focus on the happy path and later decide on the case you mentioned.

For the future, If the measurement is not fit for the active viewport it's better to have it disabled (grayed out) than filtered out. This way the user knows that the content of the SR is there but it's not valid because of reason X. Maybe later we could even allow the user to change/tune the tolerance value but this is all future things.

[fedorov]

Yes, I agree with @igoroctaviano. Showing a subset of items from SR based on what can be visualized would be very misleading.

What if we change this scenario a little bit, by filtering the Measurements Table based on the current active viewport. If there is annotations they will be displayed, if not the measurements panel will be empty.

Instead, we could explore a variety of other ways to communicate this to the user (in addition to disabling, as suggested by Igor):

• show warning sign next to the items that do not map to any slice
• show projection to the closest slice, clearly indicating by the appearance of the annotation that it is a projection to a slice beyond the threshold (for example, by displaying warning sign next to the point)

I would implement the easiest of those ideas for now, and revisit later.

[rodrigobasilio2022]

Updated the code with this features. Please review. There one more change to do. Its the auto hydration issue, to avoid popping up the infamous dialog box

[igoroctaviano]

Updated the code with this features. Please review. There one more change to do. Its the auto hydration issue, to avoid popping up the infamous dialog box

Are you updating a different env? I tried with https://d3w0jbfdn2j3pj.cloudfront.net/projects/idc-sandbox-000/locations/us/datasets/idc-dicom-test-inventory/dicomStores/test-samples/study/1.3.6.1.4.1.14519.5.2.1.7310.5101.860473186348887719777907797922

First series (measurement not active for viewport)

Second series (blank / no measurement although it jumps to a frame)

Third series (measurement not active for viewport)

[rodrigobasilio2022]

Just series 6, 8, 9 has measurements. For me it goes ok, though I saw clicking with no measurement display, and I click again and it is there. Maybe its a bug in OHIF, but can you try these series and if there is no annotation displaying try again clicking in the same annotation

[igoroctaviano]

Just series 6, 8, 9 has measurements. For me it goes ok, though I saw clicking with no measurement display, and I click again and it is there. Maybe its a bug in OHIF, but can you try these series and if there is no annotation displaying try again clicking in the same annotation

@fedorov is this correct? I thought the measurements should be rendered for other planes too.

[fedorov]

I just looked, and I am confused actually. I thought those coordinates should have been SCOORD3D, but they appear to be SCOORD! If that's the case, then they should show only on the instance listed in the SR.

[rodrigobasilio2022]

They are translated from SCOORD3D to SCOORD. Each SCOORD3D can generate multiple SCOORD annotations, depending on the proximity of the slice

[fedorov]

They are translated from SCOORD3D to SCOORD

I do not understand what this means. Translated by whom and when?

[igoroctaviano]

They are translated from SCOORD3D to SCOORD

I do not understand what this means. Translated by whom and when?

@fedorov I'll have a meeting with @rodrigobasilio2022 to discuss this. We'll get back to you ASAP.

[fedorov]

Thank you Igor.

@rodrigobasilio2022 the DICOM data received from the server must be shown as received from the server in the DICOM tag viewer. It is absolutely unacceptable to modify and present modified data in the DICOM tag viewer! The details of how you manage internal to the implementation data structures must be completely separate from the DICOM data representation.

[rodrigobasilio2022]

Understood. I will change the implementation.

[fedorov]

Per discussion today, we will test this once a sandbox instance with these features is deployed.

[mrceresa]

Dears, any updates on this?

[Joeycho]

@dclunie I hope you remember this issue. Why does Arrow need to be "single" point" rather than "two points"?

Because of this, everytime changing viewport (resizing), arrows are randomly rotated.. Or is there any logics behind it?

[dclunie]

Because SR is about semantics, not appearance.

I.e., for an arrow, what matters is what is at the tip, not the direction of the arrow.

[Joeycho]

@dclunie Thank you for the prompt response :) Yes, but in my case, a user saves the status with the screenshot, and they load next time. Their expectation is that the arrow should be loaded exactly as they saved. And I think that makes sense as well. And I think if I refer how the line (length) is saved as SR, probably I can edit how arrow is saved and may achieve what I expect. What do you think?

[sedghi]

@seandoyle FYI seems like a response to our recent discussion

[seandoyle]

I agree with David here about the interpretation of the SR spec. I'm not sure how the data could be encoded in the SR that is consistent with it being a point. Another approach would be to keep it as a point in the SR but somehow have rendering rules that constrain the presentation but I don't have any solid ideas on what these rules would be. Right now if draw an arrow that is pointed up at a 45 degree angle from the horizontal it will be displayed at roughly 45 + 90 degrees when restored from the SR (the point that is being pointed at by the arrow is correct).

The only way I can see making this consistent is that when the user has entered the text for the arrow that it automatically moves to the location constrained by the rendering algorithm. This would make it consistent when the SR is re-rendered. But users will find it startling that their arrow moved.

[Joeycho]

@seandoyle @dclunie @sedghi

I think I found a way to fix this arrow issue and made the following PRs several weeks ago. Fortunately, A point for TID300 can store additional point, which makes the implementation easier. If you can find some issues in the following PRs, please let me know :)

cornerstonejs/cornerstone3D#1450
#4356