Results gets wrong label

[grzanka]

I've defined a simulation from scrach, loosely based on "Treatment plan example".
Then I've executed it on yaptide.c3.plgrid.pl (which runs now on master branches from ui and yaptide).

Results gets wrong labels. First plot should be titled dose, not fluence:

Grouping doesn't work:

Here is the output JSON, saved with the button in upper right corner:

I was able to load the results into https://yaptide.github.io/web_dev/ instance:

The results seemed to be loaded correctly, but the editor project state is missing:

[grzanka]

Backend DB status:

ubuntu@yaptide:~$ docker exec -it yaptide_flask bash -c "flask --app yaptide.application db show"
Rev: 5003b9acb1f4 (head)
Parent: 41149fa64bd2
Path: /usr/local/app/migrations/versions/5003b9acb1f4_.py

    empty message
    
    Revision ID: 5003b9acb1f4
    Revises: 41149fa64bd2
    Create Date: 2024-12-10 22:51:14.262553

ubuntu@yaptide:~$ docker exec -it yaptide_flask bash -c "flask --app yaptide.application db check"
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.ddl.postgresql] Detected sequence named 'Estimator_id_seq' as owned by integer column 'Estimator(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'Cluster_id_seq' as owned by integer column 'Cluster(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'Logfiles_id_seq' as owned by integer column 'Logfiles(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'Page_id_seq' as owned by integer column 'Page(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'Input_id_seq' as owned by integer column 'Input(id)', assuming SERIAL and omitting
No new upgrade operations detected.
ubuntu@yaptide:~$ date
Thu Jan  2 10:44:59 UTC 2025

[jagodek]

@grzanka. Json with project would be helpful to recreate issue.

[grzanka]

Here is the results JSON - is it enough ?
treatment-plan-160-mev-mono-result.json.json

[jagodek]

I meant json saved from 3d editor

[grzanka]

Something is weird, once I click "Load to Editor" I get nothing in the editor pane. Then if I save this to a hard drive I get following, almost empty file:
plan160mev.json

You can find all necessary data (bodies, zones, filters, scorers) in the JSON from this comment:
#1927 (comment)
Unfortunately for some reason it doesn't load into editor :(

I will try to recreate from scratch a smaller example to reproduce this issue.

[grzanka]

Here is a new one. Check results for "ZPROFILE"
wrong.json

Results.-.Google.Chrome.2025-01-02.16-51-01.mp4

[jagodek]

So far i noticed that something wrong is with customMaterial when loading project from results
YaptideEditor.js:711 Error: SimulationMaterial not found
at rZ.reconstructMaterialFromJSON (SimulationZone.ts:122:47)
at rZ.fromJSON (SimulationZone.ts:204:8)
at rZ.fromJSON (BooleanZone.ts:203:9)
at rZ.fromJSON (BooleanZone.ts:216:15)
at pZ._loader (ZoneManager.ts:25:54)
at SimulationContainer.ts:87:23
at Array.forEach ()
at pZ.fromJSON (SimulationContainer.ts:86:8)
at fZ.fromJSON (ZoneManager.ts:220:22)
at _Q.fromJSON (YaptideEditor.js:677:38)
and it's from this material:
"customMaterial": {
"color": 9744820,
"density": 0.00120479,
"icru": 104,
"name": "AIR, DRY (NEAR SEA LEVEL)",
"opacity": 0,
"originalMaterialUuid": "be881070-c34e-4a41-83ca-2110cb0395b7",
"sanitizedName": "air_dry_near_sea_level",
"transparent": true,
"uuid": "8753929f-be20-4958-8edc-bba0b9cc3ba4"
},

[jagodek]

Yup. Seems like AIR, DRY (NEAR SEA LEVEL) has no entry in materials unlike WATER, LIQUID does.

[grzanka]

Yup. Seems like AIR, DRY (NEAR SEA LEVEL) has no entry in materials unlike WATER, LIQUID does.

Why not ? Air is here:

ui/src/ThreeEditor/Simulation/Materials/materials.ts

Lines 170 to 175 in 1df3e53

	{
	icru: 104,
	name: 'AIR, DRY (NEAR SEA LEVEL)',
	sanitized_name: 'air_dry_near_sea_level',
	density: 0.00120479
	},

[jagodek]

Here is the results JSON - is it enough ? treatment-plan-160-mev-mono-result.json.json

I'm talking about this file, the customMaterial has originalMaterialUuid field which refers to material in materialManager>materials which is not there.

[grzanka]

Here is the results JSON - is it enough ? treatment-plan-160-mev-mono-result.json.json

I'm talking about this file, the customMaterial has originalMaterialUuid field which refers to material in materialManager>materials which is not there.

Can you check as well this one ?
wrong-result.json.json

These results correspond to #1927 (comment)

When I open this JSON file in https://yaptide.github.io/web_dev/ then the results are loaded correctly:

The same simulation card on which I clicked "Save to file" on https://yaptide.c3.plgrid.pl/ loads them incorrectly.

Main difference between https://yaptide.github.io/web_dev/ and https://yaptide.c3.plgrid.pl/ is that the latter one doesn't have access to the backend.

[grzanka]

This is how yaptide.c3.plgrid.pl loads results, when I click on ZPROFILE it makes a call to backend:

[grzanka]

What I suspect is that there was a nice fix by @SzymanskiBartlomiej in #1898 , merged on 12.12

Then same day, but later, then came a feature by @matuszsmig in #1884, merged same day, but after the "filter-fix".

Both PRs modified src/services/ShSimulatorService.tsx which contains crucial logic.

Can you check if after changes by @matuszsmig recreateRefToScoringManagerOutputs and recreateRefToFilters is still in place and works ?

[grzanka]

After digging a bit I see a difference: getEstimatorsPages was introduced in #1884
It calls recreateRefsInResults on a single element list containing one estimator

	const getEstimatorsPages = useCallback(
		async (...[info, signal, cache = true, beforeCacheWrite]: RequestGetJobResult) => {
                        (....)
			const cachePromise = resultsCache.createPromise(
                        (....)
					const estimator: Estimator[] = [
						{
							name: estimatorName,
							pages: response.pages
						}
					];

					updateEstimators(estimator);

					const jobInputs = await getJobInputs(info, signal, cache);

					const refsInResults =
						jobInputs?.input.inputJson &&
						recreateRefsInResults(jobInputs.input.inputJson, estimator);

					const data: SpecificEstimator = {
						jobId,
						estimators: refsInResults ?? estimator,
						message: response.message
					};

Previously used method to fetch the result (prior to #1884), used recreateRefsInResults with list of all estimators.

From quick look on recreateRefsInResults and how it calls recreateRefToScoringManagerOutputs I guess that it needs to get JSON with same arrays of same length containing estimators. There is a lot of index-based logic which gets wrong in some cases if we pass different number of estimators.

See recreateRefsInResults:

ui/src/services/ShSimulatorService.tsx

Lines 118 to 135 in e534ac2

	export const recreateRefsInResults = (inputJson: EditorJson, estimators: Estimator[]) => {
	if (!inputJson) throw new Error('No editor data');
	if (!estimators) throw new Error('No esitamtors data');
	const { scoringManager }: EditorJson = inputJson;
	const estimatorsWithScoringManagerOutputs = recreateRefToScoringManagerOutputs(
	estimators,
	scoringManager
	);
	const estimatorsWithFixedFilters = recreateRefToFilters(
	estimatorsWithScoringManagerOutputs,
	scoringManager.filters
	);
	return estimatorsWithFixedFilters;
	};

And recreateRefToScoringManagerOutputs

ui/src/services/ShSimulatorService.tsx

Lines 93 to 102 in e534ac2

	const recreateRefToScoringManagerOutputs = (
	estimators: Estimator[],
	scoringManagerJSON: ScoringManagerJSON
	): Estimator[] => {
	estimators.forEach((estimator, index) => {
	estimator.scoringOutputJsonRef = scoringManagerJSON.outputs[index];
	});
	return estimators;
	};