Order of media files after migration is not correct #5518

andre-hohmann · 2023-01-23T14:31:34Z

andre-hohmann
Jan 23, 2023
Collaborator

After the migration of processes from Kitodo.Production 2 to Kitodo.Production 3 the following behavior regarding the order of media files is observed.

Metadata editor
The unstructured media files are shown

in the gallery: in an own section "Unstructured media" and
in the structure tree: at the end

METS file
In the METS file, the order of the METS attribute ORDER differs from the order of the media files in the structure tree.
The order of the pagination in METS attribute ORDERLABEL is the same as in Kitodo.Production 2 and correct.

    <mets:structMap TYPE="PHYSICAL">
        <mets:div ID="PHYS_0000">
            <mets:div ID="PHYS_0001" TYPE="page" ORDER="1" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0001"/>
            </mets:div>
            <mets:div ID="PHYS_0002" TYPE="page" ORDER="2" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0002"/>
            </mets:div>
            <mets:div ID="PHYS_0003" TYPE="page" ORDER="3" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0003"/>
            </mets:div>
            <mets:div ID="PHYS_0004" TYPE="page" ORDER="4" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0004"/>
            </mets:div>
            <mets:div ID="PHYS_0005" TYPE="page" ORDER="5" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0005"/>
            </mets:div>
            <mets:div ID="PHYS_0006" TYPE="page" ORDER="6" ORDERLABEL="I">
                <mets:fptr FILEID="FILE_0006"/>
            </mets:div>
            <mets:div ID="PHYS_0007" TYPE="page" ORDER="7" ORDERLABEL="II">
                <mets:fptr FILEID="FILE_0007"/>
            </mets:div>
            <mets:div ID="PHYS_0008" TYPE="page" ORDER="8" ORDERLABEL="III">
                <mets:fptr FILEID="FILE_0008"/>
            </mets:div>
            <mets:div ID="PHYS_0009" TYPE="page" ORDER="9" ORDERLABEL="IV">
                <mets:fptr FILEID="FILE_0009"/>
            </mets:div>
            <mets:div ID="PHYS_0010" TYPE="page" ORDER="10" ORDERLABEL="V">

[...]

           <mets:div ID="PHYS_0016" TYPE="page" ORDER="16" ORDERLABEL="XI">
                <mets:fptr FILEID="FILE_0016"/>
            </mets:div>
            <mets:div ID="PHYS_0017" TYPE="page" ORDER="17" ORDERLABEL="XII">
                <mets:fptr FILEID="FILE_0017"/>
            </mets:div>
            <mets:div ID="PHYS_0018" TYPE="page" ORDER="18" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0018"/>
            </mets:div>
            <mets:div ID="PHYS_0019" TYPE="page" ORDER="19" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0019"/>
            </mets:div>
            <mets:div ID="PHYS_0020" TYPE="page" ORDER="20" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0020"/>
            </mets:div>
            <mets:div ID="PHYS_0021" TYPE="page" ORDER="21" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0021"/>
            </mets:div>
            <mets:div ID="PHYS_0022" TYPE="page" ORDER="22" ORDERLABEL="1">
                <mets:fptr FILEID="FILE_0022"/>
            </mets:div>
            <mets:div ID="PHYS_0023" TYPE="page" ORDER="23" ORDERLABEL="2">
                <mets:fptr FILEID="FILE_0023"/>
            </mets:div>
            <mets:div ID="PHYS_0024" TYPE="page" ORDER="24" ORDERLABEL="3">

[...]

            <mets:div ID="PHYS_0539" TYPE="page" ORDER="539" ORDERLABEL="47">
                <mets:fptr FILEID="FILE_0539"/>
            </mets:div>
            <mets:div ID="PHYS_0540" TYPE="page" ORDER="540" ORDERLABEL="48">
                <mets:fptr FILEID="FILE_0540"/>
            </mets:div>
            <mets:div ID="PHYS_0541" TYPE="page" ORDER="541" ORDERLABEL="49">
                <mets:fptr FILEID="FILE_0541"/>
            </mets:div>
            <mets:div ID="PHYS_0542" TYPE="page" ORDER="542" ORDERLABEL="50">
                <mets:fptr FILEID="FILE_0542"/>
            </mets:div>
            <mets:div ID="PHYS_0543" TYPE="page" ORDER="543" ORDERLABEL="51">
                <mets:fptr FILEID="FILE_0543"/>
            </mets:div>
            <mets:div ID="PHYS_0544" TYPE="page" ORDER="544" ORDERLABEL="52">
                <mets:fptr FILEID="FILE_0544"/>
            </mets:div>
            <mets:div ID="PHYS_0545" TYPE="page" ORDER="545" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0545"/>
            </mets:div>
            <mets:div ID="PHYS_0546" TYPE="page" ORDER="546" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0546"/>
            </mets:div>
            <mets:div ID="PHYS_0547" TYPE="page" ORDER="547" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0547"/>
            </mets:div>
            <mets:div ID="PHYS_0548" TYPE="page" ORDER="548" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0548"/>
            </mets:div>
            <mets:div ID="PHYS_0549" TYPE="page" ORDER="549" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0549"/>
            </mets:div>
            <mets:div ID="PHYS_0550" TYPE="page" ORDER="550" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0550"/>
            </mets:div>
        </mets:div>
    </mets:structMap>

See: OAI: 20050696Z

After saving the process, the values of the METS attribute ORDER is modified according to the order of the files in the structure tree.
In the METS file, the values of the METS attribute ORDER is not correct anymore. The order of the pagination in METS attribute ORDERLABEL is not the same anymore as in Kitodo.Production 2.

    <mets:structMap TYPE="PHYSICAL">
        <mets:div ID="PHYS_0000">
            <mets:div ID="PHYS_0004" TYPE="page" ORDER="1" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0004"/>
            </mets:div>
            <mets:div ID="PHYS_0006" TYPE="page" ORDER="2" ORDERLABEL="I">
                <mets:fptr FILEID="FILE_0006"/>
            </mets:div>
            <mets:div ID="PHYS_0008" TYPE="page" ORDER="3" ORDERLABEL="III">
                <mets:fptr FILEID="FILE_0008"/>
            </mets:div>
            <mets:div ID="PHYS_0009" TYPE="page" ORDER="4" ORDERLABEL="IV">
                <mets:fptr FILEID="FILE_0009"/>
            </mets:div>
            <mets:div ID="PHYS_0010" TYPE="page" ORDER="5" ORDERLABEL="V">
                <mets:fptr FILEID="FILE_0010"/>
            </mets:div>
            <mets:div ID="PHYS_0011" TYPE="page" ORDER="6" ORDERLABEL="VI">
                <mets:fptr FILEID="FILE_0011"/>
            </mets:div>
            <mets:div ID="PHYS_0012" TYPE="page" ORDER="7" ORDERLABEL="VII">
                <mets:fptr FILEID="FILE_0012"/>
            </mets:div>
            <mets:div ID="PHYS_0013" TYPE="page" ORDER="8" ORDERLABEL="VIII">
                <mets:fptr FILEID="FILE_0013"/>
            </mets:div>
            <mets:div ID="PHYS_0014" TYPE="page" ORDER="9" ORDERLABEL="IX">
                <mets:fptr FILEID="FILE_0014"/>
            </mets:div>
            <mets:div ID="PHYS_0015" TYPE="page" ORDER="10" ORDERLABEL="X">
                <mets:fptr FILEID="FILE_0015"/>
            </mets:div>
            <mets:div ID="PHYS_0016" TYPE="page" ORDER="11" ORDERLABEL="XI">
                <mets:fptr FILEID="FILE_0016"/>
            </mets:div>
            <mets:div ID="PHYS_0017" TYPE="page" ORDER="12" ORDERLABEL="XII">
                <mets:fptr FILEID="FILE_0017"/>
            </mets:div>
            <mets:div ID="PHYS_0023" TYPE="page" ORDER="13" ORDERLABEL="2">
                <mets:fptr FILEID="FILE_0023"/>
            </mets:div>
            <mets:div ID="PHYS_0024" TYPE="page" ORDER="14" ORDERLABEL="3">
                <mets:fptr FILEID="FILE_0024"/>
            </mets:div>
            <mets:div ID="PHYS_0025" TYPE="page" ORDER="15" ORDERLABEL="4">
                <mets:fptr FILEID="FILE_0025"/>
            </mets:div>
            <mets:div ID="PHYS_0026" TYPE="page" ORDER="16" ORDERLABEL="5">
                <mets:fptr FILEID="FILE_0026"/>
            </mets:div>
            <mets:div ID="PHYS_0027" TYPE="page" ORDER="17" ORDERLABEL="6">
                <mets:fptr FILEID="FILE_0027"/>
            </mets:div>
            <mets:div ID="PHYS_0028" TYPE="page" ORDER="18" ORDERLABEL="7">
                <mets:fptr FILEID="FILE_0028"/>
            </mets:div>
            <mets:div ID="PHYS_0029" TYPE="page" ORDER="19" ORDERLABEL="8">
                <mets:fptr FILEID="FILE_0029"/>
            </mets:div>

[...]

            <mets:div ID="PHYS_0538" TYPE="page" ORDER="517" ORDERLABEL="46">
                <mets:fptr FILEID="FILE_0538"/>
            </mets:div>
            <mets:div ID="PHYS_0539" TYPE="page" ORDER="518" ORDERLABEL="47">
                <mets:fptr FILEID="FILE_0539"/>
            </mets:div>
            <mets:div ID="PHYS_0540" TYPE="page" ORDER="519" ORDERLABEL="48">
                <mets:fptr FILEID="FILE_0540"/>
            </mets:div>
            <mets:div ID="PHYS_0541" TYPE="page" ORDER="520" ORDERLABEL="49">
                <mets:fptr FILEID="FILE_0541"/>
            </mets:div>
            <mets:div ID="PHYS_0542" TYPE="page" ORDER="521" ORDERLABEL="50">
                <mets:fptr FILEID="FILE_0542"/>
            </mets:div>
            <mets:div ID="PHYS_0543" TYPE="page" ORDER="522" ORDERLABEL="51">
                <mets:fptr FILEID="FILE_0543"/>
            </mets:div>
            <mets:div ID="PHYS_0544" TYPE="page" ORDER="523" ORDERLABEL="52">
                <mets:fptr FILEID="FILE_0544"/>
            </mets:div>
            <mets:div ID="PHYS_0546" TYPE="page" ORDER="524" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0546"/>
            </mets:div>
            <mets:div ID="PHYS_0547" TYPE="page" ORDER="525" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0547"/>
            </mets:div>
            <mets:div ID="PHYS_0001" TYPE="page" ORDER="526" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0001"/>
            </mets:div>
            <mets:div ID="PHYS_0002" TYPE="page" ORDER="527" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0002"/>
            </mets:div>
            <mets:div ID="PHYS_0003" TYPE="page" ORDER="528" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0003"/>
            </mets:div>
            <mets:div ID="PHYS_0005" TYPE="page" ORDER="529" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0005"/>
            </mets:div>
            <mets:div ID="PHYS_0007" TYPE="page" ORDER="530" ORDERLABEL="II">
                <mets:fptr FILEID="FILE_0007"/>
            </mets:div>
            <mets:div ID="PHYS_0018" TYPE="page" ORDER="531" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0018"/>
            </mets:div>
            <mets:div ID="PHYS_0019" TYPE="page" ORDER="532" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0019"/>
            </mets:div>
            <mets:div ID="PHYS_0020" TYPE="page" ORDER="533" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0020"/>
            </mets:div>
            <mets:div ID="PHYS_0021" TYPE="page" ORDER="534" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0021"/>
            </mets:div>
            <mets:div ID="PHYS_0046" TYPE="page" ORDER="535" ORDERLABEL="uncounted">
                <mets:fptr FILEID="FILE_0046"/>
            </mets:div>

[...]

        </mets:div>
    </mets:structMap>

Evaluation of the result
In my opinion, this behavior is not correct - due to the following reasons:

The order (METS-attribute ORDER) of the media files is wrong, after the process is saved. The order should only be changed, when the files are modified by the users.
A correction of the pagination (METS-attribute ORDERLABEL) is difficult, because the media files in the section "Unstructured media" must be paginated in some cases file by file.

Expectation
After the migration, i would expect the order of the media files as it is applied when structure elements are created in the gallery:

The order of the media files of unstructured media is kept and the structure elements are created according to the position of their pages.

Questions

Are there other users who are expecting the behavior described in the section Expectation?
Are there users who are expecting the current behavior?
@m-kotzyba: Did you encounter any problems after the migration?

Thanks in advance for your feedback.

Answered by andre-hohmann

Feb 6, 2023

@subhhwendt: Thanks a lot for your investigation with @solth, @oliver-stoehr and @henning-gerhardt!
I try to summarize the current state of knowledge. The following can be stated:

The above-mentioned behavior is not the expected one and all media files should be sorted according to their @ORDER attribute
The reason for the behavior is the missing ORDER attribute for structure elements in <mets:structMap TYPE="LOGICAL">, because the values in the ORDER attribute are needed to sort the @ORDER attribute of the <mets:structMap TYPE="PHYSICAL">
A possible solution seems to be to assign the value of the @ORDER attribute of the first assigned page to the structure element during the migration o…

View full answer

m-kotzyba · 2023-01-25T07:56:48Z

m-kotzyba
Jan 25, 2023
Collaborator

@andre-hohmann: Yes, we encountered a quite similar problem after the migration and could not solve it. I wrote "similar" because in your very first picture, our result would be different to yours in so far that the "7: III" would not be in the lower list (of unstructured media) but the items "546: uncounted" and "547: uncounted" would be in the lower list. That is, essentially, we have always problems, if processes (be it an "old" -migrated- or a new one) have uncounted media (even if they are in a structure element).

Our migrated processes very often have uncounted at the beginning because in Kitodo 2 we have not used a Cover-Element and almost always started with the Title of a work.

Hence, to "repair" migrated but also for all new processes in Kitodo 3, we use a Cover and avoid unstructured media. And so, since we have to touch the old processes anyway, we'll correct the rest right away.

1 reply

andre-hohmann Jan 26, 2023
Collaborator Author

@m-kotzyba : Thanks for your response!
I hope i understood you correctly: You avoid unstructured media by assigning them to a structure element.
In my opinion this is possible, but not the right way - although i can understand the decisions for workarounds.
The object should be assigned to the structure elements based on their content, not to avoid unstructured media.

I will think a bit more about the consequences of this behavior.

m-kotzyba · 2023-01-26T06:47:02Z

m-kotzyba
Jan 26, 2023
Collaborator

@andre-hohmann: Yes, you understod right, and yes, this workaround is -in general- not the (right) way to go.
However, in our case, only the first and last pages were affected, so that using the Cover element was/is semantically still corrct and solved our problem pragmatically. :)

0 replies

andre-hohmann · 2023-02-06T05:55:08Z

andre-hohmann
Feb 6, 2023
Collaborator Author

@subhhwendt: Thanks a lot for your investigation with @solth, @oliver-stoehr and @henning-gerhardt!
I try to summarize the current state of knowledge. The following can be stated:

The above-mentioned behavior is not the expected one and all media files should be sorted according to their @ORDER attribute
The reason for the behavior is the missing ORDER attribute for structure elements in <mets:structMap TYPE="LOGICAL">, because the values in the ORDER attribute are needed to sort the @ORDER attribute of the <mets:structMap TYPE="PHYSICAL">
A possible solution seems to be to assign the value of the @ORDER attribute of the first assigned page to the structure element during the migration of the metadata from Kitodo.Production 2 to Kitodo.Production 3

One option to add the ORDER attribute to the structure elements is the following XSLT:

<xsl:template match="mets:div[@TYPE='Article' or @TYPE='Chapter' or @TYPE='Index' or @TYPE='TableOfContents' or @TYPE='TitlePage']/@ID">
    <xsl:variable name="LOGID" select="."/>
    <xsl:variable name="PHYSID" select="/mets:mets/mets:structLink/mets:smLink[@xlink:from=$LOGID][1]/@xlink:to"/>
    <xsl:variable name="ORDER" select="/mets:mets/mets:structMap[@TYPE='PHYSICAL']/mets:div/mets:div[@ID=$PHYSID]/@ORDER"/>
    <xsl:if test="$ORDER">
        <xsl:attribute name="ORDER">
           <xsl:value-of select="$ORDER"/>
        </xsl:attribute>
    </xsl:if>
     <xsl:copy>
        <xsl:apply-templates select="@*"/>
    </xsl:copy>
</xsl:template>

In the attribute @TYPE of the mets:div element, all structure elements must be declared, to which the @ORDER attribute should be assigned. Additionaly, the namespace for xlink must be declared:

xmlns:xlink=http://www.w3.org/1999/xlink

There might be better solutions, instead of defining the necessary structure elements for @TYPE of the mets:div, as the list can become quite long. Suggestions to improve the XSLT are welcome.

1 reply

andre-hohmann Feb 10, 2023
Collaborator Author

We were able to successfully migrate some test-processes in the SLUB. The structure elements as well as the unstructured elements ("pages") are sorted correctly.

I made a correction to the XSLT on 10/02/2023. The older version did not work with all Saxon engines.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Order of media files after migration is not correct #5518

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Order of media files after migration is not correct #5518

andre-hohmann Jan 23, 2023 Collaborator

Replies: 3 comments · 2 replies

m-kotzyba Jan 25, 2023 Collaborator

andre-hohmann Jan 26, 2023 Collaborator Author

m-kotzyba Jan 26, 2023 Collaborator

andre-hohmann Feb 6, 2023 Collaborator Author

andre-hohmann Feb 10, 2023 Collaborator Author

andre-hohmann
Jan 23, 2023
Collaborator

Replies: 3 comments 2 replies

m-kotzyba
Jan 25, 2023
Collaborator

andre-hohmann Jan 26, 2023
Collaborator Author

m-kotzyba
Jan 26, 2023
Collaborator

andre-hohmann
Feb 6, 2023
Collaborator Author

andre-hohmann Feb 10, 2023
Collaborator Author