Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: move tests from dsp-tools repo to here (DEV-3173) #3

Merged
merged 4 commits into from
Jan 12, 2024
Merged

test: move tests from dsp-tools repo to here (DEV-3173) #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c87ca71
edit
jnussbaum Jan 12, 2024
7f6ecf7
renaming of workflows
jnussbaum Jan 12, 2024
bb2ead3
fix test collection
jnussbaum Jan 12, 2024
5005cfb
add check=True
jnussbaum Jan 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 6 additions & 10 deletions .github/workflows/compatibility_check.yml → .github/workflows/automated-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---

name: test compatibility with DSP-TOOLS
name: automated-tests

on:
schedule:
Expand All @@ -10,7 +10,7 @@ on:
- main

jobs:
compatibility:
automated-tests:
runs-on: ubuntu-latest
steps:
- name: Checkout repo
Expand All @@ -19,15 +19,11 @@ jobs:
uses: actions/setup-python@v4
with:
python-version: 3.12
- name: Install DSP-TOOLS
run: pip3 install dsp-tools
- name: run the import script
run: python3 import_script.py
- name: Install dependencies
run: pip3 install -r requirements.txt
- name: start stack
run: dsp-tools start-stack --no-prune
- name: create data model
run: dsp-tools create import_project.json
- name: upload data
run: dsp-tools xmlupload data-processed.xml
- name: run tests
run: pytest test


5 changes: 5 additions & 0 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
dsp-tools
lxml
regex
pandas
pytest
Empty file added test/__init__.py
View file
Open in desktop
Empty file.
205 changes: 205 additions & 0 deletions test/expected.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
<?xml version="1.0" encoding="UTF-8"?>
<knora xmlns="https://dasch.swiss/schema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://dasch.swiss/schema https://raw.githubusercontent.com/dasch-swiss/dsp-tools/main/src/dsp_tools/resources/schema/data.xsd" shortcode="00A1" default-ontology="import">
<permissions id="res-default">
<allow group="UnknownUser">V</allow>
<allow group="KnownUser">V</allow>
<allow group="ProjectMember">D</allow>
<allow group="ProjectAdmin">CR</allow>
<allow group="Creator">CR</allow>
</permissions>
<permissions id="res-restricted">
<allow group="ProjectMember">M</allow>
<allow group="ProjectAdmin">CR</allow>
<allow group="Creator">CR</allow>
</permissions>
<permissions id="prop-default">
<allow group="UnknownUser">V</allow>
<allow group="KnownUser">V</allow>
<allow group="ProjectMember">D</allow>
<allow group="ProjectAdmin">CR</allow>
<allow group="Creator">CR</allow>
</permissions>
<permissions id="prop-restricted">
<allow group="ProjectMember">M</allow>
<allow group="ProjectAdmin">CR</allow>
<allow group="Creator">CR</allow>
</permissions>
<resource label="Anubis.jpg" restype=":Image2D" id="Anubis.jpg_37fa98fd-a2e4-4031-bbf6-57da0ff82750" permissions="res-default">
<bitstream permissions="prop-default">images/Anubis.jpg</bitstream>
<text-prop name=":hasTitle">
<text permissions="prop-default" encoding="utf8">Anubis.jpg</text>
</text-prop>
</resource>
<resource label="Basho_Horohoroto.jpg" restype=":Image2D" id="Basho_Horohoroto.jpg_71a7e332-56cf-4e3b-a717-c34b2463e2c6" permissions="res-default">
<bitstream permissions="prop-default">images/Basho_Horohoroto.jpg</bitstream>
<text-prop name=":hasTitle">
<text permissions="prop-default" encoding="utf8">Basho_Horohoroto.jpg</text>
</text-prop>
</resource>
<resource label="BM1888-0601-716.png" restype=":Image2D" id="BM1888-0601-716.png_dbf7cc27-582a-4fc2-a28e-d5b66d29693c" permissions="res-default">
<bitstream permissions="prop-default">images/BM1888-0601-716.png</bitstream>
<text-prop name=":hasTitle">
<text permissions="prop-default" encoding="utf8">BM1888-0601-716.png</text>
</text-prop>
</resource>
<resource label="GibeonMeteorite.jpg" restype=":Image2D" id="GibeonMeteorite.jpg_79eeb9f5-96db-4595-addd-524306d0e586" permissions="res-default">
<bitstream permissions="prop-default">images/GibeonMeteorite.jpg</bitstream>
<text-prop name=":hasTitle">
<text permissions="prop-default" encoding="utf8">GibeonMeteorite.jpg</text>
</text-prop>
</resource>
<resource label="Anubis" restype=":Object" id="Anubis_fda7c834-692e-4e2c-b7c8-82e26a2775d9" permissions="res-default">
<resptr-prop name=":hasImage">
<resptr permissions="prop-default">Anubis.jpg_37fa98fd-a2e4-4031-bbf6-57da0ff82750</resptr>
</resptr-prop>
<text-prop name=":hasName">
<text permissions="prop-default" encoding="utf8">Bengal cat</text>
</text-prop>
<text-prop name=":hasDescription">
<text permissions="prop-restricted" comment="comment to 'Description'" encoding="xml">An example of a domesticated cat</text>
</text-prop>
<list-prop list="category" name=":hasCategory">
<list permissions="prop-default">mammals</list>
<list permissions="prop-default">humans</list>
</list-prop>
<boolean-prop name=":isPublic">
<boolean permissions="prop-default">true</boolean>
</boolean-prop>
<color-prop name=":hasColor">
<color permissions="prop-default">#f5f5dc</color>
</color-prop>
<date-prop name=":hasDate">
<date permissions="prop-default">GREGORIAN:CE:2015-01-01:CE:2015-01-01</date>
</date-prop>
<time-prop name=":hasTime">
<time permissions="prop-default">2015-01-01T13:45:12Z</time>
</time-prop>
<decimal-prop name=":hasWeight">
<decimal permissions="prop-default">4.8</decimal>
</decimal-prop>
<geoname-prop name=":hasLocation">
<geoname permissions="prop-default">2661604</geoname>
</geoname-prop>
<uri-prop name=":hasExternalLink">
<uri permissions="prop-default">https://en.wikipedia.org/wiki/Cat</uri>
</uri-prop>
</resource>
<resource label="Meteorite" restype=":Object" id="Meteorite_f726b5bb-2da3-4643-8922-70924e01d395" permissions="res-default">
<resptr-prop name=":hasImage">
<resptr permissions="prop-default">GibeonMeteorite.jpg_79eeb9f5-96db-4595-addd-524306d0e586</resptr>
</resptr-prop>
<text-prop name=":hasName">
<text permissions="prop-default" encoding="utf8">Gibeon Meteorite</text>
</text-prop>
<text-prop name=":hasDescription">
<text permissions="prop-restricted" comment="comment to 'Description'" encoding="xml">This is a piece of the so-called Gibeon Meteroite</text>
</text-prop>
<list-prop list="category" name=":hasCategory">
<list permissions="prop-default">physics</list>
</list-prop>
<boolean-prop name=":isPublic">
<boolean permissions="prop-default">true</boolean>
</boolean-prop>
<color-prop name=":hasColor">
<color permissions="prop-default">#808080</color>
</color-prop>
<date-prop name=":hasDate">
<date permissions="prop-default">GREGORIAN:CE:1908-03-05:CE:1908-03-05</date>
</date-prop>
<time-prop name=":hasTime">
<time permissions="prop-default">1908-03-05T12:00:00-05:00</time>
</time-prop>
<decimal-prop name=":hasWeight">
<decimal permissions="prop-default">0.3</decimal>
</decimal-prop>
<geoname-prop name=":hasLocation">
<geoname permissions="prop-default">11821111</geoname>
</geoname-prop>
<uri-prop name=":hasExternalLink">
<uri permissions="prop-default">https://en.wikipedia.org/wiki/Gibeon_(meteorite)</uri>
</uri-prop>
</resource>
<resource label="BM1888-0601-716" restype=":Object" id="BM1888-0601-716_97f7982b-eaed-4ef2-8813-cb93f8e2c480" permissions="res-default">
<resptr-prop name=":hasImage">
<resptr permissions="prop-default">BM1888-0601-716.png_dbf7cc27-582a-4fc2-a28e-d5b66d29693c</resptr>
</resptr-prop>
<text-prop name=":hasName">
<text permissions="prop-default" encoding="utf8">Lekythos</text>
</text-prop>
<text-prop name=":hasDescription">
<text permissions="prop-restricted" comment="comment to 'Description'" encoding="xml">Attic red-figured Lekythos BM 1888,601.716</text>
</text-prop>
<list-prop list="category" name=":hasCategory">
<list permissions="prop-default">artwork</list>
</list-prop>
<boolean-prop name=":isPublic">
<boolean permissions="prop-default">true</boolean>
</boolean-prop>
<date-prop name=":hasDate">
<date permissions="prop-default">GREGORIAN:CE:1973-12-01:CE:1974-01-06</date>
</date-prop>
<decimal-prop name=":hasWeight">
<decimal permissions="prop-default">0.5</decimal>
</decimal-prop>
<geoname-prop name=":hasLocation">
<geoname permissions="prop-default">351274</geoname>
</geoname-prop>
<uri-prop name=":hasExternalLink">
<uri permissions="prop-default">https://www.britishmuseum.org/collection/object/G_1888-0601-716</uri>
</uri-prop>
</resource>
<resource label="Horohoroto" restype=":Object" id="Horohoroto_5652b411-2c50-4086-896c-23b208ea27a8" permissions="res-default">
<resptr-prop name=":hasImage">
<resptr permissions="prop-default">Basho_Horohoroto.jpg_71a7e332-56cf-4e3b-a717-c34b2463e2c6</resptr>
</resptr-prop>
<text-prop name=":hasName">
<text permissions="prop-default" encoding="utf8">Picture and Poem by Matsuo Bashō</text>
</text-prop>
<text-prop name=":hasDescription">
<text permissions="prop-restricted" comment="comment to 'Description'" encoding="xml">ほろほろと山吹ちるかたきのおと</text>
</text-prop>
<list-prop list="category" name=":hasCategory">
<list permissions="prop-default">artwork</list>
</list-prop>
<date-prop name=":hasDate">
<date permissions="prop-default">GREGORIAN:CE:1849:CE:1850</date>
</date-prop>
<decimal-prop name=":hasWeight">
<decimal permissions="prop-default">1.0</decimal>
</decimal-prop>
<uri-prop name=":hasExternalLink">
<uri permissions="prop-default">https://en.wikipedia.org/wiki/Haiku#/media/File:Basho_Horohoroto.jpg</uri>
</uri-prop>
</resource>
<annotation label="Annotation to Anubis" id="annotation_to_anubis" permissions="res-default">
<text-prop name="hasComment">
<text permissions="prop-default" encoding="utf8">Date and time are invented, like for the other resources.</text>
</text-prop>
<resptr-prop name="isAnnotationOf">
<resptr permissions="prop-default">Anubis_fda7c834-692e-4e2c-b7c8-82e26a2775d9</resptr>
</resptr-prop>
</annotation>
<region label="Region of the Meteorite image" id="region_of_meteorite" permissions="res-default">
<text-prop name="hasComment">
<text permissions="prop-default" encoding="utf8">This is a comment</text>
</text-prop>
<color-prop name="hasColor">
<color permissions="prop-default">#5d1f1e</color>
</color-prop>
<resptr-prop name="isRegionOf">
<resptr permissions="prop-default">GibeonMeteorite.jpg_79eeb9f5-96db-4595-addd-524306d0e586</resptr>
</resptr-prop>
<geometry-prop name="hasGeometry">
<geometry permissions="prop-default">{"type": "rectangle", "lineColor": "#ff3333", "lineWidth": 2, "points": [{"x": 0.08, "y": 0.16}, {"x": 0.73, "y": 0.72}], "original_index": 0}</geometry>
</geometry-prop>
</region>
<link label="Link between BM1888-0601-716 and Horohoroto" id="link_BM1888-0601-716_horohoroto" permissions="res-default">
<text-prop name="hasComment">
<text permissions="prop-default" encoding="utf8">This is a comment</text>
</text-prop>
<resptr-prop name="hasLinkTo">
<resptr permissions="prop-default">BM1888-0601-716_97f7982b-eaed-4ef2-8813-cb93f8e2c480</resptr>
<resptr permissions="prop-default">Horohoroto_5652b411-2c50-4086-896c-23b208ea27a8</resptr>
</resptr-prop>
</link>
</knora>
87 changes: 87 additions & 0 deletions test/test_script.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
from collections.abc import Iterator
from pathlib import Path
import re
import subprocess

from dsp_tools import excel2xml
from lxml import etree
import pytest

import import_script


@pytest.fixture(scope="module")
def generated_xml_file() -> Iterator[Path]:
"""Yield the generated XML file as fixture, and delete it afterwards"""
xml_file = Path("data-processed.xml")
yield xml_file
xml_file.unlink(missing_ok=True)


@pytest.mark.filterwarnings("ignore::UserWarning")
def test_script_output(generated_xml_file: Path) -> None:
"""Execute the import script and compare the generated XML with the expected XML"""
import_script.main()
with open("test/expected.xml", encoding="utf-8") as f:
xml_expected = _derandomize_xsd_id(f.read(), multiple_occurrences=True)
with open(generated_xml_file, encoding="utf-8") as f:
xml_returned = _derandomize_xsd_id(f.read(), multiple_occurrences=True)
assert _sort_xml_by_id(xml_expected) == _sort_xml_by_id(xml_returned)
Comment on lines +25 to +29
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not particularly a fan of this kind of testing against serializations. In this context it's ok, but I would generally rather try to test for contents than for serializations



def test_create() -> None:
"""Create the project on the DSP server"""
subprocess.run("dsp-tools create import_project.json".split(), check=True)


def test_upload(generated_xml_file: Path) -> None:
"""Upload the created XML to the DSP server"""
subprocess.run(f"dsp-tools xmlupload {generated_xml_file}".split(), check=True)


def _sort_xml_by_id(xml: str) -> str:
"""Sort the elements in the XML by their ID"""
xml_tree = etree.fromstring(xml.encode("utf-8"))
for elem in xml_tree.iter():
elem[:] = sorted(elem, key=lambda x: x.attrib.get("id", ""))
return etree.tostring(xml_tree).decode("utf-8")


def _derandomize_xsd_id(
string: str,
multiple_occurrences: bool = False,
) -> str:
"""
In some contexts, the random component of the output of make_xsd_id_compatible() is a hindrance,
especially for testing.
This method removes the random part,
but leaves the other modifications introduced by make_xsd_id_compatible() in place.
This method's behaviour is defined by the example in the "Examples" section.

Args:
string: the output of make_xsd_id_compatible()
multiple_occurrences: If true, string can be an entire XML document, and all occurrences will be removed

Raises:
Exception: if the input cannot be derandomized

Returns:
the derandomized string

Examples:
>>> id_1 = make_xsd_id_compatible("Hello!")
>>> id_2 = make_xsd_id_compatible("Hello!")
>>> assert _derandomize_xsd_id(id_1) == _derandomize_xsd_id(id_2)
"""
if not isinstance(string, str) or not excel2xml.check_notna(string):
raise Exception(f"The input '{string}' cannot be derandomized.")

uuid4_regex = r"[a-f0-9]{8}-?[a-f0-9]{4}-?4[a-f0-9]{3}-?[89ab][a-f0-9]{3}-?[a-f0-9]{12}"
if multiple_occurrences:
return re.subn(uuid4_regex, "", string, flags=re.IGNORECASE)[0]
else:
return re.sub(uuid4_regex, "", string, re.IGNORECASE)


if __name__ == "__main__":
pytest.main([__file__])