Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BDRSPS-1109 Add support for multiple Surveys per dataset to SSD V3 #372

Merged
merged 8 commits into from
Dec 18, 2024
Merged

BDRSPS-1109 Add support for multiple Surveys per dataset to SSD V3 #372

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
663e447
Make test pass data as file-like object to apply_validation
Lincoln-GR Dec 12, 2024
9157d35
BDRSPS-1109 Make surveyID a required field, and allow multiple rows
Lincoln-GR Dec 12, 2024
220e788
BDRSPS-1109 Add plugin for validating surveyID matches the metadata t…
Lincoln-GR Dec 13, 2024
ace3554
BDRSPS-1109 Method to extract survey IDs for cross-validation
Lincoln-GR Dec 13, 2024
b7c883d
Add test files for new templates, copied from previous version
Lincoln-GR Dec 13, 2024
63a5058
BDRSPS-1109 Made surveyID a required field in site data file, and cro…
Lincoln-GR Dec 13, 2024
456aa79
BDRSPS-1109 Cross validate surveyID in occurrence template with metad…
Lincoln-GR Dec 13, 2024
b7df179
BDRSPS-1109 When surveyID is missing from an occurrence, treat it as …
Lincoln-GR Dec 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions abis_mapping/plugins/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
from . import required
from . import sites_geometry
from . import string_customized
from . import survey_id_validation
from . import tabular
from . import timestamp
from . import wkt
49 changes: 49 additions & 0 deletions abis_mapping/plugins/survey_id_validation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Third-Party
import attrs
import frictionless
import frictionless.errors

# Typing
from collections.abc import Iterator, Set


@attrs.define(kw_only=True, repr=False)
class SurveyIDValidation(frictionless.Check):
"""Validates that the surveyID field, if provided, is valid.

Attributes:
valid_survey_ids: surveyIDs from the metadata template.
"""

# Check Attributes
type = "survey-id-validation"
Errors = [frictionless.errors.RowConstraintError]

# Attributes specific to this check
valid_survey_ids: Set[str]

def validate_row(self, row: frictionless.Row) -> Iterator[frictionless.Error]:
"""Called to validate the given row (on every row).

Args:
row (frictionless.Row): The row to check.

Yields:
frictionless.Error: Any errors detected.
"""
# Get surveyID value, which can be None
survey_id: str | None = row["surveyID"]
# If you want surveyID to be required, use "required": true in conjunction
# with this check.

# If None, don't check anything
if survey_id is None:
return

# Otherwise check that surveyID is one of the valid values
if survey_id not in self.valid_survey_ids:
yield frictionless.errors.ConstraintError.from_row(
row=row,
note="surveyID must match a surveyID in the survey_metadata template",
field_name="surveyID",
)
1 change: 1 addition & 0 deletions abis_mapping/templates/survey_metadata_v3/examples/minimal.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
surveyID,surveyName,surveyPurpose,surveyType,surveyStart,surveyEnd,targetTaxonomicScope,targetHabitatScope,spatialCoverageWKT,geodeticDatum,surveyOrgs,surveyMethodCitation,surveyMethodDescription,surveyMethodURL,keywords
COL1,"Disentangling the effects of farmland use, habitat edges, and vegetation structure on ground beetle morphological traits - Summer",Summer sampling for peak insect diversity.,Wet pitfall trapping,21/01/2015,3/02/2015,Coleoptera | Insecta,Woodland,"POLYGON ((146.363 -33.826, 148.499 -33.826, 148.499 -34.411, 146.363 -33.826))",GDA2020,"NSW Department of Planning, Industry and Environment | CSIRO","Ng, K., Barton, P.S., Blanchard, W. et al. Disentangling the effects of farmland use, habitat edges, and vegetation structure on ground beetle morphological traits. Oecologia 188, 645–657 (2018). https://doi.org/10.1007/s00442-018-4180-9""","Our experimental design consisted of four 400 m transects running from inside each woodland patch out into four adjoining farmland uses (crop, rested, woody debris application, revegetation plantings). To quantify potential edge efects on beetle species traits, we sampled beetles at five locations along each transect: 200 and 20 m inside woodlands, 200 and 20 m inside farmlands, and at the woodland–farmland edge (0 m). Each sampling location comprised a pair of wet invertebrate pitfall traps. separated by a drift fence (60 cm long x 10 cm high) to help direct arthropods into traps. We opened a total of 220 pairs of traps for 14 days during spring (Oct–Nov 2014), and repeated sampling during summer (January–February 2015). Beetle samples from each pitfall trap pair, and across the two time periods, were pooled to provide one sample per sampling location.",https://doi.org/10.1002/9781118945568.ch11 | https://biocollect.ala.org.au/document/download/2022-01/202201%20CBR%20Flora%20and%20Vegetation%20report_draftv1.pdf ,ground beetle | habitat | morphology | traits | farmland | woodland | remnant vegetation | split-plot study
COL2,"Disentangling the effects of farmland use, habitat edges, and vegetation structure on ground beetle morphological traits - Winter",Winter sampling for peak insect diversity.,Wet pitfall trapping,21/06/2015,3/07/2015,Coleoptera | Insecta,Woodland,"POLYGON ((146.363 -33.826, 148.499 -33.826, 148.499 -34.411, 146.363 -33.826))",GDA2020,"NSW Department of Planning, Industry and Environment | CSIRO","Ng, K., Barton, P.S., Blanchard, W. et al. Disentangling the effects of farmland use, habitat edges, and vegetation structure on ground beetle morphological traits. Oecologia 188, 645–657 (2018). https://doi.org/10.1007/s00442-018-4180-9""","Our experimental design consisted of four 400 m transects running from inside each woodland patch out into four adjoining farmland uses (crop, rested, woody debris application, revegetation plantings). To quantify potential edge efects on beetle species traits, we sampled beetles at five locations along each transect: 200 and 20 m inside woodlands, 200 and 20 m inside farmlands, and at the woodland–farmland edge (0 m). Each sampling location comprised a pair of wet invertebrate pitfall traps. separated by a drift fence (60 cm long x 10 cm high) to help direct arthropods into traps. We opened a total of 220 pairs of traps for 14 days during spring (Oct–Nov 2014), and repeated sampling during summer (January–February 2015). Beetle samples from each pitfall trap pair, and across the two time periods, were pooled to provide one sample per sampling location.",https://doi.org/10.1002/9781118945568.ch11 | https://biocollect.ala.org.au/document/download/2022-01/202201%20CBR%20Flora%20and%20Vegetation%20report_draftv1.pdf ,ground beetle | habitat | morphology | traits | farmland | woodland | remnant vegetation | split-plot study
77 changes: 67 additions & 10 deletions abis_mapping/templates/survey_metadata_v3/examples/minimal.ttl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,25 +12,29 @@

<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/SurveyCollection/surveyType/Wet-pitfall-trapping> a schema:Collection ;
schema:isPartOf <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000> ;
schema:member <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL1> ;
schema:member <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL1>,
<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL2> ;
schema:name "Survey Collection - Survey Type - Wet pitfall trapping" ;
tern:hasAttribute <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/attribute/surveyType/Wet-pitfall-trapping> .

<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/SurveyCollection/targetHabitatScope/Woodland> a schema:Collection ;
schema:isPartOf <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000> ;
schema:member <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL1> ;
schema:member <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL1>,
<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL2> ;
schema:name "Survey Collection - Target Habitat Scope - Woodland" ;
tern:hasAttribute <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/attribute/targetHabitatScope/Woodland> .

<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/SurveyCollection/targetTaxonomicScope/Coleoptera> a schema:Collection ;
schema:isPartOf <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000> ;
schema:member <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL1> ;
schema:member <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL1>,
<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL2> ;
schema:name "Survey Collection - Target Taxonomic Scope - Coleoptera" ;
tern:hasAttribute <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/attribute/targetTaxonomicScope/Coleoptera> .

<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/SurveyCollection/targetTaxonomicScope/Insecta> a schema:Collection ;
schema:isPartOf <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000> ;
schema:member <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL1> ;
schema:member <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL1>,
<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL2> ;
schema:name "Survey Collection - Target Taxonomic Scope - Insecta" ;
tern:hasAttribute <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/attribute/targetTaxonomicScope/Insecta> .

Expand All @@ -40,15 +44,27 @@
schema:isPartOf <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000> ;
schema:name "Disentangling the effects of farmland use, habitat edges, and vegetation structure on ground beetle morphological traits - Summer" .

<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/project/SSD-Survey-Project/2> a abis:Project ;
schema:hasPart <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/Survey/COL2> ;
schema:identifier "COL2" ;
schema:isPartOf <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000> ;
schema:name "Disentangling the effects of farmland use, habitat edges, and vegetation structure on ground beetle morphological traits - Winter" .

<https://linked.data.gov.au/dataset/bdr/datatypes/surveyID/CSIRO> a rdfs:Datatype ;
skos:prefLabel "surveyID source" ;
prov:qualifiedAttribution [ a prov:Attribution ;
prov:agent <https://linked.data.gov.au/dataset/bdr/orgs/CSIRO> ;
prov:hadRole <https://linked.data.gov.au/def/data-roles/principalInvestigator> ],
[ a prov:Attribution ;
prov:agent <https://linked.data.gov.au/dataset/bdr/orgs/CSIRO> ;
prov:hadRole <https://linked.data.gov.au/def/data-roles/principalInvestigator> ] .

<https://linked.data.gov.au/dataset/bdr/datatypes/surveyID/NSW-Department-of-Planning-Industry-and-Environment> a rdfs:Datatype ;
skos:prefLabel "surveyID source" ;
prov:qualifiedAttribution [ a prov:Attribution ;
prov:agent <https://linked.data.gov.au/dataset/bdr/orgs/NSW-Department-of-Planning-Industry-and-Environment> ;
prov:hadRole <https://linked.data.gov.au/def/data-roles/principalInvestigator> ],
[ a prov:Attribution ;
prov:agent <https://linked.data.gov.au/dataset/bdr/orgs/NSW-Department-of-Planning-Industry-and-Environment> ;
prov:hadRole <https://linked.data.gov.au/def/data-roles/principalInvestigator> ] .

Expand Down Expand Up @@ -110,12 +126,6 @@
rdfs:label "Insecta" ;
rdf:value <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/bdr-cv/attribute/targetTaxonomicScope/Insecta> .

<https://linked.data.gov.au/dataset/bdr/orgs/CSIRO> a prov:Agent ;
schema:name "CSIRO" .

<https://linked.data.gov.au/dataset/bdr/orgs/NSW-Department-of-Planning-Industry-and-Environment> a prov:Agent ;
schema:name "NSW Department of Planning, Industry and Environment" .

<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/Survey/COL1> a tern:Survey ;
bdr:purpose "Summer sampling for peak insect diversity." ;
bdr:target "Coleoptera",
Expand All @@ -139,13 +149,49 @@
"woodland" ;
schema:name "Disentangling the effects of farmland use, habitat edges, and vegetation structure on ground beetle morphological traits - Summer" .

<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/Survey/COL2> a tern:Survey ;
bdr:purpose "Winter sampling for peak insect diversity." ;
bdr:target "Coleoptera",
"Insecta" ;
geo:hasGeometry _:N27fdbd9c9077333e51ac0d0600000001 ;
time:hasTime [ a time:TemporalEntity ;
time:hasBeginning [ a time:Instant ;
time:inXSDDate "2015-06-21"^^xsd:date ] ;
time:hasEnd [ a time:Instant ;
time:inXSDDate "2015-07-03"^^xsd:date ] ] ;
prov:hadPlan <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL2> ;
schema:identifier "COL2"^^<https://linked.data.gov.au/dataset/bdr/datatypes/surveyID/CSIRO>,
"COL2"^^<https://linked.data.gov.au/dataset/bdr/datatypes/surveyID/NSW-Department-of-Planning-Industry-and-Environment> ;
schema:keywords "farmland",
"ground beetle",
"habitat",
"morphology",
"remnant vegetation",
"split-plot study",
"traits",
"woodland" ;
schema:name "Disentangling the effects of farmland use, habitat edges, and vegetation structure on ground beetle morphological traits - Winter" .

<https://linked.data.gov.au/dataset/bdr/orgs/CSIRO> a prov:Agent ;
schema:name "CSIRO" .

<https://linked.data.gov.au/dataset/bdr/orgs/NSW-Department-of-Planning-Industry-and-Environment> a prov:Agent ;
schema:name "NSW Department of Planning, Industry and Environment" .

<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL1> a prov:Plan ;
schema:citation "Ng, K., Barton, P.S., Blanchard, W. et al. Disentangling the effects of farmland use, habitat edges, and vegetation structure on ground beetle morphological traits. Oecologia 188, 645–657 (2018). https://doi.org/10.1007/s00442-018-4180-9\"" ;
schema:description "Our experimental design consisted of four 400 m transects running from inside each woodland patch out into four adjoining farmland uses (crop, rested, woody debris application, revegetation plantings). To quantify potential edge efects on beetle species traits, we sampled beetles at five locations along each transect: 200 and 20 m inside woodlands, 200 and 20 m inside farmlands, and at the woodland–farmland edge (0 m). Each sampling location comprised a pair of wet invertebrate pitfall traps. separated by a drift fence (60 cm long x 10 cm high) to help direct arthropods into traps. We opened a total of 220 pairs of traps for 14 days during spring (Oct–Nov 2014), and repeated sampling during summer (January–February 2015). Beetle samples from each pitfall trap pair, and across the two time periods, were pooled to provide one sample per sampling location." ;
schema:isPartOf <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000> ;
schema:url "https://biocollect.ala.org.au/document/download/2022-01/202201%20CBR%20Flora%20and%20Vegetation%20report_draftv1.pdf"^^xsd:anyURI,
"https://doi.org/10.1002/9781118945568.ch11"^^xsd:anyURI .

<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/survey/plan/COL2> a prov:Plan ;
schema:citation "Ng, K., Barton, P.S., Blanchard, W. et al. Disentangling the effects of farmland use, habitat edges, and vegetation structure on ground beetle morphological traits. Oecologia 188, 645–657 (2018). https://doi.org/10.1007/s00442-018-4180-9\"" ;
schema:description "Our experimental design consisted of four 400 m transects running from inside each woodland patch out into four adjoining farmland uses (crop, rested, woody debris application, revegetation plantings). To quantify potential edge efects on beetle species traits, we sampled beetles at five locations along each transect: 200 and 20 m inside woodlands, 200 and 20 m inside farmlands, and at the woodland–farmland edge (0 m). Each sampling location comprised a pair of wet invertebrate pitfall traps. separated by a drift fence (60 cm long x 10 cm high) to help direct arthropods into traps. We opened a total of 220 pairs of traps for 14 days during spring (Oct–Nov 2014), and repeated sampling during summer (January–February 2015). Beetle samples from each pitfall trap pair, and across the two time periods, were pooled to provide one sample per sampling location." ;
schema:isPartOf <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000> ;
schema:url "https://biocollect.ala.org.au/document/download/2022-01/202201%20CBR%20Flora%20and%20Vegetation%20report_draftv1.pdf"^^xsd:anyURI,
"https://doi.org/10.1002/9781118945568.ch11"^^xsd:anyURI .

<https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000> a tern:Dataset .

[] a rdf:Statement ;
Expand All @@ -156,6 +202,17 @@
rdf:subject <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/Survey/COL1> ;
rdfs:comment "supplied as" .

[] a rdf:Statement ;
geo:hasGeometry [ a geo:Geometry ;
geo:asWKT "<http://www.opengis.net/def/crs/EPSG/0/7844> POLYGON ((-33.826 146.363, -33.826 148.499, -34.411 148.499, -33.826 146.363))"^^geo:wktLiteral ] ;
rdf:object _:N27fdbd9c9077333e51ac0d0600000001 ;
rdf:predicate geo:hasGeometry ;
rdf:subject <https://linked.data.gov.au/dataset/bdr/00000000-0000-0000-0000-000000000000/Survey/COL2> ;
rdfs:comment "supplied as" .

_:N27fdbd9c9077333e51ac0d0600000000 a geo:Geometry ;
geo:asWKT "<http://www.opengis.net/def/crs/EPSG/0/7844> POLYGON ((-33.826 146.363, -33.826 148.499, -34.411 148.499, -33.826 146.363))"^^geo:wktLiteral .

_:N27fdbd9c9077333e51ac0d0600000001 a geo:Geometry ;
geo:asWKT "<http://www.opengis.net/def/crs/EPSG/0/7844> POLYGON ((-33.826 146.363, -33.826 148.499, -34.411 148.499, -33.826 146.363))"^^geo:wktLiteral .

Loading
Loading