Update read_camtrap_dp() so it can read Camtrap DP 0.1.6 and 1.0-rc.1

[peterdesmet]

For camtraptor v0.20 we envision read_camtrap_dp() to be able to read legacy Agouti exports (Camtrap DP 0.1.6) and new Agouti exports (Camtrap DP 1.0-rc.1) without affecting the other functions (i.e. non-breaking change).

To do this, read_camtrap_dp() should:

• Detect the version in package$profile
• Ignore or deprecate the media flag (debatable): decided to not do it in v0.20 (see Update read_camtrap_dp() so it can read Camtrap DP 0.1.6 and 1.0-rc.1 #196 (comment))

The required changes are listed below.

profile

• contributor$role: author/maintainer are deprecated in 1.0-rc.1, some new roles are added. Ok to leave as is.
• organizations: deprecated in 1.0-rc.1. Ok if missing.
• project$animalTypes: deprecated in 1.0-rc.1. Ok if missing.
• project$individualAnimals: new in 1.0-rc.1. Ok to leave in.
• project$classificationLevel: renamed to observationLevel in 1.0-rc.1. Convert to old name.
• project$sequenceInterval: deprecated in 1.0-rc.1. Ok if missing.
• coordinatePrecision: new in 1.0-rc.1. Ok to leave in.
• taxonomic[x]$kingdom, etc: new in 1.0-rc.1. Ok to leave in.
• platform: deprecated in 1.0-rc.1. Look for first element in sources and assign title as platform$title
• rightsHolder: deprecated in 1.0-rc.1. Look for first contributor with role rightsHolder and assign title as rightsHolder
• relatedIdentifiers: new in 1.0-rc.1. Ok to leave in.
• references: new in 1.0-rc.1 (moved from project$references, but was never populated for Agouti). Ok to leave in.

$data$deployments

• deploymentID: same
• locationID: same
• locationName: same
• longitude: order switched
• latitude: order switched
• coordinateUncertainty: same
• start: rename from eventStart
• end: rename from eventEnd
• setupBy: same
• cameraID: same
• cameraModel: same
• cameraInterval: rename from cameraDelay
• cameraHeight: same
• cameraTilt: same
• cameraHeading: same
• detectionDistance: new, ignore
• timestampIssues: same
• baitUse: convert from boolean baitUse
• session: deprecated, leave empty
• array: deprecated, leave empty
• featureType: same
• habitat: same
• tags: rename from deploymentTags
• comments: rename from deploymentComments
• _id: deprecated, leave empty

$data$media

• mediaID: same
• deploymentID: same
• sequenceID: deprecated, leave empty? Not sure if this will break things. created by joining media and observations on deploymentID and timestamp between eventStart and eventEnd as proposed in Single observation table tdwg/camtrap-dp#314
• captureMethod: same
• timestamp: same
• filePath: same
• filePublic: new term, ignore
• fileName: same
• fileMediatype: same
• exifData: same
• favourite: rename from favorite
• comments: rename from mediaComments
• _id: deprecated, leave empty

$data$observations

• Filter on observationLevel = event
• observationID: same
• deploymentID: same
• sequenceID: rename from eventID
• mediaID: same
• timestamp: rename from eventStart
• eventEnd: new term, ignore
• observationLevel: new term, ignore
• observationType: same
• cameraSetup: rename from cameraSetupType
• taxonID: same
• scientificName: same
• count: same
• countNew: deprecated, leave empty
• lifeStage: same
• sex: same
• behaviour: rename from behavior
• individualID: same
• classificationMethod: same
• classifiedBy: same
• classificationTimestamp: same
• classificationConfidence: rename from classificationProbability
• observationTags: new term, ignore
• comments: rename from observationComments
• _id: deprecated, leave empty
• speed: rename from individualSpeed
• radius: cannot be supported before we use the new 1.0 model (decided to read them in if present, see Update read_camtrap_dp() so it can read Camtrap DP 0.1.6 and 1.0-rc.1 #196 (comment))
• angle: cannot be supported before we use the new 1.0 model (same as radius)
• bboxX: new term, ignore
• bboxY: new term, ignore
• bboxWidth: new term, ignore
• bboxHeight: new term, ignore

[damianooldoni]

I would both ignore AND deprecate media arg (flag). Done in 7bd2106

[damianooldoni]

After discussion with @peterdesmet, we will:

• support only camtrap-dp packages formatted using versions 0.1.6 and 0.6. Other versions are not supported and an informative message will be returned
• won't load media-observations as extra slot of package$data as it could be very misleading for the users. It will make things more difficult for us to test the package as pkg$data will not contain the same number of slots. Notice that we still make use of media-observations for populating the fields radius and angle in observations.

The factor value to use for baitUse field will be defined based on values dep.tags and boolean dep.baitUse:

dep.baitUse	dep.tags	baitUse
TRUE	no bait related tags	other
TRUE	bait related tag NOT among the ones allowed	other
TRUE	bait related tag among the ones allowed	bait related tag
FALSE	ignored	none
NA	ignored	NA

[damianooldoni]

@peterdesmet: about individualPosition and individualAngle, you wrote:

• radius: cannot be supported before we use the new 1.0 model
• angle: cannot be supported before we use the new 1.0 model

We provided a patch to read not-in-schema terms speed, radius and angle present in Agouti exports (#185) following v0.1.6. So, I wonder if we can at the moment still support radius and angle by renaming from individualPositionRadius- Angle even if they are not really make sense for event-based observations?

[damianooldoni]

Decided to still leave media argument as in version v1.0-rc-1 we can still "bypass" media table to get (event-based) observations.

[damianooldoni]

Decided with @peterdesmet to map "new" field deploymentGroups to "old" field session. This shouldn't have a significant impact at the moment.

[damianooldoni]

Notice that get_record_table() uses sequenceID in media. So, we cannot leave it empty. Added via workflow proposed by @peterdesmet in tdwg/camtrap-dp#314.

[damianooldoni]

Notice also that new new format has 4 bounding box related columns:

• bboxX
• bboxY
• bboxWidth
• bboxHeight

I will not remove nor rename them. List terms in original comment of @peterdesmet updated

[peterdesmet]

If you retain them, will that effect tests?

[damianooldoni]

The only affected test was the one where I check the column names. I run all package functions locally without any issue. Notice that I didn't write new tests for each test. I wrote "only" a lot of new tests for checking that the output of read_camtrap_dp() is as expected, and so very similar to the old one.

[peterdesmet]

If the bbox columns are the only ones we add, I would ignore them at this stage as they are not part of 0.1.6. Doesn’t feel right to make an exception for just these.

[damianooldoni]

Decided with @peterdesmet to remove bounding box related columns as they could be confusing for the user at this stage.

[damianooldoni]

Agreed with @peterdesmet: do not add taxonRank from taxonomic slot (new term in 1.0-rc.1) while building .data$observations.
In this way, the only difference among the columns of .$data$observations is the 2 vs 3 letters to indicate vernacular language, e.g. vernacularName.eng vs vernacularName.en.

[damianooldoni]

I see in https://github.com/tdwg/camtrap-dp/blob/ef261578d956e1d3ece78ead760e0bd2d4df81c8/camtrap-dp-profile.json#L231C16-L245 that taxonRank is a field in taxonomic element in *v0.1.6 and so it is must NOT be removed! The fact that the example datapackage we were using (mica) doesn't contain it's not a reason for removing it.

[damianooldoni]

Contrarely to taxonRank, we shouldn't add higher taxonomic classification fields (kingdom, phylum, etc.) as they are not valid in v0.1.6.

[damianooldoni]

Solved by #223