OPT20039: Test the limits of Pecha annotation transfer functionality

[tenzin3]

Description

The primary functionality of this toolkit is transferring span annotations from one text to another. Other important features, such as alignment annotation transfer, heavily depend on this capability. Therefore, we aim to thoroughly test the limits of this functionality.

Illustration of a Simple Text Annotation Transfer

Expected Output

Establish clear metrics to determine the conditions under which annotation transfer is possible.

Gold standard data brain storming

Implementation Steps

• Prepare Benchmark data.
• Research other tools for text span annotation transfer.
• Test the tools.
• Enhance the toolkit's functionality based on test results.

Reviewer

• @ta4tsering

[tenzin3]

Benchmark Data Brainstorming

When preparing gold-standard data, all possible differences between source and target texts should be considered. This ensures comprehensive annotations and precise data preparation. Below are the scenarios, examples, and annotation spans:

---

i) Segmentation Differences

Explanation: Differences in how text is segmented, such as use of newlines, spaces, or tabs.

source_text = "This is first sentence.This is      second sentence.This is third sentence."
target_text = "This is first sentence.\nThis is  second sentence.This is     third sentence."

annotation_spans_to_transfer => [
  "This is first sentence.",
  "This is      second sentence.",
  "This is third sentence."
]

---

ii) Extra Text in the Middle

Explanation: The source text contains additional content not present in the target text.

source_text = "This is first sentence.This is the second sentence.This is third sentence."
target_text = "This is first sentence.This is second sentence.This is third sentence."

annotation_spans_to_transfer => [
  "This is the second sentence."
]

---

iii) Missing Text in the Middle

Explanation: The source text is missing content that is present in the target text.

source_text = "This is first sentence.This is the sentence.This is third sentence."
target_text = "This is first sentence.This is second sentence.This is third sentence."

annotation_spans_to_transfer => [
  "This is the sentence."
]

---

iv) Extra Text at middle of sentence.

Explanation: The source text contains extra content at the beginning.

source_text = "This is first sentence.Starting from here is nice.This is second sentence.This is third sentence."
target_text = "This is first sentence.This is second sentence.This is third sentence."

annotation_spans_to_transfer => [
  "Starting from here is nice."
]

---

v) Word/Phrase Differences

Explanation: The source text contains different words or phrases compared to the target text.

source_text = "This is first sentence.This is my own second sentence.This is third sentence."
target_text = "This is first sentence.This is second sentence.This is third sentence."

annotation_spans_to_transfer => [
  "This is my own second sentence."
]

---

vi) Reordering of Content

Explanation: The order of sentences or phrases differs between the source and target texts.

source_text = "This is first sentence. This is third sentence. This is second sentence."
target_text = "This is first sentence. This is second sentence. This is third sentence."

annotation_spans_to_transfer => [
  "This is third sentence.",
  "This is second sentence."
]

---

vi) Combination of above cases.