Skip to content

Commit

Permalink
Merge branch 'main' of github.com:touche-webis-de/touche-webis-de.git…
Browse files Browse the repository at this point in the history 
…hub.io
  • Loading branch information
@johanneskiesel
johanneskiesel committed Jul 6, 2024
2 parents d6bedb6 + 6038048 commit 0838dc8
Show file tree
Hide file tree
Showing 6 changed files with 800 additions and 24 deletions.
680 changes: 680 additions & 0 deletions clef24/touche24-data/results.jsonl
View file

Large diffs are not rendered by default.

14 changes: 7 additions & 7 deletions clef24/touche24-web/human-value-detection.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ <h2 id="synopsis">Synopsis</h2>
<li>Communication: [mailing lists: <a href="https://groups.google.com/group/valueeval/subscribe">task</a>, <a href="mailto:[email protected]">organizers</a>] [<a href="https://twitter.com/ValueEval">twitter/x</a>]</li>
<!-- <li>Submission: [paper template: <a href="https://www.overleaf.com/latex/templates/valueeval23-paper-template/htprfrpkkfhh">overleaf</a>, <a href="https://github.com/touche-webis-de/touche-code/raw/main/semeval23/human-value-detection/participant-paper-template/participant-paper-template.zip">download</a>, <a href="https://github.com/touche-webis-de/touche-code/tree/main/semeval23/human-value-detection/participant-tables">tables</a>, <a href="https://github.com/touche-webis-de/touche-code/tree/main/semeval23/human-value-detection/value-radar-plot">radar plots</a>] [<a href="https://www.tira.io/task/valueeval-at-semeval-2023-human-value-detection">tira</a>] [<a href="https://softconf.com/acl2023/SemEval2023/">paper submission site</a>]</li> -->
<li>Data: [<a href="https://zenodo.org/doi/10.5281/zenodo.10396293">download</a>] [<a href="https://knowledge4policy.ec.europa.eu/projects-activities/valuesml-unravelling-expressed-values-media-informed-policy-making_en">project</a>]</li>
<li>Submission: [<a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/approaches">example approaches</a>] [<a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/evaluator">evaluator</a>] [<a href="https://www.tira.io/c/touche/">forum</a>] [<a href="https://www.tira.io/task-overview/valueeval-2024-human-value-detection/valueeval24-2024-04-15-test-20240415-test">submit</a>]</li>
<li>Submission: [<a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/approaches">example approaches</a>] [<a href="https://valueeval24.web.webis.de/">web ui</a>] [<a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/evaluator">evaluator</a>] [<a href="https://www.tira.io/c/touche/">forum</a>] [<a href="https://www.tira.io/task-overview/valueeval-2024-human-value-detection/valueeval24-2024-04-15-test-20240415-test">submit</a>]</li>
<li>ValueEval'23: [<a href="https://touche.webis.de/semeval23/touche23-web/index.html">website</a>] [<a href="https://values.args.me/">demo</a>]</li>
</ul>
<a class="uk-button uk-button-primary" href="https://clef2024-labs-registration.dei.unipd.it/">Register now</a>
Expand All @@ -102,15 +102,15 @@ <h2 id="synopsis">Synopsis</h2>
<h2 id="important-dates" class="uk-margin-large-top">Important Dates</h2>
<p>Subscribe to the <a href="https://groups.google.com/group/valueeval/subscribe">mailing list</a> to receive notifications.</p>
<ul>
<li><strong>Dec. 18, 2023:</strong> CLEF Registration opens. [<a href="https://clef2024-labs-registration.dei.unipd.it/">register</a>]</li>
<li><strong>May 6, 2024:</strong> Approaches submission deadline.</li>
<li><strong>May 31, 2024:</strong> Participant paper submission<!-- [<a href="../touche23-data/touche-paper-template.zip">template</a>] [<a href="../touche23-data/instructions.txt">short instructions</a>] [<a href="https://clef2023.clef-initiative.eu/resources/wn_instructions.pdf">full instructions</a>] [<a href="https://www.easychair.org/my/conference?conf=clef2023">submit</a>]-->.</li>
<li><strong>June 21, 2024:</strong> Peer review notification.</li>
<!-- <li><strong>Dec. 18, 2023:</strong> CLEF Registration opens. [<a href="https://clef2024-labs-registration.dei.unipd.it/">register</a>]</li> -->
<li><strong>May 20, 2024:</strong> EXTENDED approaches submission deadline. [<a href="https://www.tira.io/task-overview/valueeval-2024-human-value-detection/valueeval24-2024-04-15-test-20240415-test">register via submission system (SUBMIT)</a>]</li>
<li><strong>May 31, 2024:</strong> Participant paper submission [<a href="https://www.overleaf.com/latex/templates/notebook-for-the-touche-lab-at-clef-2024/czqbtmxjymfy">paper template + submission instructions</a>].</li>
<li><strong>June 24, 2024:</strong> Peer review notification.</li>
<li><strong>July 8, 2024:</strong> Camera-ready participant papers submission<!--[<a href="../touche23-data/touche-paper-template.zip">template</a>] [<a href="../touche23-data/instructions.txt">short instructions</a>] [<a href="https://clef2023.clef-initiative.eu/resources/wn_instructions.pdf">full instructions</a>] [<a href="https://www.easychair.org/my/conference?conf=clef2023">submit</a>]-->.</li>
<li><strong>Sep. 9-12, 2024:</strong> <a href="https://clef2024.imag.fr/">CLEF Conference</a> in Grenoble and Touché Workshop.</li>
<!-- <li><strong>September 8, 2024:</strong> Touché Workshop.</li> -->
</ul>
<p>All deadlines are 23:59 <a href="https://time.is/Anywhere_on_Earth">anywhere on earth</a> (UTC-12).</p>
<p>All deadlines are 23:59 <a href="https://time.is/CEST">CEST</a> (UTC+2).</p>

<h2 id="task">Task</h2>

Expand Down Expand Up @@ -414,7 +414,7 @@ <h2 id="data">Data</h2>

<h2 id="submission">Submission</h2>
<p>Submit your approach via <a href="https://www.tira.io/task-overview/valueeval-2024-human-value-detection">TIRA</a>. Ask in the <a href="https://www.tira.io/c/touche/">Forum</a> if you need help. You need to register your team (in addition to a registration at CLEF) and pick an alias for your team name (submission is anonymous; you can reveal you true team name after final paper acceptance). You submit a Docker image or submit from your Github repository (via automated Docker building). In case of trouble, you can also submit via run file upload (not recommended due to poor reproducibility; rather contact us if you need help with Dockerization). You can submit on the validation dataset to check how submission works or the test dataset. You will not be able to see your results on the test dataset until after the deadline. Datasets are provided multilingual or in machine-translated English (see <a href="#data">Data</a>). [<a href="https://www.tira.io/c/touche/">forum</a>] [<a href="https://www.tira.io/task-overview/valueeval-2024-human-value-detection">submit</a>]</p>
<p>We recommend to start your approach from one of our <a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/approaches">example approaches</a> (in Python), which include the code for reading and writing the files and make it easy to later deploy your approach as server or submit and distribute it as Docker image. [random baseline: <a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/approaches/random-baseline">script</a>, <a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/approaches/random-baseline-notebook">notebook</a>] [<a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/approaches/bert-baseline">bert baseline</a>] [<a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/approaches/ollama-baseline">ollama baseline</a>]</p>
<p>We recommend to start your approach from one of our <a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/approaches">example approaches</a> (in Python), which include the code for reading and writing the files and make it easy to later deploy your approach as server or submit and distribute it as Docker image. If you run them as a local server, you can use our <a href="https://valueeval24.web.webis.de/">web ui</a> to use them interactively. [random baseline: <a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/approaches/random-baseline">script</a>, <a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/approaches/random-baseline-notebook">notebook</a>] [<a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/approaches/bert-baseline">bert baseline</a>] [<a href="https://github.com/touche-webis-de/touche-code/tree/main/clef24/human-value-detection/approaches/ollama-baseline">ollama baseline</a>]</p>
<p>Approaches need to produce run files that have the same format as the <code>labels.tsv</code>, but the numbers can be between 0 and 1 and are interpreted as the confidence of the approach (employed for evaluation via ROC-curves): [<a href="https://github.com/touche-webis-de/touche-code/blob/main/clef24/human-value-detection/toy-dataset/run.tsv">toy example</a>]</p>
<ul>
<li><p>For sub-task 1: For each sentence and value, the sum of the numbers in the attained and constrained columns should be the confidence of your approach in that the sentence references the value. A sum &GreaterEqual; 0.5 is treated as a positive prediction for purposes of evaluation with precision, recall, and F1-score.<br>
Expand Down
42 changes: 34 additions & 8 deletions clef24/touche24-web/ideology-and-power-identification-in-parliamentary-debates.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ <h2 id="synopsis">Synopsis</h2>
<li>Sub-Task 2: Given a parliamentary speech in one of several languages, identify whether the speaker's party is currently governing or in opposition.</li>
<li>Communication: [mailing lists: <a href="https://groups.google.com/g/ideology-and-power-in-parliamentary-speeches">task</a>, <a href="mailto:[email protected]">organizers</a>] </li>
<li>Training data: [<a href="https://zenodo.org/doi/10.5281/zenodo.10450640">download</a>]</li>
<li>Test data: [<a href="https://zenodo.org/doi/10.5281/zenodo.11061649">download</a>]</li>
<li>Submission: [<a href="https://github.com/coltekin/ideology-power-st-baseline">baseline</a>] [<a href="https://github.com/coltekin/ideology-power-st-baseline/tree/main/evaluation">evaluator</a>] [<a href="https://www.tira.io/c/touche/">forum</a>] [<a href="https://www.tira.io/task-overview/ideology-and-power-identification-in-parliamentary-debates-2024">submit</a>]</li>

<!-- <li>Input: [<a href="../touche24-data/topics-task3.xml" download>topics</a>]</li> -->
<!-- <li>Submission: [queries: <a href="https://github.com/touche-webis-de/touche-code/tree/main/clef23/image-retrieval-for-arguments/query-submission-baselines">baselines code</a>, <a href="https://github.com/touche-webis-de/touche-code/tree/main/clef23/image-retrieval-for-arguments/query-submission-evaluator">validator</a>] [<a href="https://www.tira.io/task/touche-task-3">submit</a>]</li> -->
</ul>
Expand All @@ -52,14 +55,14 @@ <h2 id="important-dates" class="uk-margin-large-top">Important Dates</h2>
<p>Subscribe to the <a href="https://groups.google.com/g/ideology-and-power-in-parliamentary-speeches">mailing list</a> to receive notifications.</p>
<ul>
<li><strong>Dec. 18, 2023:</strong> CLEF Registration opens. [<a href="https://clef2024-labs-registration.dei.unipd.it/">register</a>]</li>
<li><strong>May 6, 2024:</strong> Approaches submission deadline.</li>
<li><strong>May 20, 2024:</strong> EXTENDED approaches submission deadline.</li>
<li><strong>May 31, 2024:</strong> Participant paper submission<!-- [<a href="../touche23-data/touche-paper-template.zip">template</a>] [<a href="../touche23-data/instructions.txt">short instructions</a>] [<a href="https://clef2023.clef-initiative.eu/resources/wn_instructions.pdf">full instructions</a>] [<a href="https://www.easychair.org/my/conference?conf=clef2023">submit</a>]-->.</li>
<li><strong>June 21, 2024:</strong> Peer review notification.</li>
<li><strong>June 24, 2024:</strong> Peer review notification.</li>
<li><strong>July 8, 2024:</strong> Camera-ready participant papers submission<!--[<a href="../touche23-data/touche-paper-template.zip">template</a>] [<a href="../touche23-data/instructions.txt">short instructions</a>] [<a href="https://clef2023.clef-initiative.eu/resources/wn_instructions.pdf">full instructions</a>] [<a href="https://www.easychair.org/my/conference?conf=clef2023">submit</a>]-->.</li>
<li><strong>Sep. 9-12, 2024:</strong> <a href="https://clef2024.imag.fr/">CLEF Conference</a> in Grenoble and Touché Workshop.</li>
<!-- <li><strong>September 8, 2024:</strong> Touché Workshop.</li> -->
</ul>
<p>All deadlines are 23:59 <a href="https://time.is/Anywhere_on_Earth">anywhere on earth</a> (UTC-12).</p>
<p>All deadlines are 23:59 <a href="https://time.is/CEST">CEST</a> (UTC+2).</p>

<h2 id="task">Task</h2>
Debates in national parliaments do not only affect
Expand Down Expand Up @@ -109,8 +112,8 @@ <h2 id="data">Data</h2>
non-English speeches.
<li> <b>label</b> is the binary/numeric label. For political
orientation, <b>0</b> is <b>left</b> and <b>1</b> is right.
For power identification <b>0</b> indicates <b>opposition</b>
and <b>1</b> indicates <b>coalition</b> (or governing party).
For power identification <b>1</b> indicates <b>opposition</b>
and <b>0</b> indicates <b>coalition</b> (or governing party).
</ul>

Participants are not required to use the first four fields,
Expand Down Expand Up @@ -176,10 +179,33 @@ <h2 id="submission">Submission</h2>
The participants are allowed to use any external datasets,
except the source data from ParlaMint.

The submission system will open soon.
Register on the mailing list to get notified.
Submission are now accepted through
<a href="https://www.tira.io/task-overview/ideology-and-power-identification-in-parliamentary-debates-2024">TIRA</a>.
You need to register your team on TIRA (in addition to a registration at CLEF) and pick an alias for your team name (submission is anonymous; you can reveal you true team name after final paper acceptance).
You can submit both predictoins
and dockerized software submisions (for better reproducibility).
Please note that you will not be able to see your results on the test dataset until after the deadline.
However, your will be informed if your submission failed because of a
formatting or software error.

We provide <a href="https://github.com/coltekin/ideology-power-st-baseline">
a simple linear baseline</a> with code for reading and writing the files.
a simple linear baseline and evaluation script</a>,
which also include a toy example, and examples of how to dockerize your submission.

<p>The predictions should be formatted as simple TSV files with two
columns, id, and the prediction. The prediction files should not
contain a header row. The names of the files should be formatted as
<code>team-task-pcode-runname.tsv</code>.
The <code>task</code> is either 'orientation', or 'power', and the
<code>pcode</code> is the lowercase code of the parliament (e.g., at, or
es-ct). The <code>team</code> and <code>runname</code> can be helpful
for identifying the team and the approach or the run information, but
they are not significant for the evaluation script. You can set them
to arbitrary strings, but they cannot be empty, and they should not
contain a dash (<code>-</code>). All files should be placed in the
same directory/folder without further hierarchy. If you are
participating in only a subset of the tasks and/or parliaments, then
you can submit files only for the combinations that you participate in.

<!--
<h2 id="tira-quickstart">TIRA Quickstart</h2>
Expand Down
Loading

0 comments on commit 0838dc8

Please sign in to comment.