From 17fc66258d15a4f1b3afa83610519bf69f5af31d Mon Sep 17 00:00:00 2001 From: kmax-tech <64914021+kmax-tech@users.noreply.github.com> Date: Wed, 24 Apr 2024 07:20:58 +0200 Subject: [PATCH 01/38] Update image-retrieval-for-arguments.html --- .../image-retrieval-for-arguments.html | 63 +++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/clef24/touche24-web/image-retrieval-for-arguments.html b/clef24/touche24-web/image-retrieval-for-arguments.html index ea43db5..89f1fd0 100644 --- a/clef24/touche24-web/image-retrieval-for-arguments.html +++ b/clef24/touche24-web/image-retrieval-for-arguments.html @@ -193,7 +193,70 @@

Submission

Images alone can be ambiguous and difficult to understand without context, e.g. if they refer to symbolism. That's why we offer the option to submit a rationale along with the image. The rationale is a piece of text that helps us understand the image. For example, it could be a caption or contextual information about the image. The image and rationale will be evaluated together to see how this combination conveys the premise.

+

Submission - Format

+ You can either submit a docker file that reproduces your results, or you can submit the results directly. In the latter case, the submission format depends on whether you chose the retrieval or the generation approach. If you want to use both methods, please submit separate files for each approach. + You can assign up to 10 images for one argument. Use the rank key to indicate the order of your submissions, with 1 being the most relevant submission. + The result files should be in JSON Lines text format, JSON Lines, where each line is a JSON Object conforming to the conventions below. + + + 
+
+            Image Retrieval
+        
+            If you have chosen the retrieval approach, please submit your results in a file called "results.jsonl". 
+        
+            Each JSON object in your "results.jsonl" file should have the following keys:
+            
+            argument_id - id of the argument in the arguments.xml file
+            method - retrieval
+            image_id - the image's ID - it corresponds to the name of the image's directory in the released dataset
+            rationale - additional info or caption for the image to understand how it conveys the premise (optional)
+            rank - specifies the preference of your submissions - 1 is highest
+            tag - tag defined by you and your group, identifies your group and the method you used to obtain the results
+        
+            An example submission for argument "65302-a-2" would look like the following:
+        
+            { "argument_id" : "65302-a-2 ",
+             "method " : "retrieval",
+            "image_id" : "Iffdea3cd664722c736d7d667",
+            "rationale" : "space is the final frontier",
+            "rank" : 1,
+            "tag " : "touche organizers - example submission for image retrieval; manual selection of images"
+            }
+        
+            Image Generation
+        
+            If you are using image generation, submit a .zip file, which should contain a JSONL file called "results.jsonl" and a directory called "images "containing the generated images.
+        
+            Please use the following keys for your JSON Objects in the JSONL file:
+        
+            argument_id - id of the arguments in the arguments.xml file
+            method - generation 
+            prompt - the prompt that you have used to generate the image
+            image : - name of the generated image, which can be found in the images directory
+            rationale - additional info or caption for the image to understand how it conveys the premise (optional)
+            rank - specifies the preference of your submissions  - 1 is highest
+            tag - tag defined by you and your group, identifies your group and the method you used to obtain the results
+        
+            An example looks like this:
+        
+            { "argument_id" : "65302-a-2 ",
+             "method " : "generation",
+            "prompt" : "cat looking into the stars",
+            "image_name" : "space-pic1.jpg", 
+            "rationale" : "space is fascinating",
+            "rank" : 1,
+            "tag : "touche organizers - example submission for image generation; manual prompt engineering"
+            }
+        
+            Therefore the corresponding zip would have the following structure:
+        
+            - submissions.jsonl
+            - generated_images 
+              - space-pic1.jpg
+
+

Important Dates

@@ -176,10 +179,15 @@

Submission

The participants are allowed to use any external datasets, except the source data from ParlaMint. -The submission system will open soon. -Register on the mailing list to get notified. +Submission are accepted through +TIRA. +You can submit both predictoins +and dockerized software submisions (for better reproducibility). + We provide - a simple linear baseline with code for reading and writing the files. +a simple linear baseline and evaluation script, +which also include a toy example, and examples of how to dockerize +your submission. +
  • May 6, 2024: Approaches submission deadline. [register via submission system (SUBMIT)]
  • May 31, 2024: Participant paper submission.
  • June 21, 2024: Peer review notification.
  • July 8, 2024: Camera-ready participant papers submission.
    • From 3522f2dd33e3ad332c8dfc2f52462eb280198d16 Mon Sep 17 00:00:00 2001 From: Johannes Kiesel Date: Mon, 29 Apr 2024 13:20:51 +0200 Subject: [PATCH 21/38] late registration notice --- clef24/touche24-web/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/clef24/touche24-web/index.html b/clef24/touche24-web/index.html index 4a62e1c..86bf968 100644 --- a/clef24/touche24-web/index.html +++ b/clef24/touche24-web/index.html @@ -47,7 +47,7 @@

    Shared Tasks

    Important Dates

    From 397318017aaa6920b243ab640fa45125c2200926 Mon Sep 17 00:00:00 2001 From: Johannes Kiesel Date: Fri, 3 May 2024 00:21:32 +0200 Subject: [PATCH 26/38] add web ui --- clef24/touche24-web/human-value-detection.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/clef24/touche24-web/human-value-detection.html b/clef24/touche24-web/human-value-detection.html index 24651d1..51cb696 100644 --- a/clef24/touche24-web/human-value-detection.html +++ b/clef24/touche24-web/human-value-detection.html @@ -93,7 +93,7 @@

    Synopsis

  • Communication: [mailing lists: task, organizers] [twitter/x]
  • Data: [download] [project]
    • -
  • Submission: [example approaches] [evaluator] [forum] [submit]
    • +
  • Submission: [example approaches] [web ui] [evaluator] [forum] [submit]
  • ValueEval'23: [website] [demo]
    • Register now @@ -414,7 +414,7 @@

    Data

    Submission

    Submit your approach via TIRA. Ask in the Forum if you need help. You need to register your team (in addition to a registration at CLEF) and pick an alias for your team name (submission is anonymous; you can reveal you true team name after final paper acceptance). You submit a Docker image or submit from your Github repository (via automated Docker building). In case of trouble, you can also submit via run file upload (not recommended due to poor reproducibility; rather contact us if you need help with Dockerization). You can submit on the validation dataset to check how submission works or the test dataset. You will not be able to see your results on the test dataset until after the deadline. Datasets are provided multilingual or in machine-translated English (see Data). [forum] [submit]

    -

    We recommend to start your approach from one of our example approaches (in Python), which include the code for reading and writing the files and make it easy to later deploy your approach as server or submit and distribute it as Docker image. [random baseline: script, notebook] [bert baseline] [ollama baseline]

    +

    We recommend to start your approach from one of our example approaches (in Python), which include the code for reading and writing the files and make it easy to later deploy your approach as server or submit and distribute it as Docker image. If you run them as a local server, you can use our web ui to use them interactively. [random baseline: script, notebook] [bert baseline] [ollama baseline]

    Approaches need to produce run files that have the same format as the labels.tsv, but the numbers can be between 0 and 1 and are interpreted as the confidence of the approach (employed for evaluation via ROC-curves): [toy example]