This is the process documentation for crowdAI. It is a work in progress, and portions may be incomplete or out of date.
If you see something that needs some improved documentation, please go ahead and make a pull request.
crowdAI is a public website and large sections of the application are publicly viewable. To contribute to crowdAI or engage in a challege, users can sign up for a free participant account.
crowdAI has support for GDPR, fine-grained email preferences.
Organizers are groups such as universities or companies which launch competitions using crowdAI, define the competition rules and provide data and other information to competitors.
The organizers also judge the competitors and, at the end of the competition, award the winner.
An Organizer has one or more user accounts associated with it, these accounts are equivalent in authority and can perform the same actions.
The Organizer account is created in the admin panel. Participant accounts are associated with it via the Participant record in the admin panel.
Once created with associated accounts, the Organizer Participants can create new challenges via the organizer page. Currently Organizers are accessed via links on the challenges themselves (insert screen shot), but they can also be accessed via direct url links.
Related open issues:
Challenges run on crowdAI to crowd-source machine learning solutions to open data problems.
- New Challenge Creation
- Ordering of Challenges
- Challenge Registrations
- CLEF Challenges
- Dynamic challenge content
- Leaderboard and Submissions visibility
- Post Challenge Submissions
- Private Challenges
- Submission Limits
The crowdAI Leaderboard is dynamic and may be configured via the Challenge configuration. The leaderboard rows are populated with:
- The highest scoring entry for each participant, for that challenge round
- Only graded scores are considered
- A clickable link displays the graded submission entries used to make up the leaderboard score
Submission information may be seen in four places:
TODO - insert screen shots
- Leaderboard row
- Leaderboard accordion submission row
- Submissions tab
- Submission Detail page
Access to submission information dependent on challenge configuration and running status, and is detailed here.
A challenge can be configured to allow submissions via the Python Client or via the online submissions page (or both). Submissions can also be made after the challenge round is over.
This page has a shortened url and buttons to share the page on Facebook, Twitter and other social media. For graded submissions, it is publicly viewable.
The page can have a number of components, depending on configuration settings and data stored on the submission row.
The links to the page are from:
- Leaderboard row
- Submissions row
- Media thumbnail on the leaderboard
The links will only be active if the Submissions Details Visible? switch is set in the Challenge configuration.
The possible elements, in the order the will appear are:
- always displayed
- participant name, admin / organizer badge, scores
- grading status badge
- media
-
- Show scores only if challenges.show_leaderboard is true
Media is displayed only if the Media on leaderboard? configuration switch is set on the Challenge. If no media is available on the Submission, then a placeholder image (missing avatar) will be displayed.
Includes sharable buttons for Facebook and Twitter. Also has a copy button to access a shortened url for this page.
This section is displayed if the submissions.description fields has content, and is rendered as HTML. This field is populated via the submissions.description_markdown field.
If the Grading History Visible? setting is on, then the scoring history for all entries for this participant will be displayed in a table. The current row is highlighted in bold.
If the Grader Logs Visible? flag is set on the challenge, and the submissions.meta field is set, then the page will try to load a logfile from S3 into a CodeMirror instance. The key is in the format:
grader_logs/{challenge.slug}/grader_logs_submission_{submission.id}.txt