Skip to content

Releases: facebookresearch/Mephisto

v1.5.4

13 Dec 21:32
@meta-paul meta-paul
v1.5.4
887eac2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.5.4 Latest
Latest

Overview

A housekeeping release devoted to paying down urrent tech debt, e.g.:

  • port UI code to Typescript
  • upgrade Python version to 3.10
  • making UI fully mobile-compatible for standard task generators
  • update existing examples with InHouse provider
  • add missing doc pages for providers
  • remove deprecated packages

What's Changed

  • Updated UI validation and fixed error handling in remote procedures by @meta-paul in #1259
  • Make FormComposer andVideoAnnotator render well on mobile by @meta-paul in #1260
  • Removed deprecated mephisto-task and mephisto-worker-addons libs by @meta-paul in #1261
  • Port existing Task examples to use In-House provider by @meta-paul in #1262
  • Added doc pages for In-House and Mock providers by @meta-paul in #1263
  • Auto-submit partial Task results right before Unit expiration by @meta-paul in #1264
  • Ported core packages and examples to Typecript by @meta-paul in #1265
  • Fixed auto-redirect upon Task completion in In-House provider by @meta-paul in #1266

Full Changelog: v1.5.3...v1.5.4

Contributors

meta-paul
Assets 2
Loading

v1.5.3

05 Nov 15:47
@meta-paul meta-paul
v1.5.3
70bf630
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.5.3

Overview

This release offers clarity and ease of use for Model-in-the-Loop tasks. The pattern of communicating with the server over the course of the task (called "remote procedures") was added to FormComposer, and an example called "Interactive Generation of a Look-Alike Image" was provided. Other examples of remote procedure tasks were brought up-to-date as well.

FormComposer gainded a few useful additions:

  • dynamic fieldsets (to save multiple instances of the same response fieldset)
  • additional flexibility for coding up UI triggers
  • buttons as an element of a form fieldset

This is useful when:

  • Worker should react to continuously generated content
  • Worker should guide your algorithm to achieve a goal

What's Changed

  • Renamed React component files to jsx; fixed VideoAnnotator bug by @meta-paul in #1256
  • Added example task 'Interactive Image Generation' by @meta-paul in #1257
  • Updated remote procedure examples; Refactored remote procedure blueprint by @meta-paul in #1258

Full Changelog: v1.5.2...v1.5.3

Contributors

meta-paul
Assets 2
Loading

v1.5.2

17 Oct 02:29
@meta-paul meta-paul
v1.5.2
672a8f2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.5.2

Overview

Mephisto now offers a UI for managing qualifications and qualification assigns. We've nested it under TaskReview app since it already can run a local web server. Some new capabilities were also added:

  • Now workers can be granted qualifications directly (without context of a specific Task Unit)
  • Qualifications now support not only name, but also description
  • Multiple qualifications can now be assigned to a single Task Unit
  • Screenshots and all details are described in the docs

Note that this code change includes a database migration. It should port all your existing qualifications data automatically. Though just in case we'd recommend backing up the qualifications part of your database (using mephisto db export --qualification-only command as described here)

Some other changes include bug fixes and reorganization of local Mephisto packages for more transparent naming and versioning.

What's Changed

Full Changelog: v1.5.1...v1.5.2

Contributors

meta-paul
Assets 2
Loading

v1.5.1

02 Oct 18:17
@meta-paul meta-paul
v1.5.1
f6037c8
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.5.1

Overview

Patch with small fixes to VideoAnnotator and docs update.

VideoAnnotator

  • Display validation error for overlapping video segments on the same track
  • Add support for multiple video formats (not just mp4)
  • Fix a few outdated unittests for generators

For docs

  • Add "Known Issues" page in the docs
  • Add FormComposer screenshots

What's Changed

Full Changelog: v1.5.0...v1.5.1

Contributors

meta-paul
Assets 2
Loading

v1.5.0

26 Sep 16:13
@meta-paul meta-paul
v1.5.0
641994a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.5.0

Overview

This release introduces a couple of key features, and a number of smaller improvements.

Video Annotator

VideoAnnotator is a new Task Generator for convenient and functional video annotation.

Key features:

  • Worker can indicate multiple video segments along multiple tracks, and annotate each one
  • For each segment, you can specify (via JSON config) desired annotation input (as a list of standard form fields)
  • VideoAnnotator JSON config works similar to FormComposer (including pre-signed S3 URLs, and support for custom JS triggers)
  • Workers inputs are downloadable in JSON
  • When viewing a completed unit, worker inputs are also available in WebVTT format

InHouse provider

The new InHouse provider enables using Mephisto with your own "in-house" crowd - and basically crowdsourcing your task to people you know. (This used to be "mock" provider last year, but now we're using "mock" only for testing).

The initial implementation is quite simple: a worker navigates to your Task page, types in their worker nickname, and proceeds to the next available Task unit (if any). Afterwards you can review their results and take notes or grant qualifications, just like you do that with the 3rd party workers.

Future roadmap for InHouse provider would include: proper worker authorization; allowing workers to view researcher notes; restricting worker acces based on their qualification.

Feature improvements

TaskReview app

  • Visualized selection-based response stats in Task Review
  • Added ability to view completed task units, without assessing them

DataPorter feature

  • Now you can port only granted worker qualifications (without all other data), to easily build pool of known workers collaboratively

FormComposer

  • Enabled CSS insertions in FormComposer

Grafana

  • Updated Grafana dockerized deployment and docs
  • Added access to Grafana dashboard from TaskReview app

General housekeeping

  • Updated and added missing architecture diagrams in Mephisto docs
  • Refactored increasingly nested CLI commands, and updated their Help strings
  • Add example of GoldUnit and Worker Onboarding
  • upgraded NodeJS to v20

Included PRs

Full Changelog: v1.4.0...v1.5.0

Contributors

JackUrb, dependabot, and meta-paul
Assets 2
Loading

v1.4.0

13 May 02:38
@meta-paul meta-paul
v1.4.0
5ac0e4b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.4.0

Overview

In this release we added Data Porter feature that allows to move full/partial data dumps between different isntances of Mephisto. This enables things like:

  • backups/restores of Mephisto data
  • consolidating task data across different Mephisto instances
  • sharing task data with your colleagues

We also introduced a simple (forward-only) workflow for database migrations, enabled custom code snippets in Form Composer, and updated numerous library versions.

Last but not least, mephisto.ai documentation got a major update as well.

Included PRs

Full Changelog: v1.3.0...v1.4.0

Contributors

JackUrb, dependabot, and meta-paul
Assets 2
Loading

v1.3.0

27 Feb 22:01
@meta-paul meta-paul
v1.3.0
2dd08af
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.3.0

Overview

This release introduces FormComposer task generator. It aims to simplify creating React front-end for form-based Tasks. Main highlights:

  • Task front-end can be defined via JSON configuration (no need for custom code)
  • Multiple form versions are supported within a single Task
  • Multi-version forms can be generated within the same Task, based on variables' values
  • Expiring (presigned) URLs are supported for S3 files (for data safety)
  • Task Review app support includes preview of uploaded media files

Included PRs

Full Changelog: v1.2.1...v1.3.0

Contributors

meta-paul
Assets 2
Loading

v1.2.1

01 Feb 22:16
@meta-paul meta-paul
v1.2.1
5686c28
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.2.1

Overview

This release rounded out TaskReview app functionality. Now you can conveniently give feedback and bonuses to individual workers while performing review of their submissions. This feature works with all currently supported providers.

Included PRs

Full Changelog: v1.2.0...v1.2.1

Contributors

meta-paul
Assets 2
Loading

Mephisto 1.2.0

19 Jan 20:54
@meta-paul meta-paul
v1.2.0
aac76d7
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Mephisto 1.2.0

Overview

This release is focused on adding a new provider (Prolific) and better data review experience.
It has a larger diff than usual, and we will return to a more frequent release cadence afterwards.

See the list of changes when upgrading from v1.1.0 below. Note that only essential dependency updates are listed.

Big Changes

  • Added integration with Prolific provider, as an alternative to Mturk for data collection. Expect some minor additions in the subsequent releases, but the core functionality is already there. #1008
  • Added UI-driven review of collected task results, a major upgrade over the previously used command-line script. Support for bonuses and feedback for workers to be added in the subsequent releases. When jobs have had no activity for a set duration, they'll be shut down automatically (suspecting an issue). #1058
  • Added dockerization support for running Mephisto. #1008

Included PRs

Feature + UX Improvements

Bugfixes

Docs + Examples

Dependencies

New Contributors

Full Changelog: v1.1.0...v1.2.0

Contributors

benmatselby, pringshia, and 8 other contributors
Assets 2
Loading

Mephisto 1.1.0

28 Oct 17:47
@JackUrb JackUrb
v1.1.0
66be164
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Mephisto 1.1.0

Overview

This release contains a number of Stability and Quality of Life improvements from v1.0.0 and v1.0.2. The big focus has been on collection success rate, documentation, and various bugfixes.

For those updating from 1.0, see the 1.0.2 changes as well for a complete changelog.

Breaking changes

  • Mephisto no longer supports Python 3.7. This is as of #898, wherein to get the ParlAI peer imports working we needed to bump to ParlAI 1.7.0, which no longer supports Python 3.7.

Big Changes

  • When launching on MTurk, Mephisto will request that you run the mephisto scripts mturk cleanup script inline before launching. This exercise prevents having long-running tasks that haven't been cleaned up properly. #918
  • When jobs have had no activity for a set duration, they'll be shut down automatically (suspecting an issue). #919

Big Fixes

  • Lots of documentation, thanks @Etesam913!
  • Significant fixes to proper execution and state tracking in Mephisto, which should improve stability across all job types (#875, #878, #902, #922)
  • Major fix for Heroku tasks using the useMephistoLiveTask hook, which was causing all messages to be double sent in both directions. (#921)

Included PRs

Feature + UX Improvements

Bugfixes

Testing & DevX

Docs + Examples

Dependencies

New Contributors

Full Changelog: v1.0.3...v1.1.0

Contributors

pringshia, JackUrb, and 7 other contributors
Assets 2
Loading