Paper: AI Model Share - subtitle: An Integrated Toolkit for Collaborative Machine Learning Model Development, Provenance Tracking, and Deployment in Python #911
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
|
Curvenote Preview
|
|
@hp2500 Hi Heinrich, thanks for your submission! I have a few questions about the supporting material you submitted. It seems like the main manuscript doesn’t refer the supporting material, and I am wondering what its purpose is. It also seems that the supporting material includes screenshots of your software and I would really appreciate what value they have regarding the manuscript. We would like the authors to be judicious about the use of supporting material because it is an extra burden for everyone including reviewers and proceeding chairs. If you can answer these questions, I would really appreciate it! |
|
Hi @hongsupshin , thanks for checking in! We are referring to the SI in the manuscript. However, we had to manually include the references bc the \ref{} links didn't seem to work across documents. I am happy to add additional references wherever it might be useful. The screenshots are indeed examples of the different functionalities of our software. We felt like a pure description might be a bit dry, so we tried to give the reader a better idea of what the different components look like. If you feel strongly about this I can remove the SI or attach the most important pieces to the main document. Alternatively, I can upload the SI to a separate repo and simply provide a link for those who are curious. Your guidance is much appreciated. Maybe the reviewers want to chime in on this, too? On a different note, can I continue to update the PR as we go or should I refrain from making changes before the reviewer's feedback is in? |
There was a problem hiding this comment.
@hp2500 Thanks for the prompt reply! (I am moving the conversation as a formal review)
Regarding the supporting material in general, I went to your website and I was wondering whether your team has online examples of the figures you shared in the supporting material. If this is the case, you can probably just cite the online material, which might be more useful then static figures (because it would be interactive and especially since you have many tutorials).
If this already exists, you can consider this suggestion. But if it doesn't and you have to create additional material, let's just keep the supporting material with figures as is. I made a comment about one of the figures in the SI, so take a look and see if it makes sense. Thank you!
We haven't assigned the reviewers yet, and it's likely that we will continue accepting papers until the end of next week, please keep updating your PR if needed! |
We have reviewers assigned to this paper @hp2500 We haven't heard from you for the past 3 weeks. Can you give us an update about the status of the paper? |
My apologies, @hongsupshin - I didn't realize you were holding off on reviews because of my comment. The paper has been ready since my latest commit. |
|
Hi everyone, I am one of the reviewers. I have had a first look at the paper, and I believe it is already in very good shape. I have only a few minor comments as outlined below:
|
|
@ankurankan Hi, would you mind leaving the comments directly on the corresponding file? You can press the + button to leave a comment and this will start a former review. |
papers/heinrich_peters/main.tex
Outdated
| \end{abstract} | ||
|
|
||
| \section{Introduction} | ||
| Machine learning (ML) is revolutionizing a wide range of research areas and industries, providing data-driven solutions to important societal problems. However, researchers and practitioners lack easy-to-use, structured pathways to collaboratively develop and rapidly deploy ML models. Traditionally, researchers have been using version-control systems like GitHub in combination with custom model evaluation and benchmarking experiments to ensure reproducibility and to compare models. However, larger-scale collaboration and crowd-sourcing are severely limited in the absence of standardized tasks and standardized processes for model sharing and evaluation. Additionally, most models developed by data scientists do not progress past the proof-of-concept stage and are never deployed \citep{davenport_is_2022, siegel_models_2022}, preventing the wider audience from participating in the promise of applied ML research. While the recent rise of platforms and tools like Hugging Face Hub \citep{noauthor_hugging_2023}, TensorFlow Hub \citep{noauthor_tensorflow_2023}, and MLflow \citep{noauthor_mlflow_2023,chen_developments_2020, zaharia_accelerating_2018}, illustrates the demand for open-source model repositories and MLOps solutions, barriers of entry are still high for researchers, educators, and practitioners from non-technical disciplines. Model Share AI (AIMS) addresses this problem by providing a lightweight, easy-to-use alternative. In a few lines of code, users can create Model Playgrounds - standardized ML project spaces that offer an all-in-one MLOps toolkit for collaborative model improvement, experiment tracking, model metadata analytics, and instant model deployment, allowing researchers and other data scientists to rapidly share, improve, and learn from ML models in one streamlined workflow. |
There was a problem hiding this comment.
This paragraph mentions that the current existing tools are difficult to use and have a high barrier to entry, but do not provide any reasoning behind why this is the case and what exactly the main pain points are. I think adding the reasons explicitly (or a table of features) would make the argument stronger and also help in highlighting the features of AIMS.
There was a problem hiding this comment.
Thank you for this suggestion. We have adjusted the related work section to explains how AIMS uniquely fit into the ecosystem of existing solutions.
papers/heinrich_peters/main.tex
Outdated
|
|
||
|
|
||
| \subsection{Key Functions} | ||
| \paragraph{Collaborative Model Development} A key feature of AIMS is its focus on collaborative model development and crowd-sourced model improvement, enabling teams to iterate quickly by allowing collaborators to build on each other's progress, even across libraries. For supervised learning tasks, users can collaboratively submit models into Experiments or Competitions associated with a Model Playground project in order to track model performance and rank submissions in standardized leaderboards Experiments and Competitions are set up by providing evaluation data against which the predictions of submitted models are evaluated. Standardized model evaluations allow collaborators to track the testing performance of their models along with a wide range of model metadata that are automatically extracted from submitted models and added to the model registry (see section below). Out of the box, AIMS calculates accuracy, f1-score, precision, and recall for classification tasks, and mean squared error, root mean squared error, mean absolute error, and $R^{2}$-scores for regression tasks, but users can submit custom evaluation functions for more flexibility. The main difference between Experiments and Competitions is that a proportion of the evaluation data is kept secret for Competitions, preventing participants from deliberately overfitting on evaluation data. Being able to submit models into shared Experiments enables ML teams to standardize tasks, rigorously track their progress, and build on each other's success, while Competitions facilitate crowd-sourced solutions for any ML task. Both Experiments and Competitions can be either public (any AIMS user can submit) or private (only designated team members can submit). Users can deploy any model from an Experiment or Competition into the REST API associated with their Model Playground with a single line of code. |
There was a problem hiding this comment.
"standardized leaderboards Experiments and Competitions." It seems there might be a missing period.
There was a problem hiding this comment.
Thank you for catching this. Fixed.
papers/heinrich_peters/main.tex
Outdated
| \end{figure} | ||
|
|
||
| \subsection{Architecture} | ||
| AIMS consists of three main components: an open-source Python library, user-owned cloud backend resources, and the AIMS website. The AIMS Python library is the main interface allowing users to set up Model Playground pages (including Experiments and Competitions), submit and deploy models, analyze model metadata, and reproduce model artifacts. It provides an accessible layer that facilitates the creation of the cloud backend resources that power REST APIs, as well as model evaluations and model metadata extraction. The ModelPlayground() class acts as a local representation of a Model Playground page and its associated REST API. It provides a range of methods to configure, change, and query Model Playground resources. A detailed overview of the Python library is provided below (AIMS Workflow). |
There was a problem hiding this comment.
Writing class and function names such as ModelPlayground, input_type, etc., in italics would improve readability.
There was a problem hiding this comment.
This is a good suggestion. We have changed the manuscript accordingly.
|
@AmadiGabriel Good to meet you at SciPy. I am inviting you to review this paper. You were sent an invitation from GitHub to be a collaborator on this repository. Please accept the invitation. Your review should be in the form of GitHub review comments: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/commenting-on-a-pull-request |
|
I would like to see more details on the support of experiments. Can you store certain metrics? |
@cbcunc The authors have made further improvements on the paper based on my comments. Kindly proceed with subsequent stages of the review process. |
The metrics are stored and can be queried by users through the Python library or on the AIMS website. In case you mean additional metrics, there is the possibility of submitting a dictionary of key-value pairs to the "custom_metadata" argument of the "submit_model" method. We now mention this in the "AIMS Workflow" section. |
If you are creating this PR in order to submit a draft of your paper, please name your PR with
Paper: <title>. An editor will then add apaperlabel and GitHub Actions will be run to check and build your paper.See the project readme for more information.
Editor: Charles Lindsey @cdlindsey
Reviewers: