One-time review may not be appropriate for software projects

[lassoan]

@Kevin-Mattheus-Moerman recommended in #153 to open a ticket with our feedbacks on the review process.

Traditionally, journals publish static papers that are reviewed once. I don't think that this can be meaningfully applied to software projects, mainly because the software is continuously changing. Even things that are simple for a traditional paper, like list of authors is really difficult to determine for software projects (who contributed, how much, to which parts of the software, ...).

I think "Amazon product review" or "Expedia hotel review" style reviews would be much more appropriate. Number of reviews, ratings in different categories, etc. would be the living proof about importance and quality of a software tool.

Asking a couple of people to review a new submission is a good start, but:

1. The review should be structured so that it can be easily searched and results can be aggregated (the current free discussion format is very inconvenient and the content is quite useless - who would ever want to read a long discussion thread between a couple of people happened a few years ago when a particular software was submitted?)
2. Users should be encouraged to keep submitting/updating reviews (people who submitted reviews could be asked yearly or when there are new releases if they still use the software, how would they rate the current state of the software, etc)

[pjotrp]

With JOSS people can publish new papers when the software gets updated. This is particularly useful when new authors join and do relevant work. The review is tied to one submission, with the source code in a certain 'state'. What happens after is not tied to the publication. I think should not look at the review process as something that gets 'updated' over time. @arfon if you agree we can close this issue.

[lassoan]

With JOSS people can publish new papers when the software gets updated

You could do that, but that would mean that the number of citations would be dispersed among a number of papers. So, instead of having a "landmark" paper that has an outstanding number of citations that gets on the top of citation lists, make your career move forward, etc., you would end up having many papers with mediocre number of citations.

Keeping publishing new papers for each new major release may be good for new contributors, but unfair to authors that made contributions earlier. People may still rely on code that was implemented years before, yet the author who implemented that code may not be recognized anymore if he has not contributed to the repository lately.

Does JOSS try to address limitations of traditional journal publications? Or it is there only to give a DOI to software developers without requiring writing a paper? If the latter, then I agree that this issue can be closed.

[pjotrp]

If you want a land mark paper it is your choice to publish one paper. I think it is totally unfair on the biopython developers, for example, that only the one paper gets cited. In fact, it is detrimental to future contributors and not in line with FOSS concepts. So when more authors contribute work, they should republish and get citations too. It is up to others to cite the correct paper.

JOSS is not an old style journal. You have a choice in these matters.

[tracykteal]

Thanks @lassoan for this feedback. I think the goal with the JOSS papers is more focused on making the software citable and giving the authors credit, than on it being a place for ongoing reviews of the software. At least in bioinformatics, a forum like Biostars is better suited for that, where there are things more like Amazon or Expedia reviews and a community that already exists who both contribute and read the reviews. Here, as you point out, the infrastructure isn't really set up for that right now.

To the point about updating authors, that is a challenge. You want to continually be giving new and current authors credit, but not diminishing contributions of original core contributors, and you don't want to spread your citations across multiple references. That is something there is ongoing conversation and thinking about. Since DOIs don't really seem to be set up for 'versions' right now, we're still continuing to work on solutions. Ideas would be welcome!

[laughedelic]

At least in bioinformatics [...]

I think the main problem of the bioinformatics software is that people mostly don't care about proper versioning. Stable releases with fixed versions is the cornerstone of reproducible research. And I agree with @pjotrp that it's unfair and discouraging to the new contributors if they are not added to the republished next-version paper.
What about forks? What if I'm an independent contributor, I don't even need to push my changes to the upstream repo. I can (depending on the license, of course) just keep my own fork separate and submit it to JOSS. How such scenario is going to work?

Since DOIs don't really seem to be set up for 'versions' right now, we're still continuing to work on solutions

I think Zenodo issues new DOI for each release.

[pjotrp]

I can't speak for JOSS as there is no clear policy yet, but my view it is that it is up to the author and the reviewer to assess whether a submission is worth publishing.

We have not had a software update paper on JOSS yet. Would be good to try. I don't see a difference between an update and a fork, really. It is really what has been added (and how it is described) which makes the paper. The author(s) should give all due credits.

[arfon]

Thanks for your input @lassoan. I think there are a couple of important ideas in this thread that I'll try to address separately:

Authorship/updating publications/reviewing software more than once

Ever since the inception of JOSS, we've wanted to be able to publish more fine-grained publications (see #52). The idea being that a single publication for a software project is a pretty bad solution and definitely penalizes contributors active after an initial JOSS publication.

This problem isn't unique to JOSS, it's more a mis-match between the static model of traditional publications and the fact that many of the most successful software projects are living entities where the authorship is in a continual state of flux.

We absolutely want to support dynamic authorship and multiple reviews for the same software project but I don't think we have a complete understanding about exactly how to do this right now and I'm pretty sure there isn't a 'one-size-fits-all' solution here.

Marketplace-style reviews of JOSS submissions

I think "Amazon product review" or "Expedia hotel review" style reviews would be much more appropriate. Number of reviews, ratings in different categories, etc. would be the living proof about importance and quality of a software tool.

I'm sympathetic to this idea but I don't think it's what we should be doing with JOSS. I think there are simply better options for metrics than some kind of voting/multiple reviews system built by us. Examples of better options include Depsy and Libraries.io that look at software dependencies, in addition there are metrics such as citations to JOSS publications, GitHub stars etc.

I'm going to close this issue at this point as I think the issue we are interested in addressing is already captured in #52.