Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW]: PsychoPhysioCollector #40

Closed
15 of 16 tasks
whedon opened this issue Jul 20, 2016 · 23 comments
Closed
15 of 16 tasks

[REVIEW]: PsychoPhysioCollector #40

whedon opened this issue Jul 20, 2016 · 23 comments
Labels
accepted published Papers published in JOSS recommend-accept Papers recommended for acceptance in JOSS. review

Comments

@whedon
Copy link

whedon commented Jul 20, 2016

Submitting author: @sbogutzky (Simon Bogutzky)
Repository: https://github.com/sbogutzky/PsychoPhysioCollector
Version: v2.0.5
Editor: @arfon
Reviewer: @davclark
Archive: 10.5281/zenodo.59387

Status

status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/aacbdea63ce8d4896a3c84d89f4c5ee0"><img src="http://joss.theoj.org/papers/aacbdea63ce8d4896a3c84d89f4c5ee0/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/aacbdea63ce8d4896a3c84d89f4c5ee0/status.svg)](http://joss.theoj.org/papers/aacbdea63ce8d4896a3c84d89f4c5ee0)

Reviewer questions

Conflict of interest

  • As the reviewer I confirm that there are no conflicts of interest for me to review this work (such as being a major contributor to the software).

General checks

  • Repository: Is the source code for this software available at the repository url?
  • License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
  • Version: Does the release version given match the GitHub release (v2.0.4)?

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: Have any performance claims of the software been confirmed?

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g. API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

Paper PDF: 10.21105.joss.00040.pdf

  • Authors: Does the paper.md file include a list of authors with their affiliations?
  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • References: Do all archival references that should have a DOI list one (e.g. papers, datasets, software)?
@whedon whedon added the review label Jul 20, 2016
@arfon
Copy link
Member

arfon commented Jul 20, 2016

/ cc @openjournals/joss-reviewers - would anyone be willing to review this submission?

If you would like to review this submission then please comment on this thread so that others know you're doing a review (so as not to duplicate effort). Something as simple as :hand: I am reviewing this will suffice.

Reviewer instructions

  • Please work through the checklist at the start of this issue.
  • If you need any further guidance/clarification take a look at the reviewer guidelines here http://joss.theoj.org/about#reviewer_guidelines
  • Please make a publication recommendation at the end of your review

Any questions, please ask for help by commenting on this issue! 🚀

Please note #41 looks to be a related submission. It may make sense to have both of these reviewed by the same individual.

@arfon arfon mentioned this issue Jul 20, 2016
16 tasks
@arfon
Copy link
Member

arfon commented Jul 29, 2016

@openjournals/joss-editors - any suggestions for potential reviewers here?

@labarba
Copy link
Member

labarba commented Jul 29, 2016

I pinged a friend that I think has the right expertise ...
https://twitter.com/lorenaabarba/status/758974889680261121

@davclark
Copy link
Member

So you just need someone to answer the reviewer questions on this page for the PDF? Here on this issue?

@arfon
Copy link
Member

arfon commented Jul 29, 2016

@davclark - basically yes. There's an associated GitHub repository (https://github.com/sbogutzky/PsychoPhysioCollector) which is the main focus of this review together with the short PDF. See our reviewer guidelines here http://joss.theoj.org/about#reviewer_guidelines

If you think you might be able to help out, I'll add you to the @openjournals/joss-reviewers team so that you have permissions to update the checkboxes at the top of this issue.

@davclark
Copy link
Member

Sounds doable. Go ahead and add me to the team :)

@arfon
Copy link
Member

arfon commented Jul 29, 2016

@davclark 🎉 . You should have an invite to join the '@joss-reviewers' team in your inbox. This link should also work.

Please shout if anything isn't clear about the review process here.

@davclark
Copy link
Member

davclark commented Aug 1, 2016

@arfon do we verify that the license is enforceable? I.e., that the copyright holder indeed has copyright?

@whedon I see you've copyrighted the work to yourself. If produced at an american university, this is often not legal - we need to assign copyright to the institution if we worked on it as a student. For free software, this is usually not a big deal - especially for a very liberal license.

@davclark
Copy link
Member

davclark commented Aug 1, 2016

Cool stuff. This is important for social science research - and warrants a change in the language above from the JOSS review checklist:

(ideally to solve real-world analysis problems)

Data collection is also part of science - much harder than analysis in many experiments. To this end, other guidelines are strained. What is the significance of "API documentation?"

A few more notes:

  • The tag 2.0.4 does not include the paper.
  • I cannot verify all functionality of this software as it requires specialized hardware. I do have an android phone.
  • I don't think I have time to install android development tooling. Is there a ready-made APK that I can directly access via the web for installation on my phone? I know I can use android file transfer, but this is a barrier for some users (and would make my life faster too).
  • I would clarify that anyone can install the existing APK easily, but that an android development environment is needed to create new questionnaires. In the paper proper, it would also be good to clarify that a few self-report surveys are provided, and that more can be created with proper android development tools.
  • Link to android developer tools / docs?
  • Could a brief developer doc be provided that points to top-level classes / files for major functionality? This would greatly help with contributions.
  • There is no advise on getting help or contributing, except that you are welcome.
  • Please spell out acronyms. These include ESM, IMU and ECG.
  • Your audience is almost certainly larger than people running "ESM protocols." Please provide a general level of technical aptitude required to use the app and get the data off an analyze it. Give a general sense of quality of biosignals.
  • What are existing solutions like? Why wouldn't I just use them?
  • Could you say a little more about how "its visualization component allows checking the experimental setup in real time." This would address the question of manual steps to verify operation.
  • Is there more written about Flow-Machines?
  • The paper appears to be missing the required link to the repo.

@davclark
Copy link
Member

davclark commented Aug 1, 2016

While the above seems like a lot, I'd call this "minor revisions" modulo some clarification on the things that break the "analysis software" model - most critically that I can't actually test this on the biosensor hardware.

@arfon
Copy link
Member

arfon commented Aug 1, 2016

@arfon do we verify that the license is enforceable? I.e., that the copyright holder indeed has copyright?

@davclark - If you know of a reason why the author may have go this wrong then I think it's worth raising the question but we don't want to give legal advice to those submitting.

@whedon I see you've copyrighted the work to yourself. If produced at an american university, this is often not legal - we need to assign copyright to the institution if we worked on it as a student. For free software, this is usually not a big deal - especially for a very liberal license.

@davclark - @whedon is actually the bot that made the submission issue. @sbogutzky - this question is for you 😉

@arfon
Copy link
Member

arfon commented Aug 1, 2016

(ideally to solve real-world analysis problems)

Data collection is also part of science - much harder than analysis in many experiments. To this end, other guidelines are strained. What is the significance of "API documentation?"

@davclark agreed and thanks for raising this point. Would you be willing to open an issue on https://github.com/openjournals/joss outlining your concerns with the language used in the checklist?

@simonbogutzky
Copy link

simonbogutzky commented Aug 1, 2016

The tag 2.0.4 does not include the paper.

Yes, this is right. tag 2.0.4 is the latest version of the PPC without the paper.

I cannot verify all functionality of this software as it requires specialized hardware. I do have an android phone.

But you can test the basic functionality, because data will always collect from the internal sensors of the smartphone. In this way, you can for example track the arm motion of a participant with a sport armband while running.

I don't think I have time to install android development tooling. Is there a ready-made APK that I can directly access via the web for installation on my phone? I know I can use android file transfer, but this is a barrier for some users (and would make my life faster too).

I would clarify that anyone can install the existing APK easily, but that an android development environment is needed to create new questionnaires.

So, you find the APK. See Installation Instructions of the readme.md

In the paper proper, it would also be good to clarify that a few self-report surveys are provided, and that more can be created with proper android development tools.

Ok.

Link to android developer tools / docs?

See Used Libraries in the readme.md

Could a brief developer doc be provided that points to top-level classes / files for major functionality? This would greatly help with contributions.
There is no advise on getting help or contributing, except that you are welcome.

Please spell out acronyms. These include ESM, IMU and ECG.

Ok.

Your audience is almost certainly larger than people running "ESM protocols." Please provide a general level of technical aptitude required to use the app and get the data off an analyze it. Give a general sense of quality of biosignals.

Ok, but this depends on the sensors and the configuration, which you use.

What are existing solutions like? Why wouldn't I just use them?

Maybe there are other solutions, but I these solutions do not support the hardware of our equipment.

Could you say a little more about how "its visualization component allows checking the experimental setup in real time." This would address the question of manual steps to verify operation.

E. g. if you place electrodes on the human body, you can check the leakage of the ECG based on the realtime visualization of the heart activity and the pattern of the signal.

Is there more written about Flow-Machines?

Yes, of course, unfortunately the most is in german. There one paper of the beginning: "Gait Biometrics in Interaction Design: A Work in Progress Report" and presentation: "Analysis of ‚flow while walking and running’: psychophysiological correlates and activity-based results".
In this year we will also published some scientific papers about our research in the project.

The paper appears to be missing the required link to the repo.

Is that needed?

@davclark
Copy link
Member

davclark commented Aug 1, 2016

I think I may have inter-mixed some different kinds of issues above, so I'll respond to a few things again here. Let me also add that assuming it works (as I do), this is a very useful contribution. My comments are simply directed at making the publication / documentation better, and opening this up to more potential users and perhaps developers.

Things that are up to the editors

The things in this section will have no bearing on my recommendation.

The tag 2.0.4 does not include the paper.

Yes, this is right. tag 2.0.4 is the latest version of the PPC without the paper.

I'm not sure if there are rules here, and I'll leave this to the editors. Seems better to have the paper in the tagged release, though.

The paper appears to be missing the required link to the repo.

Is that needed?

This is in the guidelines for reviewers (emphasis mine):

JOSS papers should contain the following:

...

  • A list of key references including a link to the software archive

As a reviewer, I feel my job is done by pointing out that this is not done.

Things that are "mine"

I cannot verify all functionality of this software as it requires specialized hardware. I do have an android phone.

But you can test the basic functionality, because data will always collect from the internal sensors of the smartphone. In this way, you can for example track the arm motion of a participant with a sport armband while running.

I've opened openjournals/joss#156 on this point. Given the current discussion, I think we're gonna let the things I can't test slide.

I don't think I have time to install android development tooling. Is there a ready-made APK that I can directly access via the web for installation on my phone? I know I can use android file transfer, but this is a barrier for some users (and would make my life faster too).

I would clarify that anyone can install the existing APK easily, but that an android development environment is needed to create new questionnaires.

So, you find the APK. See Installation Instructions of the readme.md

It seems like there is either a disagreement or lack of understanding here. Let me give a more concrete example:

I am at work with my fancy/annoying new macbook retina. To connect my phone, I need to unplug the hub from the single USB-C port, which means I need to rearrange my desk and move my ergo keyboard out of the way so I can plug into the USB-C port on my phone. I then grab the cable from the power adapter to connect my phone to my computer. The future is going to look more like this for more people.

What I'm suggesting is easy enough that I just installed the APK by creating a
fork and a new release here. Just like what you have, except including a direct download of the APK.

Link to android developer tools / docs?

See Used Libraries in the readme.md

Those don't include https://developer.android.com/studio/index.html

Compiling this would not be hard for most people with rudimentary development skills. You are not currently giving the basic pieces that would facilitate this.

What are existing solutions like? Why wouldn't I just use them?

Maybe there are other solutions, but I these solutions do not support the hardware of our equipment.

Here is another solution. Why wouldn't I just use that? I can at least partially answer this question, but I'm asking you to at a minimum cite primary existing solutions in your paper per JOSS guidelines.

Could you say a little more about how "its visualization component allows checking the experimental setup in real time." This would address the question of manual steps to verify operation.

E. g. if you place electrodes on the human body, you can check the leakage of the ECG based on the realtime visualization of the heart activity and the pattern of the signal.

It seems there is no similar readout for phone sensors? Or am I missing something?

I am able to verify the contents of the CSV files manually. Note that there is a slight type-o in your documentation. You mention psychophysiocollector/PARTICIPANT_NAME/ACTIVITY_NAME, but you actually get them in psychophysiocollector/ACTIVITY_NAME/PARTICIPANT_NAME/DATETIME. Maybe just say PsychoPhysioCollector and leave the rest to the user (it's obvious from there).

Is there more written about Flow-Machines?

Yes, of course, unfortunately the most is in german. There one paper of the beginning: "Gait Biometrics in Interaction Design: A Work in Progress Report" and presentation: "Analysis of ‚flow while walking and running’: psychophysiological correlates and activity-based results".
In this year we will also published some scientific papers about our research in the project.

You should cite these - It doesn't hurt anything, and it facilitates discovery of your work! But I won't demand it.

@simonbogutzky
Copy link

@davclark - First of all, that's for your feedback.

@arfon - Now some questions to the editors.

The tag 2.0.4 does not include the paper.

Yes, this is right. tag 2.0.4 is the latest version of the PPC without the paper.

I'm not sure if there are rules here, and I'll leave this to the editors. Seems better to have the paper in the tagged release, though.

Is there a rule?

The paper appears to be missing the required link to the repo.

Is that needed?

This is in the guidelines for reviewers (emphasis mine):

JOSS papers should contain the following:

...

A list of key references including a link to the software archive
As a reviewer, I feel my job is done by pointing out that this is not done.

I do not get it, sorry? Should I at the repo as webpage to the bib file and reference the entry in the paper? I do not see this in other paper of the journal.

There, I see Paper DOI; Software repository and Software Archive. But these things will be added by the editors, right? After accepting?

@simonbogutzky
Copy link

@davclark -

It seems like there is either a disagreement or lack of understanding here. Let me give a more concrete example:

I am at work with my fancy/annoying new macbook retina. To connect my phone, I need to unplug the hub from the single USB-C port, which means I need to rearrange my desk and move my ergo keyboard out of the way so I can plug into the USB-C port on my phone. I then grab the cable from the power adapter to connect my phone to my computer. The future is going to look more like this for more people.

What I'm suggesting is easy enough that I just installed the APK by creating a
fork and a new release here. Just like what you have, except including a direct download of the APK.

Sorry, there is a lack of understanding. What is the difference of my APK and your fork? In my case, I downloaded the APK from the repo and put it in my Google Drive or Dropbox folder and installed it on the smartphone.

@simonbogutzky
Copy link

@davclark -

What are existing solutions like? Why wouldn't I just use them?

Maybe there are other solutions, but I these solutions do not support the hardware of our equipment.
Here is another solution. Why wouldn't I just use that? I can at least partially answer this question, but I'm asking you to at a minimum cite primary existing solutions in your paper per JOSS guidelines.

Ok. Right. This is a quite new solution of Shimmer for a newer type of IMUs. We began our studies in 2013 and build the existing solution for an older type of Shimmer IMUs (R2). These solution seems not to meet our requirements of mobility and automatic time synchronization of more data streams. A solution, which can you use, if you find it, is that: "Psychophysiological Correlates of Flow During Daily Activities" by Gaggioli et al. (2013)

Could you say a little more about how "its visualization component allows checking the experimental setup in real time." This would address the question of manual steps to verify operation.
E. g. if you place electrodes on the human body, you can check the leakage of the ECG based on the realtime visualization of the heart activity and the pattern of the signal.
It seems there is no similar readout for phone sensors? Or am I missing something?

No, because we worked in our studies primary with the Shimmer R2 IMUs.

I am able to verify the contents of the CSV files manually. Note that there is a slight type-o in your documentation. You mention psychophysiocollector/PARTICIPANT_NAME/ACTIVITY_NAME, but you actually get them in psychophysiocollector/ACTIVITY_NAME/PARTICIPANT_NAME/DATETIME. Maybe just say PsychoPhysioCollector and leave the rest to the user (it's obvious from there).

That is right, I fixed it!

@davclark
Copy link
Member

davclark commented Aug 2, 2016

@sbogutzky I am pretty sure at this point that I've communicated my concerns / recommendations - so please make any remaining revisions and I'll wait for you to tell me you're done. I'll also wait for @arfon or another editor to chime in on the questions for them.

Regarding the google drive solution, that makes good sense. You've illustrated that there is a pretty easy way for individuals to install the APK without a wire. My solution was the same APK as yours, just providing a direct link from the github release page. I still think that'd be a nice-to-have, but I don't think this is required.

@simonbogutzky
Copy link

@davclark - Thank for your recommendations. I'm ready with my revision.

DOI

@davclark
Copy link
Member

davclark commented Aug 3, 2016

Looks good to me, and I hope the changes are helpful to others and (perhaps more importantly) for you substantiating this work as a genuine academic contribution.

So as reviewer, I recommend acceptance. It's up to the editors about the verification issue described above (that I can't verify all functionality without the sensors), but I recommend the "low bar" of accepting based on verification "as possible" and trusting the authors for the rest (including evidence from the provided references in which this software was used with the sensors).

Editors, please also note that the tag is updated to 2.0.5.

@simonbogutzky
Copy link

Editors, please also note that change the title to "PsychoPhysioCollector: A Smartphone-Based Data Collection App for Psychophysiological Research"

It is possible to change that on the Paper page?

@arfon
Copy link
Member

arfon commented Aug 4, 2016

Thanks for the review @davclark and for being responsive @sbogutzky. Some responses to questions that have come up in this thread:

Is there a rule? (about the tagged releases)

Ideally the release should include the paper but I don't think this is a hard rule right now. More important to me is that the release matches the version that has been archived (e.g. with Zenodo).

There, I see Paper DOI; Software repository and Software Archive. But these things will be added by the editors, right? After accepting?

That's correct. The final version of the PDF will include a direct link to the software repository and the archive of the code.

Editors, please also note that change the title to "PsychoPhysioCollector: A Smartphone-Based Data Collection App for Psychophysiological Research"

This is now changed.

So as reviewer, I recommend acceptance. It's up to the editors about the verification issue described above (that I can't verify all functionality without the sensors), but I recommend the "low bar" of accepting based on verification "as possible" and trusting the authors for the rest (including evidence from the provided references in which this software was used with the sensors).

👍

@arfon arfon added the accepted label Aug 4, 2016
@arfon
Copy link
Member

arfon commented Aug 4, 2016

@sbogutzky you paper is now accepted into JOSS and your DOI is http://dx.doi.org/10.21105/joss.00040 🎉 🚀 💥

Thanks for all of your help @davclark!

@arfon arfon closed this as completed Aug 4, 2016
@arfon arfon changed the title Submission: PsychoPhysioCollector [REVIEW]: PsychoPhysioCollector Aug 19, 2018
@whedon whedon added the published Papers published in JOSS label Mar 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted published Papers published in JOSS recommend-accept Papers recommended for acceptance in JOSS. review
Projects
None yet
Development

No branches or pull requests

5 participants