different versions of CR mkfastq/(count-vdj)

[manugarciaquismondo]

Added documentation for using different versions of Cellranger mkfastq and count/vdj

[cornhundred]

@manugarciaquismondo please explain why this would happen in the first place:

Occasionally, multiple cellranger mkfastq processes running on the same machine

Should we be doing this or is it a sign that we're not running things correctly? Conceptually, we should only ever need to run one instance of mkfastq to de-multiplex a BCL file - if we're running more than one we're probably doing something wrong (e.g. trying to avoid an index clash, right, telling two simultaneous jobs to write to the same directory when they could be run sequentially).

[manugarciaquismondo]

We are running a single instance of mkfastq. It is just that the image of the containers that we are using to run cellranger mkfastq and cellranger count is different. As it says here:

Occasionally, A single call to cellranger mkfastq spawns multiple processes running on the same machine.

these processes are running on the same machine, so there is only a single container obtaining the FASTQs from the BCL file.

[manugarciaquismondo]

(e.g. trying to avoid an index clash, right, telling two simultaneous jobs to write to the same directory when they could be run sequentially).

This is how cellranger mkfastq normally operates (it is not a patch that we did to avoid the index clash); it spawns multiple processes to finish the conversion from BCL to FASTQs faster. However, this normal operation mode fails on Cellranger version 3.1.0 when the processes write on the same directory (that is also completely normal for all versions of Cellranger; in fact, a call to cellranger mkfastq on 3.0.2 spawns multiple processes that write on the same directory and works perfectly fine), that it is why it could be convenient to use 3.0.2 for calls to cellranger mkfastq.

The index clash is a different problem, it is solved setting --barcode-mismatches=0 when cellranger mkfastq is called.

[cornhundred]

That is not clear to me. Why would we want to use more than one version? Why is it normal for it to fail?

[manugarciaquismondo]

It is not normal to fail, what is normal is "telling two simultaneous jobs to write to the same directory when they could be run sequentially". It is done that way to finish the generation of FASTQs faster.
Also, suppose the case that we have a pooled sample with no Gene Expression data. If we use cellranger mkfastq version 3.1.0, then it can fail due to this concurrency problem. If we use cellranger count version 3.0.2, it cannot be done. Therefore, an option is to use cellranger mkfastq version 3.0.2 and cellranger count version 3.1.0, that is, using different version of Cellranger for mkfastq and count.

[cornhundred]

I don't understand what you are saying. Have you checked with 10x tech help about this?

[manugarciaquismondo]

I did; they asked me to:

1. Upload the mri.tgz file
cellranger upload [email protected] sample_id.mri.tgz
2. Please send us your samplesheet file.
3. List of all FASTQ files in the directory used for --fastqs 

I have not done it yet; I will get to it this week. Given that this is a non-deterministic error, it may require to launch a large number of cellranger mkfastq runs until this error manifests.