Kaguya Ancillary Data download

[jmk0502]

Hello all,
I am trying to download Kaguya Ancillary data (using downloadIsisData kaguya $ISISDATA --exclude="kernels/**")but this is the error I am getting:

2024/11/08 10:02:05 ERROR : Local file system at /home/iirs/anaconda3/envs/isis/data/kaguya: Ignoring --track-renames as the source and destination do not have a common hash
2024/11/08 10:02:05 ERROR : Local file system at /home/iirs/anaconda3/envs/isis/data/kaguya: Ignoring --track-renames as it doesn't work with copy or move, only sync
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         1.2s

This same command had worked perfectly when I downloaded LRO Ancillary data and base data area just after installing ISIS. I suppose, all data need not be downloaded immediately after installation and so I am a little baffled. Any idea why I am receiving this error and a possible workaround/solution?

P.S. Should I try to use rclone (using conda install –c conda-forge rclone), I get the following message:

Channels:
 - usgs-astrogeology
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - c
  - conda-forge

Current channels:

  - https://conda.anaconda.org/usgs-astrogeology
  - https://conda.anaconda.org/conda-forge
  - defaults

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

Maybe because of this,I couldn't download the TestData even the first time around?
Could anyone maybe help me with what I am missing here?
Any help is much appreciated ! Thanks !

[KrisBecker]

Hi @jmk0502 ...

It's not clear what you are trying to accomplish with the command you are running as it filters out the entire Kaguya kernel set.

It does not appear to be a problem with a missing rclone command, rather its how downloadIsisData filters data files.

Some robust verbosity in the resulting command shows this:

% ./downloadIsisData kaguya $PWD/TempDir --config=rclone.conf -vv --exclude="kernels/**"
Message: 'args: '
Arguments: (Namespace(mission='kaguya', dest='/Volumes/ISISDATA/TempDir', verbose=2, num_transfers=10, config='rclone.conf', filter=None, include=None, exclude=['kernels/**'], no_kernels=False, command='copy'),)
2024-11-08 05:25:45 DEBUG    Additional Args:
2024-11-08 05:25:45 INFO     ---Script starting---
2024-11-08 05:25:45 DEBUG    Using config: rclone.conf
2024-11-08 05:25:45 DEBUG    rclone() : using config file rclone.conf
2024-11-08 05:25:45 DEBUG    Invoking : rclone listremotes --config=rclone.conf
2024-11-08 05:25:45 DEBUG    Invoking : rclone listremotes --config=rclone.conf
2024-11-08 05:25:45 DEBUG    Remote Sources: ['asc_s3:', 'esa:', 'naif:', 'jaxa:', 'sbn:', 'base:', 'legacybase:', 'control:', 'apollo15:', 'apollo16:', 'apollo17:', 'lro:', 'tgo_naifKernels:', 'tgo_usgs:', 'tgo:', 'dawn_naifKernels:', 'dawn_usgs:', 'dawn_pck:', 'dawn_kernel_combine:', 'dawn:', 'cassini_naifKernels:', 'cassini_usgs:', 'cassini:', 'hayabusa2_naifKernels:', 'hayabusa2_usgs:', 'hayabusa2:', 'galileo_naifKernels:', 'galileo_usgs:', 'galileo:', 'juno_naifKernels:', 'juno_usgs:', 'juno:', 'lo:', 'odyssey_naifKernels:', 'odyssey_usgs:', 'odyssey:', 'mro_naifKernels:', 'mro_usgs:', 'mro:', 'mex_naifKernels:', 'mex_usgs:', 'mex:', 'hayabusa_naifKernels:', 'hayabusa_usgs:', 'hayabusa:', 'chandrayaan1_naifKernels:', 'chandrayaan1_usgs:', 'chandrayaan1:', 'clementine1_naifKernels:', 'clementine1_usgs:', 'clementine1:', 'kaguya_naifKernels:', 'kaguya_usgs:', 'kaguya:', 'mariner10_naifKernels:', 'mariner10_usgs:', 'mariner10:', 'mer:', 'messenger_naifKernels:', 'messenger_usgs:', 'messenger:', 'mgs_naifKernels:', 'mgs_usgs:', 'mgs:', 'near_naifKernels:', 'near_usgs:', 'near:', 'newhorizons_naifKernels:', 'newhorizons_usgs:', 'newhorizons:', 'osirisrex_naifKernels:', 'osirisrex_usgs:', 'osirisrex:', 'rolo:', 'rosetta_naifKernels:', 'rosetta_usgs:', 'rosetta:', 'smart1_naifKernels:', 'smart1_usgs:', 'smart1:', 'viking1_naifKernels:', 'viking1_usgs:', 'viking1:', 'viking2_naifKernels:', 'viking2_usgs:', 'viking2:', 'voyager1_naifKernels:', 'voyager1_usgs:', 'voyager1:', 'voyager2_naifKernels:', 'voyager2_usgs:', 'voyager2:', 'msl_usgs:', 'msl_naifKernels:', 'msl:', '']
2024-11-08 05:25:45 DEBUG    Supported missions:
 ['base', 'legacybase', 'control', 'apollo15', 'apollo16', 'apollo17', 'lro', 'tgo', 'dawn', 'cassini', 'hayabusa2', 'galileo', 'juno', 'lo', 'odyssey', 'mro', 'mex', 'hayabusa', 'chandrayaan1', 'clementine1', 'kaguya', 'mariner10', 'mer', 'messenger', 'mgs', 'near', 'newhorizons', 'osirisrex', 'rolo', 'rosetta', 'smart1', 'viking1', 'viking2', 'voyager1', 'voyager2', 'msl']
2024-11-08 05:25:45 DEBUG    Creating RClone command for kaguya
2024-11-08 05:25:45 DEBUG    Args created: ['kaguya:', '/Volumes/ISISDATA/TempDir/kaguya', '--progress', '--checkers=10', '--transfers=10', '--track-renames', '--log-level=DEBUG', '--filter=+ calibration/**', '--filter=- source/', '--filter=- /a_older_versions/', '--filter=- /former_versions/', '--filter=- corrupt_files/', '--filter=- zzarchive/', '--filter=- /original/', '--filter=- ck/prime_mission/', '--filter=- extended_mission/', '--filter=- /prime_mission/', '--filter=- ck/GEM/', '--filter=- ck/save/', '--filter=- spk/SAVE_SCS_2017-11-22/', '--filter=- spk/spk_psp_rec*', '--filter=- release*/', '--filter=- Archive/', '--filter=- ek/', '--filter=- *.lbl', '--filter=- *.txt', '--filter=- *.log', '--filter=- *.temp', '--filter=- *.history', '--filter=- misc/', '--filter=- document/', '--filter=- *.csv', '--filter=- toolkit/', '--filter=- kernels_ORG/', '--filter=- orbnum/**', '--filter=- old/**', '--filter=- SpecialProducts/**', '--filter=- kernels/**']
2024-11-08 05:25:45 DEBUG    rclone() : using config file rclone.conf
2024-11-08 05:25:45 DEBUG    Invoking : rclone copy --config=rclone.conf kaguya: /Volumes/ISISDATA/TempDir/kaguya --progress --checkers=10 --transfers=10 --track-renames --log-level=DEBUG --filter=+ calibration/** --filter=- source/ --filter=- /a_older_versions/ --filter=- /former_versions/ --filter=- corrupt_files/ --filter=- zzarchive/ --filter=- /original/ --filter=- ck/prime_mission/ --filter=- extended_mission/ --filter=- /prime_mission/ --filter=- ck/GEM/ --filter=- ck/save/ --filter=- spk/SAVE_SCS_2017-11-22/ --filter=- spk/spk_psp_rec* --filter=- release*/ --filter=- Archive/ --filter=- ek/ --filter=- *.lbl --filter=- *.txt --filter=- *.log --filter=- *.temp --filter=- *.history --filter=- misc/ --filter=- document/ --filter=- *.csv --filter=- toolkit/ --filter=- kernels_ORG/ --filter=- orbnum/** --filter=- old/** --filter=- SpecialProducts/** --filter=- kernels/**
2024-11-08 05:25:45 DEBUG    Invoking : rclone copy --config=rclone.conf kaguya: /Volumes/ISISDATA/TempDir/kaguya --progress --checkers=10 --transfers=10 --track-renames --log-level=DEBUG --filter=+ calibration/** --filter=- source/ --filter=- /a_older_versions/ --filter=- /former_versions/ --filter=- corrupt_files/ --filter=- zzarchive/ --filter=- /original/ --filter=- ck/prime_mission/ --filter=- extended_mission/ --filter=- /prime_mission/ --filter=- ck/GEM/ --filter=- ck/save/ --filter=- spk/SAVE_SCS_2017-11-22/ --filter=- spk/spk_psp_rec* --filter=- release*/ --filter=- Archive/ --filter=- ek/ --filter=- *.lbl --filter=- *.txt --filter=- *.log --filter=- *.temp --filter=- *.history --filter=- misc/ --filter=- document/ --filter=- *.csv --filter=- toolkit/ --filter=- kernels_ORG/ --filter=- orbnum/** --filter=- old/** --filter=- SpecialProducts/** --filter=- kernels/**
2024/11/08 05:25:45 DEBUG : rclone: Version "v1.68.1-DEV" starting with parameters ["rclone" "copy" "--config=rclone.conf" "kaguya:" "/Volumes/ISISDATA/TempDir/kaguya" "--progress" "--checkers=10" "--transfers=10" "--track-renames" "--log-level=DEBUG" "--filter=+ calibration/**" "--filter=- source/" "--filter=- /a_older_versions/" "--filter=- /former_versions/" "--filter=- corrupt_files/" "--filter=- zzarchive/" "--filter=- /original/" "--filter=- ck/prime_mission/" "--filter=- extended_mission/" "--filter=- /prime_mission/" "--filter=- ck/GEM/" "--filter=- ck/save/" "--filter=- spk/SAVE_SCS_2017-11-22/" "--filter=- spk/spk_psp_rec*" "--filter=- release*/" "--filter=- Archive/" "--filter=- ek/" "--filter=- *.lbl" "--filter=- *.txt" "--filter=- *.log" "--filter=- *.temp" "--filter=- *.history" "--filter=- misc/" "--filter=- document/" "--filter=- *.csv" "--filter=- toolkit/" "--filter=- kernels_ORG/" "--filter=- orbnum/**" "--filter=- old/**" "--filter=- SpecialProducts/**" "--filter=- kernels/**"]
2024/11/08 05:25:45 DEBUG : Creating backend with remote "kaguya:"
2024/11/08 05:25:45 DEBUG : Using config file from "/Volumes/ISISDATA/rclone.conf"
2024/11/08 05:25:45 DEBUG : Creating backend with remote "kaguya_naifKernels:"
2024/11/08 05:25:45 DEBUG : Creating backend with remote "kaguya_usgs:"
2024/11/08 05:25:45 DEBUG : Creating backend with remote "asc_s3:asc-isisdata/usgs_data/kaguya/"
2024/11/08 05:25:45 DEBUG : Creating backend with remote "jaxa:/pub/spice/SELENE/kernels/"
2024/11/08 05:25:45 DEBUG : Using anonymous credentials - did you mean to set env_auth=true?
2024/11/08 05:25:45 DEBUG : Root: http://www.darts.isas.jaxa.jp/pub/spice/SELENE/kernels/
2024/11/08 05:25:45 DEBUG : fs cache: renaming cache item "asc_s3:asc-isisdata/usgs_data/kaguya/" to be canonical "asc_s3:asc-isisdata/usgs_data/kaguya"
2024/11/08 05:25:45 DEBUG : fs cache: renaming cache item "kaguya_usgs:" to be canonical "asc_s3:asc-isisdata/usgs_data/kaguya"
2024/11/08 05:25:45 DEBUG : fs cache: switching user supplied name "kaguya_usgs:" for canonical name "asc_s3:asc-isisdata/usgs_data/kaguya"
2024/11/08 05:25:45 DEBUG : union root '': actionPolicy = *policy.EpAll, createPolicy = *policy.EpMfs, searchPolicy = *policy.FF
2024/11/08 05:25:45 DEBUG : Creating backend with remote "/Volumes/ISISDATA/TempDir/kaguya"
2024/11/08 05:25:45 ERROR : Local file system at /Volumes/ISISDATA/TempDir/kaguya: Ignoring --track-renames as the source and destination do not have a common hash
2024/11/08 05:25:45 ERROR : Local file system at /Volumes/ISISDATA/TempDir/kaguya: Ignoring --track-renames as it doesn't work with copy or move, only sync
2024/11/08 05:25:45 DEBUG : kernels: Excluded
2024/11/08 05:25:45 DEBUG : Local file system at /Volumes/ISISDATA/TempDir/kaguya: Waiting for checks to finish
2024/11/08 05:25:45 DEBUG : Local file system at /Volumes/ISISDATA/TempDir/kaguya: Waiting for transfers to finish
2024/11/08 05:25:45 INFO  : There was nothing to transfer
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.3s
2024/11/08 05:25:45 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.3s

2024/11/08 05:25:45 DEBUG : 7 go routines active

The ERRORS you see are due to the incompatible parameter --track-renames, which is only valid for the rclone sync command - downloadIsisData uses the rclone copy command, and as indicated, is ignored in his case. It's really more of a warning in this context.

To get the Kaguya kernels downloaded, remove the --exclude="kernels/**".

[jmk0502]

Sorry for the late reply. I will try it again after removing the --exclude="kernels/**".

Also, do you know if the Test Data is required to be downloaded for non-developers?

Thanks for your response!

[Kelvinrr]

@jmk0502 for non developers, test data is not needed, you only need it if you plan to run the test suite.