Upload of package version of same name causes error and package removal #57

matthewfeickert · 2024-01-24T20:27:11Z

Anaconda Cloud seems to have changed how they handle uploads of packages where there is already a package of the same name and version on Anaconda Cloud. In the past, this simply just resulted in the artifact being uploaded overwriting the previous artifact. networkx is one of the core packages that used this approach. However, I noticed that networkx's nightly workflow is failing to do so

...
File type is "Standard Python"
Extracting standard python attributes for upload
Creating package "networkx"
Creating release "3.3rc0.dev0"
Uploading file "scientific-python-nightly-wheels/networkx/3.3rc0.dev0/networkx-3.3rc0.dev0-py3-none-any.whl"
Warning:  Distribution "networkx-3.3rc0.dev0-py3-none-any.whl" already exists. Removing.
Error:  ("release version='3.3rc0.dev0' does not exist", 404)
##[debug]Docker Action run completed with exit code 1
##[debug]Finishing: Upload nighlty wheel

(The success after failure is due to the package being removed and so there is no conflict the next upload.)

This started happening on 2024-01-04 for networkx, and was not happening before that. For example, this 2024-01-02 upload ran successfully

+ anaconda --token *** upload --force --user scientific-python-nightly-wheels dist/networkx-3.3rc0.dev0-py3-none-any.whl
...
Using Anaconda API: https://api.anaconda.org/
Using "scientific-python-nightly-wheels" as upload username
Processing "dist/networkx-3.3rc0.dev0-py3-none-any.whl"
Detecting file type...
File type is "Standard Python"
Extracting standard python attributes for upload
Creating package "networkx"
Creating release "3.3rc0.dev0"
Uploading file "scientific-python-nightly-wheels/networkx/3.3rc0.dev0/networkx-3.3rc0.dev0-py3-none-any.whl"
Warning:  Distribution "networkx-3.3rc0.dev0-py3-none-any.whl" already exists. Removing.

  0%|          | 0.00/1.59M [00:00<?, ?B/s]
1.59MB [00:00, 5.26MB/s]                   
Upload complete

standard python located at:
  https://anaconda.org/scientific-python-nightly-wheels/networkx

We know that as we're using a lock file

upload-nightly-action/cmd.sh

Lines 57 to 60 in 6f1e2b9

    
           micromamba create \ 
        
             --yes \ 
        
             --name upload-nightly-action \ 
        
             --file /conda-lock.yml

that nothing has changed from the upload-nightly-action action side, and so this has to be on the Anaconda Cloud side.

cc as a heads up to @MridulS @jarrodmillman

The text was updated successfully, but these errors were encountered:

matthewfeickert · 2024-01-24T20:31:23Z

Opened anaconda/anaconda-client#702 to try to get input from Anaconda.

tupui · 2024-01-24T21:17:26Z

While we are at it, could we update the token used by NetworkX? It's still using a global token instead of a personal one which is less than ideal. (Xarray is the only other one doing that, and we should also migrate)

matthewfeickert · 2024-01-24T21:34:39Z

While we are at it, could we update the token used by NetworkX? It's still using a global token instead of a personal one which is less than ideal. (Xarray is the only other one doing that, and we should also migrate)

@MridulS @jarrodmillman can you please take care of this for NetworkX? I don't remember who is responsible for the Xarray uploads, so tagging @martinfleis to see if they can do it or point us to the correct people to ask. (@tupui if you know please tag them in. 👍)

tupui · 2024-01-24T21:46:41Z

That's the thing I don't know who they are as the team on Anaconda is empty 😅 (sorry for hijacking the issue, hum...)

martinfleis · 2024-01-24T22:05:19Z

@matthewfeickert I believe that @keewis is the one with right permissions over in xarray.

mattip · 2024-01-31T12:24:00Z

Also hitting this in scipy-openblas32 and scipy-openblas64, with @honno's API key, see the latest merge CI run which all failed to upload.

mattip · 2024-01-31T15:55:27Z

I wonder what permissions are missing on the existing token? Here are the possible ones. This action recommends "Allow write access to the API site" and "Allow uploads to Standard Python repositories"

keewis · 2024-01-31T15:57:51Z

From https://scientific-python.org/specs/spec-0004/#process-for-adding-new-projects:

The token should only have the “Allow uploads to Standard Python repositories”, “Allow read access to the API site” and “Allow write access to the API site” scope.

(see #60 for an issue on updating the readme)

mattip · 2024-02-01T11:04:47Z

Could an admin check that the tokens in question have the "Allow read access to the API site" permission?

matthewfeickert · 2024-02-02T06:35:06Z

Also hitting this in scipy-openblas32 and scipy-openblas64, with @honno's API key, see the latest merge CI run which all failed to upload.

@mattip This is a different issue, so can you please move the discussion to Issue #61?

dholth · 2024-02-06T20:21:21Z

Are you using the --force option? We are able to run a command like this one.
anaconda upload /opt/anaconda/conda-bld/osx-64/argh-0.24.1-py39_0.tar.bz2 --force
Multiple archives are being uploaded with one command?

matthewfeickert · 2024-02-06T21:35:57Z

Are you using the --force option? We are able to run a command like this one. anaconda upload /opt/anaconda/conda-bld/osx-64/argh-0.24.1-py39_0.tar.bz2 --force.

@dholth, yes. This is the command that is being run

upload-nightly-action/cmd.sh

Lines 75 to 79 in 66bc1b6

    
           anaconda --token "${ANACONDA_TOKEN}" upload \ 
        
             --force \ 
        
             --user "${ANACONDA_ORG}" \ 
        
             ${LABEL_ARGS} \ 
        
             "${INPUT_ARTIFACTS_PATH}"/*.whl

Multiple archives are being uploaded with one command?

This works with multiple wheels, yes, but in this case only 1 wheel is being uploaded. I assume that's what you mean by archives here.

The problem here is not that the command can not run, it is that there has been a behavior change in how Anaconda Cloud seems to be dealing with the isntance when there already exists a package release of the same name. To step through (the perceived order of events) for what is causing the fail/pass alternation in

First, networkx always uploads a nightly wheel with the exact same name (networkx-3.3rc0.dev0-py3-none-any.whl), so they are always going to try overwrite the previous wheels they uploaded.

In https://github.com/networkx/networkx/blob/31ef96361fc95ce8789d780cbc5bb4b3c4e27eb9/.github/workflows/nightly.yml a wheel is built and then

      - name: Upload nighlty wheel
        uses: scientific-python/upload-nightly-action@6e9304f7a3a5501c6f98351537493ec898728299 # 0.3.0
        with:
          anaconda_nightly_upload_token: ${{ secrets.ANACONDA_NIGHTLY }}
          artifacts_path: dist/

, which is going to call the anaconda upload --force command above, is run.

The uploaded creates the package and release and begins the upload to Anaconda cloud (e.g. https://anaconda.org/scientific-python-nightly-wheels/networkx/)

+ anaconda --token *** upload --force --user scientific-python-nightly-wheels --label main dist/networkx-3.3rc0.dev0-py3-none-any.whl
Uploading wheels to anaconda.org...
Using Anaconda API: https://api.anaconda.org
Using "scientific-python-nightly-wheels" as upload username
Processing "dist/networkx-3.3rc0.dev0-py3-none-any.whl"
Detecting file type...
File type is "Standard Python"
Extracting standard python attributes for upload
Creating package "networkx"
Creating release "3.3rc0.dev0"
Uploading file "scientific-python-nightly-wheels/networkx/3.3rc0.dev0/networkx-3.3rc0.dev0-py3-none-any.whl"

It detects that there already is a distribution of the same name at the upload location and removes it

Warning:  Distribution "networkx-3.3rc0.dev0-py3-none-any.whl" already exists. Removing.

This is also removing the only file under the https://anaconda.org/scientific-python-nightly-wheels/networkx package.

Anaconda cloud throws a 404 that the distribution does not exist (I don't know why this happens)

Error:  ("release version='3.3rc0.dev0' does not exist", 404)

and despite the --force option being used at this point things stop and exit out with exit code 1.

##[debug]Docker Action run completed with exit code 1
##[debug]Finishing: Upload nighlty wheel

The next day (the workflow runs on a nightly schedule) steps 1 and 2 are repeated, but as there is no existing distribution of that name given the upload failed the previous day, step 3 never happens and the upload succeeds.

+ anaconda --token *** upload --force --user scientific-python-nightly-wheels --label main dist/networkx-3.3rc0.dev0-py3-none-any.whl
Using Anaconda API: https://api.anaconda.org
Using "scientific-python-nightly-wheels" as upload username
Processing "dist/networkx-3.3rc0.dev0-py3-none-any.whl"
Detecting file type...
File type is "Standard Python"
Extracting standard python attributes for upload
Creating package "networkx"
Creating release "3.3rc0.dev0"
Uploading file "scientific-python-nightly-wheels/networkx/3.3rc0.dev0/networkx-3.3rc0.dev0-py3-none-any.whl"

  0%|          | 0.00/1.60M [00:00<?, ?B/s]
 62%|██████▏   | 0.98M/1.60M [00:00<00:00, 10.3MB/s]
1.60MB [00:00, 6.89MB/s]
Upload complete

standard python located at:
  https://anaconda.org/scientific-python-nightly-wheels/networkx

This started happening on 2024-01-04. The last change to our lock file for our software environment and upload command before 2024 (5fb764c) was on 2023-10-06.

You can see from the networkx workflow run history that before 2024-01-04 this exact same workflow ran without errors every day. (Earliest successful run in this screenshot is 2023-12-24 and the first failing run at the top is 2024-01-04)

Given all this, my take away is that there was some backend (well, backend from the user point of view) change in Anaconda Cloud on 2024-01-04 that caused this breaking behavior.

matthewfeickert · 2024-02-06T21:49:58Z

If it is helpful at all, I can also provide the passing 2024-01-03 and failing 2024-01-04 full run logs (for the same workflow).

matthewfeickert · 2024-02-09T05:40:16Z

Multiple archives are being uploaded with one command?

So in anaconda/anaconda-client#702 (comment) it seems that there is different behavior with one archive/wheel being uploaded vs. multiple. :? Or more accurately, with multiple archives being uploaded it is more unclear what is causing the original 404 error that is causing everything to break.

edit: From more testing, I can see that what is happening is that at the

Warning:  Distribution "networkx-3.3rc0.dev0-py3-none-any.whl" already exists. Removing.

stage, as there is only 1 release (networkx-3.3rc0.dev0-py3-none-any.whl) in the package list when the release is removed, the package is removed from the org too. So I guess the

Error:  ("release version='3.3rc0.dev0' does not exist", 404)

is because the package is gone, there isn't a package directory to upload the release to. This doesn't happen with Awkward testing above with multiple wheels, as when one wheel is removed there are others that keep the package directory from being removed mid-upload.

matthewfeickert added the bug Something isn't working label Jan 24, 2024

matthewfeickert mentioned this issue Jan 24, 2024

Anaconda Cloud now causes error and package removal for upload of package version of same name anaconda/anaconda-client#702

Closed

keewis mentioned this issue Jan 24, 2024

switching xarray away from a global token #58

Closed

matthewfeickert mentioned this issue Feb 2, 2024

scipy-openblas32 and scipy-openblas64 have wrong permissions for their tokens #61

Closed

matthewfeickert mentioned this issue Feb 15, 2024

fix: Guard against removing package while uploading to it #64

Merged

matthewfeickert closed this as completed in #64 Feb 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upload of package version of same name causes error and package removal #57

Upload of package version of same name causes error and package removal #57

matthewfeickert commented Jan 24, 2024 •

edited

Loading

matthewfeickert commented Jan 24, 2024

tupui commented Jan 24, 2024

matthewfeickert commented Jan 24, 2024

tupui commented Jan 24, 2024 •

edited

Loading

martinfleis commented Jan 24, 2024

mattip commented Jan 31, 2024

mattip commented Jan 31, 2024

keewis commented Jan 31, 2024 •

edited

Loading

mattip commented Feb 1, 2024

matthewfeickert commented Feb 2, 2024

dholth commented Feb 6, 2024 •

edited

Loading

matthewfeickert commented Feb 6, 2024 •

edited

Loading

matthewfeickert commented Feb 6, 2024

matthewfeickert commented Feb 9, 2024 •

edited

Loading

Upload of package version of same name causes error and package removal #57

Upload of package version of same name causes error and package removal #57

Comments

matthewfeickert commented Jan 24, 2024 • edited Loading

matthewfeickert commented Jan 24, 2024

tupui commented Jan 24, 2024

matthewfeickert commented Jan 24, 2024

tupui commented Jan 24, 2024 • edited Loading

martinfleis commented Jan 24, 2024

mattip commented Jan 31, 2024

mattip commented Jan 31, 2024

keewis commented Jan 31, 2024 • edited Loading

mattip commented Feb 1, 2024

matthewfeickert commented Feb 2, 2024

dholth commented Feb 6, 2024 • edited Loading

matthewfeickert commented Feb 6, 2024 • edited Loading

matthewfeickert commented Feb 6, 2024

matthewfeickert commented Feb 9, 2024 • edited Loading

matthewfeickert commented Jan 24, 2024 •

edited

Loading

tupui commented Jan 24, 2024 •

edited

Loading

keewis commented Jan 31, 2024 •

edited

Loading

dholth commented Feb 6, 2024 •

edited

Loading

matthewfeickert commented Feb 6, 2024 •

edited

Loading

matthewfeickert commented Feb 9, 2024 •

edited

Loading