Trying to remove a zarr tmp files on s3, but won't remove?

[rsignell-usgs]

What am I doing wrong here?
I wrote these temp files using the same fs_write filesystem.

fs_write = fsspec.filesystem('s3', anon=False, skip_instance_cache=True, 
                             use_listings_cache=False, 
                config_kwargs={'connect_timeout':5, 'read_timeout':5})

temp_name = 'esip-qhub/noaa/maucoin/rechunk/test01.tmp'

fs_write.rm(temp_name, recursive=True, clear_instance_cache=True)

fs_write.ls(temp_name)

produces:

 'esip-qhub/noaa/maucoin/rechunk/test01.tmp/APCP_surface',
 'esip-qhub/noaa/maucoin/rechunk/test01.tmp/DLWRF_surface',
 'esip-qhub/noaa/maucoin/rechunk/test01.tmp/DSWRF_surface',
 'esip-qhub/noaa/maucoin/rechunk/test01.tmp/PRES_surface',
 'esip-qhub/noaa/maucoin/rechunk/test01.tmp/SPFH_2maboveground',
 'esip-qhub/noaa/maucoin/rechunk/test01.tmp/TMP_2maboveground',
 'esip-qhub/noaa/maucoin/rechunk/test01.tmp/UGRD_10maboveground',
 'esip-qhub/noaa/maucoin/rechunk/test01.tmp/VGRD_10maboveground']

[martindurant]

I have ... absolutely no idea! I am trying this too and seeing the same. I may create some branches on s3fs for debugging. It's OK to kill these files?

[martindurant]

It seems like "access denied" is being silently ignored for each file. I tried setting the profile to "esip-qhub" (from within the hub) and then it actually raises with PermissionError .

[rsignell-usgs]

Okay I went to the AWS console and found an extra policy added by AWS 10 days ago triggered by a security issue -- apparently the keys were leaked. this policy does not allow deletion! so not an fsspec problem after all. I will rotate the keys

[martindurant]

I am changing the rm() code to return the list of deleted keys, so at least you would have been informed that s3fs knew it didn't succeed in deleting things. I'm not sure if we can automatically parse the message that comes back from AWS to raise an appropriate error.