Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scheduled monthly dependency update for August #212

Closed
wants to merge 14 commits into from
Closed

Scheduled monthly dependency update for August #212

wants to merge 14 commits into from

Conversation

pyup-bot
Copy link
Collaborator

@pyup-bot pyup-bot commented Aug 1, 2024

Update toil[cwl] from 5.0.0 to 7.0.0.

Changelog

6.1.0

Highlighted Features Added

* WDL and CWL task standard output and standard error logs that are not captured by the workflow will now be logged at INFO level and stored in the `--writeLogs`/`--writeLogsGzip` directory. (4657)
* Use a default log limit of 100MiB (4788)

Breaking Changes

* Stats and logging system again uses job display name (4755)
* `--disableProgress` is once again a flag that doesn't take an argument (4758)


CWL

* Don't clear out user-provided values for the --default-container option (4730)

WDL

* WDL job names now include numbers for scatters (4755)
* Multi-line WDL placeholder substitutions no longer interfere with de-indenting WDL command blocks (https://github.com/chanzuckerberg/miniwdl/issues/665)
* Standard error for failed tasks is now always logged to the worker log somewhere (4781)

Kubernetes

Dependencies

* Deps: removed the ruaml.yaml.string plugin dependency for a simpler solution (4760)

Misc

* Toil will no longer warn about a missing XDG_RUNTIME_DIR (4769)
* Read the Docs and CI docs builds should have Graphviz installed (pending CI image rebuild) (4734)
* Add more Python3.12 compatibility by replacing the one function from distutils that we use, `strtobool()`. (4765)
* Set default cache folders to be accessible between toil-wdl-runner workflows (Same as MiniWDL/Singularity defaults) (4761)
* Set toil-wdl-runner cache folders on Toil managed clusters to be at `/var/lib/toil` (4761)
* Fall back to assuming machine has 1 core when CPU count is unavailable. (4545)
* `FileJobStore` now supports filenames that get modified when percent-encoded (4779)

Thank you to our contributors:

DailyDreaming mr-c stxue1 adamnovak app/dependabot 

**Full Changelog**: https://github.com/DataBiosphere/toil/compare/releases/6.0.0...releases/6.1.0

6.0.0

NOTE!
We now have a config file!  https://toil.readthedocs.io/en/latest/running/cliOptions.html#the-config-file


Breaking Changes
* Removed the parasol batch system
* Removed the TES batch system (this is now a plugin)
* Removed our WDL compiler in favor of an interpreter (we still support WDL, we just do it differently now)
* We no longer support python3.7


CWL
* Support CWL 1.2.1 (4682)
* CWL Pipefish compatibility (4636)
* Support per-task preemptibility in CWL (4551)
* Fix configargparse in CWL (4618)
* cwl: use the latest commit from the proposed CWL v1.2.1 branch (4565)
* Upgrade cwltool to avoid broken galaxy-tool-util release. (4639)
* Implement a better config file system for CWL/WDL options (4666)
* Allow working with remote files in CWL and WDL workflows (4690)
* Make cwl mutually exclusive groups exist only when cwl is not suppressed (4725)
* Log more usefully for CWL workflows (4736)


WDL
* Simplify WDL Toil job graphs (4524)
* More WDL and Slurm documentation (4558)
* Improve WDL documentation (4732)
* Add String to File functionality into toil-wdl-runner (4589)
* Run WDL output through Toil export system to support URIs (4579)
* Allow the WDL output section to reference itself (4592)
* Ensure sibling files in toil-wdl-runner (4610)
* Make WDLOutputJob collect all task outputs (4602)
* Report errors in WDL using MiniWDL's error location printer (4637)
* Remove the WDL compiler. (4679)
* Implement a better config file system for CWL/WDL options (4666)
* Allow working with remote files in CWL and WDL workflows (4690)
* Strip leading whitespace from WDL commands (4720)


Misc
* Add config file support (4569)
* Support Python3.11 and drop Python 3.7 (4646)
* Move TES batch system to a plugin (4650)
* Turn batch system tests back on (4649)
* Separate out integration tests to run on a schedule (4612)
* Avoid concurrent modification in cluster scaler tests (4600)
* Remove old buckets from AWS (4588)
* Tests: only request a single core (4572)
* Reduce the number of assert statements (4590)
* take any nvidia-smi exception as not having gpu (4611)
* More resiliancy (4395)
* Remove useage of the deprecated pkg_resources (4701)
* Make sure cwltool always knows we have an outdir to fix 4698 (4699)
* AWS jobStoreTest: re-use delete_s3_bucket from toil.lib.aws (4700)
* Only count output file usage when using the file store (4692)
* Remove the parasol batch system. (4678)
* Move around reqs and move aws dev libraries to aws (4664)
* Make sure the `--batchLogsDir` exists if it is set (4635)
* Update EC2 instances and EC2 update script. (4745)
* remove extraneous dependency on old 'mock' (4739)
* Point CI at the new public URLs for stuff we host
* Add __init__.py to options folder (4723)


_Bug Fixes_
* Lower redirect log level to fix 4526 (4578)
* Fix mypy from being broken by new boto types (4577)
* Fix CI on local Gitlab runners (4571)
* Banish ghost jobs (4563)
* Stop deleting chained-to jobs which fail as orphaned jobs (4557)
* Fix pickling error when jobstate file doesnt exist and fix threading error when lock file exists then disappears (4575)
* Fix 3867 and try to explain but not crash when bad things happen to our mutex file (4656)
* Fix CI Appliance Builds (4655)
* Tolerate a failed AMI polling attempt (4727)* Add pure Python fallback for getDirSizeRecursively() (4753)
* Don't mark inputs (or outputs) executable for no reason (4728)
* Fix scheduled CI tests (4742)
* Fix --printJobInfo (4709)


Thank you to our contributors: stxue1 , w-gao, DailyDreaming , mr-c , adamnovak , glennhickey, misterbrandonwalker, and a-detiste !

5.12.0

WDL
* Virtualize filenames as in-container paths from point of view of WDL command (4527)
* Add WDL conformance tests to CI (4530)
* Use less memory in the Giraffe WDL test (4541)

Version Upgrades
* Upgrade to cwltool 3.1.20230601100705 (4500)
* Update mock requirement from <5,>=4.0.3 to >=4.0.3,<6 (4366)

Misc
* Anonymous access to Google Storage (4518)
* Reorder config so that default settings are applied first (4528)
* Add a way to forward accelerators to Docker containers (4492)

_Bug Fixes_
* Fix test failures without docker installed (4544)
* Prevent certain tests from being run twice in CI (4529)
* Drop external Docker builder (4523)
* Fix CI lint test (4533)
* Grab AWS group policies on top of user (4505)
* Grab accelerator set off the end of the list instead of by index (4506)
* Fix RtD build (4491)
* Include tests (4499)

Thank you to our contributors: stxue1 , DailyDreaming , mr-c , adamnovak , and tjni !

5.11.0

Breaking Changes
* Imported files will be symlinked by default, unless the user sets `--noLinkImports` or the workflow imports with `symlink=False`. (3949)

WDL
* Toil will now stop if it encounters an error polling a possible import URL for a WDL workflow input file. (4479)
* WDL workflows will be protected against imported files with no basenames. (4477)

Misc
* Toil batch system ID numbers for issued jobs now start at 1. (4482)
* Attempts to import files from URLs when the implementing job store is missing an extra are now better reported. (4479)
* Include tests in the source distribution that gets published to PyPI (4499)

_Bug Fixes_
* Toil should no longer crash when a delete wins a race against a load in `FileJobStore` (4484)
* Prevent local root jobs (such as WDLRootJob) from being run twice. (4482)
* Slurm and other grid batch system jobs will now have more informative names (4472)
* WDL workflows can no longer import `""` as a File. (4477)

Thank you to our contributors: stxue1, DailyDreaming, mr-c, adamnovak

5.10.0

Changelog

Highlighted Features Added
* Add a `--caching` option which explicitly states whether to use caching with a workflow. Uses a default value depending on whether or not we are using the file job store if not specified. (4218)
* New prototype WDL runner `python -m toil.wdl.wdltoil` using MiniWDL (3468)
* MiniWDL-based WDL implementation can now run the vg Giraffe WDL workflow ( 4353)
* Toil now tests against our own tiny set of WDL conformance tests (4351)
* Toil can run the HPRC assembly WDL workflows (4435)
* Toil can now use Mesos roles (4455)

Breaking Changes

* Replace "preemptable" with "preemptible", add example of using --defaultPreemptible flag to Preemptibility documentation (1951)

CWL
* CWL: run all ExpressionTools on the Leader node, instead of submitting separate jobs (4157)

Kubernetes
* Kubernetes batch system: Delete jobs individually when batch delete fails (3403)
* Documentation for running a Toil leader for a Kubernetes workflow outside Kubernetes now covers examples and common problems for running CWL workflows (document toil-cwl-runner + "Running the Leader Outside Kubernetes" 3422)
* Kubernetes batch system: support `--maxCores`, `--maxDisk`, and `--maxMemory` (2864)
* Add tutorial for Kubernetes launch cluster (3743)


Dependencies

* Require htcondor 10 exactly (4315)
* Toil jobs now have a `local` parameter which determines if they should run on the leader. (4388)

Misc

* The offline tests can now be run in parallel (3493)
* Code updated to be more idiomatic for Python3.7 (4295)
* Support for a `--network` for `toil launch-cluster` for Google cloud (4196)
* Support for a `--use_private_ip` for `toil launch-cluster` to dial nodes by private IP instead of public IP (4196)
* GPU scheduling should now be supported on Slurm (4308)
* Toil now supports a `--batchLogsDir` option and `TOIL_BATCH_LOGS_DIR` environment variable, to provide a directory other than the work dir where Toil will instruct HPC batch systems to save their captured job logs.
* `htcondor` batch system should now work again, and will retry connections
*  Updated the --coalesceStatusCalls help documentation to reflect the current state of https://github.com/DataBiosphere/toil/issues/4431 (#4437)
* Toil no longer trusts XDG_RUNTIME_DIR under Slurm (fixes some of the issues behind 4395 when Slurm is configured not to follow the XDG spec) (4435)
* Toil now puts it lock files for Singularity cache directories for WDL in those directories (4435)
* Toil's WDL interpreter can now use local-to-the-leader jobs for evaluating WDL code that doesn't need appreciable resources (4388)
* Toil now tolerates more possible exceptions related to the panasas network file system (4440)
* Type hinting to functions in resource.py (938)
* Added return type to inVirtualEnv() in `__init__.py` (938)
* Added None checks to some function bodies (938)



_Bug Fixes_
* Stop crashing when predefined batch job exit reasons are used and need to go into the message bus log file (4321)
* Added `import subprocess` to restore the behavior of 588. (4429)
* Toil will no longer use the stored message bus path from an old execution of a workflow when deciding where to save the message bus log when restarting a workflow (4438)
* Fix --custom-net mutual exclusivity bug. (4458)


Thank you to our contributors: stxue1 , DailyDreaming , mr-c , adamnovak , jfennick , misterbrandonwalker , w-gao , stephanaime , glennhickey , Hexotical , manabuishii gmloose , boukn , and thiagogenez !

5.9.2

Changelog

_Bug Fixes_
* Change build tag import (4329)

Thank you to our contributors: adamnovak , Hexotical !

5.9.0

Changelog

_Bug Fixes_
* Fix --provisioner and --metrics together (4328)
* Ignore incorrect type hint from boto3, remove json.loads (4330)
* Warn about missing --bypass-file-store with in-place update (4337)
* Replace prepareHTSubmission with prepareSubmission in HTCondor (4319)
* Merge "Google fixes" (4293)
* Support (only) current htcondor (4320)
* Delete k8s jobs individually when batch delete fails (4306)

Misc
* Update aws spot documentation (4310)
* Enable parallel testing (3493)
* Add documentation for running CWL workflows on non-Toil-managed Kubernetes clusters (4332)
* Export all slurm args by default (4237)
* Allow for subclasses of base types in messages (4322)
* Non cache default (4299)

Dependencies
* Bump mypy from 0.982 to 0.991 (4345)
* Bump schema-salad>=8.4.20230128170514,<9 to schema-salad>=8.3.20220913105718,<8.4 (4342) (4341)
* Bump cwltool from 3.1.20221008225030 to 3.1.20221201130942 (4338)
* Bump pyupgrade to 3.7 (4295)

Thank you to our contributors: adamnovak , Hexotical , w-gao, mr-c , gmloose , boukn , and thiagogenez !

5.8.0

Changelog

Highlighted Features Added

* Toil server now exposes workflow tasks via WES (4046).
* Toil server now has a `--wes_dialect agc` option that will hide any tasks that don't have Amazon Batch job IDs, and put the IDs in the task names for those that do (4047).
* Toil jobs now accept an `accelerators` requirement, like `accelerators=1` or `accelerators={'kind': 'gpu', 'brand': 'nvidia', 'count': 2}` (4163)
* Include total requested cores for each job type in `toil stats` (4173)
* Toil jobs now expose `job.accelerators` to workflow 
* Add prefix suffix params to `AbstractFileStore.getLocalTempFile` and `AbstractFileStore.getLocalTempFileName` (4273)
* CWL: `--no-compute-checksum`, `--strict-cpu-limit`, `--disable-validate`, and `--fast-parser` are now available


Breaking Changes
* Toil's built-in autoscaler now guesses that some memory and disk space on nodes will not actually be available for jobs; pass `--assumeZeroOverhead` to revert to the old behavior (2103) 

CWL
* CWL job unit and display names have been changed to make more sense as task names, and management of them has been unified into a `CWLNamedJob`. (4046/4047)
* CWL `CUDARequirement` is parsed by `cwltool` and turned into a requirement for the minimum requested number of nvidia GPU accelerators (3982)
* fix false warning when outputSource contains only one None value (4300)


Kubernetes
* `KubernetesBatchSystem` can add `nvidia.com/gpu` and `amd.com/gpu` resource requests for jobs that request those accelerators (4163)
* `KubernetesBatchSystem` can request GPUs by `model` key, if nodes are labeled appropriately (4163)

Dependencies

Misc
* Toil WES server now accepts requests that leave out workflow_params. (4037)
* The `MessageBus` has been expanded to use `pypubsub`, and now has `MessageInbox` and `MessageOutbox` objects to represent connections to it. (4046/4047)
* `ToilMetrics` now rides on the `MessageBus` rails. (4046/4047)
* Toil workflows now have a `--writeMessages` option, which takes a file to which a line-oriented stream of `MessageBus` messages will be written. Reading this file will allow you to recover the current state of the workflow. (4046/4047)
* Add code for warning check to be used when launching cluster with AWS. (3514)
* Use a CI prebake image for gitlab testing. (4185)
* Toil clusters now have `/var/tmp` as the default temporary directory, since they often make large temporary files (4148)
* Adds basic testing for slurm using a slurm docker cluster by running sample workflows. (3856)
* Add message bus documentation (4239)
* `SingleMachineBatchSystem` can schedule nvidia GPU accelerators, limiting the concurrent jobs to no more than there are accelerators to support, and setting `CUDA_VISIBLE_DEVICES` in the tasks' environments to tell them which nvidia GPU(s) to use. (4163)
* `AWSBatchBatchSystem` can use AWS Batch's GPU resource to provide nvidia GPU accelerators (4163)
* Toil jobs no longer need to re-run after their child/followOn/service jobs in order to delete themselves. (3188)
* Message bus is now thread safe (4276)
* Docker build has been updated with new Aventer Mesos deb URL (fixes 4290)
* `docker` binary in the container has been updated to that included in the Ubuntu repos (fixes 4282)
* Singularity in the appliance has been updated to 3.10 which is >=3.9, for cgroups v2 support.
* Base Ubuntu container image for the appliance has been updated to 22.04, which has a new enough libc for Debian's Singularity 3.10 debs.
* Safer type usage checking for systems without boto3 installed
* Tests are now more runnable post-installation. Temporary paths are not selected based upon the location of the tests themselves. (4287)

_Bug Fixes_
* Only use `/var/run/user` if XDG tells us we have it in our session. Otherwise we will try other places, including `/run/lock/toil`. (4170)
* `toil destroy-cluster`: terminate stopped instances when destroying the cluster (4271)
* fileJobStore: handle arbitrary `os.link` errors to work on some filesystems (2232)

Thank you to our contributors!

5.7.1

Changelog


Highlighted Features Added

AWS Batch Batch System (3956)
AGC Integration (4039) + More AGC integration (4067) + AGC megabranch (4113)
Scale TES to be able to run reasonably-sized workflows on Funnel on Kubernetes with the AWS job store (3927)

CWL

Run CWL conformance tests via WES (4052)
Implement and test CWL loadContents from URLs to fix 4125 (4126)
Add CWL tests under ARM (4038)
Cache results of cwltool version lookup (4141)

Misc

SGE batch system change to support serial jobs. (4022)
Performance testing for Graviton instances (4123)
Stop waiting on hostpath volumes to exist (4146)
Catch and warn about jobs going away too slowly on FileJobStore (4149)
Add documentation for the type-checking hooks (4117)
Pod murder bot (4060)
Contrib hook scripts (4105)
Allow newer google-cloud-storage (4114)
Use environment variable to set parallel partition name (4096)
Register pytest markers (4103)
Mention --export=ALL for SLURM environments (4100) (4102)
Allow persisting workflow state in WES server across container recreation (4082)
Change `toil kill` to use the job store shared file API to find `pig.log` (4075)
Bring back kill loop in the single_machine batch system but with a timeout (4070)
Reorganize Locking (4059)
Add and test preemptability constraints (4044)
Enhanced types (3975)
Use an `init` process that reaps zombies on toil clusters (3974)
Add launch cluster support for ARM (3971)
Feat: square bracket to period separator (4008)
Add AGC health check endpoint (3997)
Tolerate and require typed Werkzeug (4011)
Add more static URLs for Singularity debs (4007)

_Bug Fixes_

Update WES set up docs (4027)
Add real time logs (4031)
Fail fast if Docker builder is missing (4001)
Make Toil version be reported as a string in WES (4013)
Fix assorted typos within assorted comments (4023)
Make file store case insensitive (4153)
Pre-lex commands for qsub (4150)
Update Cactus and exclude broken networkx (4107)
Make `toil kill` work when the leader is on another machine (4084)
Wrong filename in output (4139)
Tolerate a missing VersionID key to fix 4129 (4130)
Only import from typing_extensions on old Python where we install it (4090)
Allow missing username and fix Docker build (4077)
Leave more time for concurrency measurement to fix 4012 (4068)
Stop people asking for ARM Mesos clusters to fix 4057 (4058)

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, Hexotical, gmloose, kannon92, douglowe, gcapes, and pmiddend!

5.6.0

Changelog


Highlighted Features Added
* Integrate ARM Docker builds to make multi-arch images. (3802)
* WES support and server mode. (3779)
* TES batch system prototype. (3821)
* Support for new resource syntax in PBSPro. (3048)
* Toil now looks for lost jobs every minute instead of every hour (3948)

Breaking Changes
* Remove --disableCaching's true/false argument. (3869)

CWL
* CWL helper jobs: better disk & memory requirements. (3834)
* CWL: safer test path generation for post-install testing. (3818)
* More detailed names, which show up in job names sent to BatchSystem schedulers. 3941 (was 3893)
* If you use scatter and collect files,  duplicates are correctly dealt with and renamed. (3968)
* CWL: at the end of a job, ask cwltool to cleanup (3965)

Kubernetes
* Assign Kubernetes jobs an explicit TTL. (3936)
* Wait for node creation. (3934)

Dependencies
* Enable Dependabot updates. (3827)
* Multiple consolidated dependabot updates. (3851)
* Update addict requirement from <2.3,>=2.2.1 to >=2.2.1,<2.5. (3861)
* Bump cwltool to 3.1.20211107152837. (3833 3866 3909)
* Bump cwltest from 2.1.20210626101542 to 2.2.20210901154959. (3848)
* Bump flake8 from 3.8.4 to 4.0.1. (3847)
* Allow more docker-py versions. (3860)
* Remove pyyaml dependency. (3858)

Misc
* Update cleanup script. (3937)
* Remove use of sys.maxsize. (3824)
* Spelling fixes. (3814)
* Add codeql-analysis for Python. (3825)
* Update jobstore function names. (3809)
* Move AMI functions to lib. (3810)
* Add "make pyupgrade" (py36-plus). (3805)
* Type hints.  (3930)
* Coalesce status calls in slurm. (3822)
* Python logging takes format values as *args. (3852)
* Change quick test to a 10 minute timeout. (3843)
* Add make uninstall to makefile. (3883)
* Fix toil kill to find shared pid.log file (with unit test). 3941 (was 3932)
* Update documentation. (3947)
+ Remove remains of Travis. (3976)

_Bug Fixes_
* Stop checkpoints from being reissued multiple times. (3931)
* Don't consult LSF config when explicitly defining memory units. (3820)
* Robustly remove state dirs. (3836)
* Fix exception checking for exit_code. (3830)
* Use exitStatus instead of exitReason for batch exit type comparison. (3839)
* Update cwltest to improve K8 runs. (3935)
* Consolidated CI Fixes. (3887)
* Fix CWL conformance tests. (3891)
* Toil-managed cluster scaling should work again with `--metrics`. (3943)

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, Hexotical, tmooney, nikhil, kannon92, douglowe, mhpopescu, Phhere, and gmloose!

5.5.0

Changelog

CWL
* Add podman support; and other fixes from recent cwltool 3799
* Add streaming feature for cwltoil 3694
* Warn users if a different cwltool version is installed 3686
* Turn on all Kubernetes CWL tests that are expected to work on Singularity 3720
* Fix CWL in toil docs jobstore usage 3728
* DOC: update versions of CWL support
* Allow filestore bypass 3652

Misc
* Numerous Type Hints. 3705 3701 3693 3691 3663 3688 3642 3684 3682 3680 3666 3675
* Single source of truth for job state 3776
* Do not set default for statePollingWait 3774
* Use absolute local paths when exportFile/importFile do not detect a schema 3767
* Multi-zone balancing within regions for AWS autoscaling groups 3746
* Migrate cloud-config to ignition 3488
* 🎡 Wheel Of Issues 🎰 3760
* Add back addBatchSystemFactory function 3754
* Redirect stderr to /dev/null of lsf conf queries 3751
* Set number of cores based on job.cores for OpenMP applications 3739
* Google jobstore batching 3740
* Locations of CLI option docs
* Add AWS provisioner storage system 3727
* Set cls.bucket 3726
* Stream vs dowload jobs 3722
* Update Toil's main python test version to 3.8. 3669
* Move Travis tests to Gitlab 3675

_Bug Fixes_
* Don't leak symlinks 3795
* Prevent exception from being raised when modifying dir permissions for clean up 3778
* Fix scontrol output parsing 3793
* Fix AttributeError 3742
* Workaround for S3 in us-east-1 3710
* Time data format 3708
* Fix leader.py batch system std files prefix glob 3679

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, Hexotical, ionox0, gmloose, juanesarango, mhpopescu, mberacochea, nikhil!

5.4.0

Changelog

CWL
* Fix cwl and wdl dependency bleed and add stand alone tests. 3582
* Use MpiConfig.load() to handle MPI config file 3574
* Add support for MPI with CWL. 3525

Misc
* Numerous Type Hints. 3571 3634 3626 3625 3616 3614 3601 3592 3590 3581
* Configurable Grafana port 3597
* Handle streaming reads from the cache when the data isn't written back yet 3595
* Balance over pools using AWS ASGs 3490
* Change how Kubernetes schedules and scales for hopefully better scale-down behavior 3587
* Add Owner tags to our AWS buckets when testing. 3577
* Allow workdir override 3586
* Additional info for "Permission denied" error 3579
* Consolidate memory functions 3529
* Pen children 3482
* Sniff raid better 3526
* Add a propagation policy to batch delete 3522
* Cleanup script for buckets, sdb domains, instance profiles, and roles. 3373
* Add decorator for flaky tests. 3510

_Bug Fixes_
* Enforce valid uuid from getNodeId() 3611
* Removed extra memory multiplication 3608
* Fix non integers lsf memory requested 3609
* Fixes jobCommand unpack 3610
* Fix CWL test 20 on Kubernetes 3572
* Fix 3579 breaking AWS docs 3596
* Update cactus test to fix broken bucket links. 3594
* Allow cleaning up a job whose overlargeID fell off 3584
* Only deploy a user script if we can deploy user scripts 3518
* Reduce path name lengths. 3438
* Insist on credentials for testing AMI finding 3524

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, thiagogenez, julian-klode, darafferty, nikhil, mhpopescu!

5.3.0

Changelog

CWL
* Run CWL conformance tests on Kubernetes. 3323
* CWL symlinking files into work directory 3445
* set resource reqs for all CWL_INTERNAL_JOBS 3442

WDL
* Add WDL flatten() function + tests. 3485
* Add WDL collect_by_key() function + tests. 3476
* Add WDL keys() function + tests. 3460
* Add WDL as_pairs() function + tests. 3364
* Add as_map() WDL function 3448
* Add basic WDL development support 3434
* Add basic WDL 1.0 support 3421
* Update dictionary structure in AnalyzeWDL. 3416

Misc
* Add extra context for disk usage warning. 3495
* Update developing.rst 3479
* Allow text/encode options for read and write functions in file store 3428
* Type hinting and mypy checking additions 3470  3458 3456
* Quality of life improvements for cactus. 3463
* Add checklists for reviewing and merging PRs 3432
* Ensure that python3 and not python (2) is used 3446
* Support Kubernetes clusters in toil launch-cluster 3357
* Additional check for predecessors. 3417
* Allow setOptions to accept an argparse group. 3426

_Bug Fixes_
* AWS: Handle socket timeout during AWS discovery 3503
* Use back-up flatcar AMI if not found. 3513
* Lsf command parser 3475
* Retry on s3 throttling. 3504
* Stop deleting all ASGs with tags 3474
* Only determine execute permissions on files we copy/move. 3437
* cope with an invalid HTTP_PROXY 3447
* Fix failing Google jobstore test 3420
* Update Google job store 3412

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, Arthur Rand, mberacochea, Jessime, thiagogenez, julian-klode!

5.2.0

Changelog

CWL
* Update to the latest cwltool (3.0.20201121085451 -> 3.0.20201203173111).  3375
* Add better handling for potentially mis-ordered CWL args. 3395
* Confirm CommandInputParameter expression can receive a File object. 3350

WDL
* Make AnalyzeWDL abstract to allow implementation for different WDL versions 3391

Misc
* Clean up memoize.py.  3374
* Refactor bioio library, simplify logging, and small adjustments to utils.  3351
* Use regular division for memory calculation on LSF. 3387
* Remove unused --nodeOptions arg. 3397
* Get boto S3 args for minio from environment variables. (3370)
* Update EC2 lists. 3376
* Add a retry to AWS destroy. 3379
* Regular division for memory calculation in LSF. 3387
* Add option to blank mem allocation for SLURM. 3399
* Preserve file permissions on imported/exported files 3322

_Bug Fixes_
* Preserve file permissions on imported/exported files. 3322
* Make building a flattened list of live jobs non-recursive. 3394
* Move AWS job store uploads/downloads to boto3 (reopened) 3400
* Use a Docker Hub mirror when pulling Docker images. 3411
* Corrected gridengine maxMEM check to use memorystring 3410

Thank you to our contributors: mr-c, adamnovak, w-gao, jonathanxu18, douglowe, stevekm , thiagogenez!

5.1.0

Changelog

WDL
* Add WDL cross function.  3360

_Bug Fixes_
* Better error message for insufficient disk.  3362
* Fix maxMEM comparison in LSF.  3369

Thank you to our contributors: mr-c , adamnovak , w-gao , jonathanxu18 !
Links

Update pytz from 2020.1 to 2024.1.

The bot wasn't able to find a changelog for this release. Got an idea?

Links

Update typing from 3.7.4.3 to 3.10.0.0.

The bot wasn't able to find a changelog for this release. Got an idea?

Links

Update ruamel.yaml from 0.16.5 to 0.18.6.

The bot wasn't able to find a changelog for this release. Got an idea?

Links

Update pip from 20.2.3 to 24.2.

Changelog

24.2

=================

Deprecations and Removals
-------------------------

- Deprecate ``pip install --editable`` falling back to ``setup.py develop``
when using a setuptools version that does not support :pep:`660`
(setuptools v63 and older). (`11457 <https://github.com/pypa/pip/issues/11457>`_)

Features
--------

- Check unsupported packages for the current platform. (`11054 <https://github.com/pypa/pip/issues/11054>`_)
- Use system certificates *and* certifi certificates to verify HTTPS connections on Python 3.10+.
Python 3.9 and earlier only use certifi.

To revert to previous behaviour, pass the flag ``--use-deprecated=legacy-certs``. (`11647 <https://github.com/pypa/pip/issues/11647>`_)
- Improve discovery performance of installed packages when the ``importlib.metadata``
backend is used to load distribution metadata (used by default under Python 3.11+). (`12656 <https://github.com/pypa/pip/issues/12656>`_)
- Improve performance when the same requirement string appears many times during
resolution, by consistently caching the parsed requirement string. (`12663 <https://github.com/pypa/pip/issues/12663>`_)
- Minor performance improvement of finding applicable package candidates by not
repeatedly calculating their versions (`12664 <https://github.com/pypa/pip/issues/12664>`_)
- Disable pip's self version check when invoking a pip subprocess to install
PEP 517 build requirements. (`12683 <https://github.com/pypa/pip/issues/12683>`_)
- Improve dependency resolution performance by caching platform compatibility
tags during wheel cache lookup. (`12712 <https://github.com/pypa/pip/issues/12712>`_)
- ``wheel`` is no longer explicitly listed as a build dependency of ``pip``.
``setuptools`` injects this dependency in the ``get_requires_for_build_wheel()``
hook and no longer needs it on newer versions. (`12728 <https://github.com/pypa/pip/issues/12728>`_)
- Ignore ``--require-virtualenv`` for ``pip check`` and ``pip freeze`` (`12842 <https://github.com/pypa/pip/issues/12842>`_)
- Improve package download and install performance.

Increase chunk sizes when downloading (256 kB, up from 10 kB) and reading files (1 MB, up from 8 kB).
This reduces the frequency of updates to pip's progress bar. (`12810 <https://github.com/pypa/pip/issues/12810>`_)
- Improve pip install performance.

Files are now extracted in 1MB blocks, or in one block matching the file size for
smaller files. A decompressor is no longer instantiated when extracting 0 bytes files,
it is not necessary because there is no data to decompress. (`12803 <https://github.com/pypa/pip/issues/12803>`_)

Bug Fixes
---------

- Set ``no_color`` to global ``rich.Console`` instance. (`11045 <https://github.com/pypa/pip/issues/11045>`_)
- Fix resolution to respect ``--python-version`` when checking ``Requires-Python``. (`12216 <https://github.com/pypa/pip/issues/12216>`_)
- Perform hash comparisons in a case-insensitive manner. (`12680 <https://github.com/pypa/pip/issues/12680>`_)
- Avoid ``dlopen`` failure for glibc detection in musl builds (`12716 <https://github.com/pypa/pip/issues/12716>`_)
- Avoid keyring logging crashes when pip is run in verbose mode. (`12751 <https://github.com/pypa/pip/issues/12751>`_)
- Fix finding hardlink targets in tar files with an ignored top-level directory. (`12781 <https://github.com/pypa/pip/issues/12781>`_)
- Improve pip install performance by only creating required parent
directories once, instead of before extracting every file in the wheel. (`12782 <https://github.com/pypa/pip/issues/12782>`_)
- Improve pip install performance by calculating installed packages printout
in linear time instead of quadratic time. (`12791 <https://github.com/pypa/pip/issues/12791>`_)

Vendored Libraries
------------------

- Remove vendored tenacity.
- Update the preload list for the ``DEBUNDLED`` case, to replace ``pep517`` that has been renamed to ``pyproject_hooks``.
- Use tomllib from the stdlib if available, rather than tomli
- Upgrade certifi to 2024.7.4
- Upgrade platformdirs to 4.2.2
- Upgrade pygments to 2.18.0
- Upgrade setuptools to 70.3.0
- Upgrade typing_extensions to 4.12.2

Improved Documentation
----------------------

- Correct ``—-ignore-conflicts`` (including an em dash) to ``--ignore-conflicts``. (`12851 <https://github.com/pypa/pip/issues/12851>`_)

24.1.2

===================

Bug Fixes
---------

- Fix finding hardlink targets in tar files with an ignored top-level directory. (`12781 <https://github.com/pypa/pip/issues/12781>`_)

24.1.1

===================

Bug Fixes
---------

- Actually use system trust stores when the truststore feature is enabled.

Vendored Libraries
------------------

- Upgrade requests to 2.32.3

24.1

=================

Vendored Libraries
------------------

- Upgrade truststore to 0.9.1.

24.1b2

===================

Features
--------

- Report informative messages about invalid requirements. (`12713 <https://github.com/pypa/pip/issues/12713>`_)

Bug Fixes
---------

- Eagerly import the self version check logic to avoid crashes while upgrading or downgrading pip at the same time. (`12675 <https://github.com/pypa/pip/issues/12675>`_)
- Accommodate for mismatches between different sources of truth for extra names, for packages generated by ``setuptools``. (`12688 <https://github.com/pypa/pip/issues/12688>`_)
- Accommodate for development versions of CPython ending in ``+`` in the version string. (`12691 <https://github.com/pypa/pip/issues/12691>`_)

Vendored Libraries
------------------

- Upgrade packaging to 24.1
- Upgrade requests to 2.32.0
- Remove vendored colorama
- Remove vendored six
- Remove vendored webencodings
- Remove vendored charset_normalizer

``requests`` provides optional character detection support on some APIs when processing ambiguous bytes. This isn't relevant for pip to function and we're able to remove it due to recent upstream changes.

24.1b1

===================

Deprecations and Removals
-------------------------

- Drop support for EOL Python 3.7. (`11934 <https://github.com/pypa/pip/issues/11934>`_)
- Remove support for legacy versions and dependency specifiers.

Packages with non standard-compliant versions or dependency specifiers are now ignored by the resolver.
Already installed packages with non standard-compliant versions or dependency specifiers
must be uninstalled before upgrading them. (`12063 <https://github.com/pypa/pip/issues/12063>`_)

Features
--------

- Improve performance of resolution of large dependency trees, with more caching. (`12453 <https://github.com/pypa/pip/issues/12453>`_)
- Further improve resolution performance of large dependency trees, by caching hash calculations. (`12657 <https://github.com/pypa/pip/issues/12657>`_)
- Reduce startup time of commands (e.g. show, freeze) that do not access the network by 15-30%. (`4768 <https://github.com/pypa/pip/issues/4768>`_)
- Reword and improve presentation of uninstallation errors. (`10421 <https://github.com/pypa/pip/issues/10421>`_)
- Add a 'raw' progress_bar type for simple and parsable download progress reports (`11508 <https://github.com/pypa/pip/issues/11508>`_)
- ``pip list`` no longer performs the pip version check unless ``--outdated`` or ``--uptodate`` is given. (`11677 <https://github.com/pypa/pip/issues/11677>`_)
- Use the ``data_filter`` when extracting tarballs, if it's available. (`12111 <https://github.com/pypa/pip/issues/12111>`_)
- Display the Project-URL value under key "Home-page" in ``pip show`` when the Home-Page metadata field is not set.

The Project-URL key detection is case-insensitive, and ignores any dashes and underscores. (`11221 <https://github.com/pypa/pip/issues/11221>`_)

Bug Fixes
---------

- Ensure ``-vv`` gets passed to any ``pip install`` build environment subprocesses. (`12577 <https://github.com/pypa/pip/issues/12577>`_)
- Deduplicate entries in the ``Requires`` field of ``pip show``. (`12165 <https://github.com/pypa/pip/issues/12165>`_)
- Fix error on checkout for subversion and bazaar with verbose mode on. (`11050 <https://github.com/pypa/pip/issues/11050>`_)
- Fix exception with completions when COMP_CWORD is not set (`12401 <https://github.com/pypa/pip/issues/12401>`_)
- Fix intermittent "cannot locate t64.exe" errors when upgrading pip. (`12666 <https://github.com/pypa/pip/issues/12666>`_)
- Remove duplication in invalid wheel error message (`12579 <https://github.com/pypa/pip/issues/12579>`_)
- Remove the incorrect pip3.x console entrypoint from the pip wheel. This console
script continues to be generated by pip when it installs itself. (`12536 <https://github.com/pypa/pip/issues/12536>`_)
- Gracefully skip VCS detection in pip freeze when PATH points to a non-directory path. (`12567 <https://github.com/pypa/pip/issues/12567>`_)
- Make the ``--proxy`` parameter take precedence over environment variables. (`10685 <https://github.com/pypa/pip/issues/10685>`_)

Vendored Libraries
------------------

- Add charset-normalizer 3.3.2
- Remove chardet
- Remove pyparsing
- Upgrade CacheControl to 0.14.0
- Upgrade certifi to 2024.2.2
- Upgrade distro to 1.9.0
- Upgrade idna to 3.7
- Upgrade msgpack to 1.0.8
- Upgrade packaging to 24.0
- Upgrade platformdirs to 4.2.1
- Upgrade pygments to 2.17.2
- Upgrade rich to 13.7.1
- Upgrade setuptools to 69.5.1
- Upgrade tenacity to 8.2.3
- Upgrade typing_extensions to 4.11.0
- Upgrade urllib3 to 1.26.18

Improved Documentation
----------------------

- Document UX research done on pip. (`10745 <https://github.com/pypa/pip/issues/10745>`_)
- Fix the direct usage of zipapp showing up as ``python -m pip.pyz`` rather than ``./pip.pyz`` / ``.\pip.pyz`` (`12043 <https://github.com/pypa/pip/issues/12043>`_)
- Add a warning explaining that the snippet in "Fallback behavior" is not a valid
``pyproject.toml`` snippet for projects, and link to setuptools documentation
instead. (`12122 <https://github.com/pypa/pip/issues/12122>`_)
- The Python Support Policy has been updated. (`12529 <https://github.com/pypa/pip/issues/12529>`_)
- Document the environment variables that correspond with CLI options. (`12576 <https://github.com/pypa/pip/issues/12576>`_)
- Update architecture documentation for command line interface. (`6831 <https://github.com/pypa/pip/issues/6831>`_)

Process
-------

- Remove ``setup.py`` since all the pip project metadata is now declared in
``pyproject.toml``.
- Move remaining pip development tools configurations to ``pyproject.toml``.

24.0

=================

Features
--------

- Retry on HTTP status code 502 (`11843 <https://github.com/pypa/pip/issues/11843>`_)
- Automatically use the setuptools PEP 517 build backend when ``--config-settings`` is
used for projects without ``pyproject.toml``. (`11915 <https://github.com/pypa/pip/issues/11915>`_)
- Make pip freeze and pip uninstall of legacy editable installs of packages whose name
contains ``_`` compatible with ``setuptools>=69.0.3``. (`12477 <https://github.com/pypa/pip/issues/12477>`_)
- Support per requirement ``--config-settings`` for editable installs. (`12480 <https://github.com/pypa/pip/issues/12480>`_)

Bug Fixes
---------

- Optimized usage of ``--find-links=<path-to-dir>``, by only scanning the relevant directory once, only considering file names that are valid wheel or sdist names, and only considering files in the directory that are related to the install. (`12327 <https://github.com/pypa/pip/issues/12327>`_)
- Removed ``wheel`` from the ``[build-system].requires`` list fallback
that is used when ``pyproject.toml`` is absent. (`12449 <https://github.com/pypa/pip/issues/12449>`_)

Vendored Libraries
------------------

- Upgrade distlib to 0.3.8

Improved Documentation
----------------------

- Fix explanation of how PIP_CONFIG_FILE works (`11815 <https://github.com/pypa/pip/issues/11815>`_)
- Fix outdated pip install argument description in documentation. (`12417 <https://github.com/pypa/pip/issues/12417>`_)
- Replace some links to PEPs with links to the canonical specifications on the :doc:`pypug:index` (`12434 <https://github.com/pypa/pip/issues/12434>`_)
- Updated the ``pyproject.toml`` document to stop suggesting
to depend on ``wheel`` as a build dependency directly. (`12449 <https://github.com/pypa/pip/issues/12449>`_)
- Update supported interpreters in development docs (`12475 <https://github.com/pypa/pip/issues/12475>`_)

Process
-------

- Most project metadata is now defined statically via pip's ``pyproject.toml`` file.

23.3.2

===================

Bug Fixes
---------

- Fix a bug in extras handling for link requirements (`12372 <https://github.com/pypa/pip/issues/12372>`_)
- Fix mercurial revision "parse error": use ``--rev={ref}`` instead of ``-r={ref}`` (`12373 <https://github.com/pypa/pip/issues/12373>`_)

23.3.1

===================

Bug Fixes
---------

- Handle a timezone indicator of Z when parsing dates in the self check. (`12338 <https://github.com/pypa/pip/issues/12338>`_)
- Fix bug where installing the same package at the same time with multiple pip processes could fail. (`12361 <https://github.com/pypa/pip/issues/12361>`_)

23.3

=================

Process
-------

- Added reference to `vulnerability reporting guidelines <https://www.python.org/dev/security/>`_ to pip's security policy.

Deprecations and Removals
-------------------------

- Drop a fallback to using SecureTransport on macOS. It was useful when pip detected OpenSSL older than 1.0.1, but the current pip does not support any Python version supporting such old OpenSSL versions. (`12175 <https://github.com/pypa/pip/issues/12175>`_)

Features
--------

- Improve extras resolution for multiple constraints on same base package. (`11924 <https://github.com/pypa/pip/issues/11924>`_)
- Improve use of datastructures to make candidate selection 1.6x faster. (`12204 <https://github.com/pypa/pip/issues/12204>`_)
- Allow ``pip install --dry-run`` to use platform and ABI overriding options. (`12215 <https://github.com/pypa/pip/issues/12215>`_)
- Add ``is_yanked`` boolean entry to the installation report (``--report``) to indicate whether the requirement was yanked from the index, but was still selected by pip conform to :pep:`592`. (`12224 <https://github.com/pypa/pip/issues/12224>`_)

Bug Fixes
---------

- Ignore errors in temporary directory cleanup (show a warning instead). (`11394 <https://github.com/pypa/pip/issues/11394>`_)
- Normalize extras according to :pep:`685` from package metadata in the resolver
for comparison. This ensures extras are correctly compared and merged as long
as the package providing the extra(s) is built with values normalized according
to the standard. Note, however, that this *does not* solve cases where the
package itself contains unnormalized extra values in the metadata. (`11649 <https://github.com/pypa/pip/issues/11649>`_)
- Prevent downloading sdists twice when :pep:`658` metadata is present. (`11847 <https://github.com/pypa/pip/issues/11847>`_)
- Include all requested extras in the install report (``--report``). (`11924 <https://github.com/pypa/pip/issues/11924>`_)
- Removed uses of ``datetime.datetime.utcnow`` from non-vendored code. (`12005 <https://github.com/pypa/pip/issues/12005>`_)
- Consistently report whether a dependency comes from an extra. (`12095 <https://github.com/pypa/pip/issues/12095>`_)
- Fix completion script for zsh (`12166 <https://github.com/pypa/pip/issues/12166>`_)
- Fix improper handling of the new onexc argument of ``shutil.rmtree()`` in Python 3.12. (`12187 <https://github.com/pypa/pip/issues/12187>`_)
- Filter out yanked links from the available versions error message: "(from versions: 1.0, 2.0, 3.0)" will not contain yanked versions conform PEP 592. The yanked versions (if any) will be mentioned in a separate error message. (`12225 <https://github.com/pypa/pip/issues/12225>`_)
- Fix crash when the git version number contains something else than digits and dots. (`12280 <https://github.com/pypa/pip/issues/12280>`_)
- Use ``-r=...`` instead of ``-r ...`` to specify references with Mercurial. (`12306 <https://github.com/pypa/pip/issues/12306>`_)
- Redact password from URLs in some additional places. (`12350 <https://github.com/pypa/pip/issues/12350>`_)
- pip uses less memory when caching large packages. As a result, there is a new on-disk cache format stored in a new directory ($PIP_CACHE_DIR/http-v2). (`2984 <https://github.com/pypa/pip/issues/2984>`_)

Vendored Libraries
------------------

- Upgrade certifi to 2023.7.22
- Add truststore 0.8.0
- Upgrade urllib3 to 1.26.17

Improved Documentation
----------------------

- Document that ``pip search`` support has been removed from PyPI (`12059 <https://github.com/pypa/pip/issues/12059>`_)
- Clarify --prefer-binary in CLI and docs (`12122 <https://github.com/pypa/pip/issues/12122>`_)
- Document that using OS-provided Python can cause pip's test suite to report false failures. (`12334 <https://github.com/pypa/pip/issues/12334>`_)

23.2.1

===================

Bug Fixes
---------

- Disable :pep:`658` metadata fetching with the legacy resolver. (`12156 <https://github.com/pypa/pip/issues/12156>`_)

23.2

=================

Process
-------

- Deprecate support for eggs for Python 3.11 or later, when the new ``importlib.metadata`` backend is used to load distribution metadata. This only affects the egg *distribution format* (with the ``.egg`` extension); distributions using the ``.egg-info`` *metadata format* (but are not actually eggs) are not affected. For more information about eggs, see `relevant section in the setuptools documentation <https://setuptools.pypa.io/en/stable/deprecated/python_eggs.html>`__.

Deprecations and Removals
-------------------------

- Deprecate legacy version and version specifiers that don't conform to the
:ref:`specification <pypug:version-specifiers>`.
(`12063 <https://github.com/pypa/pip/issues/12063>`_)
- ``freeze`` no longer excludes the ``setuptools``, ``distribute``, and ``wheel``
from the output when running on Python 3.12 or later, where they are not
included in a virtual environment by default. Use ``--exclude`` if you wish to
exclude any of these packages. (`4256 <https://github.com/pypa/pip/issues/4256>`_)

Features
--------

- make rejection messages slightly different between 1 and 8, so the user can make the difference. (`12040 <https://github.com/pypa/pip/issues/12040>`_)

Bug Fixes
---------

- Fix ``pip completion --zsh``. (`11417 <https://github.com/pypa/pip/issues/11417>`_)
- Prevent downloading files twice when :pep:`658` metadata is present (`11847 <https://github.com/pypa/pip/issues/11847>`_)
- Add permission check before configuration (`11920 <https://github.com/pypa/pip/issues/11920>`_)
- Fix deprecation warnings in Python 3.12 for usage of shutil.rmtree (`11957 <https://github.com/pypa/pip/issues/11957>`_)
- Ignore invalid or unreadable ``origin.json`` files in the cache of locally built wheels. (`11985 <https://github.com/pypa/pip/issues/11985>`_)
- Fix installation of packages with :pep:`658` metadata using non-canonicalized names (`12038 <https://github.com/pypa/pip/issues/12038>`_)
- Correctly parse ``dist-info-metadata`` values from JSON-format index data. (`12042 <https://github.com/pypa/pip/issues/12042>`_)
- Fail with an error if the ``--python`` option is specified after the subcommand name. (`12067 <https://github.com/pypa/pip/issues/12067>`_)
- Fix slowness when using ``importlib.metadata`` (the default way for pip to read metadata in Python 3.11+) and there is a large overlap between already installed and to-be-installed packages. (`12079 <https://github.com/pypa/pip/issues/12079>`_)
- Pass the ``-r`` flag to mercurial to be explicit that a revision is passed and protect
against ``hg`` options injection as part of VCS URLs. Users that do not have control on
VCS URLs passed to pip are advised to upgrade. (`12119 <https://github.com/pypa/pip/issues/12119>`_)

Vendored Libraries
------------------

- Upgrade certifi to 2023.5.7
- Upgrade platformdirs to 3.8.1
- Upgrade pygments to 2.15.1
- Upgrade pyparsing to 3.1.0
- Upgrade Requests to 2.31.0
- Upgrade rich to 13.4.2
- Upgrade setuptools to 68.0.0
- Updated typing_extensions to 4.6.0
- Upgrade typing_extensions to 4.7.1
- Upgrade urllib3 to 1.26.16

23.1.2

===================

Vendored Libraries
------------------

- Upgrade setuptools to 67.7.2

23.1.1

===================

Bug Fixes
---------

- Revert `11487 <https://github.com/pypa/pip/pull/11487>`_, as it causes issues with virtualenvs created by the Windows Store distribution of Python. (`#11987 <https://github.com/pypa/pip/issues/11987>`_)

Vendored Libraries
------------------

- Revert pkg_resources (via setuptools) back to 65.6.3

Improved Documentation
----------------------

- Update documentation to reflect the new behavior of using the cache of locally
built wheels in hash-checking mode. (`11967 <https://github.com/pypa/pip/issues/11967>`_)

23.1

=================

Deprecations and Removals
-------------------------

- Remove support for the deprecated ``--install-options``. (`11358 <https://github.com/pypa/pip/issues/11358>`_)
- ``--no-binary`` does not imply ``setup.py install`` anymore. Instead a wheel will be
built locally and installed. (`11451 <https://github.com/pypa/pip/issues/11451>`_)
- ``--no-binary`` does not disable the cache of locally built wheels anymore. It only
means "don't download wheels". (`11453 <https://github.com/pypa/pip/issues/11453>`_)
- Deprecate ``--build-option`` and ``--global-option``. Users are invited to switch to
``--config-settings``. (`11859 <https://github.com/pypa/pip/issues/11859>`_)
- Using ``--config-settings`` with projects that don't have a ``pyproject.toml`` now prints
a deprecation warning. In the future the presence of config settings will automatically
enable the default build backend for legacy projects and pass the settings to it. (`11915 <https://github.com/pypa/pip/issues/11915>`_)
- Remove ``setup.py install`` fallback when building a wheel failed for projects without
``pyproject.toml``. (`8368 <https://github.com/pypa/pip/issues/8368>`_)
- When the ``wheel`` package is not installed, pip now uses the default build backend
instead of ``setup.py install`` and ``setup.py develop`` for project without
``pyproject.toml``. (`8559 <https://github.com/pypa/pip/issues/8559>`_)

Features
--------

- Specify egg-link location in assertion message when it does not match installed location to provide better error message for debugging. (`10476 <https://github.com/pypa/pip/issues/10476>`_)
- Present conflict information during installation after each choice that is rejected (pass ``-vv`` to ``pip install`` to show it) (`10937 <https://github.com/pypa/pip/issues/10937>`_)
- Display dependency chain on each Collecting/Processing log line. (`11169 <https://github.com/pypa/pip/issues/11169>`_)
- Support a per-requirement ``--config-settings`` option in requirements files. (`11325 <https://github.com/pypa/pip/issues/11325>`_)
- The ``--config-settings``/``-C`` option now supports using the same key multiple
times. When the same key is specified multiple times, all values are passed to
the build backend as a list, as opposed to the previous behavior, where pip would
only pass the last value if the same key was used multiple times. (`11681 <https://github.com/pypa/pip/issues/11681>`_)
- Add ``-C`` as a short version of the ``--config-settings`` option. (`11786 <https://github.com/pypa/pip/issues/11786>`_)
- Reduce the number of resolver rounds, since backjumping makes the resolver more efficient in finding solutions. This also makes pathological cases fail quicker. (`11908 <https://github.com/pypa/pip/issues/11908>`_)
- Warn if ``--hash`` is used on a line without requirement in a requirements file. (`11935 <https://github.com/pypa/pip/issues/11935>`_)
- Stop propagating CLI ``--config-settings`` to the build dependencies. They already did
not propagate to requirements provided in requirement files. To pass the same config
settings to several requirements, users should provide the requirements as CLI
arguments. (`11941 <https://github.com/pypa/pip/issues/11941>`_)
- Support wheel cache when using ``--require-hashes``. (`5037 <https://github.com/pypa/pip/issues/5037>`_)
- Add ``--keyring-provider`` flag. See the Authentication page in the documentation for more info. (`8719 <https://github.com/pypa/pip/issues/8719>`_)
- In the case of virtual environments, configuration files are now also included from the base installation. (`9752 <https://github.com/pypa/pip/issues/9752>`_)

Bug Fixes
---------

- Fix grammar by changing "A new release of pip available:" to "A new release of pip is available:" in the notice used for indicating that. (`11529 <https://github.com/pypa/pip/issues/11529>`_)
- Normalize paths before checking if installed scripts are on PATH. (`11719 <https://github.com/pypa/pip/issues/11719>`_)
- Correct the way to decide if keyring is available. (`11774 <https://github.com/pypa/pip/issues/11774>`_)
- More consistent resolution backtracking by removing legacy hack related to setuptools resolution (`11837 <https://github.com/pypa/pip/issues/11837>`_)
- Include ``AUTHORS.txt`` in pip's wheels. (`11882 <https://github.com/pypa/pip/issues/11882>`_)
- The ``uninstall`` and ``install --force-reinstall`` commands no longer call
``normalize_path()`` repeatedly on the same paths. Instead, these results are
cached for the duration of an uninstall operation, resulting in improved
performance, particularly on Windows. (`11889 <https://github.com/pypa/pip/issues/11889>`_)
- Fix and improve the parsing of hashes embedded in URL fragments. (`11936 <https://github.com/pypa/pip/issues/11936>`_)
- When package A depends on package B provided as a direct URL dependency including a hash
embedded in the link, the ``--require-hashes`` option did not warn when user supplied hashes
were missing for package B. (`11938 <https://github.com/pypa/pip/issues/11938>`_)
- Correctly report ``requested_extras`` in the installation report when extras are
specified for a local directory installation. (`11946 <https://github.com/pypa/pip/issues/11946>`_)
- When installing an archive from a direct URL or local file, populate
``download_info.info.hashes`` in the installation report, in addition to the legacy
``download_info.info.hash`` key. (`11948 <https://github.com/pypa/pip/issues/11948>`_)

Vendored Libraries
------------------

- Upgrade msgpack to 1.0.5
- Patch pkg_resources to remove dependency on ``jaraco.text``.
- Upgrade platformdirs to 3.2.0
- Upgrade pygments to 2.14.0
- Upgrade resolvelib to 1.0.1
- Upgrade rich to 13.3.3
- Upgrade setuptools to 67.6.1
- Upgrade tenacity to 8.2.2
- Upgrade typing_extensions to 4.5.0
- Upgrade urllib3 to 1.26.15

Improved Documentation
----------------------

- Cross-reference the ``--python`` flag from the ``--prefix`` flag,
and mention limitations of ``--prefix`` regarding script installation. (`11775 <https://github.com/pypa/pip/issues/11775>`_)
- Add SECURITY.md to make the policy official. (`11809 <https://github.com/pypa/pip/issues/11809>`_)
- Add username to Git over SSH example. (`11838 <https://github.com/pypa/pip/issues/11838>`_)
- Quote extras in the pip install docs to guard shells with default glob
qualifiers, like zsh. (`11842 <https://github.com/pypa/pip/issues/11842>`_)
- Make it clear that requirements/constraints file can be a URL (`11954 <https://github.com/pypa/pip/issues/11954>`_)

23.0.1

===================

Features
--------

- Ignore PIP_REQUIRE_VIRTUALENV for ``pip index`` (`11671 <https://github.com/pypa/pip/issues/11671>`_)
- Implement ``--break-system-packages`` to permit installing packages into
``EXTERNALLY-MANAGED`` Python installations. (`11780 <https://github.com/pypa/pip/issues/11780>`_)

Bug Fixes
---------

- Improve handling of isolated build environments on platforms that
customize the Python's installation schemes, such as Debian and
Homebrew. (`11740 <https://github.com/pypa/pip/issues/11740>`_)
- Do not crash in presence of misformatted hash field in ``direct_url.json``. (`11773 <https://github.com/pypa/pip/issues/11773>`_)

23.0

=================

Features
--------

- Change the hashes in the installation report to be a mapping. Emit the
``archive_info.hashes`` dictionary in ``direct_url.json``. (`11312 <https://github.com/pypa/pip/issues/11312>`_)
- Implement logic to read the ``EXTERNALLY-MANAGED`` file as specified in :pep:`668`.
This allows a downstream Python distributor to prevent users from using pip to
modify the externally managed environment. (`11381 <https://github.com/pypa/pip/issues/11381>`_)
- Enable the use of ``keyring`` found on ``PATH``. This allows ``keyring``
installed using ``pipx`` to be used by ``pip``. (`11589 <https://github.com/pypa/pip/issues/11589>`_)
- The inspect and installation report formats are now declared stable, and their version
has been bumped from ``0`` to ``1``. (`11757 <https://github.com/pypa/pip/issues/11757>`_)

Bug Fixes
---------

- Wheel cache behavior is restored to match previous versions, allowing the
cache to find existing entries. (`11527 <https://github.com/pypa/pip/issues/11527>`_)
- Use the "venv" scheme if available to obtain prefixed lib paths. (`11598 <https://github.com/pypa/pip/issues/11598>`_)
- Deprecated a historical ambiguity in how ``egg`` fragments in URL-style
requirements are formatted and handled. ``egg`` fragments that do not look
like :pep:`508` names now produce a deprecation warning. (`11617 <https://github.com/pypa/pip/issues/11617>`_)
- Fix scripts path in isolated build environment on Debian. (`11623 <https://github.com/pypa/pip/issues/11623>`_)
- Make ``pip show`` show the editable location if package is editable (`11638 <https://github.com/pypa/pip/issues/11638>`_)
- Stop checking that ``wheel`` is present when ``build-system.requires``
is provided without ``build-system.build-backend`` as ``setuptools``
(which we still check for) will inject it anyway. (`11673 <https://github.com/pypa/pip/issues/11673>`_)
- Fix an issue when an already existing in-memory distribution would cause
exceptions in ``pip install`` (`11704 <https://github.com/pypa/pip/issues/11704>`_)

Vendored Libraries
------------------

- Upgrade certifi to 2022.12.7
- Upgrade chardet to 5.1.0
- Upgrade colorama to 0.4.6
- Upgrade distro to 1.8.0
- Remove pep517 from vendored packages
- Upgrade platformdirs to 2.6.2
- Add pyproject-hooks 1.0.0
- Upgrade requests to 2.28.2
- Upgrade rich to 12.6.0
- Upgrade urllib3 to 1.26.14

Improved Documentation
----------------------

- Fixed the description of the option "--install-options" in the documentation (`10265 <https://github.com/pypa/pip/issues/10265>`_)
- Remove mention that editable installs are necessary for pip freeze to report the VCS
URL. (`11675 <https://github.com/pypa/pip/issues/11675>`_)
- Clarify that the egg URL fragment is only necessary for editable VCS installs, and
otherwise not necessary anymore. (`11676 <https://github.com/pypa/pip/issues/11676>`_)

22.3.1

===================

Bug Fixes
---------

- Fix entry point generation of ``pip.X``, ``pipX.Y``, and ``easy_install-X.Y``
to correctly account for multi-digit Python version segments (e.g. the "11"
part of 3.11). (`11547 <https://github.com/pypa/pip/issues/11547>`_)

22.3

=================

Deprecations and Removals
-------------------------

- Deprecate ``--install-options`` which forces pip to use the deprecated ``install``
command of ``setuptools``. (`11358 <https://github.com/pypa/pip/issues/11358>`_)
- Deprecate installation with 'setup.py install' when no-binary is enabled for
source distributions without 'pyproject.toml'. (`11452 <https://github.com/pypa/pip/issues/11452>`_)
- Deprecate --no-binary`` disabling the wheel cache. (`11454 <https://github.com/pypa/pip/issues/11454>`_)
- Remove ``--use-feature=2020-resolver`` opt-in flag. This was supposed to be removed in 21.0, but missed during that release cycle. (`11493 <https://github.com/pypa/pip/issues/11493>`_)
- Deprecate installation with 'setup.py install' when the 'wheel' package is absent for
source distributions without 'pyproject.toml'. (`8559 <https://github.com/pypa/pip/issues/8559>`_)
- Remove the ability to use ``pip list --outdated`` in combination with ``--format=freeze``. (`9789 <https://github.com/pypa/pip/issues/9789>`_)

Features
--------

- Use ``shell=True`` for opening the editor with ``pip config edit``. (`10716 <https://github.com/pypa/pip/issues/10716>`_)
- Use the ``data-dist-info-metadata`` attribute from :pep:`658` to resolve distribution metadata without downloading the dist yet. (`11111 <https://github.com/pypa/pip/issues/11111>`_)
- Add an option to run the test suite with pip built as a zipapp. (`11250 <https://github.com/pypa/pip/issues/11250>`_)
- Add a ``--python`` option to allow pip to manage Python environments other
than the one pip is installed in. (`11320 <https://github.com/pypa/pip/issues/11320>`_)
- Document the new (experimental) zipapp distribution of pip. (`11459 <https://github.com/pypa/pip/issues/11459>`_)
- Use the much faster 'bzr co --lightweight' to obtain a copy of a Bazaar tree. (`5444 <https://github.com/pypa/pip/issues/5444>`_)

Bug Fixes
---------

- Fix ``--no-index`` when ``--index-url`` or ``--extra-index-url`` is specified
inside a requirements file. (`11276 <https://github.com/pypa/pip/issues/11276>`_)
- Ensure that the candidate ``pip`` executable exists, when checking for a new version of pip. (`11309 <https://github.com/pypa/pip/issues/11309>`_)
- Ignore distributions with invalid ``Name`` in metadata instead of crashi

@pyup-bot pyup-bot mentioned this pull request Aug 1, 2024
Closed
@pyup-bot
Copy link
Collaborator Author

pyup-bot commented Sep 1, 2024

Closing this in favor of #213

@pyup-bot pyup-bot closed this Sep 1, 2024
@rhshah rhshah deleted the pyup-scheduled-update-2024-08-01 branch September 1, 2024 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@pyup-bot