gh-126180: Remove getopt and optparse deprecation notices #126227

ncoghlan · 2024-10-31T05:35:32Z

getopt and optparse are in no way at risk of being removed,
as they have genuinely valid use cases. Those use cases
(mimicing the C getopt API, implementing third party
command line argument processing libraries) are just
far less common than the ones that argparse addresses.

Relevant issue is #126180.

Issue: Undeprecate getopt and optparse #126180

📚 Documentation preview 📚: https://cpython-previews--126227.org.readthedocs.build/

Co-authored-by: Serhiy Storchaka <[email protected]>

ncoghlan · 2024-10-31T08:11:29Z

Doc/library/optparse.rst

-more declarative style of command-line parsing: you create an instance of
-:class:`OptionParser`, populate it with options, and parse the command
-line. :mod:`optparse` allows users to specify options in the conventional
+command-line options than the minimalist :mod:`getopt` module.


Replacing old with minimalist is the only actual wording change here (the rest is just reflowing text)

serhiy-storchaka

You are focused on one particular peculiarity in argparse, but there are many others. The difference is fundamental in the way how optional and positional arguments are parsed.

serhiy-storchaka · 2024-10-31T08:53:56Z

Doc/howto/argparse.rst

@@ -15,7 +15,7 @@ recommended command-line parsing module in the Python standard library.

   There are two other modules that fulfill the same task, namely
   :mod:`getopt` (an equivalent for ``getopt()`` from the C
-   language) and the deprecated :mod:`optparse`.
+   language) and the lower level :mod:`optparse` module.


This may create an impression that optparse is lower level than getopt.

I'll reorder it to make it clear the "lower level" comparison is relative to argparse

serhiy-storchaka · 2024-10-31T08:57:44Z

Doc/library/argparse.rst

+   conventions around the handling of option parameter values that start
+   with ``-`` is desired.


Not only this. There are other peculiarities that make implementation of compatible interface with argparse impossible (unlike to getopt and optparse).

OK, I had gotten the impression from the Discourse thread that this was the primary quirk, but I can reword these caveats to be less specific to that one issue.

serhiy-storchaka · 2024-10-31T09:00:55Z

Doc/library/argparse.rst

@@ -11,6 +11,18 @@

 **Source code:** :source:`Lib/argparse.py`

+.. note::
+
+   While :mod:`argparse` is the recommended standard library module for


I would not recommend it for now. You can use it on your own risk. optparse may need more work, but it works as expected. I would recommend it for beginners, which cannot distinguish their own errors from peculiarities of the library.

For this PR, I'd like to stick with the status quo as far as recommendations go (that is, only taking the step back from "argparse is the only non-deprecated argument processing option in the standard library").

Taking the extra step to saying "argparse is not recommended over optparse anymore, even for end user applications" would then be a separate follow up question.

Indeed. Keep the argparse recommendation. We're not going to dis-recommend the most widely used Python argument parsing library in the world (that causes churn and angst from users). Nor are we going to be adding yet another one to the stdlib in the future. argparse supports most users quite well, "warts" and all, with plenty of popular third party options for yet other behaviors beyond our stdlib options.

serhiy-storchaka · 2024-10-31T09:03:08Z

Doc/library/cmdlinelibs.rst

+   argparse.rst
+   optparse.rst


I would place optparse above argparse.

If anything, this list sure be in purely alphabetical order. passive aggressive opinions via list ordering aren't helpful. 😄

Doc/library/getopt.rst

serhiy-storchaka · 2024-10-31T09:07:52Z

Doc/library/getopt.rst

+       opts, args = parser.parse_args()
+       process(args, output=opts.output, verbose=opts.verbose)
+
+An equivalent command line interface for this simple case can also be produced


Even in this simple example this is not an exact equivalent. For example, -o -v and -o -- do not work, -o=foo works differently, you cannot interleave options and positional arguments. I said this is "a roughly equivalent".

I think if we elaborate on the discrepancies here, it may help lay the foundation for weakening the current recommendation to use argparse in a future PR. I'll incorporate more of your notes here tomorrow.

serhiy-storchaka · 2024-10-31T09:09:46Z

Doc/library/getopt.rst

       # ... do something with args.output ...
       # ... do something with args.verbose ..


I added this to show the difference in the API between three modules

parser.add_argument('rest', nargs='*') args = parser.parse_args() process(args.rest, output=args.output, verbose=args.verbose)

serhiy-storchaka · 2024-10-31T09:15:57Z

Doc/library/getopt.rst

+``optparse`` versions due to the way ``argparse`` handles parameter
+values that start with ``-``.


The difference is not only in this.

ncoghlan · 2024-10-31T10:09:02Z

@serhiy-storchaka I think several of your suggestions are good improvements. I should have time to incorporate them tomorrow (Nov 1 AEST)

LilyFoote · 2024-10-31T12:43:02Z

Doc/library/optparse.rst

+.. seealso::
+
+    The :pypi:`"click" package <click>` is an ``optparse`` based third party
+    argument processing library which allows command line applications to be
+    developed as a set of appropriately decorated command implementation
+    functions.
+
+.. seealso::
+
+    The :pypi:`"Typer" package <click>` is a ``click`` based third party
+    argument processing library which allows the use of annotated Python
+    type hints to define an application's command line interface.


It might make sense to duplicate these seealso notes in Doc/library/argparse.rst and Doc/howto/argparse.rst.

I'm wary of the potential for confusion for folks that are running through "My first command line application" style tutorials, so I'd like to avoid having too much "Should you be using this?" text at the start of the argparse docs.

Having an off-ramp from argparse to optparse and then from there to PyPI alternatives seems like it could be a reasonable middle ground.

ncoghlan · 2024-11-01T09:57:10Z

@serhiy-storchaka I won't get back to this today, and this weekend isn't looking good either. Early next week will probably be the next update.

…se-deprecations

ncoghlan · 2024-11-02T12:55:33Z

@serhiy-storchaka I ended up procrastinating on my assignment by working on this (amongst other things), so I've pushed some updates for you to take a look at when you're back at work.

ncoghlan · 2024-11-02T13:13:04Z

Doc/library/getopt.rst

+
+Whether these differing behaviors in the ``argparse`` version are
+considered desirable or a problem will depend on the specific command line
+application use case.


@serhiy-storchaka ^^^ is the main reason I don't think we should walk back the default argparse recommendation. Many (probably even most) Python devs would consider argparse to be in the right for these 3 examples, and getopt/optparse the ones doing something dubious (even if what they're doing is the common Unix convention).

The discrepancies are valid reasons for reverting the soft deprecation (optparse can genuinely be used to define command line behaviours that argparse doesn't support), but it will take less arguable examples than these ones to make the case that optparse should be preferred over argparse in general.

It would be better to write about these peculiarities in the argparse module documentation. The getopt module documentation is not the place where users will search information about the argparse module.

agreed. I like that we're giving a list of differences in behavior, but I think it makes more sense to put this in a common section :ref:'d from all three of our command line parsing module docs. I don't expect anyone reading argparse (most likely) or optparse (likely) docs to find this in the getopt docs (presumed least likely to be read).

serhiy-storchaka

I do not agree with some of these changes, but overall this is a step in the right direction. Let's not drag it in bikeshedding.

serhiy-storchaka · 2024-11-04T16:08:43Z

Doc/library/getopt.rst

+A roughly equivalent command line interface for this case can also be
+produced by using the :mod:`argparse` module::
+
+import argparse


Wrong indentation. Also, add .. testcode::.

serhiy-storchaka · 2024-11-04T16:12:54Z

Doc/library/getopt.rst

+
+Whether these differing behaviors in the ``argparse`` version are
+considered desirable or a problem will depend on the specific command line
+application use case.


It would be better to write about these peculiarities in the argparse module documentation. The getopt module documentation is not the place where users will search information about the argparse module.

mwichmann · 2024-11-08T15:57:20Z

Doc/howto/argparse-optparse.rst

+``nargs=`` specifiers and better usage messages.  When most everything in
+:mod:`optparse` had either been copy-pasted over or monkey-patched, it no
+longer seemed practical to try to maintain the backwards compatibility.
+


Let me try squinting at this from a slightly different angle, feel free to ignore...

:mod:argparse was originally created to address some needs that the design of :mod:optparse made difficult to implement. While initially the majority of the :mod:optparse interface was imported as a transitional aid, maintaining backwards compatibility was never a goal.

mwichmann · 2024-11-08T15:57:33Z

Doc/howto/argparse.rst

+   module (which may require more code to configure for a given application,
+   but also allows an application to request behaviors that ``argparse``
+   doesn't support), and the very low level :mod:`getopt` (which specifically
+   serves as an equivalent to :c:func:`!getopt` from the C language).


Python getopt provides behavior that combines the classic getopt and the GNU getopt_long, so the reference (I realize it was there in the original) is incomplete. And to be a little nitpicky, the ISO C standard says nothing about getopt (POSIX does, but only the single-char getopt) so From C could be mildly confusing. Maybe something like: (which specifically serves as an equivalent to the getopt family of functions familiar to C language programmers)?

gpshead · 2024-11-08T19:41:56Z

Doc/library/argparse.rst

+   based on it) may also be worth considering for applications where
+   ``argparse`` doesn't support behaviors that the application requires
+   (such as entirely disabling support for interspersed options and
+   positional arguments, or stricter adherence to common Unix and Linux


I'd editorialize a little less and elide "stricter" & "common Unix and Linux" and simplify this to "adherence to command line interface concentions related to the handling of values that start with -"

gpshead · 2024-11-08T19:44:29Z

Doc/library/argparse.rst

@@ -11,6 +11,18 @@

 **Source code:** :source:`Lib/argparse.py`

+.. note::
+
+   While :mod:`argparse` is the recommended standard library module for


Indeed. Keep the argparse recommendation. We're not going to dis-recommend the most widely used Python argument parsing library in the world (that causes churn and angst from users). Nor are we going to be adding yet another one to the stdlib in the future. argparse supports most users quite well, "warts" and all, with plenty of popular third party options for yet other behaviors beyond our stdlib options.

gpshead · 2024-11-08T19:45:13Z

Doc/library/cmdlinelibs.rst

+   argparse.rst
+   optparse.rst


If anything, this list sure be in purely alphabetical order. passive aggressive opinions via list ordering aren't helpful. 😄

gpshead · 2024-11-08T19:49:41Z

Doc/library/getopt.rst

+
+Whether these differing behaviors in the ``argparse`` version are
+considered desirable or a problem will depend on the specific command line
+application use case.


agreed. I like that we're giving a list of differences in behavior, but I think it makes more sense to put this in a common section :ref:'d from all three of our command line parsing module docs. I don't expect anyone reading argparse (most likely) or optparse (likely) docs to find this in the getopt docs (presumed least likely to be read).

merwok · 2024-11-11T17:13:57Z

Doc/library/getopt.rst

-   Module :mod:`argparse`
-      Alternative command line option and argument parsing library.
+   Module :mod:`optparse`
+      More object-oriented command line option parsing.


IMO the salient feature of optparse over getopt is that it’s more convenient and automated! «more OO» is not a selling point

Remove getopt and optparse deprecation notices

a560986

bedevere-app bot added docs Documentation in the Doc dir awaiting core review labels Oct 31, 2024

bedevere-app bot mentioned this pull request Oct 31, 2024

Revisit getopt and optparse soft deprecations #126225

Closed

Wording tweaks

0a568f1

ncoghlan changed the title ~~gh-126225: Remove getopt and optparse deprecation notices~~ gh-126180: Remove getopt and optparse deprecation notices Oct 31, 2024

bedevere-app bot mentioned this pull request Oct 31, 2024

Undeprecate getopt and optparse #126180

Open

ncoghlan added the DO-NOT-MERGE label Oct 31, 2024

ncoghlan mentioned this pull request Oct 31, 2024

gh-126180: Undeprecate getopt and optparse #126186

Closed

ncoghlan and others added 9 commits October 31, 2024 16:14

Incorporate improvements from Serhiy's PR

55b7215

Co-authored-by: Serhiy Storchaka <[email protected]>

Grammar fix

0971780

Fix issue number reference

09b0237

Another wording tweak

4cdebde

Update heading and description for new chapter

693a979

fileinput belongs in the new chapter

50d1ee4

Move getopt back to the superseded section

4e50c74

Reword superseded chapter intro

0db2c88

Simplify wording

e474e77

ncoghlan commented Oct 31, 2024

View reviewed changes

serhiy-storchaka self-requested a review October 31, 2024 08:50

serhiy-storchaka reviewed Oct 31, 2024

View reviewed changes

LilyFoote reviewed Oct 31, 2024

View reviewed changes

ncoghlan added 2 commits November 2, 2024 17:47

Merge remote-tracking branch 'origin/main' into reverse-getopt-optpar…

43234de

…se-deprecations

Updates based on Serhiy's feedback

93d7161

ncoghlan added 2 commits November 2, 2024 23:04

Also reword What's New entry

bb91b52

Consistent getopt markup

7cc72ad

ncoghlan commented Nov 2, 2024

View reviewed changes

serhiy-storchaka reviewed Nov 4, 2024

View reviewed changes

mwichmann reviewed Nov 8, 2024

View reviewed changes

gpshead reviewed Nov 8, 2024

View reviewed changes

merwok reviewed Nov 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-126180: Remove getopt and optparse deprecation notices #126227

gh-126180: Remove getopt and optparse deprecation notices #126227

ncoghlan commented Oct 31, 2024 •

edited

Loading

ncoghlan Oct 31, 2024

serhiy-storchaka left a comment

serhiy-storchaka Oct 31, 2024

ncoghlan Oct 31, 2024

serhiy-storchaka Oct 31, 2024

ncoghlan Oct 31, 2024

serhiy-storchaka Oct 31, 2024

ncoghlan Oct 31, 2024 •

edited

Loading

gpshead Nov 8, 2024

serhiy-storchaka Oct 31, 2024

gpshead Nov 8, 2024

serhiy-storchaka Oct 31, 2024

ncoghlan Oct 31, 2024

serhiy-storchaka Oct 31, 2024

serhiy-storchaka Oct 31, 2024

ncoghlan commented Oct 31, 2024

LilyFoote Oct 31, 2024

ncoghlan Nov 2, 2024

ncoghlan commented Nov 1, 2024

ncoghlan commented Nov 2, 2024

ncoghlan Nov 2, 2024

serhiy-storchaka Nov 4, 2024

gpshead Nov 8, 2024

serhiy-storchaka left a comment

serhiy-storchaka Nov 4, 2024

serhiy-storchaka Nov 4, 2024

mwichmann Nov 8, 2024

mwichmann Nov 8, 2024 •

edited

Loading

gpshead Nov 8, 2024

gpshead Nov 8, 2024

gpshead Nov 8, 2024

gpshead Nov 8, 2024

merwok Nov 11, 2024

		conventions around the handling of option parameter values that start
		with ``-`` is desired.

		# ... do something with args.output ...
		# ... do something with args.verbose ..

		``optparse`` versions due to the way ``argparse`` handles parameter
		values that start with ``-``.

gh-126180: Remove getopt and optparse deprecation notices #126227

Are you sure you want to change the base?

gh-126180: Remove getopt and optparse deprecation notices #126227

Conversation

ncoghlan commented Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncoghlan Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncoghlan commented Oct 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncoghlan commented Nov 1, 2024

ncoghlan commented Nov 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mwichmann Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncoghlan commented Oct 31, 2024 •

edited

Loading

ncoghlan Oct 31, 2024 •

edited

Loading

mwichmann Nov 8, 2024 •

edited

Loading