Skip to content

Commit

Permalink
bpo-42260: PyConfig_Read() only parses argv once (pythonGH-23168)
Browse files Browse the repository at this point in the history 
The PyConfig_Read() function now only parses PyConfig.argv arguments
once: PyConfig.parse_argv is set to 2 after arguments are parsed.
Since Python arguments are strippped from PyConfig.argv, parsing
arguments twice would parse the application options as Python
options.

* Rework the PyConfig documentation.
* Fix _testinternalcapi.set_config() error handling.
* SetConfigTests no longer needs parse_argv=0 when restoring the old
  configuration.
  • Loading branch information
@vstinner
vstinner authored Nov 5, 2020
1 parent f3cb814 commit dc42af8
Show file tree
Hide file tree
Showing 6 changed files with 134 additions and 105 deletions.
196 changes: 104 additions & 92 deletions Doc/c-api/init_config.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,55 +8,68 @@ Python Initialization Configuration

.. versionadded:: 3.8

Structures:

* :c:type:`PyConfig`
* :c:type:`PyPreConfig`
* :c:type:`PyStatus`
* :c:type:`PyWideStringList`

Functions:

* :c:func:`PyConfig_Clear`
* :c:func:`PyConfig_InitIsolatedConfig`
* :c:func:`PyConfig_InitPythonConfig`
* :c:func:`PyConfig_Read`
* :c:func:`PyConfig_SetArgv`
* :c:func:`PyConfig_SetBytesArgv`
* :c:func:`PyConfig_SetBytesString`
* :c:func:`PyConfig_SetString`
* :c:func:`PyConfig_SetWideStringList`
* :c:func:`PyPreConfig_InitIsolatedConfig`
* :c:func:`PyPreConfig_InitPythonConfig`
* :c:func:`PyStatus_Error`
* :c:func:`PyStatus_Exception`
* :c:func:`PyStatus_Exit`
* :c:func:`PyStatus_IsError`
* :c:func:`PyStatus_IsExit`
* :c:func:`PyStatus_NoMemory`
* :c:func:`PyStatus_Ok`
* :c:func:`PyWideStringList_Append`
* :c:func:`PyWideStringList_Insert`
* :c:func:`Py_ExitStatusException`
* :c:func:`Py_InitializeFromConfig`
* :c:func:`Py_PreInitialize`
* :c:func:`Py_PreInitializeFromArgs`
* :c:func:`Py_PreInitializeFromBytesArgs`
* :c:func:`Py_RunMain`
* :c:func:`Py_GetArgcArgv`

The preconfiguration (``PyPreConfig`` type) is stored in
``_PyRuntime.preconfig`` and the configuration (``PyConfig`` type) is stored in
``PyInterpreterState.config``.
Python can be initialized with :c:func:`Py_InitializeFromConfig` and the
:c:type:`PyConfig` structure. It can be preinitialized with
:c:func:`Py_PreInitialize` and the :c:type:`PyPreConfig` structure.

There are two kinds of configuration:

* The :ref:`Python Configuration <init-python-config>` can be used to build a
customized Python which behaves as the regular Python. For example,
environments variables and command line arguments are used to configure
Python.

* The :ref:`Isolated Configuration <init-isolated-conf>` can be used to embed
Python into an application. It isolates Python from the system. For example,
environments variables are ignored, the LC_CTYPE locale is left unchanged and
no signal handler is registred.

See also :ref:`Initialization, Finalization, and Threads <initialization>`.

.. seealso::
:pep:`587` "Python Initialization Configuration".

Example
=======

Example of customized Python always running in isolated mode::

int main(int argc, char **argv)
{
PyStatus status;

PyConfig config;
PyConfig_InitPythonConfig(&config);
config.isolated = 1;

/* Decode command line arguments.
Implicitly preinitialize Python (in isolated mode). */
status = PyConfig_SetBytesArgv(&config, argc, argv);
if (PyStatus_Exception(status)) {
goto exception;
}

status = Py_InitializeFromConfig(&config);
if (PyStatus_Exception(status)) {
goto exception;
}
PyConfig_Clear(&config);

return Py_RunMain();

exception:
PyConfig_Clear(&config);
if (PyStatus_IsExit(status)) {
return status.exitcode;
}
/* Display the error message and exit the process with
non-zero exit code */
Py_ExitStatusException(status);
}


PyWideStringList
----------------
================

.. c:type:: PyWideStringList
Expand Down Expand Up @@ -95,7 +108,7 @@ PyWideStringList
List items.
PyStatus
--------
========
.. c:type:: PyStatus
Expand Down Expand Up @@ -187,7 +200,7 @@ Example::
PyPreConfig
-----------
===========
.. c:type:: PyPreConfig
Expand Down Expand Up @@ -317,7 +330,7 @@ PyPreConfig
.. _c-preinit:
Preinitialize Python with PyPreConfig
-------------------------------------
=====================================
The preinitialization of Python:
Expand All @@ -326,26 +339,35 @@ The preinitialization of Python:
* Set the :ref:`Python UTF-8 Mode <utf8-mode>`
(:c:member:`PyPreConfig.utf8_mode`)
The current preconfiguration (``PyPreConfig`` type) is stored in
``_PyRuntime.preconfig``.
Functions to preinitialize Python:
.. c:function:: PyStatus Py_PreInitialize(const PyPreConfig *preconfig)
Preinitialize Python from *preconfig* preconfiguration.
*preconfig* must not be ``NULL``.
.. c:function:: PyStatus Py_PreInitializeFromBytesArgs(const PyPreConfig *preconfig, int argc, char * const *argv)
Preinitialize Python from *preconfig* preconfiguration.
Parse *argv* command line arguments (bytes strings) if
:c:member:`~PyPreConfig.parse_argv` of *preconfig* is non-zero.
*preconfig* must not be ``NULL``.
.. c:function:: PyStatus Py_PreInitializeFromArgs(const PyPreConfig *preconfig, int argc, wchar_t * const * argv)
Preinitialize Python from *preconfig* preconfiguration.
Parse *argv* command line arguments (wide strings) if
:c:member:`~PyPreConfig.parse_argv` of *preconfig* is non-zero.
*preconfig* must not be ``NULL``.
The caller is responsible to handle exceptions (error or exit) using
:c:func:`PyStatus_Exception` and :c:func:`Py_ExitStatusException`.
Expand Down Expand Up @@ -388,7 +410,7 @@ the :ref:`Python UTF-8 Mode <utf8-mode>`::
PyConfig
--------
========
.. c:type:: PyConfig
Expand Down Expand Up @@ -449,8 +471,20 @@ PyConfig
Fields which are already initialized are left unchanged.
The :c:func:`PyConfig_Read` function only parses
:c:member:`PyConfig.argv` arguments once: :c:member:`PyConfig.parse_argv`
is set to ``2`` after arguments are parsed. Since Python arguments are
strippped from :c:member:`PyConfig.argv`, parsing arguments twice would
parse the application options as Python options.
:ref:`Preinitialize Python <c-preinit>` if needed.
.. versionchanged:: 3.10
The :c:member:`PyConfig.argv` arguments are now only parsed once,
:c:member:`PyConfig.parse_argv` is set to ``2`` after arguments are
parsed, and arguments are only parsed if
:c:member:`PyConfig.parse_argv` equals ``1``.
.. c:function:: void PyConfig_Clear(PyConfig *config)
Release configuration memory.
Expand Down Expand Up @@ -833,7 +867,7 @@ PyConfig
If :c:member:`~PyConfig.orig_argv` list is empty and
:c:member:`~PyConfig.argv` is not a list only containing an empty
string, :c:func:`PyConfig_Read()` copies :c:member:`~PyConfig.argv` into
string, :c:func:`PyConfig_Read` copies :c:member:`~PyConfig.argv` into
:c:member:`~PyConfig.orig_argv` before modifying
:c:member:`~PyConfig.argv` (if :c:member:`~PyConfig.parse_argv` is
non-zero).
Expand All @@ -849,12 +883,22 @@ PyConfig
Parse command line arguments?
If non-zero, parse :c:member:`~PyConfig.argv` the same way the regular
If equals to ``1``, parse :c:member:`~PyConfig.argv` the same way the regular
Python parses :ref:`command line arguments <using-on-cmdline>`, and strip
Python arguments from :c:member:`~PyConfig.argv`.
The :c:func:`PyConfig_Read` function only parses
:c:member:`PyConfig.argv` arguments once: :c:member:`PyConfig.parse_argv`
is set to ``2`` after arguments are parsed. Since Python arguments are
strippped from :c:member:`PyConfig.argv`, parsing arguments twice would
parse the application options as Python options.
Default: ``1`` in Python mode, ``0`` in isolated mode.
.. versionchanged:: 3.10
The :c:member:`PyConfig.argv` arguments are now only parsed if
:c:member:`PyConfig.parse_argv` equals to ``1``.
.. c:member:: int parser_debug
Parser debug mode. If greater than 0, turn on parser debugging output (for expert only, depending
Expand Down Expand Up @@ -1108,7 +1152,7 @@ the :option:`-X` command line option.
Initialization with PyConfig
----------------------------
============================
Function to initialize Python:
Expand All @@ -1123,6 +1167,9 @@ If :c:func:`PyImport_FrozenModules`, :c:func:`PyImport_AppendInittab` or
:c:func:`PyImport_ExtendInittab` are used, they must be set or called after
Python preinitialization and before the Python initialization.
The current configuration (``PyConfig`` type) is stored in
``PyInterpreterState.config``.
Example setting the program name::
void init_python(void)
Expand All @@ -1136,17 +1183,17 @@ Example setting the program name::
status = PyConfig_SetString(&config, &config.program_name,
L"/path/to/my_program");
if (PyStatus_Exception(status)) {
goto fail;
goto exception;
}
status = Py_InitializeFromConfig(&config);
if (PyStatus_Exception(status)) {
goto fail;
goto exception;
}
PyConfig_Clear(&config);
return;
fail:
exception:
PyConfig_Clear(&config);
Py_ExitStatusException(status);
}
Expand Down Expand Up @@ -1202,7 +1249,7 @@ configuration, and then override some parameters::
.. _init-isolated-conf:
Isolated Configuration
----------------------
======================
:c:func:`PyPreConfig_InitIsolatedConfig` and
:c:func:`PyConfig_InitIsolatedConfig` functions create a configuration to
Expand All @@ -1223,7 +1270,7 @@ configuration.
.. _init-python-config:
Python Configuration
--------------------
====================
:c:func:`PyPreConfig_InitPythonConfig` and :c:func:`PyConfig_InitPythonConfig`
functions create a configuration to build a customized Python which behaves as
Expand All @@ -1237,46 +1284,11 @@ and :ref:`Python UTF-8 Mode <utf8-mode>`
(:pep:`540`) depending on the LC_CTYPE locale, :envvar:`PYTHONUTF8` and
:envvar:`PYTHONCOERCECLOCALE` environment variables.
Example of customized Python always running in isolated mode::
int main(int argc, char **argv)
{
PyStatus status;
PyConfig config;
PyConfig_InitPythonConfig(&config);
config.isolated = 1;
/* Decode command line arguments.
Implicitly preinitialize Python (in isolated mode). */
status = PyConfig_SetBytesArgv(&config, argc, argv);
if (PyStatus_Exception(status)) {
goto fail;
}
status = Py_InitializeFromConfig(&config);
if (PyStatus_Exception(status)) {
goto fail;
}
PyConfig_Clear(&config);
return Py_RunMain();
fail:
PyConfig_Clear(&config);
if (PyStatus_IsExit(status)) {
return status.exitcode;
}
/* Display the error message and exit the process with
non-zero exit code */
Py_ExitStatusException(status);
}
.. _init-path-config:
Path Configuration
------------------
==================
:c:type:`PyConfig` contains multiple fields for the path configuration:
Expand Down Expand Up @@ -1356,7 +1368,7 @@ The ``__PYVENV_LAUNCHER__`` environment variable is used to set
Py_RunMain()
------------
============
.. c:function:: int Py_RunMain(void)
Expand All @@ -1376,7 +1388,7 @@ customized Python always running in isolated mode using
Py_GetArgcArgv()
----------------
================
.. c:function:: void Py_GetArgcArgv(int *argc, wchar_t ***argv)
Expand All @@ -1386,7 +1398,7 @@ Py_GetArgcArgv()
Multi-Phase Initialization Private Provisional API
--------------------------------------------------
==================================================
This section is a private provisional API introducing multi-phase
initialization, the core feature of the :pep:`432`:
Expand Down
8 changes: 7 additions & 1 deletion Lib/test/_test_embed_set_config.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ def setUp(self):
self.sys_copy = dict(sys.__dict__)

def tearDown(self):
self.set_config(parse_argv=0)
_testinternalcapi.set_config(self.old_config)
sys.__dict__.clear()
sys.__dict__.update(self.sys_copy)

Expand Down Expand Up @@ -234,6 +234,12 @@ def test_argv(self):
self.assertEqual(sys.argv, ['python_program', 'args'])
self.assertEqual(sys.orig_argv, ['orig', 'orig_args'])

self.set_config(parse_argv=0,
argv=[],
orig_argv=[])
self.assertEqual(sys.argv, [''])
self.assertEqual(sys.orig_argv, [])

def test_pycache_prefix(self):
self.check(pycache_prefix=None)
self.check(pycache_prefix="pycache_prefix")
Expand Down
Loading

0 comments on commit dc42af8

Please sign in to comment.