# embeddings

id: pep-0233.rst
content:
PEP: 233
Title: Python Online Help
Author: Paul Prescod <paul@prescod.net>
Status: Deferred
Type: Standards Track
Created: 11-Dec-2000
Python-Version: 2.1
Post-History:


Abstract
========

This PEP describes a command-line driven online help facility for
Python.  The facility should be able to build on existing
documentation facilities such as the Python documentation and
docstrings.  It should also be extensible for new types and
modules.


Interactive use
===============

Simply typing ``help`` describes the help function (through ``repr()``
overloading).

``help`` can also be used as a function.

The function takes the following forms of input:

* ``help( "string" )`` -- built-in topic or global
* ``help( <ob> )`` -- docstring from object or type
* ``help( "doc:filename" )`` -- filename from Python documentation

If you ask for a global, it can be a fully-qualified name, such as::

    help("xml.dom")

You can also use the facility from a command-line::

    python --help if

In either situation, the output does paging similar to the ``more``
command.


Implementation
==============

The help function is implemented in an ``onlinehelp`` module which is
demand-loaded.

There should be options for fetching help information from
environments other than the command line through the ``onlinehelp``
module::

    onlinehelp.gethelp(object_or_string) -> string

It should also be possible to override the help display function
by assigning to ``onlinehelp.displayhelp(object_or_string)``.

The module should be able to extract module information from
either the HTML or LaTeX versions of the Python documentation.
Links should be accommodated in a "lynx-like" manner.

Over time, it should also be able to recognize when docstrings are
in "special" syntaxes like structured text, HTML and LaTeX and
decode them appropriately.

A prototype implementation is available with the Python source
distribution as ``nondist/sandbox/doctools/onlinehelp.py``.


Built-in Topics
===============

* ``help( "intro" )`` -- What is Python? Read this first!

* ``help( "keywords" )`` -- What are the keywords?

* ``help( "syntax" )`` -- What is the overall syntax?

* ``help( "operators" )`` -- What operators are available?

* ``help( "builtins" )`` -- What functions, types, etc. are built-in?

* ``help( "modules" )`` -- What modules are in the standard library?

* ``help( "copyright" )`` -- Who owns Python?

* ``help( "moreinfo" )`` -- Where is there more information?

* ``help( "changes" )`` -- What changed in Python 2.0?

* ``help( "extensions" )`` -- What extensions are installed?

* ``help( "faq" )`` -- What questions are frequently asked?

* ``help( "ack" )`` -- Who has done work on Python lately?


Security Issues
===============

This module will attempt to import modules with the same names as
requested topics.  Don't use the modules if you are not confident
that everything in your ``PYTHONPATH`` is from a trusted source.


id: pep-0569.rst
content:
PEP: 569
Title: Python 3.8 Release Schedule
Author: Åukasz Langa <lukasz@python.org>
Status: Final
Type: Informational
Topic: Release
Created: 27-Jan-2018
Python-Version: 3.8


Abstract
========

This document describes the development and release schedule for
Python 3.8.  The schedule primarily concerns itself with PEP-sized
items.

.. Small features may be added up to the first beta
   release.  Bugs may be fixed until the final release,
   which is planned for end of October 2019.


Release Manager and Crew
========================

- 3.8 Release Manager: Åukasz Langa
- Windows installers: Steve Dower
- Mac installers: Ned Deily
- Documentation: Julien Palard


3.8 Lifespan
============

3.8 will receive bugfix updates approximately every 2 months for
approximately 18 months.  Some time after the release of 3.9.0 final,
the ninth and final 3.8 bugfix update was released.  After that,
security updates (source only) were released for 5 years until the
release of Python 3.13.0 final.

As of 2024-10-07, 3.8 has reached the
`end-of-life phase <https://devguide.python.org/devcycle/#end-of-life-branches>`_
of its release cycle. 3.8.20 was the final security release.
The codebase for 3.8 is now frozen and no further updates will be
provided nor issues of any kind will be accepted on the bug tracker.


Release Schedule
================

3.8.0 schedule
--------------

- 3.8 development begins: Monday, 2018-01-29
- 3.8.0 alpha 1: Sunday, 2019-02-03
- 3.8.0 alpha 2: Monday, 2019-02-25
- 3.8.0 alpha 3: Monday, 2019-03-25
- 3.8.0 alpha 4: Monday, 2019-05-06
- 3.8.0 beta 1: Tuesday, 2019-06-04
  (No new features beyond this point.)

- 3.8.0 beta 2: Thursday, 2019-07-04
- 3.8.0 beta 3: Monday, 2019-07-29
- 3.8.0 beta 4: Friday, 2019-08-30
- 3.8.0 candidate 1: Tuesday, 2019-10-01
- 3.8.0 final: Monday, 2019-10-14

Bugfix releases
---------------

- 3.8.1rc1: Tuesday, 2019-12-10
- 3.8.1: Wednesday, 2019-12-18
- 3.8.2rc1: Monday, 2020-02-10
- 3.8.2rc2: Monday, 2020-02-17
- 3.8.2: Monday, 2020-02-24
- 3.8.3rc1: Wednesday, 2020-04-29
- 3.8.3: Wednesday, 2020-05-13
- 3.8.4rc1: Tuesday, 2020-06-30
- 3.8.4: Monday, 2020-07-13
- 3.8.5: Monday, 2020-07-20 (security hotfix)
- 3.8.6rc1: Tuesday, 2020-09-08
- 3.8.6: Thursday, 2020-09-24
- 3.8.7rc1: Monday, 2020-12-07
- 3.8.7: Monday, 2020-12-21
- 3.8.8rc1: Tuesday, 2021-02-16
- 3.8.8: Friday, 2021-02-19
- 3.8.9: Friday, 2021-04-02 (security hotfix)
- 3.8.10: Monday, 2021-05-03 (final regular bugfix release with binary
  installers)

Source-only security fix releases
---------------------------------

Provided irregularly on an "as-needed" basis until October 7th 2024.

- 3.8.11: Monday, 2021-06-28
- 3.8.12: Monday, 2021-08-30
- 3.8.13: Wednesday, 2022-03-16
- 3.8.14: Tuesday, 2022-09-06
- 3.8.15: Tuesday, 2022-10-11
- 3.8.16: Tuesday, 2022-12-06
- 3.8.17: Tuesday, 2023-06-06
- 3.8.18: Thursday, 2023-08-24
- 3.8.19: Tuesday, 2024-03-19
- 3.8.20: Friday, 2024-09-06 (final security release)


Features for 3.8
================

Some of the notable features of Python 3.8 include:

* :pep:`570`, Positional-only arguments
* :pep:`572`, Assignment Expressions
* :pep:`574`, Pickle protocol 5 with out-of-band data
* :pep:`578`, Runtime audit hooks
* :pep:`587`, Python Initialization Configuration
* :pep:`590`, Vectorcall: a fast calling protocol for CPython
* Typing-related: :pep:`591` (Final qualifier), :pep:`586` (Literal types),
  and :pep:`589` (TypedDict)
* Parallel filesystem cache for compiled bytecode
* Debug builds share ABI as release builds
* f-strings support a handy ``=`` specifier for debugging
* ``continue`` is now legal in ``finally:`` blocks
* on Windows, the default ``asyncio`` event loop is now
  ``ProactorEventLoop``
* on macOS, the *spawn* start method is now used by default in
  ``multiprocessing``
* ``multiprocessing`` can now use shared memory segments to avoid
  pickling costs between processes
* ``typed_ast`` is merged back to CPython
* ``LOAD_GLOBAL`` is now 40% faster
* ``pickle`` now uses Protocol 4 by default, improving performance

There are many other interesting changes, please consult the
"What's New" page in the documentation for a full list.


Copyright
=========

This document has been placed in the public domain.


id: pep-0582.rst
content:
PEP: 582
Title: Python local packages directory
Author: Kushal Das <mail@kushaldas.in>, Steve Dower <steve.dower@python.org>,
        Donald Stufft <donald@stufft.io>, Alyssa Coghlan <ncoghlan@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-582-python-local-packages-directory/963/
Status: Rejected
Type: Standards Track
Topic: Packaging
Created: 16-May-2018
Python-Version: 3.12
Post-History: `01-Mar-2019 <https://discuss.python.org/t/pep-582-python-local-packages-directory/963>`__,
Resolution: https://discuss.python.org/t/pep-582-python-local-packages-directory/963/430


Abstract
========

This PEP proposes extending the existing mechanism for setting up ``sys.path``
to include a new ``__pypackages__`` directory, in addition to the existing
locations. The new directory will be added at the start of ``sys.path``, after
the current working directory and just before the system site-packages, to give
packages installed there priority over other locations.

This is similar to the existing mechanism of adding the current directory (or
the directory the script is located in), but by using a subdirectory,
additional libraries are kept separate from the user's work.


Motivation
==========

New Python programmers can benefit from being taught the value of isolating an
individual project's dependencies from their system environment. However, the
existing mechanism for doing this, virtual environments, is known to be complex
and error-prone for beginners to understand. Explaining virtual environments is
often a distraction when trying to get a group of beginners set up - differences
in platform and shell environments require individual assistance, and the need
for activation in every new shell session makes it easy for students to make
mistakes when coming back to work after a break. This proposal offers a lightweight
solution that gives isolation without the user needing to understand more
advanced concepts.

Furthermore, standalone Python applications usually need 3rd party libraries to
function. Typically, they are either designed to be run from a virtual environment,
where the dependencies are installed into the environment alongside the application,
or they bundle their dependencies in a subdirectory, and modify ``sys.path`` at
application startup. Virtual environments, while a common and effective solution
(used, for example, by the ``pipx`` tool), are somewhat awkward to set up and manage,
and are not relocatable. On the other hand, manual manipulation of ``sys.path`` is
boilerplate that developers need to get right, and (being a runtime behaviour)
it is not understood by tools like linters and type checkers. The ``__pypackages__``
proposal formalises the idea of a "bundled dependencies" location, avoiding the
boilerplate and providing a standard location that development tools can be taught
to recognise.

It should be noted that in general, Python libraries cannot be simply copied
between machines, platforms, or even necessarily between Python versions. This
proposal does nothing to change that fact, and while it is tempting to assume
that bundling a script and its ``__pypackages__`` is a mechanism for
distributing applications, this is explicitly *not* a goal of this proposal.
Developers remain responsible for the portability of their code.

Rationale
=========

While ``sys.path`` can be manipulated at runtime, the default value is important, as
it establishes a common baseline that users and tools can agree on. The current default
does not include a location that could be viewed as "private to the current project",
and yet that is a useful concept.

This is similar to the npm ``node_modules`` directory, which is popular in the
Javascript community, and something that developers familiar with that
ecosystem often ask for from Python.


Specification
=============


This PEP proposes to add a new step in the process of calculating ``sys.path`` at
startup.

When the interactive interpreter starts, if a ``__pypackages__`` directory is
found in the current working directory, then it will be included in
``sys.path`` after the entry for current working directory and just before the
system site-packages.

When the interpreter runs a script, Python will try to find ``__pypackages__``
in the same directory as the script. If found (along with the current Python
version directory inside), then it will be used, otherwise Python will behave
as it does currently.

The behaviour should work exactly the same as the way the existing mechanism
for adding the current working directory or script directory to ``sys.path``
works. For example, ``__pypackages__`` will be ignored if the ``-P`` option or
the ``PYTHONSAFEPATH`` environment variable is set.

In order to be recognised, the ``__pypackages__`` directory must be laid out
according to a new ``localpackages`` scheme in the sysconfig module.
Specifically, both of the ``purelib`` and ``platlib`` directories must be
present, using the following code to determine the locations of those
directories::

    scheme = "localpackages"
    purelib = sysconfig.get_path("purelib", scheme, vars={"base": "__pypackages__", "platbase": "__pypackages__"})
    platlib = sysconfig.get_path("platlib", scheme, vars={"base": "__pypackages__", "platbase": "__pypackages__"})

These two locations will be added to ``sys.path``, other directories or
files in the ``__pypackages__`` directory will be silently ignored. The
paths will be based on Python versions.

.. note:: There is a possible option of having a separate new API, it is documented at `issue #3013 <https://github.com/python/peps/issues/3013>`_.


Example
-------

The following shows an example project directory structure, and different ways
the Python executable and any script will behave. The example is for Unix-like
systems - on Windows the subdirectories will be different.

::

    foo
        __pypackages__
            lib
                python3.10
                           site-packages
                                         bottle
        myscript.py

    /> python foo/myscript.py
    sys.path[0] == 'foo'
    sys.path[1] == 'foo/__pypackages__/lib/python3.10/site-packages/'


    cd foo

    foo> /usr/bin/ansible
        #! /usr/bin/env python3
    foo> python /usr/bin/ansible

    foo> python myscript.py

    foo> python
    sys.path[0] == '.'
    sys.path[1] == './__pypackages__/lib/python3.10/site-packages'

    foo> python -m bottle

We have a project directory called ``foo`` and it has a ``__pypackages__``
inside of it. We have ``bottle`` installed in that
``__pypackages__/lib/python3.10/site-packages/``, and have a ``myscript.py``
file inside of the project directory. We have used whatever tool we generally
use to install ``bottle`` in that location.

For invoking a script, Python will try to find a ``__pypackages__`` inside of
the directory that the script resides [1]_, ``/usr/bin``.  The same will happen
in case of the last example, where we are executing ``/usr/bin/ansible`` from
inside of the ``foo`` directory. In both cases, it will **not** use the
``__pypackages__`` in the current working directory.

Similarly, if we invoke ``myscript.py`` from the first example, it will use the
``__pypackages__`` directory that was in the ``foo`` directory.

If we go inside of the ``foo`` directory and start the Python executable (the
interpreter), it will find the ``__pypackages__`` directory inside of the
current working directory and use it in the ``sys.path``. The same happens if we
try to use the ``-m`` and use a module. In our example, ``bottle`` module will
be found inside of the ``__pypackages__`` directory.

The above two examples are only cases where ``__pypackages__`` from current
working directory is used.

In another example scenario, a trainer of a Python class can say "Today we are
going to learn how to use Twisted! To start, please checkout our example
project, go to that directory, and then run a given command to install Twisted."

That will install 


id: pep-0596.rst
content:
PEP: 596
Title: Python 3.9 Release Schedule
Author: Åukasz Langa <lukasz@python.org>
Discussions-To: https://discuss.python.org/t/pep-596-python-3-9-release-schedule-doubling-the-release-cadence/1828
Status: Active
Type: Informational
Topic: Release
Created: 04-Jun-2019
Python-Version: 3.9


Abstract
========

This document describes the development and release schedule for
Python 3.9.  The schedule primarily concerns itself with PEP-sized
items.

.. Small features may be added up to the first beta
   release.  Bugs may be fixed until the final release,
   which is planned for beginning of October 2020.


Release Manager and Crew
========================

- 3.9 Release Manager: Åukasz Langa
- Windows installers: Steve Dower
- Mac installers: Ned Deily
- Documentation: Julien Palard


Release Schedule
================

3.9.0 schedule
--------------

Note: the dates below use a 17-month development period that results
in a 12-month release cadence between feature versions, as defined by
:pep:`602`.

Actual:

- 3.9 development begins: Tuesday, 2019-06-04
- 3.9.0 alpha 1: Tuesday, 2019-11-19
- 3.9.0 alpha 2: Wednesday, 2019-12-18
- 3.9.0 alpha 3: Saturday, 2020-01-25
- 3.9.0 alpha 4: Wednesday, 2020-02-26
- 3.9.0 alpha 5: Monday, 2020-03-23
- 3.9.0 alpha 6: Tuesday, 2020-04-28
- 3.9.0 beta 1: Monday, 2020-05-18
  (No new features beyond this point.)
- 3.9.0 beta 3: Tuesday, 2020-06-09
  (beta 2 was recalled.)
- 3.9.0 beta 4: Friday, 2020-07-03
- 3.9.0 beta 5: Monday, 2020-07-20
- 3.9.0 candidate 1: Tuesday, 2020-08-11
- 3.9.0 candidate 2: Thursday, 2020-09-17
- 3.9.0 final: Monday, 2020-10-05


Bugfix releases
---------------

Actual:

- 3.9.1 candidate 1: Tuesday, 2020-11-24
- 3.9.1 final: Monday, 2020-12-07
- 3.9.2 candidate 1: Tuesday, 2021-02-16
- 3.9.2 final: Friday, 2021-02-19
- 3.9.3: Friday, 2021-04-02 (security hotfix; recalled due to bpo-43710)
- 3.9.4: Sunday, 2021-04-04 (ABI compatibility hotfix)
- 3.9.5: Monday, 2021-05-03
- 3.9.6: Monday, 2021-06-28
- 3.9.7: Monday, 2021-08-30
- 3.9.8: Friday, 2021-11-05 (recalled due to bpo-45235)
- 3.9.9: Monday, 2021-11-15
- 3.9.10: Friday, 2022-01-14
- 3.9.11: Wednesday, 2022-03-16
- 3.9.12: Wednesday, 2022-03-23
- 3.9.13: Tuesday, 2022-05-17 (final regular bugfix release with binary
  installers)


Source-only security fix releases
---------------------------------

Provided irregularly on an "as-needed" basis until October 2025.

- 3.9.14: Tuesday, 2022-09-06
- 3.9.15: Tuesday, 2022-10-11
- 3.9.16: Tuesday, 2022-12-06
- 3.9.17: Tuesday, 2023-06-06
- 3.9.18: Thursday, 2023-08-24
- 3.9.19: Tuesday, 2024-03-19
- 3.9.20: Friday, 2024-09-06
- 3.9.21: Tuesday, 2024-12-03
- 3.9.22: Tuesday, 2025-04-08


3.9 Lifespan
------------

3.9 received bugfix updates approximately every 2 months for
approximately 18 months.  Some time after the release of 3.10.0 final,
the ninth and final 3.9 bugfix update was released.  After that,
it is expected that security updates (source only) will be released
until 5 years after the release of 3.9 final, so until approximately
October 2025.


Features for 3.9
================

Implemented PEPs for 3.9:

* :pep:`573`, Module State Access from C Extension Methods
* :pep:`584`, Add Union Operators To dict
* :pep:`585`, Type Hinting Generics In Standard Collections
* :pep:`593`, Flexible function and variable annotations
* :pep:`614`, Relaxing Grammar Restrictions On Decorators
* :pep:`615`, Support for the IANA Time Zone Database in the Standard Library
* :pep:`616`, String methods to remove prefixes and suffixes
* :pep:`617`, New PEG parser for CPython


Copyright
=========

This document has been placed in the public domain.


id: pep-0743.rst
content:
PEP: 743
Title: Add Py_COMPAT_API_VERSION to the Python C API
Author: Victor Stinner <vstinner@python.org>,
        Petr Viktorin <encukou@gmail.com>,
PEP-Delegate: C API Working Group
Discussions-To: https://discuss.python.org/t/pep-743-add-py-compat-api-version-to-the-python-c-api-take-2/59323
Status: Draft
Type: Standards Track
Created: 11-Mar-2024
Python-Version: 3.14

.. highlight:: c


Abstract
========

Add ``Py_COMPAT_API_VERSION`` C macro that hides some deprecated and
soft-deprecated symbols, allowing users to opt out of using API with known
issues that other API solves.
The macro is versioned, allowing users to update (or not) on their own pace.

Also, add namespaced alternatives for API without the ``Py_`` prefix,
and soft-deprecate the original names.


Motivation
==========

Some of Python's C API has flaws that are only obvious in hindsight.

If an API prevents adding features or optimizations, or presents a serious
security risk or maintenance burden, we can deprecate and remove it as
described in :pep:`387`.

However, this leaves us with some API that has âsharp edgesâ -- it works fine
for its current users, but should be avoided in new code.
For example:

- API that cannot signal an exception, so failures are either ignored or
  exit the process with a fatal error. For example ``PyObject_HasAttr``.
- API that is not thread-safe, for example by borrowing references from
  mutable objects, or exposing unfinished mutable objects. For example
  ``PyDict_GetItemWithError``.
- API with names that don't use the ``Py``/``_Py`` prefix, and so can clash
  with other code. For example: ``setter``.

It is important to note that despite such flaws, it's usually possible
to use the API correctly. For example, in a single-threaded environment,
thread safety is not an issue.
We do not want to break working code, even if it uses API that would be wrong
in some -- or even *most* -- other contexts.

On the other hand, we want to steer users away from such âundesirableâ API
in *new* code, especially if a safer alternative exists.


Adding the ``Py`` prefix
------------------------

Some names defined in CPython headers is not namespaced: it that lacks the
``Py`` prefix (or a variant: ``_Py``, and alternative capitalizations).
For example, we declare a function type named simply ``setter``.

While such names are not exported in the ABI (as checked by ``make smelly``),
they can clash with user code and, more importantly, with libraries linked
to third-party extensions.

While it would be possible to provide namespaced aliases and (soft-)deprecate
these names, the only way to make them not clash with third-party code is to
not define them in Python headers at all.


Rationale
=========

We want to allow an easy way for users to avoid âundesirableâ API if they
choose to do so.

It might be be sufficient to leave this to third-party linters.
For that we'd need a good way to expose a list of (soft-)deprecated
API to such linters.
While adding that, we can -- rather easily -- do the linter's job directly
in CPython headers, avoiding the neel for an extra tool.
Unlike Python, C makes it rather easy to limit available API -- for a whole
project or for each individual source file -- by having users define
an âopt-inâ macro.

We already do something similar with ``Py_LIMITED_API``, which limits the
available API to a subset that compiles to stable ABI. (In hindsight, we should
have used a different macro name for that particular kind of limiting, but it's
too late to change that now.)

To prevent working code from breaking as we identify more âundesirableâ API
and add safer alternatives to it, the opt-in macro should be *versioned*.
Users can choose a version they need based on their compatibility requirements,
and update it at their own pace.

To be clear, this mechanism is *not* a replacement for deprecation.
Deprecation is for API that prevents new features or optimizations, or
presents a security risk or maintenance burden.
This mechanism, on the other hand, is meant for cases where âwe found
a slightly better way of doing thingsâ -- perhaps one that's harder to misuse,
or just has a less misleading name.
(On a lighter note: many people configure a code quality checker to shout at
them about the number of blank lines between functions. Let's help them
identify more substantial âcode smellsâ!)

The proposed macro does not *change* any API definitions; it only *hides* them.
So, if code compiles with the macro, it'll also compile without it, with
identical behaviour.
This has implications for core devs: to deal with undesirable behaviour,
we'll need to introduce new, better API, and *then* discourage the old one.
In turn, this implies that we should look at an individual API and fix all its
known issues at once, rather than do codebase-wide sweeps for a single kind of
issue, so that we avoid multiple renames of the same function.


Adding the ``Py`` prefix
------------------------

An opt-in macro allows us to omit definitions that could clash with
third-party libraries.


Specification
=============

We introduce a ``Py_COMPAT_API_VERSION`` macro.
If this macro is defined before ``#include <Python.h>``, some API definitions
-- as described below -- will be omitted from the Python header files.

The macro only omits complete top-level definitions exposed from ``<Python.h>``.
Other things (the ABI, structure definitions, macro expansions, static inline
function bodies, etc.) are not affected.

The C API working group (:pep:`731`) has authority over the set of omitted
definitions.

The set of omitted definitions will be tied to a particular feature release
of CPython, and is finalized in each 3.x.0 Beta 1 release.
In rare cases, entries can be removed (i.e. made available for use) at any
time.

The macro should be defined to a version in the format used by
``PY_VERSION_HEX``, with the âmicroâ, âreleaseâ and âserialâ fields
set to zero.
For example, to omit API deemed undesirable in 3.14.0b1, users should define
``Py_COMPAT_API_VERSION`` to ``0x030e0000``.


Requirements for omitted API
----------------------------

An API that is omitted with ``Py_COMPAT_API_VERSION`` must:

- be soft-deprecated (see :pep:`387`);
- for all known use cases of the API, have a documented alternative
  or workaround;
- have tests to ensure it keeps working (except for 1:1 renames using
  ``#define`` or ``typedef``);
- be documented (except if it was never mentioned in previous versions of the
  documentation); and
- be approved by the C API working group. (The WG may give blanket approvals
  for groups of related API; see *Initial set* below for examples.)

Note that ``Py_COMPAT_API_VERSION`` is meant for API that can be trivially
replaced by a better alternative.
API without a replacement should generally be deprecated instead.


Location
--------

All API definitions omitted by ``Py_COMPAT_API_VERSION`` will be moved to
a new header, ``Include/legacy.h``.

This is meant to help linter authors compile lists, so they can flag the API
with warnings rather than errors.

Note that for simple renaming of source-only constructs (macros, types), we
expect names to be omitted in the same version -- or the same PR -- that adds
a replacement.
This means that the original definition will be renamed, and a ``typedef``
or ``#define`` for the old name added to ``Include/legacy.h``.


Documentation
-------------

Documentation for omitted API should generally:

- appear after the recommended replacement,
- reference the replacement (e.g. âSimilar to X, butâ¦â), and
- focus on differences from the replacement and migration advice.

Exceptions are possible if there is a good reason for them.


Initial set
-----------

The following API will be omitted with ``Py_COMPAT_API_VERSION`` set to
``0x030e0000`` (3.14) or greater:

- Omit API returning borrowed references:

  ====================================  ==============================
  Omitted API      


id: pep-0757.rst
content:
PEP: 757
Title: C API to import-export Python integers
Author: Sergey B Kirpichev <skirpichev@gmail.com>,
        Victor Stinner <vstinner@python.org>
Discussions-To: https://discuss.python.org/t/63895
Status: Final
Type: Standards Track
Created: 13-Sep-2024
Python-Version: 3.14
Post-History: `14-Sep-2024 <https://discuss.python.org/t/63895>`__
Resolution: `08-Dec-2024 <https://discuss.python.org/t/63895/79>`__


.. canonical-doc:: the `Export API <https://docs.python.org/dev/c-api/long.html#export-api>`_ and the `PyLongWriter API <https://docs.python.org/dev/c-api/long.html#pylongwriter-api>`_

.. highlight:: c


Abstract
========

Add a new C API to import and export Python integers, :class:`int` objects:
especially :c:func:`PyLongWriter_Create()` and :c:func:`PyLong_Export()` functions.


Rationale
=========

Projects such as `gmpy2 <https://github.com/aleaxit/gmpy>`_, `SAGE
<https://www.sagemath.org/>`_ and `Python-FLINT
<https://github.com/flintlib/python-flint>`_ access directly Python
"internals" (the :c:type:`PyLongObject` structure) or use an inefficient
temporary format (hex strings for Python-FLINT) to import and
export Python :class:`int` objects.  The Python :class:`int` implementation
changed in Python 3.12 to add a tag and "compact values".

In the 3.13 alpha 1 release, the private undocumented :c:func:`!_PyLong_New()`
function had been removed, but it is being used by these projects to
import Python integers. The private function has been restored in 3.13
alpha 2.

A public efficient abstraction is needed to interface Python with these
projects without exposing implementation details. It would allow Python
to change its internals without breaking these projects.  For example,
implementation for gmpy2 was changed recently for CPython 3.9 and
for CPython 3.12.


Specification
=============

Layout API
----------

Data needed by `GMP <https://gmplib.org/>`_-like `import
<https://gmplib.org/manual/Integer-Import-and-Export#index-mpz_005fimport>`_-`export
<https://gmplib.org/manual/Integer-Import-and-Export#index-mpz_005fexport>`_
functions.

.. c:struct:: PyLongLayout

   Layout of an array of "digits" ("limbs" in the GMP terminology), used to
   represent absolute value for arbitrary precision integers.

   Use :c:func:`PyLong_GetNativeLayout` to get the native layout of Python
   :class:`int` objects, used internally for integers with "big enough"
   absolute value.

   See also :data:`sys.int_info` which exposes similar information to Python.

   .. c:member:: uint8_t bits_per_digit

      Bits per digit.  For example, a 15 bit digit means that bits 0-14
      contain meaningful information.

   .. c:member:: uint8_t digit_size

      Digit size in bytes.  For example, a 15 bit digit will require at least
      2 bytes.

   .. c:member:: int8_t digits_order

      Digits order:

      - ``1`` for most significant digit first
      - ``-1`` for least significant digit first

   .. c:member:: int8_t digit_endianness

      Digit endianness:

      - ``1`` for most significant byte first (big endian)
      - ``-1`` for least significant byte first (little endian)


.. c:function:: const PyLongLayout* PyLong_GetNativeLayout(void)

   Get the native layout of Python :class:`int` objects.

   See the :c:struct:`PyLongLayout` structure.

   The function must not be called before Python initialization nor after
   Python finalization. The returned layout is valid until Python is
   finalized. The layout is the same for all Python sub-interpreters and
   so it can be cached.


Export API
----------

.. c:struct:: PyLongExport

   Export of a Python :class:`int` object.

   There are two cases:

   * If :c:member:`digits` is ``NULL``, only use the :c:member:`value` member.
   * If :c:member:`digits` is not ``NULL``, use :c:member:`negative`,
     :c:member:`ndigits` and :c:member:`digits` members.

   .. c:member:: int64_t value

      The native integer value of the exported :class:`int` object.
      Only valid if :c:member:`digits` is ``NULL``.

   .. c:member:: uint8_t negative

      1 if the number is negative, 0 otherwise.
      Only valid if :c:member:`digits` is not ``NULL``.

   .. c:member:: Py_ssize_t ndigits

      Number of digits in :c:member:`digits` array.
      Only valid if :c:member:`digits` is not ``NULL``.

   .. c:member:: const void *digits

      Read-only array of unsigned digits. Can be ``NULL``.


If :c:member:`PyLongExport.digits` is not ``NULL``, a private field of the
:c:struct:`PyLongExport` structure stores a strong reference to the Python
:class:`int` object to make sure that that structure remains valid until
:c:func:`PyLong_FreeExport()` is called.


.. c:function:: int PyLong_Export(PyObject *obj, PyLongExport *export_long)

   Export a Python :class:`int` object.

   *export_long* must point to a :c:struct:`PyLongExport` structure allocated
   by the caller.  It must not be ``NULL``.

   On success, fill in *\*export_long* and return 0.
   On error, set an exception and return -1.

   :c:func:`PyLong_FreeExport` must be called when the export is no longer
   needed.

   **CPython implementation detail**: This function always succeeds if *obj* is
   a Python :class:`int` object or a subclass.


On CPython 3.14, no memory copy is needed in :c:func:`PyLong_Export`, it's just
a thin wrapper to expose Python :class:`int` internal digits array.


.. c:function:: void PyLong_FreeExport(PyLongExport *export_long)

   Release the export *export_long* created by :c:func:`PyLong_Export`.

   **CPython implementation detail**: Calling :c:func:`PyLong_FreeExport` is
   optional if *export_long->digits* is ``NULL``.


Import API
----------

The :c:type:`PyLongWriter` API can be used to import an integer.

.. c:struct:: PyLongWriter

   A Python :class:`int` writer instance.

   The instance must be destroyed by :c:func:`PyLongWriter_Finish` or
   :c:func:`PyLongWriter_Discard`.


.. c:function:: PyLongWriter* PyLongWriter_Create(int negative, Py_ssize_t ndigits, void **digits)

   Create a :c:type:`PyLongWriter`.

   On success, allocate *\*digits* and return a writer.
   On error, set an exception and return ``NULL``.

   *negative* is ``1`` if the number is negative, or ``0`` otherwise.

   *ndigits* is the number of digits in the *digits* array.  It must be
   greater than 0.

   *digits* must not be NULL.

   After a successful call to this function, the caller should fill in the
   array of digits *digits* and then call :c:func:`PyLongWriter_Finish` to get
   a Python :class:`int`.
   The layout of *digits* is described by :c:func:`PyLong_GetNativeLayout`.

   Digits must be in the range [``0``; ``(1 << bits_per_digit) - 1``]
   (where the :c:struct:`~PyLongLayout.bits_per_digit` is the number of bits
   per digit).
   Any unused most significant digits must be set to ``0``.

   Alternately, call :c:func:`PyLongWriter_Discard` to destroy the writer
   instance without creating an :class:`~int` object.


On CPython 3.14, the :c:func:`PyLongWriter_Create` implementation is a thin
wrapper to the private :c:func:`!_PyLong_New()` function.


.. c:function:: PyObject* PyLongWriter_Finish(PyLongWriter *writer)

   Finish a :c:type:`PyLongWriter` created by :c:func:`PyLongWriter_Create`.

   On success, return a Python :class:`int` object.
   On error, set an exception and return ``NULL``.

   The function takes care of normalizing the digits and converts the
   object to a compact integer if needed.

   The writer instance and the *digits* array are invalid after the call.


.. c:function:: void PyLongWriter_Discard(PyLongWriter *writer)

   Discard a :c:type:`PyLongWriter` created by :c:func:`PyLongWriter_Create`.

   *writer* must not be ``NULL``.

   The writer instance and the *digits* array are invalid after the call.


Optimize import for small integers
==================================

Proposed import API is efficient for large integers. Compared to
accessing directly Python internals, the proposed import API can


id: pep-0780.rst
content:
PEP: 780
Title: ABI features as environment markers
Author: Klaus Zimmermann <klaus_zimmermann@gmx.de>,
        Ralf Gommers <ralf.gommers@gmail.com>
Sponsor: Lysandros Nikolaou <lisandrosnik@gmail.com>
Discussions-To: https://discuss.python.org/t/86013
Status: Draft
Type: Standards Track
Topic: Packaging
Created: 21-Mar-2025
Python-Version: 3.14
Post-History:
  `05-Aug-2024 <https://discuss.python.org/t/60007>`__,
  `26-Mar-2025 <https://discuss.python.org/t/86013>`__,

..
    If accepted, the contents of this PEP should be integrated into the
    canonical documentation in the Python Packaging User Guide and the
    following directive should be added here:
    .. canonical-pypa-spec:: :ref:`packaging:dependency-specifiers`

Abstract
========

This PEP defines using ABI features as environment markers for project
dependencies, through a new ``sys_abi_features`` environment marker. :pep:`508`
(later moved to :ref:`packaging:dependency-specifiers`) introduced environment
markers to specify dependencies based on rules that describe when the
dependency should be used. This PEP extends the environment markers to allow
specifying dependencies based on specific ABI features of the Python
interpreter. For this, it defines a set of `ABI Features`_ and specifies how
they are made available for `environment markers <pep-780-sys_abi_features_>`_
as a new marker variable, ``sys_abi_features``.

Motivation
==========

In 2015, :pep:`508` established environment markers to specify dependencies
based on environment conditions. The development of free-threaded CPython
[#python-free-threading]_ has underlined the need for an environment marker to
discriminate between different ABI features that the interpreter was built
with. For example, currently there is no way to distinguish between a
GIL-enabled and a free-threaded CPython interpreter with an environment marker.
This leads to real world issues for the adoption of free-threading and its
incremental rollout. When a Python package is being made compatible with
free-threaded CPython, it also needs all its build and runtime dependencies to
be compatible. Capturing the first version of a dependency that is compatible
precisely in metadata is currently not possible, and increasing the minimum
version of a dependency also for the GIL-enabled build is usually undesirable
since it unnecessarily limits compatibility between packages.

Some concrete examples of such issues have been discussed in the `Environment
marker for free-threading`_ Discourse thread:

- Cython has (experimental) support for free-threading only in its master
  branch, and is used by a lot of projects that already publish ``cp313t``
  wheels. Picking up the wrong Cython version is causing a lot of obscure build
  failures and runtime crashes. It would be beneficial if the metadata could
  express that (c.f. `Require Cython Pre-release for Free-Threaded Python`_).
- CFFI has no support for free-threading yet, and `Armin Rigo, one of the
  maintainers, has stated
  <https://github.com/python-cffi/cffi/pull/143#issuecomment-2580781899>`__
  that it may be a good idea to fork ``cffi``, implement support for
  free-threading, and only come back to the CFFI project with a single large PR
  that adds support after the functionality "is reasonably well-tested (either
  as tests or, better in this case, tested by being in use in various other
  projects)". There are a lot of projects that depend on ``cffi``. They are
  likely fine to start depending on a fork for free-threading only, however
  depending on a fork for >=3.13 or for all Python versions seems like a much
  larger ask, and more disruptive for distribution packagers.

While these concrete examples may be addressed later this year by Cython and
CFFI making compatible releases, the same issue is going to repeat further up
the stack. The free-threading rollout is expected to take several years, and an
environment marker for free-threading will make that rollout significantly
easier.

Another important ABI feature that is not yet covered by environment markers is
the bitness of the interpreter. In most cases, the ``sys_platform`` or
``platform_system`` markers are enough, because there is only a single bitness
in use per platform. This is not the case on Windows however: both 32-bit and
64-bit Python interpreters are widely used on x86-64 Windows. Not being able to
distinguish between the two may be relevant for packages that provide compiled
extensions. For example, SciPy does not provide ``win32`` wheels (it isn't able
to due to the lack of a suitable 32-bit compiler toolchain with Fortran
support). Those wheels lacking can be awkward especially for projects where
SciPy is an optional dependency only. In that case, it would be useful to be
able to specify that SciPy is required *unless* the interpreter is a 32-bit one
on Windows (c.f. `Require SciPy Unless on 32-bit win32`_), to avoid failed
from-source installations due to the missing wheels.


Rationale
=========

The intention of this PEP is to introduce its core features with minimal impact
on the existing ecosystem. The existing grammar proposed in :pep:`508` lends
itself to a straightforward extension to include the new environment marker.

A Forward Looking View on Free-Threaded Python
----------------------------------------------

:pep:`703`, the accepted proposal for free threading, states that the rollout
of free-threaded Python should be gradual, which has been clarified by the
Python Steering Council in `the PEP 703 acceptance post`_ to mean a three stage
process over multiple releases. It is therefore important to make sure that the
mechanisms in this PEP are useable for Python interpreters where either
free-threading or non-free-threading could be the default or the only option.

At the time of writing, free-threaded Python is in Phase I: experimental phase.
In this phase, there is an acute need for the proposed environment markers to
help with the transition to free-threaded Python as package authors gradually
add support.

As the number of packages with support increases, and particularly during
Phase II: Supported-but-not-default phase, we still anticipate a strong need
for the environment markers to help with the transition.

As free-threaded Python enters into Phase III: Default phase, the need for the
environment markers will decrease, though at this point it is not clear that
the GIL-enabled Python will be completely phased out (it may remain available
as a non standard build option). If it persists, the inverse need for the ABI
feature detection may arise.

Indeed, in all three phases it may be necessary for package authors to choose
specific versions of their dependencies based on the ABI features, with a shift
from GIL-enabled as default to free-threading as default over time.

The ABI features are designed with this in mind to guarantee usefulness and
simplicity for the foreseeable future in a changing Python ecosystem.

Relation to Other PEPs
----------------------

This PEP extends environment markers with set semantics for ABI features.
:pep:`751#additions-to-marker-expression-syntax` includes a similar extension
for lock file specific environment markers; although the two have been
developed indepedently, they are compatible where they overlap in terms of the
new set semantics.

Specification
=============

The keywords "**MUST**", "**MUST NOT**", "**REQUIRED**", "**SHALL**",
"**SHALL NOT**", "**SHOULD**", "**SHOULD NOT**", "**RECOMMENDED**", "**MAY**",
and "**OPTIONAL**"" in this document are to be interpreted as described in
:rfc:`RFC 2119 <2119>`.

ABI Features
------------

ABI features are intrinsic properties of the Python interpreter, expressed as
simple, understandable strings. However, not all features are equally
applicable to all Python interpreters or Python versions. For example, the
distinction between free-threaded and GIL-enabled interpreters is only relevant
for CPython 3.13 onwards, but the bitness of


id: pep-0637.rst
content:
PEP: 637
Title: Support for indexing with keyword arguments
Author: Stefano Borini
Sponsor: Steven D'Aprano
Discussions-To: python-ideas@python.org
Status: Rejected
Type: Standards Track
Created: 24-Aug-2020
Python-Version: 3.10
Post-History: 23-Sep-2020
Resolution: https://mail.python.org/archives/list/python-dev@python.org/thread/6TAQ2BEVSJNV4JM2RJYSSYFJUT3INGZD/


.. note::
   This PEP has been rejected. In general, the cost of introducing new syntax
   was not outweighed by the perceived benefits. See the link in the Resolution
   header field for details.

Abstract
========

At present keyword arguments are allowed in function calls, but not in
item access. This PEP proposes that Python be extended to allow keyword
arguments in item access.

The following example shows keyword arguments for ordinary function calls::

    >>> val = f(1, 2, a=3, b=4)

The proposal would extend the syntax to allow a similar construct
to indexing operations::

    >>> val = x[1, 2, a=3, b=4]  # getitem
    >>> x[1, 2, a=3, b=4] = val  # setitem
    >>> del x[1, 2, a=3, b=4]    # delitem

and would also provide appropriate semantics. Single- and double-star unpacking of
arguments is also provided::

    >>> val = x[*(1, 2), **{a=3, b=4}]  # Equivalent to above.

This PEP is a successor to :pep:`472`, which was rejected due to lack of
interest in 2019. Since then there's been renewed interest in the feature.

Overview
========

Background
----------

:pep:`472` was opened in 2014. The PEP detailed various use cases and was created by
extracting implementation strategies from a broad discussion on the
python-ideas mailing list, although no clear consensus was reached on which strategy
should be used.  Many corner cases have been examined more closely and felt
awkward, backward incompatible or both.

The PEP was eventually rejected in 2019 [#rejection]_ mostly
due to lack of interest for the feature despite its 5 years of existence.

However, with the introduction of type hints in :pep:`484` the
square bracket notation has been used consistently to enrich the typing
annotations, e.g. to specify a list of integers as Sequence[int]. Additionally,
there has been an expanded growth of packages for data analysis such as pandas
and xarray, which use names to describe columns in a table (pandas) or axis in
an nd-array (xarray). These packages allow users to access specific data by
names, but cannot currently use index notation ([]) for this functionality.

As a result, a renewed interest in a more flexible syntax that would allow for
named information has been expressed occasionally in many different threads on
python-ideas, recently by Caleb Donovick [#request-1]_ in 2019 and Andras
Tantos [#request-2]_ in 2020. These requests prompted a strong activity on the
python-ideas mailing list, where the various options have been re-discussed and
a general consensus on an implementation strategy has now been reached.

Use cases
---------

The following practical use cases present different cases where a keyword
specification would improve notation and provide additional value:

1. To provide a more communicative meaning to the index, preventing e.g. accidental
   inversion of indexes::

       >>> grid_position[x=3, y=5, z=8]
       >>> rain_amount[time=0:12, location=location]
       >>> matrix[row=20, col=40]

2. To enrich the typing notation with keywords, especially during the use of generics::

       def function(value: MyType[T=int]):

3. In some domain, such as computational physics and chemistry, the use of a
   notation such as ``Basis[Z=5]`` is a Domain Specific Language notation to represent
   a level of accuracy::

       >>> low_accuracy_energy = computeEnergy(molecule, BasisSet[Z=3])

4. Pandas currently uses a notation such as::

       >>> df[df['x'] == 1]

   which could be replaced with ``df[x=1]``.

5. xarray has named dimensions. Currently these are handled with functions .isel::

       >>> data.isel(row=10)  # Returns the tenth row

   which could also be replaced with ``data[row=10]``. A more complex example::

       >>> # old syntax
       >>> da.isel(space=0, time=slice(None, 2))[...] = spam
       >>> # new syntax
       >>> da[space=0, time=:2] = spam

   Another example::

       >>> # old syntax
       >>> ds["empty"].loc[dict(lon=5, lat=6)] = 10
       >>> # new syntax
       >>> ds["empty"][lon=5, lat=6] = 10

       >>> # old syntax
       >>> ds["empty"].loc[dict(lon=slice(1, 5), lat=slice(3, None))] = 10
       >>> # new syntax
       >>> ds["empty"][lon=1:5, lat=6:] = 10

6. Functions/methods whose argument is another function (plus its
   arguments) need some way to determine which arguments are destined for
   the target function, and which are used to configure how they run the
   target. This is simple (if non-extensible) for positional parameters,
   but we need some way to distinguish these for keywords. [#trio-run]_

   An indexed notation would afford a Pythonic way to pass keyword
   arguments to these functions without cluttering the caller's code.

   ::

     >>> # Let's start this example with basic syntax without keywords.
     >>> # the positional values are arguments to `func` while
     >>> # `name=` is processed by `trio.run`.
     >>> trio.run(func, value1, value2, name="func")
     >>> # `trio.run` ends up calling `func(value1, value2)`.

     >>> # If we want/need to pass value2 by keyword (keyword-only argument,
     >>> # additional arguments that won't break backwards compatibility ...),
     >>> # currently we need to resort to functools.partial:
     >>> trio.run(functools.partial(func, param2=value2), value1, name="func")
     >>> trio.run(functools.partial(func, value1, param2=value2), name="func")

     >>> # One possible workaround is to convert `trio.run` to an object
     >>> # with a `__call__` method, and use an "option" helper,
     >>> trio.run.option(name="func")(func, value1, param2=value2)
     >>> # However, foo(bar)(baz) is uncommon and thus disruptive to the reader.
     >>> # Also, you need to remember the name of the `option` method.

     >>> # This PEP allows us to replace `option` with `__getitem__`.
     >>> # The call is now shorter, more mnemonic, and looks+works like typing
     >>> trio.run[name="func"](func, value1, param2=value2)

7. Availability of star arguments would benefit :pep:`646` Variadic Generics,
   especially in the forms ``a[*x]`` and ``a[*x, *y, p, q, *z]``. The PEP details
   exactly this notation in its "Unpacking: Star Operator" section.

It is important to note that how the notation is interpreted is up to the
implementation. This PEP only defines and dictates the behavior of Python
regarding passed keyword arguments, not how these arguments should be
interpreted and used by the implementing class.

Current status of indexing operation
------------------------------------

Before detailing the new syntax and semantics to the indexing notation, it is
relevant to analyse how the indexing notation works today, in which contexts,
and how it is different from a function call.

Subscripting ``obj[x]`` is, effectively, an alternate and specialised form of
function call syntax with a number of differences and restrictions compared to
``obj(x)``. The current Python syntax focuses exclusively on position to express
the index, and also contains syntactic sugar to refer to non-punctiform
selection (slices). Some common examples::

    >>> a[3]       # returns the fourth element of 'a'
    >>> a[1:10:2]  # slice notation (extract a non-trivial data subset)
    >>> a[3, 2]    # multiple indexes (for multidimensional arrays)

This translates into a ``__(get|set|del)item__`` dunder call which is passed a single
parameter containing the index (for ``__getitem__`` and ``__delitem__``) or two parameters
containing index and value (for ``__setitem__``).

The behavior of the indexing call is fundamentally different from a function call
in various aspects:

The first difference is in meaning to 


id: pep-0623.rst
content:
PEP: 623
Title: Remove wstr from Unicode
Author: Inada Naoki <songofacandy@gmail.com>
BDFL-Delegate: Victor Stinner <vstinner@python.org>
Discussions-To: https://mail.python.org/archives/list/python-dev@python.org/thread/BO2TQHSXWL2RJMINWQQRBF5LANDDJNHH/
Status: Final
Type: Standards Track
Created: 25-Jun-2020
Python-Version: 3.10
Resolution: https://mail.python.org/archives/list/python-dev@python.org/thread/VQKDIZLZ6HF2MLTNCUFURK2IFTXVQEYA/


Abstract
========

:pep:`393` deprecated some unicode APIs, and introduced ``wchar_t *wstr``,
and ``Py_ssize_t wstr_length`` in the Unicode structure to support
these deprecated APIs.

This PEP is planning removal of ``wstr``, and ``wstr_length`` with
deprecated APIs using these members by Python 3.12.

Deprecated APIs which doesn't use the members are out of scope because
they can be removed independently.


Motivation
==========

Memory usage
------------

``str`` is one of the most used types in Python. Even most simple ASCII
strings have a ``wstr`` member. It consumes 8 bytes per string on 64-bit
systems.


Runtime overhead
----------------

To support legacy Unicode object, many Unicode APIs must call
``PyUnicode_READY()``.

We can remove this overhead too by dropping support of legacy Unicode
object.


Simplicity
----------

Supporting legacy Unicode object makes the Unicode implementation more
complex.
Until we drop legacy Unicode object, it is very hard to try other
Unicode implementation like UTF-8 based implementation in PyPy.


Rationale
=========

Python 4.0 is not scheduled yet
-------------------------------

:pep:`393` introduced efficient internal representation of Unicode and
removed border between "narrow" and "wide" build of Python.

:pep:`393` was implemented in Python 3.3 which is released in 2012. Old
APIs were deprecated since then, and the removal was scheduled in
Python 4.0.

Python 4.0 was expected as next version of Python 3.9 when :pep:`393`
was accepted. But the next version of Python 3.9 is Python 3.10,
not 4.0. This is why this PEP schedule the removal plan again.


Python 2 reached EOL
--------------------

Since Python 2 didn't have :pep:`393` Unicode implementation, legacy
APIs might help C extension modules supporting both of Python 2 and 3.

But Python 2 reached the EOL in 2020. We can remove legacy APIs kept
for compatibility with Python 2.


Plan
====

Python 3.9
----------

These macros and functions are marked as deprecated, using
``Py_DEPRECATED`` macro.

* ``Py_UNICODE_WSTR_LENGTH()``
* ``PyUnicode_GET_SIZE()``
* ``PyUnicode_GetSize()``
* ``PyUnicode_GET_DATA_SIZE()``
* ``PyUnicode_AS_UNICODE()``
* ``PyUnicode_AS_DATA()``
* ``PyUnicode_AsUnicode()``
* ``_PyUnicode_AsUnicode()``
* ``PyUnicode_AsUnicodeAndSize()``
* ``PyUnicode_FromUnicode()``


Python 3.10
-----------

* Following macros, enum members are marked as deprecated.
  ``Py_DEPRECATED(3.10)`` macro are used as possible. But they
  are deprecated only in comment and document if the macro can
  not be used easily.

  * ``PyUnicode_WCHAR_KIND``
  * ``PyUnicode_READY()``
  * ``PyUnicode_IS_READY()``
  * ``PyUnicode_IS_COMPACT()``

* ``PyUnicode_FromUnicode(NULL, size)`` and
  ``PyUnicode_FromStringAndSize(NULL, size)`` emit
  ``DeprecationWarning`` when ``size > 0``.

* ``PyArg_ParseTuple()`` and ``PyArg_ParseTupleAndKeywords()`` emit
  ``DeprecationWarning`` when ``u``, ``u#``, ``Z``, and ``Z#`` formats are used.


Python 3.12
-----------

* Following members are removed from the Unicode structures:

  * ``wstr``
  * ``wstr_length``
  * ``state.compact``
  * ``state.ready``

* The ``PyUnicodeObject`` structure is removed.

* Following macros and functions, and enum members are removed:

  * ``Py_UNICODE_WSTR_LENGTH()``
  * ``PyUnicode_GET_SIZE()``
  * ``PyUnicode_GetSize()``
  * ``PyUnicode_GET_DATA_SIZE()``
  * ``PyUnicode_AS_UNICODE()``
  * ``PyUnicode_AS_DATA()``
  * ``PyUnicode_AsUnicode()``
  * ``_PyUnicode_AsUnicode()``
  * ``PyUnicode_AsUnicodeAndSize()``
  * ``PyUnicode_FromUnicode()``
  * ``PyUnicode_WCHAR_KIND``
  * ``PyUnicode_READY()``
  * ``PyUnicode_IS_READY()``
  * ``PyUnicode_IS_COMPACT()``

* ``PyUnicode_FromStringAndSize(NULL, size))`` raises
  ``RuntimeError`` when ``size > 0``.

* ``PyArg_ParseTuple()`` and ``PyArg_ParseTupleAndKeywords()`` raise
  ``SystemError`` when ``u``, ``u#``, ``Z``, and ``Z#`` formats are used,
  as other unsupported format character.


Discussion
==========

* `Draft PEP: Remove wstr from Unicode
  <https://mail.python.org/archives/list/python-dev@python.org/thread/BO2TQHSXWL2RJMINWQQRBF5LANDDJNHH/#BO2TQHSXWL2RJMINWQQRBF5LANDDJNHH>`_
* `When can we remove wchar_t* cache from string?
  <https://mail.python.org/archives/list/python-dev@python.org/thread/7JVC3IKS2V73K36ISEJAAWMRFN2T4KKR/#7JVC3IKS2V73K36ISEJAAWMRFN2T4KKR>`_
* `PEP 623: Remove wstr from Unicode object #1462
  <https://github.com/python/peps/pull/1462>`_


References
==========

* `bpo-38604: Schedule Py_UNICODE API removal
  <https://bugs.python.org/issue38604>`_
* `bpo-36346: Prepare for removing the legacy Unicode C API
  <https://bugs.python.org/issue36346>`_
* `bpo-30863: Rewrite PyUnicode_AsWideChar() and
  PyUnicode_AsWideCharString() <https://bugs.python.org/issue30863>`_:
  They no longer cache the ``wchar_t*`` representation of string
  objects.


Copyright
=========

This document has been placed in the public domain.


id: pep-0353.rst
content:
PEP: 353
Title: Using ssize_t as the index type
Author: Martin von LÃ¶wis <martin@v.loewis.de>
Status: Final
Type: Standards Track
Created: 18-Dec-2005
Python-Version: 2.5
Post-History:


Abstract
========

In Python 2.4, indices of sequences are restricted to the C type
int. On 64-bit machines, sequences therefore cannot use the full
address space, and are restricted to 2**31 elements. This PEP proposes
to change this, introducing a platform-specific index type
Py_ssize_t. An implementation of the proposed change is in
http://svn.python.org/projects/python/branches/ssize_t.


Rationale
=========

64-bit machines are becoming more popular, and the size of main memory
increases beyond 4GiB. On such machines, Python currently is limited,
in that sequences (strings, unicode objects, tuples, lists,
array.arrays, ...)  cannot contain more than 2GiElements.

Today, very few machines have memory to represent larger lists: as
each pointer is 8B (in a 64-bit machine), one needs 16GiB to just hold
the pointers of such a list; with data in the list, the memory
consumption grows even more.  However, there are three container types
for which users request improvements today:

* strings (currently restricted to 2GiB)
* mmap objects (likewise; plus the system typically
  won't keep the whole object in memory concurrently)
* Numarray objects (from Numerical Python)

As the proposed change will cause incompatibilities on 64-bit
machines, it should be carried out while such machines are not in wide
use (IOW, as early as possible).


Specification
=============

A new type Py_ssize_t is introduced, which has the same size as the
compiler's size_t type, but is signed. It will be a typedef for
ssize_t where available.

The internal representation of the length fields of all container
types is changed from int to ssize_t, for all types included in the
standard distribution.  In particular, PyObject_VAR_HEAD is changed to
use Py_ssize_t, affecting all extension modules that use that macro.

All occurrences of index and length parameters and results are changed
to use Py_ssize_t, including the sequence slots in type objects, and
the buffer interface.

New conversion functions PyInt_FromSsize_t and PyInt_AsSsize_t, are
introduced. PyInt_FromSsize_t will transparently return a long int
object if the value exceeds the LONG_MAX; PyInt_AsSsize_t will
transparently process long int objects.

New function pointer typedefs ssizeargfunc, ssizessizeargfunc,
ssizeobjargproc, ssizessizeobjargproc, and lenfunc are introduced. The
buffer interface function types are now called readbufferproc,
writebufferproc, segcountproc, and charbufferproc.

A new conversion code 'n' is introduced for PyArg_ParseTuple
Py_BuildValue, PyObject_CallFunction and PyObject_CallMethod.
This code operates on Py_ssize_t.

The conversion codes 's#' and 't#' will output Py_ssize_t
if the macro PY_SSIZE_T_CLEAN is defined before Python.h
is included, and continue to output int if that macro
isn't defined.

At places where a conversion from size_t/Py_ssize_t to
int is necessary, the strategy for conversion is chosen
on a case-by-case basis (see next section).

To prevent loading extension modules that assume a 32-bit
size type into an interpreter that has a 64-bit size type,
Py_InitModule4 is renamed to Py_InitModule4_64.


Conversion guidelines
=====================

Module authors have the choice whether they support this PEP in their
code or not; if they support it, they have the choice of different
levels of compatibility.

If a module is not converted to support this PEP, it will continue to
work unmodified on a 32-bit system.  On a 64-bit system, compile-time
errors and warnings might be issued, and the module might crash the
interpreter if the warnings are ignored.

Conversion of a module can either attempt to continue using int
indices, or use Py_ssize_t indices throughout.

If the module should continue to use int indices, care must be taken
when calling functions that return Py_ssize_t or size_t, in
particular, for functions that return the length of an object (this
includes the strlen function and the sizeof operator). A good compiler
will warn when a Py_ssize_t/size_t value is truncated into an int.
In these cases, three strategies are available:

*  statically determine that the size can never exceed an int
   (e.g. when taking the sizeof a struct, or the strlen of
   a file pathname). In this case, write::

     some_int = Py_SAFE_DOWNCAST(some_value, Py_ssize_t, int);

   This will add an assertion in debug mode that the value
   really fits into an int, and just add a cast otherwise.

*  statically determine that the value shouldn't overflow an
   int unless there is a bug in the C code somewhere. Test
   whether the value is smaller than INT_MAX, and raise an
   InternalError if it isn't.
*  otherwise, check whether the value fits an int, and raise
   a ValueError if it doesn't.

The same care must be taken for tp_as_sequence slots, in
addition, the signatures of these slots change, and the
slots must be explicitly recast (e.g. from intargfunc
to ssizeargfunc). Compatibility with previous Python
versions can be achieved with the test::

 #if PY_VERSION_HEX < 0x02050000 && !defined(PY_SSIZE_T_MIN)
 typedef int Py_ssize_t;
 #define PY_SSIZE_T_MAX INT_MAX
 #define PY_SSIZE_T_MIN INT_MIN
 #endif

and then using Py_ssize_t in the rest of the code. For
the tp_as_sequence slots, additional typedefs might
be necessary; alternatively, by replacing::

 PyObject* foo_item(struct MyType* obj, int index)
 {
   ...
 }

with::

 PyObject* foo_item(PyObject* _obj, Py_ssize_t index)
 {
    struct MyType* obj = (struct MyType*)_obj;
    ...
 }

it becomes possible to drop the cast entirely; the type
of foo_item should then match the sq_item slot in all
Python versions.

If the module should be extended to use Py_ssize_t indices, all usages
of the type int should be reviewed, to see whether it should be
changed to Py_ssize_t. The compiler will help in finding the spots,
but a manual review is still necessary.

Particular care must be taken for PyArg_ParseTuple calls:
they need all be checked for s# and t# converters, and
PY_SSIZE_T_CLEAN must be defined before including Python.h
if the calls have been updated accordingly.

Fredrik Lundh has written a scanner_ which checks the code
of a C module for usage of APIs whose signature has changed.

.. _scanner: http://svn.effbot.python-hosting.com/stuff/sandbox/python/ssizecheck.py


Discussion
==========

Why not size_t
--------------

An initial attempt to implement this feature tried to use
size_t. It quickly turned out that this cannot work: Python
uses negative indices in many places (to indicate counting
from the end). Even in places where size_t would be usable,
too many reformulations of code where necessary, e.g. in
loops like::

  for(index = length-1; index >= 0; index--)

This loop will never terminate if index is changed from
int to size_t.

Why not Py_intptr_t
-------------------

Conceptually, Py_intptr_t and Py_ssize_t are different things:
Py_intptr_t needs to be the same size as void*, and Py_ssize_t
the same size as size_t. These could differ, e.g. on machines
where pointers have segment and offset. On current flat-address
space machines, there is no difference, so for all practical
purposes, Py_intptr_t would have worked as well.

Doesn't this break much code?
-----------------------------

With the changes proposed, code breakage is fairly
minimal. On a 32-bit system, no code will break, as
Py_ssize_t is just a typedef for int.

On a 64-bit system, the compiler will warn in many
places. If these warnings are ignored, the code will
continue to work as long as the container sizes don't
exceed 2**31, i.e. it will work nearly as good as
it does currently. There are two exceptions to this
statement: if the extension module implements the
sequence protocol, it must be updated, or the calling
conventions will be wrong. The other exception is
the places where


id: pep-0435.rst
content:
PEP: 435
Title: Adding an Enum type to the Python standard library
Author: Barry Warsaw <barry@python.org>,
        Eli Bendersky <eliben@gmail.com>,
        Ethan Furman <ethan@stoneleaf.us>
Status: Final
Type: Standards Track
Created: 23-Feb-2013
Python-Version: 3.4
Post-History: 23-Feb-2013, 02-May-2013
Replaces: 354
Resolution: https://mail.python.org/pipermail/python-dev/2013-May/126112.html


Abstract
========

This PEP proposes adding an enumeration type to the Python standard library.

An enumeration is a set of symbolic names bound to unique, constant values.
Within an enumeration, the values can be compared by identity, and the
enumeration itself can be iterated over.


Status of discussions
=====================

The idea of adding an enum type to Python is not new - :pep:`354` is a
previous attempt that was rejected in 2005.  Recently a new set of discussions
was initiated [3]_ on the ``python-ideas`` mailing list.  Many new ideas were
proposed in several threads; after a lengthy discussion Guido proposed adding
``flufl.enum`` to the standard library [4]_.  During the PyCon 2013 language
summit the issue was discussed further.  It became clear that many developers
want to see an enum that subclasses ``int``, which can allow us to replace
many integer constants in the standard library by enums with friendly string
representations, without ceding backwards compatibility.  An additional
discussion among several interested core developers led to the proposal of
having ``IntEnum`` as a special case of ``Enum``.

The key dividing issue between ``Enum`` and ``IntEnum`` is whether comparing
to integers is semantically meaningful.  For most uses of enumerations, it's
a **feature** to reject comparison to integers; enums that compare to integers
lead, through transitivity, to comparisons between enums of unrelated types,
which isn't desirable in most cases.  For some uses, however, greater
interoperability with integers is desired. For instance, this is the case for
replacing existing standard library constants (such as ``socket.AF_INET``)
with enumerations.

Further discussion in late April 2013 led to the conclusion that enumeration
members should belong to the type of their enum: ``type(Color.red) == Color``.
Guido has pronounced a decision on this issue [5]_, as well as related issues
of not allowing to subclass enums [6]_, unless they define no enumeration
members [7]_.

The PEP was accepted by Guido on May 10th, 2013 [1]_.


Motivation
==========

*[Based partly on the Motivation stated in* :pep:`354`\ *]*

The properties of an enumeration are useful for defining an immutable, related
set of constant values that may or may not have a semantic meaning.  Classic
examples are days of the week (Sunday through Saturday) and school assessment
grades ('A' through 'D', and 'F').  Other examples include error status values
and states within a defined process.

It is possible to simply define a sequence of values of some other basic type,
such as ``int`` or ``str``, to represent discrete arbitrary values.  However,
an enumeration ensures that such values are distinct from any others including,
importantly, values within other enumerations, and that operations without
meaning ("Wednesday times two") are not defined for these values.  It also
provides a convenient printable representation of enum values without requiring
tedious repetition while defining them (i.e. no ``GREEN = 'green'``).


Module and type name
====================

We propose to add a module named ``enum`` to the standard library.  The main
type exposed by this module is ``Enum``.  Hence, to import the ``Enum`` type
user code will run::

    >>> from enum import Enum


Proposed semantics for the new enumeration type
===============================================

Creating an Enum
----------------

Enumerations are created using the class syntax, which makes them easy to read
and write.  An alternative creation method is described in `Functional API`_.
To define an enumeration, subclass ``Enum`` as follows::

    >>> from enum import Enum
    >>> class Color(Enum):
    ...     red = 1
    ...     green = 2
    ...     blue = 3

**A note on nomenclature**: we call ``Color`` an *enumeration* (or *enum*)
and ``Color.red``, ``Color.green`` are *enumeration members* (or
*enum members*).  Enumeration members also have *values* (the value of
``Color.red`` is ``1``, etc.)

Enumeration members have human readable string representations::

    >>> print(Color.red)
    Color.red

...while their ``repr`` has more information::

    >>> print(repr(Color.red))
    <Color.red: 1>

The *type* of an enumeration member is the enumeration it belongs to::

    >>> type(Color.red)
    <Enum 'Color'>
    >>> isinstance(Color.green, Color)
    True
    >>>

Enums also have a property that contains just their item name::

    >>> print(Color.red.name)
    red

Enumerations support iteration, in definition order::

    >>> class Shake(Enum):
    ...   vanilla = 7
    ...   chocolate = 4
    ...   cookies = 9
    ...   mint = 3
    ...
    >>> for shake in Shake:
    ...   print(shake)
    ...
    Shake.vanilla
    Shake.chocolate
    Shake.cookies
    Shake.mint

Enumeration members are hashable, so they can be used in dictionaries and sets::

    >>> apples = {}
    >>> apples[Color.red] = 'red delicious'
    >>> apples[Color.green] = 'granny smith'
    >>> apples
    {<Color.red: 1>: 'red delicious', <Color.green: 2>: 'granny smith'}


Programmatic access to enumeration members
------------------------------------------

Sometimes it's useful to access members in enumerations programmatically (i.e.
situations where ``Color.red`` won't do because the exact color is not known
at program-writing time).  ``Enum`` allows such access::

    >>> Color(1)
    <Color.red: 1>
    >>> Color(3)
    <Color.blue: 3>

If you want to access enum members by *name*, use item access::

    >>> Color['red']
    <Color.red: 1>
    >>> Color['green']
    <Color.green: 2>


Duplicating enum members and values
-----------------------------------

Having two enum members with the same name is invalid::

    >>> class Shape(Enum):
    ...   square = 2
    ...   square = 3
    ...
    Traceback (most recent call last):
    ...
    TypeError: Attempted to reuse key: square

However, two enum members are allowed to have the same value.  Given two members
A and B with the same value (and A defined first), B is an alias to A.  By-value
lookup of the value of A and B will return A.  By-name lookup of B will also
return A::

    >>> class Shape(Enum):
    ...   square = 2
    ...   diamond = 1
    ...   circle = 3
    ...   alias_for_square = 2
    ...
    >>> Shape.square
    <Shape.square: 2>
    >>> Shape.alias_for_square
    <Shape.square: 2>
    >>> Shape(2)
    <Shape.square: 2>

Iterating over the members of an enum does not provide the aliases::

    >>> list(Shape)
    [<Shape.square: 2>, <Shape.diamond: 1>, <Shape.circle: 3>]

The special attribute ``__members__`` is an ordered dictionary mapping names
to members.  It includes all names defined in the enumeration, including the
aliases::

    >>> for name, member in Shape.__members__.items():
    ...   name, member
    ...
    ('square', <Shape.square: 2>)
    ('diamond', <Shape.diamond: 1>)
    ('circle', <Shape.circle: 3>)
    ('alias_for_square', <Shape.square: 2>)

The ``__members__`` attribute can be used for detailed programmatic access to
the enumeration members.  For example, finding all the aliases::

    >>> [name for name, member in Shape.__members__.items() if member.name != name]
    ['alias_for_square']

Comparisons
-----------

Enumeration members are compared by identity::

    >>> Color.red is Color.red
    True
    >>> Color.red is Color.blue
    False
    >>> Color.red is not Color.blue
    True

Ordered comparisons between enumeration values are *not* supported.  Enums are
not integers (but see `IntEnum`_ below)::

    >>> Color.red < Color.blue
    Traceback


id: pep-0421.rst
content:
PEP: 421
Title: Adding sys.implementation
Author: Eric Snow <ericsnowcurrently@gmail.com>
BDFL-Delegate: Barry Warsaw
Status: Final
Type: Standards Track
Created: 26-Apr-2012
Python-Version: 3.3
Post-History: 26-Apr-2012
Resolution: https://mail.python.org/pipermail/python-dev/2012-May/119683.html


Abstract
========

This PEP introduces a new attribute for the ``sys`` module:
``sys.implementation``.  The attribute holds consolidated information
about the implementation of the running interpreter.  Thus
``sys.implementation`` is the source to which the standard library may
look for implementation-specific information.

The proposal in this PEP is in line with a broader emphasis on making
Python friendlier to alternate implementations.  It describes the new
variable and the constraints on what that variable contains.  The PEP
also explains some immediate use cases for ``sys.implementation``.


Motivation
==========

For a number of years now, the distinction between Python-the-language
and CPython (the reference implementation) has been growing.  Most of
this change is due to the emergence of Jython, IronPython, and PyPy as
viable alternate implementations of Python.

Consider, however, the nearly two decades of CPython-centric Python
(i.e. most of its existence).  That focus has understandably
contributed to quite a few CPython-specific artifacts both in the
standard library and exposed in the interpreter.  Though the core
developers have made an effort in recent years to address this, quite
a few of the artifacts remain.

Part of the solution is presented in this PEP: a single namespace in
which to consolidate implementation specifics.  This will help focus
efforts to differentiate the implementation specifics from the
language.  Additionally, it will foster a multiple-implementation
mindset.


Proposal
========

We will add a new attribute to the ``sys`` module, called
``sys.implementation``, as an object with attribute-access (as opposed
to a mapping).  It will contain implementation-specific information.

The attributes of this object will remain fixed during interpreter
execution and through the course of an implementation version.  This
ensures behaviors don't change between versions which depend on
attributes of ``sys.implementation``.

The object has each of the attributes described in the `Required
Attributes`_ section below.  Those attribute names will never start
with an underscore.  The standard library and the language definition
will rely only on those required attributes.

This proposal takes a conservative approach in requiring only a small
number of attributes.  As more become appropriate, they may be added
with discretion, as described in `Adding New Required Attributes`_.

While this PEP places no other constraints on ``sys.implementation``,
it also recommends that no one rely on capabilities outside those
described here.  The only exception to that recommendation is for
attributes starting with an underscore.  Implementers may use those
as appropriate to store per-implementation data.


Required Attributes
-------------------

These are attributes in ``sys.implementation`` on which the standard
library and language definition will rely, meaning implementers must
define them:

**name**
   A lower-case identifier representing the implementation.  Examples
   include 'pypy', 'jython', 'ironpython', and 'cpython'.

**version**
   The version of the implementation, as opposed to the version of the
   language it implements.  This value conforms to the format described
   in `Version Format`_.

**hexversion**
   The version of the implementation in the same hexadecimal format as
   ``sys.hexversion``.

**cache_tag**
   A string used for the :pep:`3147` cache tag.  It would
   normally be a composite of the name and version (e.g. 'cpython-33'
   for CPython 3.3).  However, an implementation may explicitly use a
   different cache tag.  If ``cache_tag`` is set to None, it indicates
   that module caching should be disabled.


Adding New Required Attributes
------------------------------

In time more required attributes will be added to
``sys.implementation``.  However, each must have a meaningful use case
across all Python implementations in order to be considered.  This is
made most clear by a use case in the standard library or language
specification.

All proposals for new required attributes will go through the normal
PEP process.  Such a PEP need not be long, just long enough.  It will
need to sufficiently spell out the rationale for the new attribute,
its use cases, and the impact it will have on the various Python
implementations.


Version Format
--------------

A main point of ``sys.implementation`` is to contain information that
will be used internally in the standard library.  In order to
facilitate the usefulness of the version attribute, its value should
be in a consistent format across implementations.

As such, the format of ``sys.implementation.version`` will follow that
of ``sys.version_info``, which is effectively a named tuple.  It is a
familiar format and generally consistent with normal version format
conventions.


Rationale
=========

The status quo for implementation-specific information gives us that
information in a more fragile, harder to maintain way.  It is spread
out over different modules or inferred from other information, as we
see with `platform.python_implementation()`_.

This PEP is the main alternative to that approach.  It consolidates
the implementation-specific information into a single namespace and
makes explicit that which was implicit.


Type Considerations
-------------------

It's very easy to get bogged down in discussions about the type of
``sys.implementation``.  However, its purpose is to support the
standard library and language definition.  As such, there isn't much
that really matters regarding its type, as opposed to a feature that
would be more generally used.  Thus characteristics like immutability
and sequence-ness have been disregarded.

The only real choice has been between an object with attribute access
and a mapping with item access.  This PEP espouses dotted access to
reflect the relatively fixed nature of the namespace.


Non-Required Attributes
-----------------------

Earlier versions of this PEP included a required attribute called
``metadata`` that held any non-required, per-implementation data
[#Alyssa]_.  However, this proved to be an unnecessary addition
considering the purpose of ``sys.implementation``.

Ultimately, non-required attributes are virtually ignored in this PEP.
They have no impact other than that careless use may collide with
future required attributes.  That, however, is but a marginal concern
for ``sys.implementation``.


Why a Part of ``sys``?
----------------------

The ``sys`` module holds the new namespace because ``sys`` is the depot
for interpreter-centric variables and functions.  Many
implementation-specific attributes are already found in ``sys``.


Why Strict Constraints on Any of the Values?
--------------------------------------------

As already noted in `Version Format`_, values in
``sys.implementation`` are intended for use by the standard library.
Constraining those values, essentially specifying an API for them,
allows them to be used consistently, regardless of how they are
otherwise implemented.  However, care should be taken to not
over-specify the constraints.


Discussion
==========

The topic of ``sys.implementation`` came up on the python-ideas list
in 2009, where the reception was broadly positive [#original]_.  I
revived the discussion recently while working on a pure-python
``imp.get_tag()`` [#revived]_.  Discussion has been ongoing
[#feedback]_.  The messages in `issue #14673`_ are also relevant.

A good part of the recent discussion centered on the type to use for
``sys.implementation``.


Use-cases
=========

platform.python_implementation()
--------------------------------

"explicit is better than implicit"

The ``plat


id: pep-0347.rst
content:
PEP: 347
Title: Migrating the Python CVS to Subversion
Author: Martin von LÃ¶wis <martin@v.loewis.de>
Discussions-To: python-dev@python.org
Status: Final
Type: Process
Created: 14-Jul-2004
Post-History: 14-Jul-2004


Abstract
========

The Python source code is currently managed in a CVS repository on
sourceforge.net.  This PEP proposes to move it to a Subversion
repository on svn.python.org.


Rationale
=========

This change has two aspects: moving from CVS to Subversion, and moving
from SourceForge to python.org.  For each, a rationale will be given.


Moving to Subversion
--------------------

CVS has a number of limitations that have been eliminated by
Subversion.  For the development of Python, the most notable
improvements are:

- the ability to rename files and directories, and to remove
  directories, while keeping the history of these files.

- support for change sets (sets of correlated changes to multiple
  files) through global revision numbers.  Change sets are
  transactional.

- atomic, fast tagging: a cvs tag might take many minutes; a
  Subversion tag (svn cp) will complete quickly, and atomically.
  Likewise, branches are very efficient.

- support for offline diffs, which is useful when creating patches.


Moving to python.org
--------------------

SourceForge has kindly provided an important infrastructure for the
past years.  Unfortunately, the attention that SF received has also
caused repeated overload situations in the past, to which the SF
operators could not always respond in a timely manner.  In particular,
for CVS, they had to reduce the load on the primary CVS server by
introducing a second, read-only CVS server for anonymous access.  This
server is regularly synchronized, but lags behind the read-write CVS
repository between synchronizations.  As a result, users without
commit access can see recent changes to the repository only after a
delay.

On python.org, it would be possible to make the repository accessible
for anonymous access.


Migration Procedure
===================

To move the Python CVS repository, the following steps need to be
executed.  The steps are elaborated upon in the following sections.

1. Collect SSH keys for all current committers, along with usernames
   to appear in commit messages.

2. At the beginning of the migration, announce that the repository on
   SourceForge closed.

3. 24 hours after the last commit, download the CVS repository.

4. Convert the CVS repository into a Subversion repository.

5. Publish the repository with write access for committers, and
   read-only anonymous access.

6. Disable CVS access on SF.


Collect SSH keys
----------------

After some discussion, svn+ssh was selected as the best method
for write access to the repository. Developers can continue to
use their SSH keys, but they must be installed on python.org.

In order to avoid having to create a new Unix user for each
developer, a single account should be used, with command=
attributes in the authorized_keys files.

The lines in the authorized_keys file should read like this
(wrapped for better readability)::

  command="/usr/bin/svnserve --root=/svnroot -t
  --tunnel-user='<username>'",no-port-forwarding,
  no-X11-forwarding,no-agent-forwarding,no-pty
  ssh-dss <key> <comment>

As the usernames, the real names should be used instead of
the SF account names, so that people can be better identified
in log messages.

Administrator Access
--------------------

Administrator access to the pythondev account should be granted
to all current admins of the Python SF project. To distinguish
between shell login and svnserve login, admins need to maintain
two keys. Using OpenSSH, the following procedure can be
used to create a second key::

  cd .ssh
  ssh-keygen -t DSA -f pythondev -C <user>@pythondev
  vi config

In the config file, the following lines need to be added::

  Host pythondev
    Hostname dinsdale.python.org
    User pythondev
    IdentityFile ~/.ssh/pythondev

Then, shell login becomes possible through "ssh pythondev".

Downloading the CVS Repository
------------------------------

The CVS repository can be downloaded from

    http://cvs.sourceforge.net/cvstarballs/python-cvsroot.tar.bz2

Since this tarball is generated only once a day, some time must pass
after the repository freeze before the tarball can be picked up.  It
should be verified that the last commit, as recorded on the
python-commits mailing list, is indeed included in the tarball.

After the conversion, the converted CVS tarball should be kept
forever on www.python.org/archive/python-cvsroot-<date>.tar.bz2


Converting the CVS Repository
-----------------------------

The Python CVS repository contains two modules: distutils and python.
The python module is further structured into dist and nondist,
where dist only contains src (the python code proper). nondist
contains various subdirectories.

These should be reorganized in the Subversion repository to get
shorter URLs, following the <project>/{trunk,tags,branches}
structure.  A project will be created for each nondist directory,
plus for src (called python), plus distutils.  Reorganizing the
repository is best done in the CVS tree, as shown below.

The fsfs backend should be used as the repository format (which
requires Subversion 1.1).  The fsfs backend has the advantage of being
more backup-friendly, as it allows incremental repository backups,
without requiring any dump commands to be run.

The conversion should be done using the cvs2svn utility, available
e.g. in the cvs2svn Debian package.  As cvs2svn does not currently
support the project/trunk structure, each project needs to be
converted separately.  To get each conversion result into a separate
directory in the target repository, svnadmin load must be used.

Subversion has a different view on binary-vs-text files than CVS.
To correctly carry the CVS semantics forward, svn:eol-style should
be set to native on all files that are not marked binary in the
CVS.

In summary, the conversion script is::

  #!/bin/sh
  rm cvs2svn-*
  rm -rf python py.new
  tar xjf python-cvsroot.tar.bz2
  rm -rf python/CVSROOT
  svnadmin create --fs-type fsfs py.new
  mv python/python python/orig
  mv python/orig/dist/src python/python
  mv python/orig/nondist/* python
  # nondist/nondist is empty
  rmdir python/nondist
  rm -rf python/orig
  for a in python/*
  do
    b=`basename $a`
    cvs2svn -q --dump-only --encoding=latin1 --force-branch=cnri-16-start \
    --force-branch=descr-branch --force-branch=release152p1-patches \
    --force-tag=r16b1 $a
    svn mkdir -m"Conversion to SVN" file:///`pwd`/py.new/$b
    svnadmin load -q --parent-dir $b py.new < cvs2svn-dump
    rm cvs2svn-dump
  done

Sample results of this conversion are available at

    http://www.dcl.hpi.uni-potsdam.de/pysvn/


Publish the Repository
------------------------

The repository should be published at http://svn.python.org/projects.
Read-write access should be granted to all current SF committers
through svn+ssh://pythondev@svn.python.org/;
read-only anonymous access through WebDAV should also be
granted.

As an option, websvn (available e.g. from the Debian websvn package)
could be provided. Unfortunately, in the test installation, websvn
breaks because it runs out of memory.

The current SF project admins should get write access to the
authorized_keys2 file of the pythondev account.


Disable CVS
-----------

It appears that CVS cannot be disabled entirely.  Only the user
interface can be removed from the project page; the repository itself
remains available.  If desired, write access to the python and
distutils modules can be disabled through a CVS commitinfo entry.


Discussion
==========

Several alternatives had been suggested to the procedure above.
The rejected alternatives are shortly discussed here:

- create multiple repositories, one for python and one for
  distutils. This would have allowed even shorter URLs, but
  was rejected because a single repository suppo


id: pep-0409.rst
content:
PEP: 409
Title: Suppressing exception context
Author: Ethan Furman <ethan@stoneleaf.us>
Status: Final
Type: Standards Track
Created: 26-Jan-2012
Python-Version: 3.3
Post-History: 30-Aug-2002, 01-Feb-2012, 03-Feb-2012
Superseded-By: 415
Resolution: https://mail.python.org/pipermail/python-dev/2012-February/116136.html


Abstract
========

One of the open issues from :pep:`3134` is suppressing context:  currently
there is no way to do it.  This PEP proposes one.


Rationale
=========

There are two basic ways to generate exceptions:

1) Python does it (buggy code, missing resources, ending loops, etc.)

2) manually (with a raise statement)

When writing libraries, or even just custom classes, it can become
necessary to raise exceptions; moreover it can be useful, even
necessary, to change from one exception to another.  To take an example
from my dbf module::

    try:
        value = int(value)
    except Exception:
        raise DbfError(...)

Whatever the original exception was (``ValueError``, ``TypeError``, or
something else) is irrelevant.  The exception from this point on is a
``DbfError``, and the original exception is of no value.  However, if
this exception is printed, we would currently see both.


Alternatives
============
Several possibilities have been put forth:

* ``raise as NewException()``

  Reuses the ``as`` keyword; can be confusing since we are not really
  reraising the originating exception

* ``raise NewException() from None``

  Follows existing syntax of explicitly declaring the originating
  exception

* ``exc = NewException(); exc.__context__ = None; raise exc``

  Very verbose way of the previous method

* ``raise NewException.no_context(...)``

  Make context suppression a class method.

All of the above options will require changes to the core.


Proposal
========

I propose going with the second option::

    raise NewException from None

It has the advantage of using the existing pattern of explicitly setting
the cause::

    raise KeyError() from NameError()

but because the cause is ``None`` the previous context is not displayed
by the default exception printing routines.


Implementation Discussion
=========================

Note: after acceptance of this PEP, a cleaner implementation mechanism
was proposed and accepted in :pep:`415`. Refer to that PEP for more
details on the implementation actually used in Python 3.3.

Currently, ``None`` is the default for both ``__context__`` and ``__cause__``.
In order to support ``raise ... from None`` (which would set ``__cause__`` to
``None``) we need a different default value for ``__cause__``.  Several ideas
were put forth on how to implement this at the language level:

* Overwrite the previous exception information (side-stepping the issue and
  leaving ``__cause__`` at ``None``).

  Rejected as this can seriously hinder debugging due to
  `poor error messages`_.

* Use one of the boolean values in ``__cause__``:  ``False`` would be the
  default value, and would be replaced when ``from ...`` was used with the
  explicitly chained exception or ``None``.

  Rejected as this encourages the use of two different objects types for
  ``__cause__`` with one of them (boolean) not allowed to have the full range
  of possible values (``True`` would never be used).

* Create a special exception class, ``__NoException__``.

  Rejected as possibly confusing, possibly being mistakenly raised by users,
  and not being a truly unique value as ``None``, ``True``, and ``False`` are.

* Use ``Ellipsis`` as the default value (the ``...`` singleton).

  Accepted.

  Ellipses are commonly used in English as place holders when words are
  omitted.  This works in our favor here as a signal that ``__cause__`` is
  omitted, so look in ``__context__`` for more details.

  Ellipsis is not an exception, so cannot be raised.

  There is only one Ellipsis, so no unused values.

  Error information is not thrown away, so custom code can trace the entire
  exception chain even if the default code does not.


Language Details
================

To support ``raise Exception from None``, ``__context__`` will stay as it is,
but ``__cause__`` will start out as ``Ellipsis`` and will change to ``None``
when the ``raise Exception from None`` method is used.

============================================  ==================  =======================================
form                                          __context__         __cause__
============================================  ==================  =======================================
raise                                         ``None``            ``Ellipsis``
reraise                                       previous exception  ``Ellipsis``
reraise from ``None`` | ``ChainedException``  previous exception  ``None`` | explicitly chained exception
============================================  ==================  =======================================

The default exception printing routine will then:

* If ``__cause__`` is ``Ellipsis`` the ``__context__`` (if any) will be
  printed.

* If ``__cause__`` is ``None`` the ``__context__`` will not be printed.

* if ``__cause__`` is anything else, ``__cause__`` will be printed.

In both of the latter cases the exception chain will stop being followed.

Because the default value for ``__cause__`` is now ``Ellipsis`` and ``raise
Exception from Cause`` is simply syntactic sugar for::

    _exc = NewException()
    _exc.__cause__ = Cause()
    raise _exc

``Ellipsis``, as well as ``None``, is now allowed as a cause::

    raise Exception from Ellipsis


Patches
=======

There is a patch for CPython implementing this attached to `Issue 6210`_.


References
==========

Discussion and refinements in this `thread on python-dev`_.

.. _poor error messages:
   http://bugs.python.org/msg152294
.. _issue 6210:
   http://bugs.python.org/issue6210
.. _Thread on python-dev:
   https://mail.python.org/pipermail/python-dev/2012-January/115838.html


Copyright
=========

This document has been placed in the public domain.


id: pep-0390.rst
content:
PEP: 390
Title: Static metadata for Distutils
Author: Tarek ZiadÃ© <tarek@ziade.org>
BDFL-Delegate: Alyssa Coghlan
Discussions-To: distutils-sig@python.org
Status: Rejected
Type: Standards Track
Topic: Packaging
Created: 10-Oct-2009
Python-Version: 2.7, 3.2
Post-History:
Resolution: https://mail.python.org/pipermail/distutils-sig/2013-April/020597.html

Abstract
========

This PEP describes a new section and a new format for the ``setup.cfg`` file,
that allows describing the Metadata of a package without using ``setup.py``.


Rejection Notice
================

As distutils2 is no longer going to be incorporated into the standard
library, this PEP was rejected by Alyssa Coghlan in late April, 2013.

A replacement PEP based on :pep:`426` (metadata 2.0) will be created that
defines the minimum amount of information needed to generate an sdist
archive given a source tarball or VCS checkout.


Rationale
=========

Today, if you want to list all the Metadata of a distribution (see :pep:`314`)
that is not installed, you need to use the ``setup.py`` command line interface.

So, basically, you download it, and run::

   $ python setup.py --name
   Distribute

   $ python setup.py --version
   0.6.4

Where ``name`` and ``version`` are metadata fields. This is working fine but
as soon as the developers add more code in ``setup.py``, this feature might
break or in worst cases might do unwanted things on the target system.

Moreover, when an OS packager wants to get the metadata of a distribution
he is re-packaging, he might encounter some problems to understand
the ``setup.py`` file he's working with.

So the rationale of this PEP is to provide a way to declare the metadata
in a static configuration file alongside ``setup.py`` that doesn't require
any third party code to run.


Adding a ``metadata`` section in ``setup.cfg``
==============================================

The first thing we want to introduce is a ``[metadata]`` section, in the
``setup.cfg`` file, that may contain any field from the Metadata::

   [metadata]
   name = Distribute
   version = 0.6.4

The ``setup.cfg`` file is used to avoid adding yet another configuration
file to work with in Distutils.

This file is already read by Distutils when a command is executed, and
if the ``metadata`` section is found, it will be used to fill the metadata
fields. If an option that corresponds to a Metadata field is given to
``setup()``, it will override the value that was possibly present in
``setup.cfg``.

Notice that ``setup.py`` is still used and can be required to define some
options that are not part of the Metadata fields. For instance, the
``sdist`` command can use options like ``packages`` or ``scripts``.


Multi-lines values
==================

Some Metadata fields can have multiple values. To keep ``setup.cfg`` compatible
with ``ConfigParser`` and the :rfc:`822` ``LONG HEADER FIELDS`` (see section 3.1.1),
these are expressed with ``,``-separated values::

    requires = pywin32, bar > 1.0, foo

When this variable is read, the values are parsed and transformed into a list:
``['pywin32', 'bar > 1.0', 'foo']``.


Context-dependant sections
==========================

The ``metadata`` section will also be able to use context-dependant sections.

A context-dependant section is a section with a condition about the execution
environment. Here's some examples::

   [metadata]
   name = Distribute
   version = 0.6.4

   [metadata:sys_platform == 'win32']
   requires = pywin32, bar > 1.0
   obsoletes = pywin31

   [metadata:os_machine == 'i386']
   requires = foo

   [metadata:python_version == '2.4' or python_version == '2.5']
   requires = bar

   [metadata:'linux' in sys_platform]
   requires = baz

Every ``[metadata:condition]`` section will be used only if the condition
is met when the file is read. The background motivation for these
context-dependant sections is to be able to define requirements that varies
depending on the platform the distribution might be installed on.
(see :pep:`314`).

The micro-language behind this is the simplest possible: it compares only
strings, with the ``==`` and ``in`` operators (and their opposites), and
with the ability to combine expressions. It makes it also easy to understand
to non-pythoneers.

The pseudo-grammar is ::

    EXPR [in|==|!=|not in]Â EXPR [or|and] ...

where ``EXPR`` belongs to any of those:

- python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1])
- os_name = os.name
- sys_platform = sys.platform
- platform_version = platform.version()
- platform_machine = platform.machine()
- a free string, like ``2.4``, or ``win32``

Notice that ``in`` is restricted to strings, meaning that it is not possible
to use other sequences like tuples or lists on the right side.

Distutils will provide a function that is able to generate the metadata
of a distribution, given a ``setup.cfg`` file, for the execution environment::

   >>> from distutils.util import local_metadata
   >>> local_metadata('setup.cfg')
   <DistributionMetadata instance>

This means that a vanilla Python will be able to read the metadata of a
package without running any third party code.

Notice that this feature is not restricted to the ``metadata`` namespace.
Consequently, any other section can be extended with such context-dependant
sections.

Impact on PKG-INFO generation and PEP 314
=========================================

When ``PKG-INFO`` is generated by Distutils, every field that relies on a
condition will have that condition written at the end of the line, after a
``;`` separator::

    Metadata-Version: 1.2
    Name: distribute
    Version: 0.6.4
    ...
    Requires: pywin32, bar > 1.0; sys_platform == 'win32'
    Requires: foo; os_machine == 'i386'
    Requires: bar; python_version == '2.4' or python_version == '2.5'
    Requires: baz; 'linux' in sys_platform
    Obsoletes = pywin31; sys_platform == 'win32'
    ...
    Classifier: Development Status :: 5 - Production/Stable
    Classifier: Intended Audience :: Developers
    Classifier: License :: OSI Approved :: Python Software Foundation License

Notice that this file can be opened with the ``DistributionMetadata`` class.
This class will be able to use the micro-language using the execution
environment.

Let's run in on a ``Python 2.5 i386 Linux``::

    >>> from distutils.dist import DistributionMetadata
    >>> metadata = DistributionMetadata('PKG_INFO')
    >>> metadata.get_requires()
    ['foo', 'bar', 'baz']

The execution environment can be overridden in case we want to get the metadata
for another environment::

    >>> env = {'python_version': '2.4',
    ...        'os_name': 'nt',
    ...        'sys_platform': 'win32',
    ...        'platform_version': 'MVCC++ 6.0'
    ...        'platform_machine': 'i386'}
    ...
    >>> metadata = DistributionMetadata('PKG_INFO', environment=env)
    >>> metadata.get_requires()
    ['bar > 1.0', 'foo', 'bar']

:pep:`314` is changed accordingly, meaning that each field will be able to
have that extra condition marker.

Compatibility
=============

This change is based on a new metadata ``1.2`` format meaning that
Distutils will be able to distinguish old PKG-INFO files from new ones.

The ``setup.cfg`` file change will stay ``ConfigParser``-compatible and
will not break existing ``setup.cfg`` files.

Limitations
===========

We are not providing ``<`` and ``>`` operators at this time, and
``python_version`` is a regular string. This implies using ``or`` operators
when a section needs to be restricted to a couple of Python versions.
Although, if :pep:`386` is accepted, ``python_version`` could be changed
internally into something comparable with strings, and
``<`` and ``>`` operators introduced.

Last, if a distribution is unable to set all metadata fields in ``setup.cfg``,
that's fine, the fields will be set to ``UNKNOWN`` when ``local_metadata`` is
called. Getting ``UNKNOWN`` values will mean that it might be necessary to
run the ``setup.py`` command line interface to


id: pep-0384.rst
content:
PEP: 384
Title: Defining a Stable ABI
Author: Martin von LÃ¶wis <martin@v.loewis.de>
Status: Final
Type: Standards Track
Created: 17-May-2009
Python-Version: 3.2
Post-History:

.. canonical-doc:: :ref:`python:stable` (user docs) and
                   :ref:`devguide:c-api` (development docs)

Abstract
========

Currently, each feature release introduces a new name for the
Python DLL on Windows, and may cause incompatibilities for extension
modules on Unix. This PEP proposes to define a stable set of API
functions which are guaranteed to be available for the lifetime
of Python 3, and which will also remain binary-compatible across
versions. Extension modules and applications embedding Python
can work with different feature releases as long as they restrict
themselves to this stable ABI.

Rationale
=========

The primary source of ABI incompatibility are changes to the lay-out
of in-memory structures. For example, the way in which string interning
works, or the data type used to represent the size of an object, have
changed during the life of Python 2.x. As a consequence, extension
modules making direct access to fields of strings, lists, or tuples,
would break if their code is loaded into a newer version of the
interpreter without recompilation: offsets of other fields may have
changed, making the extension modules access the wrong data.

In some cases, the incompatibilities only affect internal objects of
the interpreter, such as frame or code objects. For example, the way
line numbers are represented has changed in the 2.x lifetime, as has
the way in which local variables are stored (due to the introduction
of closures). Even though most applications probably never used these
objects, changing them had required to change the PYTHON_API_VERSION.

On Linux, changes to the ABI are often not much of a problem: the
system will provide a default Python installation, and many extension
modules are already provided pre-compiled for that version. If additional
modules are needed, or additional Python versions, users can typically
compile them themselves on the system, resulting in modules that use
the right ABI.

On Windows, multiple simultaneous installations of different Python
versions are common, and extension modules are compiled by their
authors, not by end users. To reduce the risk of ABI incompatibilities,
Python currently introduces a new DLL name pythonXY.dll for each
feature release, whether or not ABI incompatibilities actually exist.

With this PEP, it will be possible to reduce the dependency of binary
extension modules on a specific Python feature release, and applications
embedding Python can be made work with different releases.

Specification
=============

The ABI specification falls into two parts: an API specification,
specifying what function (groups) are available for use with the
ABI, and a linkage specification specifying what libraries to link
with. The actual ABI (layout of structures in memory, function
calling conventions) is not specified, but implied by the
compiler. As a recommendation, a specific ABI is recommended for
selected platforms.

During evolution of Python, new ABI functions will be added.
Applications using them will then have a requirement on a minimum
version of Python; this PEP provides no mechanism for such
applications to fall back when the Python library is too old.

Terminology
-----------

Applications and extension modules that want to use this ABI
are collectively referred to as "applications" from here on.

Header Files and Preprocessor Definitions
-----------------------------------------

Applications shall only include the header file Python.h (before
including any system headers), or, optionally, include pyconfig.h, and
then Python.h.

During the compilation of applications, the preprocessor macro
Py_LIMITED_API must be defined. Doing so will hide all definitions
that are not part of the ABI.

Structures
----------

Only the following structures and structure fields are accessible to
applications:

- PyObject (ob_refcnt, ob_type)
- PyVarObject (ob_base, ob_size)
- PyMethodDef (ml_name, ml_meth, ml_flags, ml_doc)
- PyMemberDef (name, type, offset, flags, doc)
- PyGetSetDef (name, get, set, doc, closure)
- PyModuleDefBase (ob_base, m_init, m_index, m_copy)
- PyModuleDef (m_base, m_name, m_doc, m_size, m_methods, m_traverse,
  m_clear, m_free)
- PyStructSequence_Field (name, doc)
- PyStructSequence_Desc (name, doc, fields, sequence)
- PyType_Slot (see below)
- PyType_Spec (see below)

The accessor macros to these fields (Py_REFCNT, Py_TYPE, Py_SIZE)
are also available to applications.

The following types are available, but opaque (i.e. incomplete):

- PyThreadState
- PyInterpreterState
- struct _frame
- struct symtable
- struct _node
- PyWeakReference
- PyLongObject
- PyTypeObject

Type Objects
------------

The structure of type objects is not available to applications;
declaration of "static" type objects is not possible anymore
(for applications using this ABI).
Instead, type objects get created dynamically. To allow an
easy creation of types (in particular, to be able to fill out
function pointers easily), the following structures and functions
are available::

  typedef struct{
    int slot;    /* slot id, see below */
    void *pfunc; /* function pointer */
  } PyType_Slot;

  typedef struct{
    const char* name;
    int basicsize;
    int itemsize;
    unsigned int flags;
    PyType_Slot *slots; /* terminated by slot==0. */
  } PyType_Spec;

  PyObject* PyType_FromSpec(PyType_Spec*);

To specify a slot, a unique slot id must be provided. New Python
versions may introduce new slot ids, but slot ids will never be
recycled. Slots may get deprecated, but continue to be supported
throughout Python 3.x.

The slot ids are named like the field names of the structures that
hold the pointers in Python 3.1, with an added ``Py_`` prefix (i.e.
Py_tp_dealloc instead of just tp_dealloc):

- tp_dealloc, tp_getattr, tp_setattr, tp_repr,
  tp_hash, tp_call, tp_str, tp_getattro, tp_setattro,
  tp_doc, tp_traverse, tp_clear, tp_richcompare, tp_iter,
  tp_iternext, tp_methods, tp_base, tp_descr_get, tp_descr_set,
  tp_init, tp_alloc, tp_new, tp_is_gc, tp_bases, tp_del
- nb_add nb_subtract nb_multiply nb_remainder nb_divmod nb_power
  nb_negative nb_positive nb_absolute nb_bool nb_invert nb_lshift
  nb_rshift nb_and nb_xor nb_or nb_int nb_float nb_inplace_add
  nb_inplace_subtract nb_inplace_multiply nb_inplace_remainder
  nb_inplace_power nb_inplace_lshift nb_inplace_rshift nb_inplace_and
  nb_inplace_xor nb_inplace_or nb_floor_divide nb_true_divide
  nb_inplace_floor_divide nb_inplace_true_divide nb_index
- sq_length sq_concat sq_repeat sq_item sq_ass_item
  sq_contains sq_inplace_concat sq_inplace_repeat
- mp_length mp_subscript mp_ass_subscript

The following fields cannot be set during type definition:
- tp_dict tp_mro tp_cache tp_subclasses tp_weaklist tp_print
- tp_weaklistoffset tp_dictoffset

typedefs
--------

In addition to the typedefs for structs listed above, the following
typedefs are available. Their inclusion in the ABI means that the
underlying type must not change on a platform (even though it may
differ across platforms).

- Py_uintptr_t Py_intptr_t Py_ssize_t
- unaryfunc binaryfunc ternaryfunc inquiry lenfunc ssizeargfunc
  ssizessizeargfunc ssizeobjargproc ssizessizeobjargproc objobjargproc
  objobjproc visitproc traverseproc
  destructor getattrfunc getattrofunc setattrfunc setattrofunc reprfunc
  hashfunc richcmpfunc getiterfunc iternextfunc descrgetfunc
  descrsetfunc initproc newfunc allocfunc
- PyCFunction PyCFunctionWithKeywords PyNoArgsFunction
  PyCapsule_Destructor
- getter setter
- PyOS_sighandler_t
- PyGILState_STATE
- Py_UCS4

Most notably, Py_UNICODE is not available as a typedef,
since the same Python version may use different definitions
of it on the same platform (depending on whether it uses narrow
or wide code units). Applications that need to access the contents
of a Unicod


id: pep-0385.rst
content:
PEP: 385
Title: Migrating from Subversion to Mercurial
Author: Dirkjan Ochtman <dirkjan@ochtman.nl>,
        Antoine Pitrou <solipsis@pitrou.net>,
        Georg Brandl <georg@python.org>
Status: Final
Type: Process
Created: 25-May-2009


Motivation
==========

After having decided to switch to the Mercurial DVCS, the actual
migration still has to be performed.  In the case of an important
piece of infrastructure like the version control system for a large,
distributed project like Python, this is a significant effort.  This
PEP is an attempt to describe the steps that must be taken for further
discussion.  It's somewhat similar to :pep:`347`, which discussed the
migration to SVN.

To make the most of hg, we would like to make a high-fidelity
conversion, such that (a) as much of the svn metadata as possible is
retained, and (b) all metadata is converted to formats that are common
in Mercurial.  This way, tools written for Mercurial can be optimally
used.  In order to do this, we want to use the `hgsubversion`_
software to do an initial conversion.  This hg extension is focused on
providing high-quality conversion from Subversion to Mercurial for use
in two-way correspondence, meaning it doesn't throw away as much
available metadata as other solutions.

Such a conversion also seems like a good time to reconsider the
contents of the repository and determine if some things are still
valuable.  In this spirit, the following sections also propose
discarding some of the older metadata.

.. _hgsubversion: http://bitbucket.org/durin42/hgsubversion/


Timeline
========

The current schedule for conversion milestones:

- 2011-02-24: availability of a test repo at hg.python.org

  Test commits will be allowed (and encouraged) from all committers to
  the Subversion repository.  The test repository and all test commits
  will be removed once the final conversion is done.  The server-side
  hooks will be installed for the test repository, in order to test
  buildbot, diff-email and whitespace checking integration.

- 2011-03-05: final conversion (tentative)

  Commits to the Subversion branches now maintained in Mercurial will
  be blocked.  Developers should refrain from pushing to the Mercurial
  repositories until all infrastructure is ensured to work after their
  switch over to the new repository.


Transition plan
===============

Branch strategy
---------------

Mercurial has two basic ways of using branches: cloned branches, where
each branch is kept in a separate repository, and named branches,
where each revision keeps metadata to note on which branch it belongs.
The former makes it easier to distinguish branches, at the expense of
requiring more disk space on the client.  The latter makes it a little
easier to switch between branches, but all branch names are a
persistent part of history. [1]_

Differences between named branches and cloned branches:

* Tags in a different (maintenance) clone aren't available in the
  local clone
* Clones with named branches will be larger, since they contain more
  data

We propose to use named branches for release branches and adopt cloned
branches for feature branches.


History management
------------------

In order to minimize the loss of information due to the conversion, we
propose to provide several repositories as a conversion result:

* A repository trimmed to the mainline trunk (and py3k), as well as
  past and present maintenance branches -- this is called the
  "working" repo and is where development continues.  This repository has
  all the history needed for development work, including annotating
  source files with changes back up to 1990 and other common history-digging
  operations.

  The ``default`` branch in that repo is what is known as ``py3k`` in
  Subversion, while the Subversion trunk lives on with the branch name
  ``legacy-trunk``; however in Mercurial this branch will be closed.
  Release branches are named after their major.minor version, e.g. ``3.2``.

* A repository with the full, unedited conversion of the Subversion
  repository (actually, its /python subdirectory) -- this is called
  the "historic" or "archive" repo and will be offered as a read-only
  resource. [2]_

* One more repository per active feature branch; "active" means that
  at least one core developer asks for the branch to be provided.  Each
  such repository will contain both the feature branch and all ancestor
  changesets from mainline (coming from ``trunk`` and/or ``py3k`` in SVN).

Since all branches are present in the historic repo, they can later be
extracted as separate repositories at any time should it prove to be
necessary.

The final revision map between SVN revision numbers, Mercurial changesets
and SVN branch names will be made available in a file stored in the ``Misc``
directory.  Its format is as following::

    [...]
    88483 e65daae6cf4499a0863cb7645109a4798c28d83e issue10276-snowleopard
    88484 835cb57abffeceaff0d85c2a3aa0625458dd3e31 py3k
    88485 d880f9d8492f597a030772c7485a34aadb6c4ece release32-maint
    88486 0c431b8c22f5dbeb591414c154acb7890c1809df py3k
    88487 82cda1f21396bbd10db8083ea20146d296cb630b release32-maint
    88488 8174d00d07972d6f109ed57efca8273a4d59302c release27-maint
    [...]


Converting tags
---------------

The SVN tags directory contains a lot of old stuff.  Some of these are
not, in fact, full tags, but contain only a smaller subset of the
repository.  All release tags will be kept; other tags will be
included based on requests from the developer community.  We propose
to make the tag naming scheme consistent, in this style: ``v3.2.1a2``.


Author map
----------

In order to provide user names the way they are common in hg (in the
'First Last <user@example.org>' format), we need an author map to map
cvs and svn user names to real names and their email addresses.  We
have a complete version of such a map in the migration tools
repository (not publicly accessible to avoid leaking addresses to
harvesters).  The email addresses in it might be out of date; that's
bound to happen, although it would be nice to try and have as many
people as possible review it for addresses that are out of date.  The
current version also still seems to contain some encoding problems.


Generating .hgignore
--------------------

The .hgignore file can be used in Mercurial repositories to help
ignore files that are not eligible for version control.  It does this
by employing several possible forms of pattern matching.  The current
Python repository already includes a rudimentary .hgignore file to
help with using the hg mirrors.

Since the current Python repository already includes a .hgignore file
(for use with hg mirrors), we'll just use that.  Generating full
history of the file was debated but deemed impractical (because it's
relatively hard with fairly little gain, since ignoring is less
important for older revisions).


Repository size
---------------

A bare conversion result of the current Python repository weighs 1.9
GB; although this is smaller than the Subversion repository (2.7 GB)
it is not feasible.

The size becomes more manageable by the trimming applied to the
working repository, and by a process called "revlog reordering" that
optimizes the layout of internal Mercurial storage very efficiently.

After all optimizations done, the size of the working repository is
around 180 MB on disk.  The amount of data transferred over the
network when cloning is estimated to be around 80 MB.


Other repositories
------------------

There are a number of other projects hosted in svn.python.org's
"projects" repository.  The "peps" directory will be converted along
with the main Python one.  Richard Tew has indicated that he'd like the
Stackless repository to also be converted.  What other projects in the
svn.python.org repository should be converted?

There's now an initial stab at converting the Jython repository.  The
current tip of hgsubversion unfortunately fails at some point.
Pending i


id: pep-0391.rst
content:
PEP: 391
Title: Dictionary-Based Configuration For Logging
Author: Vinay Sajip <vinay_sajip at red-dove.com>
Status: Final
Type: Standards Track
Created: 15-Oct-2009
Python-Version: 2.7, 3.2
Post-History:


Abstract
========

This PEP describes a new way of configuring logging using a dictionary
to hold configuration information.


Rationale
=========

The present means for configuring Python's logging package is either
by using the logging API to configure logging programmatically, or
else by means of ConfigParser-based configuration files.

Programmatic configuration, while offering maximal control, fixes the
configuration in Python code.  This does not facilitate changing it
easily at runtime, and, as a result, the ability to flexibly turn the
verbosity of logging up and down for different parts of a using
application is lost.  This limits the usability of logging as an aid
to diagnosing problems - and sometimes, logging is the only diagnostic
aid available in production environments.

The ConfigParser-based configuration system is usable, but does not
allow its users to configure all aspects of the logging package.  For
example, Filters cannot be configured using this system.  Furthermore,
the ConfigParser format appears to engender dislike (sometimes strong
dislike) in some quarters.  Though it was chosen because it was the
only configuration format supported in the Python standard at that
time, many people regard it (or perhaps just the particular schema
chosen for logging's configuration) as 'crufty' or 'ugly', in some
cases apparently on purely aesthetic grounds.

Recent versions of Python include JSON support in the standard
library, and this is also usable as a configuration format.  In other
environments, such as Google App Engine, YAML is used to configure
applications, and usually the configuration of logging would be
considered an integral part of the application configuration.
Although the standard library does not contain YAML support at
present, support for both JSON and YAML can be provided in a common
way because both of these serialization formats allow deserialization
to Python dictionaries.

By providing a way to configure logging by passing the configuration
in a dictionary, logging will be easier to configure not only for
users of JSON and/or YAML, but also for users of custom configuration
methods, by providing a common format in which to describe the desired
configuration.

Another drawback of the current ConfigParser-based configuration
system is that it does not support incremental configuration: a new
configuration completely replaces the existing configuration.
Although full flexibility for incremental configuration is difficult
to provide in a multi-threaded environment, the new configuration
mechanism will allow the provision of limited support for incremental
configuration.


Specification
=============

The specification consists of two parts: the API and the format of the
dictionary used to convey configuration information (i.e. the schema
to which it must conform).


Naming
------

Historically, the logging package has not been :pep:`8` conformant.
At some future time, this will be corrected by changing method and
function names in the package in order to conform with :pep:`8`.
However, in the interests of uniformity, the proposed additions to the
API use a naming scheme which is consistent with the present scheme
used by logging.


API
---

The logging.config module will have the following addition:

* A function, called ``dictConfig()``, which takes a single argument
  - the dictionary holding the configuration.  Exceptions will be
  raised if there are errors while processing the dictionary.

It will be possible to customize this API - see the section on `API
Customization`_.  `Incremental configuration`_ is covered in its own
section.


Dictionary Schema - Overview
----------------------------

Before describing the schema in detail, it is worth saying a few words
about object connections, support for user-defined objects and access
to external and internal objects.


Object connections
''''''''''''''''''

The schema is intended to describe a set of logging objects - loggers,
handlers, formatters, filters - which are connected to each other in
an object graph.  Thus, the schema needs to represent connections
between the objects.  For example, say that, once configured, a
particular logger has attached to it a particular handler.  For the
purposes of this discussion, we can say that the logger represents the
source, and the handler the destination, of a connection between the
two.  Of course in the configured objects this is represented by the
logger holding a reference to the handler.  In the configuration dict,
this is done by giving each destination object an id which identifies
it unambiguously, and then using the id in the source object's
configuration to indicate that a connection exists between the source
and the destination object with that id.

So, for example, consider the following YAML snippet::

    formatters:
      brief:
        # configuration for formatter with id 'brief' goes here
      precise:
        # configuration for formatter with id 'precise' goes here
    handlers:
      h1: #This is an id
       # configuration of handler with id 'h1' goes here
       formatter: brief
      h2: #This is another id
       # configuration of handler with id 'h2' goes here
       formatter: precise
    loggers:
      foo.bar.baz:
        # other configuration for logger 'foo.bar.baz'
        handlers: [h1, h2]

(Note: YAML will be used in this document as it is a little more
readable than the equivalent Python source form for the dictionary.)

The ids for loggers are the logger names which would be used
programmatically to obtain a reference to those loggers, e.g.
``foo.bar.baz``.  The ids for Formatters and Filters can be any string
value (such as ``brief``, ``precise`` above) and they are transient,
in that they are only meaningful for processing the configuration
dictionary and used to determine connections between objects, and are
not persisted anywhere when the configuration call is complete.

Handler ids are treated specially, see the section on
`Handler Ids`_, below.

The above snippet indicates that logger named ``foo.bar.baz`` should
have two handlers attached to it, which are described by the handler
ids ``h1`` and ``h2``. The formatter for ``h1`` is that described by id
``brief``, and the formatter for ``h2`` is that described by id
``precise``.


User-defined objects
''''''''''''''''''''

The schema should support user-defined objects for handlers, filters
and formatters.  (Loggers do not need to have different types for
different instances, so there is no support - in the configuration -
for user-defined logger classes.)

Objects to be configured will typically be described by dictionaries
which detail their configuration.  In some places, the logging system
will be able to infer from the context how an object is to be
instantiated, but when a user-defined object is to be instantiated,
the system will not know how to do this.  In order to provide complete
flexibility for user-defined object instantiation, the user will need
to provide a 'factory' - a callable which is called with a
configuration dictionary and which returns the instantiated object.
This will be signalled by an absolute import path to the factory being
made available under the special key ``'()'``.  Here's a concrete
example::

    formatters:
      brief:
        format: '%(message)s'
      default:
        format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
        datefmt: '%Y-%m-%d %H:%M:%S'
      custom:
          (): my.package.customFormatterFactory
          bar: baz
          spam: 99.9
          answer: 42

The above YAML snippet defines three formatters.  The first, with id
``brief``, is a standard ``logging.Formatter`` instance with the
specified format string.  The second, with id ``default``, has a


id: pep-0408.rst
content:
PEP: 408
Title: Standard library __preview__ package
Author: Alyssa Coghlan <ncoghlan@gmail.com>,
        Eli Bendersky <eliben@gmail.com>
Status: Rejected
Type: Standards Track
Created: 07-Jan-2012
Python-Version: 3.3
Post-History: 27-Jan-2012
Resolution: https://mail.python.org/pipermail/python-dev/2012-January/115962.html


Abstract
========

The process of including a new module into the Python standard library is
hindered by the API lock-in and promise of backward compatibility implied by
a module being formally part of Python.  This PEP proposes a transitional
state for modules - inclusion in a special ``__preview__`` package for the
duration of a minor release (roughly 18 months) prior to full acceptance into
the standard library.  On one hand, this state provides the module with the
benefits of being formally part of the Python distribution.  On the other hand,
the core development team explicitly states that no promises are made with
regards to the module's eventual full inclusion into the standard library,
or to the stability of its API, which may change for the next release.


PEP Rejection
=============

Based on his experience with a similar "labs" namespace in Google App Engine,
Guido has rejected this PEP [3]_ in favour of the simpler alternative of
explicitly marking provisional modules as such in their documentation.

If a module is otherwise considered suitable for standard library inclusion,
but some concerns remain regarding maintainability or certain API details,
then the module can be accepted on a provisional basis. While it is considered
an unlikely outcome, such modules *may* be removed from the standard library
without a deprecation period if the lingering concerns prove well-founded.

As part of the same announcement, Guido explicitly accepted Matthew
Barnett's 'regex' module [4]_ as a provisional addition to the standard
library for Python 3.3 (using the 'regex' name, rather than as a drop-in
replacement for the existing 're' module).


Proposal - the __preview__ package
==================================

Whenever the Python core development team decides that a new module should be
included into the standard library, but isn't entirely sure about whether the
module's API is optimal, the module can be placed in a special package named
``__preview__`` for a single minor release.

In the next minor release, the module may either be "graduated" into the
standard library (and occupy its natural place within its namespace, leaving the
``__preview__`` package), or be rejected and removed entirely from the Python
source tree.  If the module ends up graduating into the standard library after
spending a minor release in ``__preview__``, its API may be changed according
to accumulated feedback.  The core development team explicitly makes no
guarantees about API stability and backward compatibility of modules in
``__preview__``.

Entry into the ``__preview__`` package marks the start of a transition of the
module into the standard library.  It means that the core development team
assumes responsibility of the module, similarly to any other module in the
standard library.


Which modules should go through ``__preview__``
-----------------------------------------------

We expect most modules proposed for addition into the Python standard library
to go through a minor release in ``__preview__``. There may, however, be some
exceptions, such as modules that use a pre-defined API (for example ``lzma``,
which generally follows the API of the existing ``bz2`` module), or modules
with an API that has wide acceptance in the Python development community.

In any case, modules that are proposed to be added to the standard library,
whether via ``__preview__`` or directly, must fulfill the acceptance conditions
set by :pep:`2`.

It is important to stress that the aim of this proposal is not to make the
process of adding new modules to the standard library more difficult.  On the
contrary, it tries to provide a means to add *more* useful libraries.  Modules
which are obvious candidates for entry can be added as before.  Modules which
due to uncertainties about the API could be stalled for a long time now have
a means to still be distributed with Python, via an incubation period in the
``__preview__`` package.


Criteria for "graduation"
-------------------------

In principle, most modules in the ``__preview__`` package should eventually
graduate to the stable standard library.  Some reasons for not graduating are:

* The module may prove to be unstable or fragile, without sufficient developer
  support to maintain it.
* A much better alternative module may be found during the preview release

Essentially, the decision will be made by the core developers on a per-case
basis.  The point to emphasize here is that a module's appearance in the
``__preview__`` package in some release does not guarantee it will continue
being part of Python in the next release.


Example
-------

Suppose the ``example`` module is a candidate for inclusion in the standard
library, but some Python developers aren't convinced that it presents the best
API for the problem it intends to solve.  The module can then be added to the
``__preview__`` package in release ``3.X``, importable via::

    from __preview__ import example

Assuming the module is then promoted to the standard library proper in
release ``3.X+1``, it will be moved to a permanent location in the library::

    import example

And importing it from ``__preview__`` will no longer work.


Rationale
=========

Benefits for the core development team
--------------------------------------

Currently, the core developers are really reluctant to add new interfaces to
the standard library.  This is because as soon as they're published in a
release, API design mistakes get locked in due to backward compatibility
concerns.

By gating all major API additions through some kind of a preview mechanism
for a full release, we get one full release cycle of community feedback
before we lock in the APIs with our standard backward compatibility guarantee.

We can also start integrating preview modules with the rest of the standard
library early, so long as we make it clear to packagers that the preview
modules should not be considered optional.  The only difference between preview
APIs and the rest of the standard library is that preview APIs are explicitly
exempted from the usual backward compatibility guarantees.

Essentially, the ``__preview__`` package is intended to lower the risk of
locking in minor API design mistakes for extended periods of time.  Currently,
this concern can block new additions, even when the core development team
consensus is that a particular addition is a good idea in principle.


Benefits for end users
----------------------

For future end users, the broadest benefit lies in a better "out-of-the-box"
experience - rather than being told "oh, the standard library tools for task X
are horrible, download this 3rd party library instead", those superior tools
are more likely to be just be an import away.

For environments where developers are required to conduct due diligence on
their upstream dependencies (severely harming the cost-effectiveness of, or
even ruling out entirely, much of the material on PyPI), the key benefit lies
in ensuring that anything in the ``__preview__`` package is clearly under
python-dev's aegis from at least the following perspectives:

* Licensing:  Redistributed by the PSF under a Contributor Licensing Agreement.
* Documentation: The documentation of the module is published and organized via
  the standard Python documentation tools (i.e. ReST source, output generated
  with Sphinx and published on http://docs.python.org).
* Testing: The module test suites are run on the python.org buildbot fleet
  and results published via http://www.python.org/dev/buildbot.
* Issue management: Bugs and feature requests are handled on
  http://bugs.python.org
* Source control: The master re


id: pep-0420.rst
content:
PEP: 420
Title: Implicit Namespace Packages
Author: Eric V. Smith <eric@trueblade.com>
Status: Final
Type: Standards Track
Created: 19-Apr-2012
Python-Version: 3.3
Post-History:
Resolution: https://mail.python.org/pipermail/python-dev/2012-May/119651.html

Abstract
========

Namespace packages are a mechanism for splitting a single Python package
across multiple directories on disk.  In current Python versions, an algorithm
to compute the packages ``__path__`` must be formulated.  With the enhancement
proposed here, the import machinery itself will construct the list of
directories that make up the package.  This PEP builds upon previous work,
documented in :pep:`382` and :pep:`402`.  Those PEPs have since been rejected in
favor of this one.  An implementation of this PEP is at [1]_.


Terminology
===========

Within this PEP:

* "package" refers to Python packages as defined by Python's import
  statement.

* "distribution" refers to separately installable sets of Python
  modules as stored in the Python package index, and installed by
  distutils or setuptools.

* "vendor package" refers to groups of files installed by an
  operating system's packaging mechanism (e.g. Debian or Redhat
  packages install on Linux systems).

* "regular package" refers to packages as they are implemented in
  Python 3.2 and earlier.

* "portion" refers to a set of files in a single directory (possibly
  stored in a zip file) that contribute to a namespace package.

* "legacy portion" refers to a portion that uses ``__path__``
  manipulation in order to implement namespace packages.

This PEP defines a new type of package, the "namespace package".


Namespace packages today
========================

Python currently provides ``pkgutil.extend_path`` to denote a package
as a namespace package.  The recommended way of using it is to put::

    from pkgutil import extend_path
    __path__ = extend_path(__path__, __name__)

in the package's ``__init__.py``.  Every distribution needs to provide
the same contents in its ``__init__.py``, so that ``extend_path`` is
invoked independent of which portion of the package gets imported
first.  As a consequence, the package's ``__init__.py`` cannot
practically define any names as it depends on the order of the package
fragments on ``sys.path`` to determine which portion is imported
first.  As a special feature, ``extend_path`` reads files named
``<packagename>.pkg`` which allows declaration of additional portions.

setuptools provides a similar function named
``pkg_resources.declare_namespace`` that is used in the form::

    import pkg_resources
    pkg_resources.declare_namespace(__name__)

In the portion's ``__init__.py``, no assignment to ``__path__`` is
necessary, as ``declare_namespace`` modifies the package ``__path__``
through ``sys.modules``.  As a special feature, ``declare_namespace``
also supports zip files, and registers the package name internally so
that future additions to ``sys.path`` by setuptools can properly add
additional portions to each package.

setuptools allows declaring namespace packages in a distribution's
``setup.py``, so that distribution developers don't need to put the
magic ``__path__`` modification into ``__init__.py`` themselves.

See :pep:`402`'s :pep:`"The Problem" <402#the-problem>`
section for additional motivations
for namespace packages.  Note that :pep:`402` has been rejected, but the
motivating use cases are still valid.


Rationale
=========

The current imperative approach to namespace packages has led to
multiple slightly-incompatible mechanisms for providing namespace
packages.  For example, pkgutil supports ``*.pkg`` files; setuptools
doesn't.  Likewise, setuptools supports inspecting zip files, and
supports adding portions to its ``_namespace_packages`` variable,
whereas pkgutil doesn't.

Namespace packages are designed to support being split across multiple
directories (and hence found via multiple ``sys.path`` entries).  In
this configuration, it doesn't matter if multiple portions all provide
an ``__init__.py`` file, so long as each portion correctly initializes
the namespace package.  However, Linux distribution vendors (amongst
others) prefer to combine the separate portions and install them all
into the *same* file system directory.  This creates a potential for
conflict, as the portions are now attempting to provide the *same*
file on the target system - something that is not allowed by many
package managers.  Allowing implicit namespace packages means that the
requirement to provide an ``__init__.py`` file can be dropped
completely, and affected portions can be installed into a common
directory or split across multiple directories as distributions see
fit.

A namespace package will not be constrained by a fixed ``__path__``,
computed from the parent path at namespace package creation time.
Consider the standard library ``encodings`` package:

1. Suppose that ``encodings`` becomes a namespace package.

2. It sometimes gets imported during interpreter startup to
   initialize the standard io streams.

3. An application modifies ``sys.path`` after startup and wants to
   contribute additional encodings from new path entries.

4. An attempt is made to import an encoding from an ``encodings``
   portion that is found on a path entry added in step 3.

If the import system was restricted to only finding portions along the
value of ``sys.path`` that existed at the time the ``encodings``
namespace package was created, the additional paths added in step 3
would never be searched for the additional portions imported in step
4.  In addition, if step 2 were sometimes skipped (due to some runtime
flag or other condition), then the path items added in step 3 would
indeed be used the first time a portion was imported.  Thus this PEP
requires that the list of path entries be dynamically computed when
each portion is loaded.  It is expected that the import machinery will
do this efficiently by caching ``__path__`` values and only refreshing
them when it detects that the parent path has changed.  In the case of
a top-level package like ``encodings``, this parent path would be
``sys.path``.


Specification
=============

Regular packages will continue to have an ``__init__.py`` and will
reside in a single directory.

Namespace packages cannot contain an ``__init__.py``.  As a
consequence, ``pkgutil.extend_path`` and
``pkg_resources.declare_namespace`` become obsolete for purposes of
namespace package creation.  There will be no marker file or directory
for specifying a namespace package.

During import processing, the import machinery will continue to
iterate over each directory in the parent path as it does in Python
3.2.  While looking for a module or package named "foo", for each
directory in the parent path:

* If ``<directory>/foo/__init__.py`` is found, a regular package is
  imported and returned.

* If not, but ``<directory>/foo.{py,pyc,so,pyd}`` is found, a module
  is imported and returned.  The exact list of extension varies by
  platform and whether the -O flag is specified.  The list here is
  representative.

* If not, but ``<directory>/foo`` is found and is a directory, it is
  recorded and the scan continues with the next directory in the
  parent path.

* Otherwise the scan continues with the next directory in the parent
  path.

If the scan completes without returning a module or package, and at
least one directory was recorded, then a namespace package is created.
The new namespace package:

* Has a ``__path__`` attribute set to an iterable of the path strings
  that were found and recorded during the scan.

* Does not have a ``__file__`` attribute.

Note that if "import foo" is executed and "foo" is found as a
namespace package (using the above rules), then "foo" is immediately
created as a package.  The creation of the namespace package is not
deferred until a sub-level import occurs.

A namespace package is not fundamentally different from a regular
package.  It is just a d


id: pep-0346.rst
content:
PEP: 346
Title: User Defined ("``with``") Statements
Author: Alyssa Coghlan <ncoghlan@gmail.com>
Status: Withdrawn
Type: Standards Track
Created: 06-May-2005
Python-Version: 2.5
Post-History:


Abstract
========

This PEP is a combination of :pep:`310`'s "Reliable Acquisition/Release
Pairs" with the "Anonymous Block Statements" of Guido's :pep:`340`.  This
PEP aims to take the good parts of :pep:`340`, blend them with parts of
:pep:`310` and rearrange the lot into an elegant whole.  It borrows from
various other PEPs in order to paint a complete picture, and is
intended to stand on its own.


Author's Note
=============

During the discussion of :pep:`340`, I maintained drafts of this PEP as
PEP 3XX on my own website (since I didn't have CVS access to update a
submitted PEP fast enough to track the activity on python-dev).

Since the first draft of this PEP, Guido wrote :pep:`343` as a simplified
version of :pep:`340`.  :pep:`343` (at the time of writing) uses the exact
same semantics for the new statements as this PEP, but uses a slightly
different mechanism to allow generators to be used to write statement
templates.  However, Guido has indicated that he intends to accept a
new PEP being written by Raymond Hettinger that will integrate :pep:`288`
and :pep:`325`, and will permit a generator decorator like the one
described in this PEP to be used to write statement templates for PEP
343. The other difference was the choice of keyword ('with' versus
'do') and Guido has stated he will organise a vote on that in the
context of :pep:`343`.

Accordingly, the version of this PEP submitted for archiving on
python.org is to be WITHDRAWN immediately after submission.  :pep:`343`
and the combined generator enhancement PEP will cover the important
ideas.


Introduction
============

This PEP proposes that Python's ability to reliably manage resources
be enhanced by the introduction of a new ``with`` statement that
allows factoring out of arbitrary ``try``/``finally`` and some
``try``/``except``/``else`` boilerplate.  The new construct is called
a 'user defined statement', and the associated class definitions are
called 'statement templates'.

The above is the main point of the PEP.  However, if that was all it
said, then :pep:`310` would be sufficient and this PEP would be
essentially redundant. Instead, this PEP recommends additional
enhancements that make it natural to write these statement templates
using appropriately decorated generators.  A side effect of those
enhancements is that it becomes important to appropriately deal
with the management of resources inside generators.

This is quite similar to :pep:`343`, but the exceptions that occur are
re-raised inside the generators frame, and the issue of generator
finalisation needs to be addressed as a result.  The template
generator decorator suggested by this PEP also creates reusable
templates, rather than the single use templates of :pep:`340`.

In comparison to :pep:`340`, this PEP eliminates the ability to suppress
exceptions, and makes the user defined statement a non-looping
construct.  The other main difference is the use of a decorator to
turn generators into statement templates, and the incorporation of
ideas for addressing iterator finalisation.

If all that seems like an ambitious operation. . . well, Guido was the
one to set the bar that high when he wrote :pep:`340` :)


Relationship with other PEPs
============================

This PEP competes directly with :pep:`310`, :pep:`340` and :pep:`343`,
as those PEPs all describe alternative mechanisms for handling
deterministic resource management.

It does not compete with :pep:`342` which splits off :pep:`340`'s
enhancements related to passing data into iterators.  The associated
changes to the ``for`` loop semantics would be combined with the
iterator finalisation changes suggested in this PEP.  User defined
statements would not be affected.

Neither does this PEP compete with the generator enhancements
described in :pep:`288`.  While this PEP proposes the ability to
inject exceptions into generator frames, it is an internal
implementation detail, and does not require making that ability
publicly available to Python code.  :pep:`288` is, in part, about
making that implementation detail easily accessible.

This PEP would, however, make the generator resource release support
described in :pep:`325` redundant - iterators which require
finalisation should provide an appropriate implementation of the
statement template protocol.


User defined statements
=======================

To steal the motivating example from :pep:`310`, correct handling of a
synchronisation lock currently looks like this::

    the_lock.acquire()
    try:
        # Code here executes with the lock held
    finally:
        the_lock.release()

Like :pep:`310`, this PEP proposes that such code be able to be written
as::

    with the_lock:
        # Code here executes with the lock held

These user defined statements are primarily designed to allow easy
factoring of ``try`` blocks that are not easily converted to
functions.  This is most commonly the case when the exception handling
pattern is consistent, but the body of the ``try`` block changes.
With a user-defined statement, it is straightforward to factor out the
exception handling into a statement template, with the body of the
``try`` clause provided inline in the user code.

The term 'user defined statement' reflects the fact that the meaning
of a ``with`` statement is governed primarily by the statement
template used, and programmers are free to create their own statement
templates, just as they are free to create their own iterators for use
in ``for`` loops.


Usage syntax for user defined statements
----------------------------------------

The proposed syntax is simple::

    with EXPR1 [as VAR1]:
        BLOCK1


Semantics for user defined statements
-------------------------------------

::

    the_stmt = EXPR1
    stmt_enter = getattr(the_stmt, "__enter__", None)
    stmt_exit = getattr(the_stmt, "__exit__", None)
    if stmt_enter is None or stmt_exit is None:
        raise TypeError("Statement template required")

    VAR1 = stmt_enter() # Omit 'VAR1 =' if no 'as' clause
    exc = (None, None, None)
    try:
        try:
            BLOCK1
        except:
            exc = sys.exc_info()
            raise
    finally:
        stmt_exit(*exc)


Other than ``VAR1``, none of the local variables shown above will be
visible to user code.  Like the iteration variable in a ``for`` loop,
``VAR1`` is visible in both ``BLOCK1`` and code following the user
defined statement.

Note that the statement template can only react to exceptions, it
cannot suppress them.  See `Rejected Options`_ for an explanation as
to why.


Statement template protocol: ``__enter__``
------------------------------------------

The ``__enter__()`` method takes no arguments, and if it raises an
exception, ``BLOCK1`` is never executed.  If this happens, the
``__exit__()`` method is not called.  The value returned by this
method is assigned to VAR1 if the ``as`` clause is used.  Object's
with no other value to return should generally return ``self`` rather
than ``None`` to permit in-place creation in the ``with`` statement.

Statement templates should use this method to set up the conditions
that are to exist during execution of the statement (e.g. acquisition
of a synchronisation lock).

Statement templates which are not always usable (e.g. closed file
objects) should raise a ``RuntimeError`` if an attempt is made to call
``__enter__()`` when the template is not in a valid state.


Statement template protocol: ``__exit__``
-----------------------------------------

The ``__exit__()`` method accepts three arguments which correspond to
the three "arguments" to the ``raise`` statement: type, value, and
traceback.  All arguments are always supplied, and will be set to
``None`` if no exception occurred.  This method will be called exactly
once by the 


id: pep-0352.rst
content:
PEP: 352
Title: Required Superclass for Exceptions
Author: Brett Cannon, Guido van Rossum
Status: Final
Type: Standards Track
Created: 27-Oct-2005
Python-Version: 2.5
Post-History:


Abstract
========

In Python 2.4 and before, any (classic) class can be raised as an
exception.  The plan for 2.5 was to allow new-style classes, but this
makes the problem worse -- it would mean *any* class (or
instance) can be raised! This is a problem as it prevents any
guarantees from being made about the interface of exceptions.
This PEP proposes introducing a new superclass that all raised objects
must inherit from.  Imposing the restriction will allow a standard
interface for exceptions to exist that can be relied upon.  It also
leads to a known hierarchy for all exceptions to adhere to.

One might counter that requiring a specific base class for a
particular interface is unPythonic.  However, in the specific case of
exceptions there's a good reason (which has generally been agreed to
on python-dev): requiring hierarchy helps code that wants to *catch*
exceptions by making it possible to catch *all* exceptions explicitly
by writing ``except BaseException:`` instead of
``except *:``. [#hierarchy-good]_

Introducing a new superclass for exceptions also gives us the chance
to rearrange the exception hierarchy slightly for the better.  As it
currently stands, all exceptions in the built-in namespace inherit
from Exception.  This is a problem since this includes two exceptions
(KeyboardInterrupt and SystemExit) that often need to be excepted from
the application's exception handling: the default behavior of shutting
the interpreter down without a traceback is usually more desirable than
whatever the application might do (with the possible exception of
applications that emulate Python's interactive command loop with
``>>>`` prompt).  Changing it so that these two exceptions inherit
from the common superclass instead of Exception will make it easy for
people to write ``except`` clauses that are not overreaching and not
catch exceptions that should propagate up.

This PEP is based on previous work done for :pep:`348`.


Requiring a Common Superclass
=============================

This PEP proposes introducing a new exception named BaseException that
is a new-style class and has a single attribute, ``args``.  Below
is the code as the exception will work in Python 3.0 (how it will
work in Python 2.x is covered in the `Transition Plan`_ section)::

  class BaseException(object):

      """Superclass representing the base of the exception hierarchy.

      Provides an 'args' attribute that contains all arguments passed
      to the constructor.  Suggested practice, though, is that only a
      single string argument be passed to the constructor.

      """

      def __init__(self, *args):
          self.args = args

      def __str__(self):
          if len(self.args) == 1:
              return str(self.args[0])
          else:
              return str(self.args)

      def __repr__(self):
          return "%s(*%s)" % (self.__class__.__name__, repr(self.args))


No restriction is placed upon what may be passed in for ``args``
for backwards-compatibility reasons.  In practice, though, only
a single string argument should be used.  This keeps the string
representation of the exception to be a useful message about the
exception that is human-readable; this is why the ``__str__`` method
special-cases on length-1 ``args`` value.  Including programmatic
information (e.g., an error code number) should be stored as a
separate attribute in a subclass.

The ``raise`` statement will be changed to require that any object
passed to it must inherit from BaseException.  This will make sure
that all exceptions fall within a single hierarchy that is anchored at
BaseException [#hierarchy-good]_.  This also guarantees a basic
interface that is inherited from BaseException.  The change to
``raise`` will be enforced starting in Python 3.0 (see the `Transition
Plan`_ below).

With BaseException being the root of the exception hierarchy,
Exception will now inherit from it.


Exception Hierarchy Changes
===========================

With the exception hierarchy now even more important since it has a
basic root, a change to the existing hierarchy is called for.  As it
stands now, if one wants to catch all exceptions that signal an error
*and* do not mean the interpreter should be allowed to exit, you must
specify all but two exceptions specifically in an ``except`` clause
or catch the two exceptions separately and then re-raise them and
have all other exceptions fall through to a bare ``except`` clause::

 except (KeyboardInterrupt, SystemExit):
     raise
 except:
     ...

That is needlessly explicit.  This PEP proposes moving
KeyboardInterrupt and SystemExit to inherit directly from
BaseException.

::

  - BaseException
    |- KeyboardInterrupt
    |- SystemExit
    |- Exception
       |- (all other current built-in exceptions)

Doing this makes catching Exception more reasonable.  It would catch
only exceptions that signify errors.  Exceptions that signal that the
interpreter should exit will not be caught and thus be allowed to
propagate up and allow the interpreter to terminate.

KeyboardInterrupt has been moved since users typically expect an
application to exit when they press the interrupt key (usually Ctrl-C).
If people have overly broad ``except`` clauses the expected behaviour
does not occur.

SystemExit has been moved for similar reasons.  Since the exception is
raised when ``sys.exit()`` is called the interpreter should normally
be allowed to terminate.  Unfortunately overly broad ``except``
clauses can prevent the explicitly requested exit from occurring.

To make sure that people catch Exception most of the time, various
parts of the documentation and tutorials will need to be updated to
strongly suggest that Exception be what programmers want to use.  Bare
``except`` clauses or catching BaseException directly should be
discouraged based on the fact that KeyboardInterrupt and SystemExit
almost always should be allowed to propagate up.


Transition Plan
===============

Since semantic changes to Python are being proposed, a transition plan
is needed.  The goal is to end up with the new semantics being used in
Python 3.0 while providing a smooth transition for 2.x code.  All
deprecations mentioned in the plan will lead to the removal of the
semantics starting in the version following the initial deprecation.

Here is BaseException as implemented in the 2.x series::

  class BaseException(object):

      """Superclass representing the base of the exception hierarchy.

      The __getitem__ method is provided for backwards-compatibility
      and will be deprecated at some point.  The 'message' attribute
      is also deprecated.

      """

      def __init__(self, *args):
          self.args = args

      def __str__(self):
          return str(self.args[0]
                     if len(self.args) <= 1
                     else self.args)

      def __repr__(self):
          func_args = repr(self.args) if self.args else "()"
          return self.__class__.__name__ + func_args

      def __getitem__(self, index):
          """Index into arguments passed in during instantiation.

          Provided for backwards-compatibility and will be
          deprecated.

          """
          return self.args[index]

      def _get_message(self):
          """Method for 'message' property."""
          warnings.warn("the 'message' attribute has been deprecated "
                          "since Python 2.6")
          return self.args[0] if len(args) == 1 else ''

      message = property(_get_message,
                          doc="access the 'message' attribute; "
                              "deprecated and provided only for "
                              "backwards-compatibility")


Deprecation of features in Python 2.9 is optional.  This is because it
is not known at this time if Python 2.9


id: pep-0434.rst
content:
PEP: 434
Title: IDLE Enhancement Exception for All Branches
Author: Todd Rovito <rovitotv@gmail.com>,
        Terry Reedy <tjreedy@udel.edu>
BDFL-Delegate: Alyssa Coghlan
Status: Active
Type: Informational
Created: 16-Feb-2013
Post-History: 16-Feb-2013,
              03-Mar-2013,
              21-Mar-2013,
              30-Mar-2013
Resolution: https://mail.python.org/pipermail/python-dev/2013-March/125003.html


Abstract
========

Most CPython tracker issues are classified as behavior or enhancement.
Most behavior patches are backported to branches for existing
versions.  Enhancement patches are restricted to the default branch
that becomes the next Python version.

This PEP proposes that the restriction on applying enhancements be
relaxed for IDLE code, residing in .../Lib/idlelib/.  In practice,
this would mean that IDLE developers would not have to classify or
agree on the classification of a patch but could instead focus on what
is best for IDLE users and future IDLE development.  It would also
mean that IDLE patches would not necessarily have to be split into
'bugfix' changes and enhancement changes.

The PEP would apply to changes in existing features and addition of
small features, such as would require a new menu entry, but not
necessarily to possible major re-writes such as switching to themed
widgets or tabbed windows.


Motivation
==========

This PEP was prompted by controversy on both the tracker and pydev
list over adding Cut, Copy, and Paste to right-click context menus
(Issue 1207589, opened in 2005 [1]_; pydev thread [2]_).  The features
were available as keyboard shortcuts but not on the context menu.  It
is standard, at least on Windows, that they should be when applicable
(a read-only window would only have Copy), so users do not have to
shift to the keyboard after selecting text for cutting or copying or a
slice point for pasting.  The context menu was not documented until 10
days before the new options were added (Issue 10405 [5]_).

Normally, behavior is called a bug if it conflicts with documentation
judged to be correct.  But if there is no doc, what is the standard?
If the code is its own documentation, most IDLE issues on the tracker
are enhancement issues.  If we substitute reasonable user expectation,
(which can, of course, be its own subject of disagreement), many more
issues are behavior issues.

For context menus, people disagreed on the status of the additions --
bugfix or enhancement.  Even people who called it an enhancement
disagreed as to whether the patch should be backported.  This PEP
proposes to make the status disagreement irrelevant by explicitly
allowing more liberal backporting than for other stdlib modules.

Python does have many advanced features yet Python is well known for
being an easy computer language for beginners [3]_.  A major Python
philosophy is "batteries included" which is best demonstrated in
Python's standard library with many modules that are not typically
included with other programming languages [4]_.  IDLE is an important
"battery" in the Python toolbox because it allows a beginner to get
started quickly without downloading and configuring a third party IDE.
IDLE represents a commitment by the Python community to encouage the
use of Python as a teaching language both inside and outside of formal
educational settings.  The recommended teaching experience is to have
a learner start with IDLE.  This PEP and the work that it will enable
will allow the Python community to make that learner's experience with
IDLE awesome by making IDLE a simple tool for beginners to get started
with Python.

Rationale
=========

People primarily use IDLE by running the graphical user interface
(GUI) application, rather than by directly importing the effectively
private (undocumented) implementation modules in idlelib.  Whether
they use the shell, the editor, or both, we believe they will benefit
more from consistency across the latest releases of current Python
versions than from consistency within the bugfix releases for one
Python version.  This is especially true when existing behavior is
clearly unsatisfactory.

When people use the standard interpreter, the OS-provided frame works
the same for all Python versions.  If, for instance, Microsoft were to
upgrade the Command Prompt GUI, the improvements would be present
regardless of which Python were running within it.  Similarly, if one
edits Python code with editor X, behaviors such as the right-click
context menu and the search-replace box do not depend on the version
of Python being edited or even the language being edited.

The benefit for IDLE developers is mixed.  On the one hand, testing
more versions and possibly having to adjust a patch, especially for
2.7, is more work.  (There is, of course, the option on not
backporting everything.  For issue 12510, some changes to calltips for
classes were not included in the 2.7 patch because of issues with
old-style classes [6]_.)  On the other hand, bike-shedding can be an
energy drain.  If the obvious fix for a bug looks like an enhancement,
writing a separate bugfix-only patch is more work.  And making the
code diverge between versions makes future multi-version patches more
difficult.

These issue are illustrated by the search-and-replace dialog box.  It
used to raise an exception for certain user entries [7]_.  The
uncaught exception caused IDLE to exit.  At least on Windows, the exit
was silent (no visible traceback) and looked like a crash if IDLE was
started normally, from an icon.

Was this a bug?  IDLE Help (on the current Help submenu) just says
"Replace...  Open a search-and-replace dialog box", and a box *was*
opened.  It is not, in general, a bug for a library method to raise an
exception.  And it is not, in general, a bug for a library method to
ignore an exception raised by functions it calls.  So if we were to
adopt the 'code = doc' philosophy in the absence of detailed docs, one
might say 'No'.

However, IDLE exiting when it does not need to is definitely
obnoxious.  So four of us agreed that it should be prevented.  But
there was still the question of what to do instead?  Catch the
exception?  Just not raise the exception?  Beep?  Display an error
message box?  Or try to do something useful with the user's entry?
Would replacing a 'crash' with useful behavior be an enhancement,
limited to future Python releases?  Should IDLE developers have to ask
that?


Backwards Compatibility
=======================

For IDLE, there are three types of users who might be concerned about
back compatibility.  First are people who run IDLE as an application.
We have already discussed them above.

Second are people who import one of the idlelib modules.  As far as we
know, this is only done to start the IDLE application, and we do not
propose breaking such use.  Otherwise, the modules are undocumented
and effectively private implementations.  If an IDLE module were
defined as public, documented, and perhaps moved to the tkinter
package, it would then follow the normal rules.  (Documenting the
private interfaces for the benefit of people working on the IDLE code
is a separate issue.)

Third are people who write IDLE extensions.  The guaranteed extension
interface is given in idlelib/extension.txt.  This should be respected
at least in existing versions, and not frivolously changed in future
versions.  But there is a warning that "The extension cannot assume
much about this [EditorWindow] argument."  This guarantee should
rarely be an issue with patches, and the issue is not specific to
'enhancement' versus 'bugfix' patches.

As is happens, after the context menu patch was applied, it came up
that extensions that added items to the context menu (rare) would be
broken because the patch a) added a new item to standard rmenu_specs
and b) expected every rmenu_spec to be lengthened. It is not clear
whether this violates the guarantee, but there is a second patch that
fixes assumption b).  It should be applied when 


id: pep-0622.rst
content:
PEP: 622
Title: Structural Pattern Matching
Author: Brandt Bucher <brandt@python.org>,
        Daniel F Moisset <dfmoisset@gmail.com>,
        Tobias Kohn <kohnt@tobiaskohn.ch>,
        Ivan Levkivskyi <levkivskyi@gmail.com>,
        Guido van Rossum <guido@python.org>,
        Talin <viridia@gmail.com>
BDFL-Delegate:
Discussions-To: python-dev@python.org
Status: Superseded
Type: Standards Track
Created: 23-Jun-2020
Python-Version: 3.10
Post-History: 23-Jun-2020, 08-Jul-2020
Superseded-By: 634


Abstract
========

This PEP proposes to add a **pattern matching statement** to Python,
inspired by similar syntax found in Scala, Erlang, and other languages.

Patterns and shapes
-------------------

The **pattern syntax** builds on Pythonâs existing syntax for sequence
unpacking (e.g., ``a, b = value``).

A ``match`` statement compares a value (the **subject**)
to several different shapes (the **patterns**) until a shape fits.
Each pattern describes the type and structure of the accepted values
as well as the variables where to capture its contents.

Patterns can specify the shape to be:

- a sequence to be unpacked, as already mentioned
- a mapping with specific keys
- an instance of a given class with (optionally) specific attributes
- a specific value
- a wildcard

Patterns can be composed in several ways.

Syntax
------

Syntactically, a ``match`` statement contains:

- a *subject* expression
- one or more ``case`` clauses

Each ``case`` clause specifies:

- a pattern (the overall shape to be matched)
- an optional âguardâ (a condition to be checked if the pattern matches)
- a code block to be executed if the case clause is selected

Motivation
----------

The rest of the PEP:

- motivates why we believe pattern matching makes a good addition to Python
- explains our design choices
- contains a precise syntactic and runtime specification
- gives guidance for static type checkers (and one small addition to the ``typing`` module)
- discusses the main objections and alternatives that have been
  brought up during extensive discussion of the proposal, both within
  the group of authors and in the python-dev community

Finally, we discuss some possible extensions that might be considered
in the future, once the community has ample experience with the
currently proposed syntax and semantics.

Overview
========

Patterns are a new syntactical category with their own rules
and special cases. Patterns mix input (given values) and output
(captured variables) in novel ways. They may take a little time to
use effectively. The authors have provided
a brief introduction to the basic concepts here. Note that this section
is not intended to be complete or entirely accurate.

Pattern, a new syntactic construct, and destructuring
-----------------------------------------------------

A new syntactic construct called **pattern** is introduced in this
PEP. Syntactically, patterns look like a subset of expressions.
The following are examples of patterns:

- ``[first, second, *rest]``
- ``Point2d(x, 0)``
- ``{"name": "Bruce", "age": age}``
- ``42``

The above expressions may look like examples of object construction
with a constructor which takes some values as parameters and
builds an object from those components.

When viewed as a pattern, the above patterns mean the inverse operation of
construction, which we call **destructuring**. **Destructuring** takes a subject value
and extracts its components.

The syntactic similarity between object construction and destructuring is
intentional. It also follows the existing
Pythonic style of contexts which makes assignment targets (write contexts) look
like expressions (read contexts).

Pattern matching never creates objects. This is in the same way that
``[a, b] = my_list`` doesn't create a
new ``[a, b]`` list, nor reads the values of ``a`` and ``b``.


Matching process
----------------

..  **Reword**
    The intuition we are trying to build in users as they learn this is
    that matching a pattern to a subject binds the free variables (if any)
    to subject components in a way that reflects the original
    subject when read as an expression.

During this matching process,
the structure of the pattern may not fit the subject, and matching *fails*.

For example, matching the pattern ``Point2d(x, 0)`` to the subject
``Point2d(3, 0)`` successfully matches. The match also **binds**
the pattern's free variable ``x`` to the subject's value ``3``.

As another example, if the subject is ``[3, 0]``, the match fails
because the subject's type ``list`` is not the pattern's ``Point2d``.

As a third example, if the subject is
``Point2d(3, 7)``, the match fails because the
subject's second coordinate ``7`` is not the same as the pattern's ``0``.

The ``match`` statement tries to match a single subject to each of the
patterns in its ``case`` clauses. At the first
successful match to a pattern in a ``case`` clause:

- the variables in the pattern are assigned, and
- a corresponding block is executed.

Each ``case`` clause can also specify an optional boolean condition,
known as a **guard**.

Let's look at a more detailed example of a ``match`` statement. The
``match`` statement is used within a function to define the building
of 3D points. In this example, the function can accept as input any of
the following: tuple with 2 elements, tuple with 3 elements, an
existing Point2d object or an existing Point3d object::

    def make_point_3d(pt):
        match pt:
            case (x, y):
                return Point3d(x, y, 0)
            case (x, y, z):
                return Point3d(x, y, z)
            case Point2d(x, y):
                return Point3d(x, y, 0)
            case Point3d(_, _, _):
                return pt
            case _:
                raise TypeError("not a point we support")

Without pattern matching, this function's implementation would require several
``isinstance()`` checks, one or two ``len()`` calls, and a more
convoluted control flow. The ``match`` example version and the traditional
Python version without ``match`` translate into similar code under the hood.
With familiarity of pattern matching, a user reading this function using ``match``
will likely find this version clearer than the traditional approach.


Rationale and Goals
===================

Python programs frequently need to handle data which varies in type,
presence of attributes/keys, or number of elements. Typical examples
are operating on nodes of a mixed structure like an AST, handling UI
events of different types, processing structured input (like
structured files or network messages), or âparsingâ arguments for a
function that can accept different combinations of types and numbers
of parameters. In fact, the classic 'visitor' pattern is an example of this,
done in an OOP style -- but matching makes it much less tedious to write.

Much of the code to do so tends to consist of complex chains of nested
``if``/``elif`` statements, including multiple calls to ``len()``,
``isinstance()`` and index/key/attribute access. Inside those branches
users sometimes need to destructure the data further to extract the
required component values, which may be nested several objects deep.

Pattern matching as present in many other languages provides an
elegant solution to this problem. These range from statically compiled
functional languages like F# and Haskell, via mixed-paradigm languages
like Scala_ and Rust_, to dynamic languages like Elixir and
Ruby, and is under consideration for JavaScript. We are indebted to
these languages for guiding the way to Pythonic pattern matching, as
Python is indebted to so many other languages for many of its
features: many basic syntactic features were inherited from C,
exceptions from Modula-3, classes were inspired by C++, slicing came
from Icon, regular expressions from Perl, decorators resemble Java
annotations, and so on.

The usual logic for operating on heterogeneous data can be summarized
in the following way:


id: pep-0636.rst
content:
PEP: 636
Title: Structural Pattern Matching: Tutorial
Author: Daniel F Moisset <dfmoisset@gmail.com>
Sponsor: Guido van Rossum <guido@python.org>
BDFL-Delegate:
Discussions-To: python-dev@python.org
Status: Final
Type: Informational
Created: 12-Sep-2020
Python-Version: 3.10
Post-History: 22-Oct-2020, 08-Feb-2021
Resolution: https://mail.python.org/archives/list/python-committers@python.org/message/SQC2FTLFV5A7DV7RCEAR2I2IKJKGK7W3


Abstract
========

This PEP is a tutorial for the pattern matching introduced by :pep:`634`.

:pep:`622` proposed syntax for pattern matching, which received detailed discussion
both from the community and the Steering Council. A frequent concern was
about how easy it would be to explain (and learn) this feature. This PEP
addresses that concern providing the kind of document which developers could use
to learn about pattern matching in Python.

This is considered supporting material for :pep:`634` (the technical specification
for pattern matching) and :pep:`635` (the motivation and rationale for having pattern
matching and design considerations).

For readers who are looking more for a quick review than for a tutorial,
see `Appendix A <PEP 636 Appendix A_>`_.

Tutorial
========

As an example to motivate this tutorial, you will be writing a text adventure. That is
a form of interactive fiction where the user enters text commands to interact with a
fictional world and receives text descriptions of what happens. Commands will be
simplified forms of natural language like ``get sword``, ``attack dragon``, ``go north``,
``enter shop`` or ``buy cheese``.

Matching sequences
------------------

Your main loop will need to get input from the user and split it into words, let's say
a list of strings like this::

   command = input("What are you doing next? ")
   # analyze the result of command.split()

The next step is to interpret the words. Most of our commands will have two words: an
action and an object. So you may be tempted to do the following::

   [action, obj] = command.split()
   ... # interpret action, obj

The problem with that line of code is that it's missing something: what if the user
types more or fewer than 2 words? To prevent this problem you can either check the length
of the list of words, or capture the ``ValueError`` that the statement above would raise.

You can use a matching statement instead::

   match command.split():
       case [action, obj]:
           ... # interpret action, obj

The match statement evaluates the **"subject"** (the value after the ``match``
keyword), and checks it against the **pattern** (the code next to ``case``). A pattern
is able to do two different things:

* Verify that the subject has certain structure. In your case, the ``[action, obj]``
  pattern matches any sequence of exactly two elements. This is called **matching**
* It will bind some names in the pattern to component elements of your subject. In
  this case, if the list has two elements, it will bind ``action = subject[0]`` and
  ``obj = subject[1]``.

If there's a match, the statements inside the case block will be executed with the
bound variables. If there's no match, nothing happens and the statement after
``match`` is executed next.

Note that, in a similar way to unpacking assignments, you can use either parenthesis,
brackets, or just comma separation as synonyms. So you could write ``case action, obj``
or ``case (action, obj)`` with the same meaning. All forms will match any sequence (for
example lists or tuples).

Matching multiple patterns
--------------------------

Even if most commands have the action/object form, you might want to have user commands
of different lengths. For example, you might want to add single verbs with no object like
``look`` or ``quit``. A match statement can (and is likely to) have more than one
``case``::

   match command.split():
       case [action]:
           ... # interpret single-verb action
       case [action, obj]:
           ... # interpret action, obj

The match statement will check patterns from top to bottom. If the pattern doesn't
match the subject, the next pattern will be tried. However, once the *first*
matching pattern is found, the body of that case is executed, and all further
cases are ignored. This is similar to the way that an ``if/elif/elif/...``
statement works.

Matching specific values
------------------------

Your code still needs to look at the specific actions and conditionally execute
different logic depending on the specific action (e.g., ``quit``, ``attack``, or ``buy``).
You could do that using a chain of ``if/elif/elif/...``, or using a dictionary of
functions, but here we'll leverage pattern matching to solve that task. Instead of a
variable, you can use literal values in patterns (like ``"quit"``, ``42``, or ``None``).
This allows you to write::

   match command.split():
       case ["quit"]:
           print("Goodbye!")
           quit_game()
       case ["look"]:
           current_room.describe()
       case ["get", obj]:
           character.get(obj, current_room)
       case ["go", direction]:
           current_room = current_room.neighbor(direction)
       # The rest of your commands go here

A pattern like ``["get", obj]`` will match only 2-element sequences that have a first
element equal to ``"get"``. It will also bind ``obj = subject[1]``.

As you can see in the ``go`` case, we also can use different variable names in
different patterns.

Literal values are compared with the ``==`` operator except for the constants ``True``,
``False`` and ``None`` which are compared with the ``is`` operator.

Matching multiple values
------------------------

A player may be able to drop multiple items by using a series of commands
``drop key``, ``drop sword``, ``drop cheese``. This interface might be cumbersome, and
you might like to allow dropping multiple items in a single command, like
``drop key sword cheese``. In this case you don't know beforehand how many words will
be in the command, but you can use extended unpacking in patterns in the same way that
they are allowed in assignments::

   match command.split():
       case ["drop", *objects]:
           for obj in objects:
               character.drop(obj, current_room)
       # The rest of your commands go here

This will match any sequences having "drop" as its first elements. All remaining
elements will be captured in a ``list`` object which will be bound to the ``objects``
variable.

This syntax has similar restrictions as sequence unpacking: you can not have more than one
starred name in a pattern.

Adding a wildcard
------------------

You may want to print an error message saying that the command wasn't recognized when
all the patterns fail. You could use the feature we just learned and write
``case [*ignored_words]`` as your last pattern. There's however a much simpler way::

   match command.split():
       case ["quit"]: ... # Code omitted for brevity
       case ["go", direction]: ...
       case ["drop", *objects]: ...
       ... # Other cases
       case _:
           print(f"Sorry, I couldn't understand {command!r}")

This special pattern which is written ``_`` (and called wildcard) always
matches but it doesn't bind any variables.

Note that this will match any object, not just sequences. As such, it only makes
sense to have it by itself as the last pattern (to prevent errors, Python will stop
you from using it before).

Composing patterns
------------------

This is a good moment to step back from the examples and understand how the patterns
that you have been using are built. Patterns can be nested within each other, and we
have been doing that implicitly in the examples above.

There are some "simple" patterns ("simple" here meaning that they do not contain other
patterns) that we've seen:

* **Capture patterns** (stand-alone names like ``direction``, ``action``, ``objects``). We
  never discussed these separately, but used them as part of other patterns.
* **Literal pat


id: pep-0781.rst
content:
PEP: 781
Title: Make ``TYPE_CHECKING`` a built-in constant
Author: Inada Naoki <songofacandy@gmail.com>
Discussions-To: https://discuss.python.org/t/85728
Status: Draft
Type: Standards Track
Topic: Typing
Created: 24-Mar-2025
Python-Version: 3.15
Post-History: `11-Jan-2025 <https://discuss.python.org/t/76766>`__,
              `24-Mar-2025 <https://discuss.python.org/t/85728>`__,


Abstract
========

This PEP proposes adding a new built-in variable, :data:`!TYPE_CHECKING`, to
improve the experience of writing Python code with type annotations. It is
evaluated as ``True`` when the code is being analyzed by a static type checker,
and as ``False`` during normal runtime execution. Unlike
:data:`typing.TYPE_CHECKING`, which this variable replaces, it does not require
an import statement.


Motivation
==========

Type annotations were defined for Python by :pep:`484`, and have enjoyed
widespread adoption. A challenge with fully-annotated code is that many
more imports are required in order to bring the relevant name into scope,
potentially causing import cycles without careful design. This has been
recognized by :pep:`563` and later :pep:`649`, which introduce two different
mechanisms for deferred evaluation of type annotations. As PEP 563 notes,
"type hints are ... not computationally free". The :data:`typing.TYPE_CHECKING`
constant was thus introduced__, initially to aid in breaking cyclic imports.

__ https://github.com/python/typing/issues/230

In situations where startup time is critical, such as command-line interfaces,
applications, or core libraries, programmers may place all import statements
not required for runtime execution within a 'TYPE_CHECKING block', or even
defer certain imports to within functions. The ``typing`` module itself though
can take as much as 10ms to import, longer than Python takes to initialize.
The time taken to import the ``typing`` module clearly cannot be ignored.

To avoid importing ``TYPE_CHECKING`` from ``typing``, developers currently
define a module-level variable such as ``TYPE_CHECKING = False`` or use code
like ``if False:  # TYPE_CHECKING``. Providing a standard method will allow
many tools to implement the same behavior consistently. It will also allow
third-party tools in the ecosystem to standardize on a single behavior
with guaranteed semantics, as for example some static type checkers currently
do not permit local constants, only recognizing ``typing.TYPE_CHECKING``.


Specification
=============

``TYPE_CHECKING`` is a built-in constant and its value is ``False``.
Unlike ``True``, ``False``, ``None``, and ``__debug__``, ``TYPE_CHECKING`` is
not a real constant; assigning to it will not raise a ``SyntaxError``.

Static type checkers must treat ``TYPE_CHECKING`` as ``True``, similar to
:data:`typing.TYPE_CHECKING`.

If this PEP is accepted, the new ``TYPE_CHECKING`` constant will be
the preferred approach and importing ``typing.TYPE_CHECKING`` will be
deprecated.
To minimize the runtime impact of typing, this deprecation will generate
``DeprecationWarning`` no sooner than Python 3.13's end of life, scheduled
for October 2029.

Instead, type checkers may warn about such deprecated usage when the target
version of the checked program is signalled to be Python 3.14 or newer.


Backwards Compatibility
=======================

Since ``TYPE_CHECKING`` doesn't prohibit assignment, existing code using
``TYPE_CHECKING`` will continue to work.

.. code-block:: python

   # This code will continue to work
   TYPE_CHECKING = False
   from typing import TYPE_CHECKING


User can remove the assignment to ``TYPE_CHECKING`` after they stop using
Python 3.13 or older versions.


How to Teach This
=================

* Use ``if TYPE_CHECKING:`` for skipping type-checking code at runtime.
* Use ``from typing import TYPE_CHECKING`` to support Python versions before
  3.14.
* Workarounds like ``TYPE_CHECKING = False`` or ``if False:  # TYPE_CHECKING``
  will continue to work, but are not recommended.


Reference Implementation
========================

* `python/cpython#131793 <https://github.com/python/cpython/pull/131793>`__


Rejected Ideas
==============

Eliminate type-checking-only code
---------------------------------

It is considered to add real constant named ``__type_checking__``
to eliminate type-checking-only code at compile time.

However, adding real constant to language increase complexity of the language.
Benefit from eliminating type-checking-only code is estimated to be not enough
to justify the complexity.


Optimize ``import typing``
--------------------------

Future optimizations may eliminate the need to avoid importing the ``typing``
module for startup time.

Even with such optimizations, there will still be use cases where minimizing
imports is beneficial, such as running Python on embedded systems or
in browsers.

Therefore, defining a constant for skipping type-checking-only code outside
the ``typing`` module remains valuable.


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0756.rst
content:
PEP: 756
Title: Add PyUnicode_Export() and PyUnicode_Import() C functions
Author: Victor Stinner <vstinner@python.org>
PEP-Delegate: C API Working Group
Discussions-To: https://discuss.python.org/t/63891
Status: Withdrawn
Type: Standards Track
Created: 13-Sep-2024
Python-Version: 3.14
Post-History: `14-Sep-2024 <https://discuss.python.org/t/63891>`__
Resolution: `29-Oct-2024 <https://discuss.python.org/t/63891/62>`__


.. highlight:: c


Abstract
========

Add functions to the limited C API version 3.14:

* ``PyUnicode_Export()``: export a Python str object as a ``Py_buffer``
  view.
* ``PyUnicode_Import()``: import a Python str object.

On CPython, ``PyUnicode_Export()`` has an *O*\ (1) complexity: no memory
is copied and no conversion is done.


Rationale
=========

PEP 393
-------

:pep:`393` "Flexible String Representation" changed string internals in
Python 3.3 to use three formats:

* ``PyUnicode_1BYTE_KIND``: Unicode range [U+0000; U+00ff],
  UCS-1, 1 byte/character.
* ``PyUnicode_2BYTE_KIND``: Unicode range [U+0000; U+ffff],
  UCS-2, 2 bytes/character.
* ``PyUnicode_4BYTE_KIND``: Unicode range [U+0000; U+10ffff],
  UCS-4, 4 bytes/character.

A Python ``str`` object must always use the most compact format. For
example, a string which only contains ASCII characters must use the
UCS-1 format.

The ``PyUnicode_KIND()`` function can be used to know the format used by
a string.

One of the following functions can be used to access data:

* ``PyUnicode_1BYTE_DATA()`` for ``PyUnicode_1BYTE_KIND``.
* ``PyUnicode_2BYTE_DATA()`` for ``PyUnicode_2BYTE_KIND``.
* ``PyUnicode_4BYTE_DATA()`` for ``PyUnicode_4BYTE_KIND``.

To get the best performance, a C extension should have 3 code paths for
each of these 3 string native formats.

Limited C API
-------------

:pep:`393` functions such as ``PyUnicode_KIND()`` and
``PyUnicode_1BYTE_DATA()`` are excluded from the limited C API. It's not
possible to write code specialized for UCS formats. A C extension using
the limited C API can only use less efficient code paths and string
formats.

For example, the `MarkupSafe project
<https://markupsafe.palletsprojects.com/>`_ has a C extension
specialized for UCS formats for best performance, and so cannot use the
limited C API.


Specification
=============

API
---

Add the following API to the limited C API version 3.14::

    int32_t PyUnicode_Export(
        PyObject *unicode,
        int32_t requested_formats,
        Py_buffer *view);
    PyObject* PyUnicode_Import(
        const void *data,
        Py_ssize_t nbytes,
        int32_t format);

    #define PyUnicode_FORMAT_UCS1  0x01   // Py_UCS1*
    #define PyUnicode_FORMAT_UCS2  0x02   // Py_UCS2*
    #define PyUnicode_FORMAT_UCS4  0x04   // Py_UCS4*
    #define PyUnicode_FORMAT_UTF8  0x08   // char*
    #define PyUnicode_FORMAT_ASCII 0x10   // char* (ASCII string)

The ``int32_t`` type is used instead of ``int`` to have a well defined
type size and not depend on the platform or the compiler.
See `Avoid C-specific Types
<https://github.com/capi-workgroup/api-evolution/issues/10>`_ for the
longer rationale.

PyUnicode_Export()
------------------

API::

    int32_t PyUnicode_Export(
        PyObject *unicode,
        int32_t requested_formats,
        Py_buffer *view)

Export the contents of the *unicode* string in one of the *requested_formats*.

* On success, fill *view*, and return a format (greater than ``0``).
* On error, set an exception, and return ``-1``.
  *view* is left unchanged.

After a successful call to ``PyUnicode_Export()``,
the *view* buffer must be released by ``PyBuffer_Release()``.
The contents of the buffer are valid until they are released.

The buffer is read-only and must not be modified.

The ``view->len`` member must be used to get the string length. The
buffer should end with a trailing NUL character, but it's not
recommended to rely on that because of embedded NUL characters.

*unicode* and *view* must not be NULL.

Available formats:

===================================  ========  ===========================
Constant Identifier                  Value     Description
===================================  ========  ===========================
``PyUnicode_FORMAT_UCS1``            ``0x01``  UCS-1 string (``Py_UCS1*``)
``PyUnicode_FORMAT_UCS2``            ``0x02``  UCS-2 string (``Py_UCS2*``)
``PyUnicode_FORMAT_UCS4``            ``0x04``  UCS-4 string (``Py_UCS4*``)
``PyUnicode_FORMAT_UTF8``            ``0x08``  UTF-8 string (``char*``)
``PyUnicode_FORMAT_ASCII``           ``0x10``  ASCII string (``Py_UCS1*``)
===================================  ========  ===========================

UCS-2 and UCS-4 use the native byte order.

*requested_formats* can be a single format or a bitwise combination of the
formats in the table above.
On success, the returned format will be set to a single one of the requested
formats.

Note that future versions of Python may introduce additional formats.

No memory is copied and no conversion is done.



.. _export-complexity:

Export complexity
-----------------

On CPython, an export has a complexity of *O*\ (1): no memory is copied
and no conversion is done.

To get the best performance on CPython and PyPy, it's recommended to
support these 4 formats::

    (PyUnicode_FORMAT_UCS1 \
     | PyUnicode_FORMAT_UCS2 \
     | PyUnicode_FORMAT_UCS4 \
     | PyUnicode_FORMAT_UTF8)

PyPy uses UTF-8 natively and so the ``PyUnicode_FORMAT_UTF8`` format is
recommended. It requires a memory copy, since PyPy ``str`` objects can
be moved in memory (PyPy uses a moving garbage collector).


Py_buffer format and item size
------------------------------

``Py_buffer`` uses the following format and item size depending on the
export format:

==========================  ==================  ============
Export format               Buffer format       Item size
==========================  ==================  ============
``PyUnicode_FORMAT_UCS1``   ``"B"``             1 byte
``PyUnicode_FORMAT_UCS2``   ``"=H"``            2 bytes
``PyUnicode_FORMAT_UCS4``   ``"=I"``            4 bytes
``PyUnicode_FORMAT_UTF8``   ``"B"``             1 byte
``PyUnicode_FORMAT_ASCII``  ``"B"``             1 byte
==========================  ==================  ============


PyUnicode_Import()
------------------

API::

    PyObject* PyUnicode_Import(
        const void *data,
        Py_ssize_t nbytes,
        int32_t format)

Create a Unicode string object from a buffer in a supported format.

* Return a reference to a new string object on success.
* Set an exception and return ``NULL`` on error.

*data* must not be NULL. *nbytes* must be positive or zero.

See ``PyUnicode_Export()`` for the available formats.


UTF-8 format
------------

CPython 3.14 doesn't use the UTF-8 format internally and doesn't support
exporting a string as UTF-8. The ``PyUnicode_AsUTF8AndSize()`` function
can be used instead.

The ``PyUnicode_FORMAT_UTF8`` format is provided for compatibility with
alternate implementations which may use UTF-8 natively for strings.


ASCII format
------------

When the ``PyUnicode_FORMAT_ASCII`` format is request for export, the
``PyUnicode_FORMAT_UCS1`` export format is used for ASCII strings.

The ``PyUnicode_FORMAT_ASCII`` format is mostly useful for
``PyUnicode_Import()`` to validate that a string only contains ASCII
characters.


Surrogate characters and embedded NUL characters
------------------------------------------------

Surrogate characters are allowed: they can be imported and exported.

Embedded NUL characters are allowed: they can be imported and exported.


Implementation
==============

https://github.com/python/cpython/pull/123738


Backwards Compatibility
=======================

There is no impact on the backward compatibility, only new C API
functions are added.


Usage of PEP 393 C APIs
=======================

A code search on PyPI top 7,500 projects (in March 2024) shows that
there are many projects importing and exporting UCS formats with the
regular C API.

PyUnicode_


id: pep-0742.rst
content:
PEP: 742
Title: Narrowing types with TypeIs
Author: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-742-narrowing-types-with-typenarrower/45613
Status: Final
Type: Standards Track
Topic: Typing
Created: 07-Feb-2024
Python-Version: 3.13
Post-History: `11-Feb-2024 <https://discuss.python.org/t/pep-742-narrowing-types-with-typenarrower/45613>`__
Replaces: 724
Resolution: `03-Apr-2024 <https://discuss.python.org/t/pep-742-narrowing-types-with-typeis/45613/35>`__

.. canonical-typing-spec:: :ref:`typing:typeis` and
                           :external+py3.13:data:`typing.TypeIs`

Abstract
========

This PEP proposes a new special form, ``TypeIs``, to allow annotating functions that can be used
to narrow the type of a value, similar to the builtin :py:func:`isinstance`. Unlike the existing
:py:data:`typing.TypeGuard` special form, ``TypeIs`` can narrow the type in both the ``if``
and ``else`` branches of a conditional.


Motivation
==========

Typed Python code often requires users to narrow the type of a variable based on a conditional.
For example, if a function accepts a union of two types, it may use an :py:func:`isinstance` check
to discriminate between the two types. Type checkers commonly support type narrowing based on various
builtin function and operations, but occasionally, it is useful to use a user-defined function to
perform type narrowing.

To support such use cases, :pep:`647` introduced the :py:data:`typing.TypeGuard` special form, which
allows users to define type guards::

    from typing import assert_type, TypeGuard

    def is_str(x: object) -> TypeGuard[str]:
        return isinstance(x, str)

    def f(x: object) -> None:
        if is_str(x):
            assert_type(x, str)
        else:
            assert_type(x, object)

Unfortunately, the behavior of :py:data:`typing.TypeGuard` has some limitations that make it
less useful for many common use cases, as explained also in the "Motivation" section of :pep:`724`.
In particular:

* Type checkers must use exactly the ``TypeGuard`` return type as the narrowed type if the
  type guard returns ``True``. They cannot use pre-existing knowledge about the type of the
  variable.
* In the case where the type guard returns ``False``, the type checker cannot apply any
  additional narrowing.

The standard library function :py:func:`inspect.isawaitable` may serve as an example. It
returns whether the argument is an awaitable object, and
`typeshed <https://github.com/python/typeshed/blob/a4f81a67a07c18dd184dd068c459b02e71bcac22/stdlib/inspect.pyi#L219>`__
currently annotates it as::

    def isawaitable(object: object) -> TypeGuard[Awaitable[Any]]: ...

A user `reported <https://github.com/python/mypy/issues/15520>`__ an issue to mypy about
the behavior of this function. They observed the following behavior::

    import inspect
    from collections.abc import Awaitable
    from typing import reveal_type

    async def f(t: Awaitable[int] | int) -> None:
        if inspect.isawaitable(t):
            reveal_type(t)  # Awaitable[Any]
        else:
            reveal_type(t)  # Awaitable[int] | int

This behavior is consistent with :pep:`647`, but it did not match the user's expectations.
Instead, they would expect the type of ``t`` to be narrowed to ``Awaitable[int]`` in the ``if``
branch, and to ``int`` in the ``else`` branch. This PEP proposes a new construct that does
exactly that.

Other examples of issues that arose out of the current behavior of ``TypeGuard`` include:

* `Python typing issue <https://github.com/python/typing/issues/996>`__ (``numpy.isscalar``)
* `Python typing issue <https://github.com/python/typing/issues/1351>`__ (:py:func:`dataclasses.is_dataclass`)
* `Pyright issue <https://github.com/microsoft/pyright/issues/3450>`__ (expecting :py:data:`typing.TypeGuard` to work like :py:func:`isinstance`)
* `Pyright issue <https://github.com/microsoft/pyright/issues/3466>`__ (expecting narrowing in the ``else`` branch)
* `Mypy issue <https://github.com/python/mypy/issues/13957>`__ (expecting narrowing in the ``else`` branch)
* `Mypy issue <https://github.com/python/mypy/issues/14434>`__ (combining multiple TypeGuards)
* `Mypy issue <https://github.com/python/mypy/issues/15305>`__ (expecting narrowing in the ``else`` branch)
* `Mypy issue <https://github.com/python/mypy/issues/11907>`__ (user-defined function similar to :py:func:`inspect.isawaitable`)
* `Typeshed issue <https://github.com/python/typeshed/issues/8009>`__ (``asyncio.iscoroutinefunction``)

Rationale
=========

The problems with the current behavior of :py:data:`typing.TypeGuard` compel us to improve
the type system to allow a different type narrowing behavior. :pep:`724` proposed to change
the behavior of the existing :py:data:`typing.TypeGuard` construct, but we :ref:`believe <pep-742-change-typeguard>`
that the backwards compatibility implications of that change are too severe. Instead, we propose
adding a new special form with the desired semantics.

We acknowledge that this leads to an unfortunate situation where there are two constructs with
a similar purpose and similar semantics. We believe that users are more likely to want the behavior
of ``TypeIs``, the new form proposed in this PEP, and therefore we recommend that documentation
emphasize ``TypeIs`` over ``TypeGuard`` as a more commonly applicable tool. However, the semantics of
``TypeGuard`` are occasionally useful, and we do not propose to deprecate or remove it. In the long
run, most users should use ``TypeIs``, and ``TypeGuard`` should be reserved for rare cases
where its behavior is specifically desired.


Specification
=============

A new special form, ``TypeIs``, is added to the :py:mod:`typing`
module. Its usage, behavior, and runtime implementation are similar to
those of :py:data:`typing.TypeGuard`.

It accepts a single
argument and can be used as the return type of a function. A function annotated as returning a
``TypeIs`` is called a type narrowing function. Type narrowing functions must return ``bool``
values, and the type checker should verify that all return paths return
``bool``.

Type narrowing functions must accept at least one positional argument. The type
narrowing behavior is applied to the first positional argument passed to
the function. The function may accept additional arguments, but they are
not affected by type narrowing. If a type narrowing function is implemented as
an instance method or class method, the first positional argument maps
to the second parameter (after ``self`` or ``cls``).

Type narrowing behavior
-----------------------

To specify the behavior of ``TypeIs``, we use the following terminology:

* I = ``TypeIs`` input type
* R = ``TypeIs`` return type
* A = Type of argument passed to type narrowing function (pre-narrowed)
* NP = Narrowed type (positive; used when ``TypeIs`` returned ``True``)
* NN = Narrowed type (negative; used when ``TypeIs`` returned ``False``)

.. code-block:: python

    def narrower(x: I) -> TypeIs[R]: ...

    def func1(val: A):
        if narrower(val):
            assert_type(val, NP)
        else:
            assert_type(val, NN)

The return type ``R`` must be :ref:`consistent with <pep-483-gradual-typing>` ``I``. The type checker should
emit an error if this condition is not met.

Formally, type *NP* should be narrowed to :math:`A \land R`,
the intersection of *A* and *R*, and type *NN* should be narrowed to
:math:`A \land \neg R`, the intersection of *A* and the complement of *R*.
In practice, the theoretic types for strict type guards cannot be expressed
precisely in the Python type system. Type checkers should fall back on
practical approximations of these types. As a rule of thumb, a type checker
should use the same type narrowing logic -- and get results that are consistent
with -- its handling of :py:func:`isinstance`. This guidance allows for changes and
improvements if the type system is extended in the future.

Examples
--------

Type n


id: pep-0597.rst
content:
PEP: 597
Title: Add optional EncodingWarning
Author: Inada Naoki <songofacandy@gmail.com>
Status: Final
Type: Standards Track
Created: 05-Jun-2019
Python-Version: 3.10


Abstract
========

Add a new warning category ``EncodingWarning``. It is emitted when the
``encoding`` argument to ``open()`` is omitted and the default
locale-specific encoding is used.

The warning is disabled by default. A new ``-X warn_default_encoding``
command-line option and a new ``PYTHONWARNDEFAULTENCODING`` environment
variable can be used to enable it.

A ``"locale"`` argument value for ``encoding`` is added too. It
explicitly specifies that the locale encoding should be used, silencing
the warning.


Motivation
==========

Using the default encoding is a common mistake
----------------------------------------------

Developers using macOS or Linux may forget that the default encoding
is not always UTF-8.

For example, using ``long_description = open("README.md").read()`` in
``setup.py`` is a common mistake. Many Windows users cannot install
such packages if there is at least one non-ASCII character
(e.g. emoji, author names, copyright symbols, and the like)
in their UTF-8-encoded ``README.md`` file.

Of the 4000 most downloaded packages from PyPI, 489 use non-ASCII
characters in their README, and 82 fail to install from source on
non-UTF-8 locales due to not specifying an encoding for a non-ASCII
file. [1]_

Another example is ``logging.basicConfig(filename="log.txt")``.
Some users might expect it to use UTF-8 by default, but the locale
encoding is actually what is used. [2]_

Even Python experts may assume that the default encoding is UTF-8.
This creates bugs that only happen on Windows; see [3]_, [4]_, [5]_,
and [6]_ for example.

Emitting a warning when the ``encoding`` argument is omitted will help
find such mistakes.


Explicit way to use locale-specific encoding
--------------------------------------------

``open(filename)`` isn't explicit about which encoding is expected:

* If ASCII is assumed, this isn't a bug, but may result in decreased
  performance on Windows, particularly with non-Latin-1 locale encodings
* If UTF-8 is assumed, this may be a bug or a platform-specific script
* If the locale encoding is assumed, the behavior is as expected
  (but could change if future versions of Python modify the default)

From this point of view, ``open(filename)`` is not readable code.

``encoding=locale.getpreferredencoding(False)`` can be used to
specify the locale encoding explicitly, but it is too long and easy
to misuse (e.g. one can forget to pass ``False`` as its argument).

This PEP provides an explicit way to specify the locale encoding.


Prepare to change the default encoding to UTF-8
-----------------------------------------------

Since UTF-8 has become the de-facto standard text encoding,
we might default to it for opening files in the future.

However, such a change will affect many applications and libraries.
If we start emitting ``DeprecationWarning`` everywhere the ``encoding``
argument is omitted, it will be too noisy and painful.

Although this PEP doesn't propose changing the default encoding,
it will help enable that change by:

* Reducing the number of omitted ``encoding`` arguments in libraries
  before we start emitting a ``DeprecationWarning`` by default.

* Allowing users to pass ``encoding="locale"`` to suppress
  the current warning and any ``DeprecationWarning`` added in the future,
  as well as retaining consistent behavior if later Python versions
  change the default, ensuring support for any Python version >=3.10.


Specification
=============

``EncodingWarning``
-------------------

Add a new ``EncodingWarning`` warning class as a subclass of
``Warning``. It is emitted when the ``encoding`` argument is omitted and
the default locale-specific encoding is used.


Options to enable the warning
-----------------------------

The ``-X warn_default_encoding`` option and the
``PYTHONWARNDEFAULTENCODING`` environment variable are added. They
are used to enable ``EncodingWarning``.

``sys.flags.warn_default_encoding`` is also added. The flag is true when
``EncodingWarning`` is enabled.

When the flag is set, ``io.TextIOWrapper()``, ``open()`` and other
modules using them will emit ``EncodingWarning`` when the ``encoding``
argument is omitted.

Since ``EncodingWarning`` is a subclass of ``Warning``, they are
shown by default (if the ``warn_default_encoding`` flag is set), unlike
``DeprecationWarning``.


``encoding="locale"``
---------------------

``io.TextIOWrapper`` will accept ``"locale"`` as a valid argument to
``encoding``. It has the same meaning as the current ``encoding=None``,
except that ``io.TextIOWrapper`` doesn't emit ``EncodingWarning`` when
``encoding="locale"`` is specified.


``io.text_encoding()``
----------------------

``io.text_encoding()`` is a helper for functions with an
``encoding=None`` parameter that pass it to ``io.TextIOWrapper()`` or
``open()``.

A pure Python implementation will look like this::

   def text_encoding(encoding, stacklevel=1):
       """A helper function to choose the text encoding.

       When *encoding* is not None, just return it.
       Otherwise, return the default text encoding (i.e. "locale").

       This function emits an EncodingWarning if *encoding* is None and
       sys.flags.warn_default_encoding is true.

       This function can be used in APIs with an encoding=None parameter
       that pass it to TextIOWrapper or open.
       However, please consider using encoding="utf-8" for new APIs.
       """
       if encoding is None:
           if sys.flags.warn_default_encoding:
               import warnings
               warnings.warn(
                   "'encoding' argument not specified.",
                   EncodingWarning, stacklevel + 2)
           encoding = "locale"
       return encoding

For example, ``pathlib.Path.read_text()`` can use it like this:

.. code-block::

   def read_text(self, encoding=None, errors=None):
       encoding = io.text_encoding(encoding)
       with self.open(mode='r', encoding=encoding, errors=errors) as f:
           return f.read()

By using ``io.text_encoding()``, ``EncodingWarning`` is emitted for
the caller of ``read_text()`` instead of ``read_text()`` itself.


Affected standard library modules
---------------------------------

Many standard library modules will be affected by this change.

Most APIs accepting ``encoding=None`` will use ``io.text_encoding()``
as written in the previous section.

Where using the locale encoding as the default encoding is reasonable,
``encoding="locale"`` will be used instead. For example,
the ``subprocess`` module will use the locale encoding as the default
for pipes.

Many tests use ``open()`` without ``encoding`` specified to read
ASCII text files. They should be rewritten with ``encoding="ascii"``.


Rationale
=========

Opt-in warning
--------------

Although ``DeprecationWarning`` is suppressed by default, always
emitting ``DeprecationWarning`` when the ``encoding`` argument is
omitted would be too noisy.

Noisy warnings may lead developers to dismiss the
``DeprecationWarning``.


"locale" is not a codec alias
-----------------------------

We don't add "locale" as a codec alias because the locale can be
changed at runtime.

Additionally, ``TextIOWrapper`` checks ``os.device_encoding()``
when ``encoding=None``. This behavior cannot be implemented in
a codec.


Backward Compatibility
======================

The new warning is not emitted by default, so this PEP is 100%
backwards-compatible.


Forward Compatibility
=====================

Passing ``"locale"`` as the argument to ``encoding`` is not
forward-compatible. Code using it will not work on Python older than
3.10, and will instead raise ``LookupError: unknown encoding: locale``.

Until developers can drop Python 3.9 support, ``EncodingWarning``
can only be used for finding missing ``encoding="utf-8"`` arguments.


How to Teach This
=================


id: pep-0583.rst
content:
PEP: 583
Title: A Concurrency Memory Model for Python
Author: Jeffrey Yasskin <jyasskin@google.com>
Status: Withdrawn
Type: Informational
Created: 22-Mar-2008
Post-History:


Abstract
========

This PEP describes how Python programs may behave in the presence of
concurrent reads and writes to shared variables from multiple threads.
We use a *happens before* relation to define when variable accesses
are ordered or concurrent.  Nearly all programs should simply use locks
to guard their shared variables, and this PEP highlights some of the
strange things that can happen when they don't, but programmers often
assume that it's ok to do "simple" things without locking, and it's
somewhat unpythonic to let the language surprise them.  Unfortunately,
avoiding surprise often conflicts with making Python run quickly, so
this PEP tries to find a good tradeoff between the two.


Rationale
=========

So far, we have 4 major Python implementations -- CPython, Jython_,
IronPython_, and PyPy_ -- as well as lots of minor ones.  Some of
these already run on platforms that do aggressive optimizations.  In
general, these optimizations are invisible within a single thread of
execution, but they can be visible to other threads executing
concurrently.  CPython currently uses a `GIL`_ to ensure that other
threads see the results they expect, but this limits it to a single
processor.  Jython and IronPython run on Java's or .NET's threading
system respectively, which allows them to take advantage of more cores
but can also show surprising values to other threads.

.. _Jython: http://www.jython.org/

.. _IronPython: http://www.codeplex.com/Wiki/View.aspx?ProjectName=IronPython

.. _PyPy: http://codespeak.net/pypy/dist/pypy/doc/home.html

.. _GIL: http://en.wikipedia.org/wiki/Global_Interpreter_Lock

So that threaded Python programs continue to be portable between
implementations, implementers and library authors need to agree on
some ground rules.


A couple definitions
====================

Variable
    A name that refers to an object.  Variables are generally
    introduced by assigning to them, and may be destroyed by passing
    them to ``del``.  Variables are fundamentally mutable, while
    objects may not be.  There are several varieties of variables:
    module variables (often called "globals" when accessed from within
    the module), class variables, instance variables (also known as
    fields), and local variables.  All of these can be shared between
    threads (the local variables if they're saved into a closure).
    The object in which the variables are scoped notionally has a
    ``dict`` whose keys are the variables' names.

Object
    A collection of instance variables (a.k.a. fields) and methods.
    At least, that'll do for this PEP.

Program Order
    The order that actions (reads and writes) happen within a thread,
    which is very similar to the order they appear in the text.

Conflicting actions
    Two actions on the same variable, at least one of which is a write.

Data race
    A situation in which two conflicting actions happen at the same
    time.  "The same time" is defined by the memory model.


Two simple memory models
========================

Before talking about the details of data races and the surprising
behaviors they produce, I'll present two simple memory models.  The
first is probably too strong for Python, and the second is probably
too weak.


Sequential Consistency
----------------------

In a sequentially-consistent concurrent execution, actions appear to
happen in a global total order with each read of a particular variable
seeing the value written by the last write that affected that
variable.  The total order for actions must be consistent with the
program order.  A program has a data race on a given input when one of
its sequentially consistent executions puts two conflicting actions
next to each other.

This is the easiest memory model for humans to understand, although it
doesn't eliminate all confusion, since operations can be split in odd
places.


Happens-before consistency
--------------------------

The program contains a collection of *synchronization actions*, which
in Python currently include lock acquires and releases and thread
starts and joins.  Synchronization actions happen in a global total
order that is consistent with the program order (they don't *have* to
happen in a total order, but it simplifies the description of the
model).  A lock release *synchronizes with* all later acquires of the
same lock.  Similarly, given ``t = threading.Thread(target=worker)``:

* A call to ``t.start()`` synchronizes with the first statement in
  ``worker()``.

* The return from ``worker()`` synchronizes with the return from
  ``t.join()``.

* If the return from ``t.start()`` happens before (see below) a call
  to ``t.isAlive()`` that returns ``False``, the return from
  ``worker()`` synchronizes with that call.

We call the source of the synchronizes-with edge a *release* operation
on the relevant variable, and we call the target an *acquire* operation.

The *happens before* order is the transitive closure of the program
order with the synchronizes-with edges.  That is, action *A* happens
before action *B* if:

* A falls before B in the program order (which means they run in the
  same thread)
* A synchronizes with B
* You can get to B by following happens-before edges from A.

An execution of a program is happens-before consistent if each read
*R* sees the value of a write *W* to the same variable such that:

* *R* does not happen before *W*, and
* There is no other write *V* that overwrote *W* before *R* got a
  chance to see it. (That is, it can't be the case that *W* happens
  before *V* happens before *R*.)

You have a data race if two conflicting actions aren't related by
happens-before.


An example
''''''''''

Let's use the rules from the happens-before model to prove that the
following program prints "[7]"::

    class Queue:
        def __init__(self):
            self.l = []
            self.cond = threading.Condition()

        def get():
            with self.cond:
                while not self.l:
                    self.cond.wait()
                ret = self.l[0]
                self.l = self.l[1:]
                return ret

        def put(x):
            with self.cond:
                self.l.append(x)
                self.cond.notify()

    myqueue = Queue()

    def worker1():
        x = [7]
        myqueue.put(x)

    def worker2():
        y = myqueue.get()
        print y

    thread1 = threading.Thread(target=worker1)
    thread2 = threading.Thread(target=worker2)
    thread2.start()
    thread1.start()

1. Because ``myqueue`` is initialized in the main thread before
   ``thread1`` or ``thread2`` is started, that initialization happens
   before ``worker1`` and ``worker2`` begin running, so there's no way
   for either to raise a NameError, and both ``myqueue.l`` and
   ``myqueue.cond`` are set to their final objects.

2. The initialization of ``x`` in ``worker1`` happens before it calls
   ``myqueue.put()``, which happens before it calls
   ``myqueue.l.append(x)``, which happens before the call to
   ``myqueue.cond.release()``, all because they run in the same
   thread.

3. In ``worker2``, ``myqueue.cond`` will be released and re-acquired
   until ``myqueue.l`` contains a value (``x``). The call to
   ``myqueue.cond.release()`` in ``worker1`` happens before that last
   call to ``myqueue.cond.acquire()`` in ``worker2``.

4. That last call to ``myqueue.cond.acquire()`` happens before
   ``myqueue.get()`` reads ``myqueue.l``, which happens before
   ``myqueue.get()`` returns, which happens before ``print y``, again
   all because they run in the same thread.

5. Because happens-before is transitive, the list initially stored in
   ``x`` in thread1 is initialized before it is printed in thread2.

Usually, we wouldn't need to look all the way into a thread-safe
queue's implementation in order to prov


id: pep-0568.rst
content:
PEP: 568
Title: Generator-sensitivity for Context Variables
Author: Nathaniel J. Smith <njs@pobox.com>
Status: Deferred
Type: Standards Track
Created: 04-Jan-2018
Python-Version: 3.8
Post-History:


Abstract
========

Context variables provide a generic mechanism for tracking dynamic,
context-local state, similar to thread-local storage but generalized
to cope work with other kinds of thread-like contexts, such as asyncio
Tasks. :pep:`550` proposed a mechanism for context-local state that was
also sensitive to generator context, but this was pretty complicated,
so the BDFL requested it be simplified. The result was :pep:`567`, which
is targeted for inclusion in 3.7. This PEP then extends :pep:`567`'s
machinery to add generator context sensitivity.

This PEP is starting out in the "deferred" status, because there isn't
enough time to give it proper consideration before the 3.7 feature
freeze. The only goal *right now* is to understand what would be
required to add generator context sensitivity in 3.8, so that we can
avoid shipping something in 3.7 that would rule it out by accident.
(Ruling it out on purpose can wait until 3.8 ;-).)


Rationale
=========

[Currently the point of this PEP is just to understand *how* this
would work, with discussion of *whether* it's a good idea deferred
until after the 3.7 feature freeze. So rationale is TBD.]


High-level summary
==================

Instead of holding a single ``Context``, the threadstate now holds a
``ChainMap`` of ``Context``\s. ``ContextVar.get`` and
``ContextVar.set`` are backed by the ``ChainMap``. Generators and
async generators each have an associated ``Context`` that they push
onto the ``ChainMap`` while they're running to isolate their
context-local changes from their callers, though this can be
overridden in cases like ``@contextlib.contextmanager`` where
"leaking" context changes from the generator into its caller is
desirable.


Specification
=============

Review of PEP 567
-----------------

Let's start by reviewing how :pep:`567` works, and then in the next
section we'll describe the differences.

In :pep:`567`, a ``Context`` is a ``Mapping`` from ``ContextVar`` objects
to arbitrary values. In our pseudo-code here we'll pretend that it
uses a ``dict`` for backing storage. (The real implementation uses a
HAMT, which is semantically equivalent to a ``dict`` but with
different performance trade-offs.)::

   class Context(collections.abc.Mapping):
       def __init__(self):
           self._data = {}
           self._in_use = False

       def __getitem__(self, key):
           return self._data[key]

       def __iter__(self):
           return iter(self._data)

       def __len__(self):
           return len(self._data)

At any given moment, the threadstate holds a current ``Context``
(initialized to an empty ``Context`` when the threadstate is created);
we can use ``Context.run`` to temporarily switch the current
``Context``::

   # Context.run
   def run(self, fn, *args, **kwargs):
       if self._in_use:
           raise RuntimeError("Context already in use")
       tstate = get_thread_state()
       old_context = tstate.current_context
       tstate.current_context = self
       self._in_use = True
       try:
           return fn(*args, **kwargs)
       finally:
           state.current_context = old_context
           self._in_use = False

We can fetch a shallow copy of the current ``Context`` by calling
``copy_context``; this is commonly used when spawning a new task, so
that the child task can inherit context from its parent::

   def copy_context():
       tstate = get_thread_state()
       new_context = Context()
       new_context._data = dict(tstate.current_context)
       return new_context

In practice, what end users generally work with is ``ContextVar``
objects, which also provide the only way to mutate a ``Context``. They
work with a utility class ``Token``, which can be used to restore a
``ContextVar`` to its previous value::

   class Token:
       MISSING = sentinel_value()

       # Note: constructor is private
       def __init__(self, context, var, old_value):
           self._context = context
           self.var = var
           self.old_value = old_value

       # XX: PEP 567 currently makes this a method on ContextVar, but
       # I'm going to propose it switch to this API because it's simpler.
       def reset(self):
           # XX: should we allow token reuse?
           # XX: should we allow tokens to be used if the saved
           # context is no longer active?
           if self.old_value is self.MISSING:
               del self._context._data[self.context_var]
           else:
               self._context._data[self.context_var] = self.old_value

   # XX: the handling of defaults here uses the simplified proposal from
   # https://mail.python.org/pipermail/python-dev/2018-January/151596.html
   # This can be updated to whatever we settle on, it was just less
   # typing this way :-)
   class ContextVar:
       def __init__(self, name, *, default=None):
           self.name = name
           self.default = default

       def get(self):
           context = get_thread_state().current_context
           return context.get(self, self.default)

       def set(self, new_value):
           context = get_thread_state().current_context
           token = Token(context, self, context.get(self, Token.MISSING))
           context._data[self] = new_value
           return token


Changes from PEP 567 to this PEP
--------------------------------

In general, ``Context`` remains the same. However, now instead of
holding a single ``Context`` object, the threadstate stores a stack of
them. This stack acts just like a ``collections.ChainMap``, so we'll
use that in our pseudocode. ``Context.run`` then becomes::

   # Context.run
   def run(self, fn, *args, **kwargs):
       if self._in_use:
           raise RuntimeError("Context already in use")
       tstate = get_thread_state()
       old_context_stack = tstate.current_context_stack
       tstate.current_context_stack = ChainMap([self])     # changed
       self._in_use = True
       try:
           return fn(*args, **kwargs)
       finally:
           state.current_context_stack = old_context_stack
           self._in_use = False

Aside from some updated variables names (e.g.,
``tstate.current_context`` â ``tstate.current_context_stack``), the
only change here is on the marked line, which now wraps the context in
a ``ChainMap`` before stashing it in the threadstate.

We also add a ``Context.push`` method, which is almost exactly like
``Context.run``, except that it temporarily pushes the ``Context``
onto the existing stack, instead of temporarily replacing the whole
stack::

   # Context.push
   def push(self, fn, *args, **kwargs):
       if self._in_use:
           raise RuntimeError("Context already in use")
       tstate = get_thread_state()
       tstate.current_context_stack.maps.insert(0, self)  # different from run
       self._in_use = True
       try:
           return fn(*args, **kwargs)
       finally:
           tstate.current_context_stack.maps.pop(0)       # different from run
           self._in_use = False

In most cases, we don't expect ``push`` to be used directly; instead,
it will be used implicitly by generators. Specifically, every
generator object and async generator object gains a new attribute
``.context``. When an (async) generator object is created, this
attribute is initialized to an empty ``Context`` (``self.context =
Context()``). This is a mutable attribute; it can be changed by user
code. But trying to set it to anything that isn't a ``Context`` object
or ``None`` will raise an error.

Whenever we enter an generator via ``__next__``, ``send``, ``throw``,
or ``close``, or enter an async generator by calling one of those
methods on its ``__anext__``, ``asend``, ``athrow``, or ``aclose``
coroutines, then its ``.context`` attribute is checked, and if
non-``None``, is automatically pushed


id: pep-0554.rst
content:
PEP: 554
Title: Multiple Interpreters in the Stdlib
Author: Eric Snow <ericsnowcurrently@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-554-multiple-interpreters-in-the-stdlib/24855
Status: Superseded
Type: Standards Track
Created: 05-Sep-2017
Python-Version: 3.13
Post-History: `07-Sep-2017 <https://mail.python.org/archives/list/python-ideas@python.org/thread/HQQWEE527HG3ILJVKQTXVSJIQO6NUSIA/>`__,
              `08-Sep-2017 <https://mail.python.org/archives/list/python-dev@python.org/thread/NBWMA6LVD22XOUYC5ZMPBFWDQOECRP77/>`__,
              `13-Sep-2017 <https://mail.python.org/archives/list/python-dev@python.org/thread/EG4FSFG5E3O22FTIUQOXMQ6X6B5X3DP7/>`__,
              `05-Dec-2017 <https://mail.python.org/archives/list/python-dev@python.org/thread/BCSRGAMCYB3NGXNU42U66J56XNZVMQP2/>`__,
              `04-May-2020 <https://mail.python.org/archives/list/python-dev@python.org/thread/X2KPCSRVBD2QD5GP5IMXXZTGZ46OXD3D/>`__,
              `14-Mar-2023 <https://discuss.python.org/t/pep-554-multiple-interpreters-in-the-stdlib/24855/2/>`__,
              `01-Nov-2023 <https://discuss.python.org/t/pep-554-multiple-interpreters-in-the-stdlib/24855/26/>`__,
Superseded-By: 734


.. note::
   This PEP effectively continues in a cleaner form in :pep:`734`.
   This PEP is kept as-is for the sake of the various sections of
   background information and deferred/rejected ideas that have
   been stripped from :pep:`734`.


Abstract
========

CPython has supported multiple interpreters in the same process (AKA
"subinterpreters") since version 1.5 (1997).  The feature has been
available via the C-API. [c-api]_  Multiple interpreters operate in
`relative isolation from one another <Interpreter Isolation_>`_, which
facilitates novel alternative approaches to
`concurrency <Concurrency_>`_.

This proposal introduces the stdlib ``interpreters`` module.  It exposes
the basic functionality of multiple interpreters already provided by the
C-API, along with basic support for communicating between interpreters.
This module is especially relevant since :pep:`684` introduced a
per-interpreter GIL in Python 3.12.


Proposal
========

Summary:

* add a new stdlib module: "interpreters"
* add concurrent.futures.InterpreterPoolExecutor
* help for extension module maintainers


The "interpreters" Module
-------------------------

The ``interpreters`` module will provide a high-level interface
to the multiple interpreter functionality, and wrap a new low-level
``_interpreters`` (in the same way as the ``threading`` module).
See the `Examples`_ section for concrete usage and use cases.

Along with exposing the existing (in CPython) multiple interpreter
support, the module will also support a basic mechanism for
passing data between interpreters.  That involves setting "shareable"
objects in the ``__main__`` module of a target subinterpreter.  Some
such objects, like  ``os.pipe()``, may be used to communicate further.
The module will also provide a minimal implementation of "channels"
as a demonstration of cross-interpreter communication.

Note that *objects* are not shared between interpreters since they are
tied to the interpreter in which they were created.  Instead, the
objects' *data* is passed between interpreters.  See the `Shared Data`_
and `API For Communication`_ sections for more details about
sharing/communicating between interpreters.

API summary for interpreters module
-----------------------------------

Here is a summary of the API for the ``interpreters`` module.  For a
more in-depth explanation of the proposed classes and functions, see
the `"interpreters" Module API`_ section below.

For creating and using interpreters:

+----------------------------------+----------------------------------------------+
| signature                        | description                                  |
+==================================+==============================================+
| ``list_all() -> [Interpreter]``  | Get all existing interpreters.               |
+----------------------------------+----------------------------------------------+
| ``get_current() -> Interpreter`` | Get the currently running interpreter.       |
+----------------------------------+----------------------------------------------+
| ``get_main() -> Interpreter``    | Get the main interpreter.                    |
+----------------------------------+----------------------------------------------+
| ``create() -> Interpreter``      | Initialize a new (idle) Python interpreter.  |
+----------------------------------+----------------------------------------------+

|

+----------------------------------+---------------------------------------------------+
| signature                        | description                                       |
+==================================+===================================================+
| ``class Interpreter``            | A single interpreter.                             |
+----------------------------------+---------------------------------------------------+
| ``.id``                          | The interpreter's ID (read-only).                 |
+----------------------------------+---------------------------------------------------+
| ``.is_running() -> bool``        | Is the interpreter currently executing code?      |
+----------------------------------+---------------------------------------------------+
| ``.close()``                     | Finalize and destroy the interpreter.             |
+----------------------------------+---------------------------------------------------+
| ``.set_main_attrs(**kwargs)``    | Bind "shareable" objects in ``__main__``.         |
+----------------------------------+---------------------------------------------------+
| ``.get_main_attr(name)``         | Get a "shareable" object from ``__main__``.       |
+----------------------------------+---------------------------------------------------+
| ``.exec(src_str, /)``            | | Run the given source code in the interpreter    |
|                                  | | (in the current thread).                        |
+----------------------------------+---------------------------------------------------+

.. XXX Support blocking interp.exec() until the interpreter
   finishes its current work.

For communicating between interpreters:

+---------------------------------------------------------+--------------------------------------------+
| signature                                               | description                                |
+=========================================================+============================================+
| ``is_shareable(obj) -> Bool``                           | | Can the object's data be passed          |
|                                                         | | between interpreters?                    |
+---------------------------------------------------------+--------------------------------------------+
| ``create_channel() -> (RecvChannel, SendChannel)``      | | Create a new channel for passing         |
|                                                         | | data between interpreters.               |
+---------------------------------------------------------+--------------------------------------------+

concurrent.futures.InterpreterPoolExecutor
------------------------------------------

An executor will be added that extends ``ThreadPoolExecutor`` to run
per-thread tasks in subinterpreters.  Initially, the only supported
tasks will be whatever ``Interpreter.exec()`` takes (e.g. a ``str``
script).  However, we may also support some functions, as well as
eventually a separate method for pickling the task and arguments,
to reduce friction (at the expense of performance
for short-running tasks).

Help for Extension Module Maintainers
-------------------------------------

In practice, an extension that implements multi-phase init (:pep:`489`)
is considered isolated and thus compatible with multiple


id: pep-0232.rst
content:
PEP: 232
Title: Function Attributes
Author: Barry Warsaw <barry@python.org>
Status: Final
Type: Standards Track
Created: 02-Dec-2000
Python-Version: 2.1
Post-History: 20-Feb-2001


Introduction
============

This PEP describes an extension to Python, adding attribute
dictionaries to functions and methods.  This PEP tracks the status
and ownership of this feature.  It contains a description of the
feature and outlines changes necessary to support the feature.
This PEP summarizes discussions held in mailing list forums, and
provides URLs for further information, where appropriate.  The CVS
revision history of this file contains the definitive historical
record.


Background
==========

Functions already have a number of attributes, some of which are
writable, e.g. ``func_doc``, a.k.a. ``func.__doc__``.  ``func_doc``
has the interesting property that there is special syntax in
function (and method) definitions for implicitly setting the
attribute.  This convenience has been exploited over and over again,
overloading docstrings with additional semantics.

For example, John Aycock has written a system where docstrings are
used to define parsing rules. [1]_  Zope's ZPublisher ORB [2]_ uses
docstrings to signal publishable methods, i.e. methods that can
be called through the web.

The problem with this approach is that the overloaded semantics
may conflict with each other.  For example, if we wanted to add a
doctest unit test to a Zope method that should not be publishable
through the web.


Proposal
========

This proposal adds a new dictionary to function objects, called
``func_dict`` (a.k.a. ``__dict__``).  This dictionary can be set
and get using ordinary attribute set and get syntax.

Methods also gain getter syntax, and they currently access the
attribute through the dictionary of the underlying function
object.  It is not possible to set attributes on bound or unbound
methods, except by doing so explicitly on the underlying function
object.  See the `Future Directions`_ discussion below for
approaches in subsequent versions of Python.

A function object's ``__dict__`` can also be set, but only to a
dictionary object.  Deleting a function's ``__dict__``, or setting
it to anything other than a concrete dictionary object results in a
``TypeError``.  If no function attributes have ever been set, the
function's ``__dict__`` will be empty.


Examples
========

Here are some examples of what you can do with this feature.

::

    def a():
        pass

    a.publish = 1
    a.unittest = '''...'''

    if a.publish:
        print a()

    if hasattr(a, 'unittest'):
        testframework.execute(a.unittest)

    class C:
        def a(self):
            'just a docstring'
            a.publish = 1

    c = C()
    if c.a.publish:
        publish(c.a())


Other Uses
----------

Paul Prescod enumerated a bunch of other uses on the `python-dev thread`_.

.. _python-dev thread: https://mail.python.org/pipermail/python-dev/2000-April/003364.html


Future Directions
=================

Here are a number of future directions to consider.  Any adoption
of these ideas would require a new PEP, which referenced this one,
and would have to be targeted at a Python version subsequent to
the 2.1 release.

- A previous version of this PEP allowed for both setter and
  getter of attributes on unbound methods, and only getter on
  bound methods.  A number of problems were discovered with this
  policy.

  Because method attributes were stored in the underlying
  function, this caused several potentially surprising results::

      class C:
          def a(self): pass

      c1 = C()
      c2 = C()
      c1.a.publish = 1
      # c2.a.publish would now be == 1 also!

  Because a change to ``a`` bound ``c1`` also caused a change to
  ``a`` bound to ``c2``, setting of attributes on bound methods
  was disallowed.  However, even allowing setting of attributes on
  unbound methods has its ambiguities::

      class D(C): pass
      class E(C): pass

      D.a.publish = 1
      # E.a.publish would now be == 1 also!

  For this reason, the current PEP disallows setting attributes on
  either bound or unbound methods, but does allow for getting
  attributes on either -- both return the attribute value on the
  underlying function object.

  A future PEP might propose to implement setting (bound or
  unbound) method attributes by setting attributes on the instance
  or class, using special naming conventions.  I.e.::

      class C:
          def a(self): pass

      C.a.publish = 1
      C.__a_publish__ == 1 # true

      c = C()
      c.a.publish = 2
      c.__a_publish__ == 2 # true

      d = C()
      d.__a_publish__ == 1 # true

  Here, a lookup on the instance would look to the instance's
  dictionary first, followed by a lookup on the class's
  dictionary, and finally a lookup on the function object's
  dictionary.

- Currently, Python supports function attributes only on Python
  functions (i.e. those that are written in Python, not those that
  are built-in).  Should it be worthwhile, a separate patch can be
  crafted that will add function attributes to built-ins.

- ``__doc__`` is the only function attribute that currently has
  syntactic support for conveniently setting.  It may be
  worthwhile to eventually enhance the language for supporting
  easy function attribute setting.  Here are some syntaxes
  suggested by PEP reviewers: [3]_

  .. code:: python

      def a {
          'publish' : 1,
          'unittest': '''...''',
          }
          (args):
          # ...

      def a(args):
          """The usual docstring."""
          {'publish' : 1,
           'unittest': '''...''',
           # etc.
           }

      def a(args) having (publish = 1):
          # see reference [3]
          pass

  The BDFL is currently against any such special syntactic support
  for setting arbitrary function attributes.  Any syntax proposals
  would have to be outlined in new PEPs.


Dissenting Opinion
==================

When this was discussed on the python-dev mailing list in April
2000, a number of dissenting opinions were voiced.  For
completeness, the discussion thread starts on `python-dev`_.

.. _python-dev: https://mail.python.org/pipermail/python-dev/2000-April/003361.html

The dissenting arguments appear to fall under the following
categories:

- no clear purpose (what does it buy you?)
- other ways to do it (e.g. mappings as class attributes)
- useless until syntactic support is included

Countering some of these arguments is the observation that with
vanilla Python 2.0, ``__doc__`` can in fact be set to any type of
object, so some semblance of writable function attributes are
already feasible.  But that approach is yet another corruption of
``__doc__``.

And while it is of course possible to add mappings to class
objects (or in the case of function attributes, to the function's
module), it is more difficult and less obvious how to extract the
attribute values for inspection.

Finally, it may be desirable to add syntactic support, much the
same way that ``__doc__`` syntactic support exists.  This can be
considered separately from the ability to actually set and get
function attributes.


Reference Implementation
========================

This PEP has been accepted and the implementation has been
integrated into Python 2.1.


References
==========

.. [1] Aycock, "Compiling Little Languages in Python",
   https://legacy.python.org/workshops/1998-11/proceedings/papers/aycock-little/aycock-little.html

.. [2] https://web.archive.org/web/20010307022153/http://classic.zope.org:8080/Documentation/Reference/ORB

.. [3] Hudson, Michael, SourceForge patch implementing this syntax,
   https://web.archive.org/web/20010901050535/http://sourceforge.net/tracker/index.php?func=detail&aid=403441&group_id=5470&atid=305470


Copyright
=========

This document has been placed in the public domain.


id: pep-0226.rst
content:
PEP: 226
Title: Python 2.1 Release Schedule
Author: Jeremy Hylton <jeremy@alum.mit.edu>
Status: Final
Type: Informational
Topic: Release
Created: 16-Oct-2000
Python-Version: 2.1
Post-History:


Abstract
========

This document describes the post Python 2.0 development and
release schedule.  According to this schedule, Python 2.1 will be
released in April of 2001.  The schedule primarily concerns
itself with PEP-size items.  Small bug fixes and changes will
occur up until the first beta release.


Release Schedule
================

Tentative future release dates

[bugfix release dates go here]

Past release dates:

- 17-Apr-2001: 2.1 final release
- 15-Apr-2001: 2.1 release candidate 2
- 13-Apr-2001: 2.1 release candidate 1
- 23-Mar-2001: Python 2.1 beta 2 release
- 02-Mar-2001: First 2.1 beta release
- 02-Feb-2001: Python 2.1 alpha 2 release
- 22-Jan-2001: Python 2.1 alpha 1 release
- 16-Oct-2000: Python 2.0 final release


Open issues for Python 2.0 beta 2
=================================

Add a default unit testing framework to the standard library.


Guidelines for making changes for Python 2.1
============================================

The guidelines and schedule will be revised based on discussion in
the python-dev@python.org mailing list.

The PEP system was instituted late in the Python 2.0 development
cycle and many changes did not follow the process described in :pep:`1`.
The development process for 2.1, however, will follow the PEP
process as documented.

The first eight weeks following 2.0 final will be the design and
review phase.  By the end of this period, any PEP that is proposed
for 2.1 should be ready for review.  This means that the PEP is
written and discussion has occurred on the python-dev@python.org
and python-list@python.org mailing lists.

The next six weeks will be spent reviewing the PEPs and
implementing and testing the accepted PEPs.  When this period
stops, we will end consideration of any incomplete PEPs.  Near the
end of this period, there will be a feature freeze where any small
features not worthy of a PEP will not be accepted.

Before the final release, we will have six weeks of beta testing
and a release candidate or two.


General guidelines for submitting patches and making changes
============================================================

Use good sense when committing changes.  You should know what we
mean by good sense or we wouldn't have given you commit privileges
<0.5 wink>.  Some specific examples of good sense include:

- Do whatever the dictator tells you.

- Discuss any controversial changes on python-dev first.  If you
  get a lot of +1 votes and no -1 votes, make the change.  If you
  get a some -1 votes, think twice; consider asking Guido what he
  thinks.

- If the change is to code you contributed, it probably makes
  sense for you to fix it.

- If the change affects code someone else wrote, it probably makes
  sense to ask him or her first.

- You can use the SourceForge (SF) Patch Manager to submit a patch
  and assign it to someone for review.

Any significant new feature must be described in a PEP and
approved before it is checked in.

Any significant code addition, such as a new module or large
patch, must include test cases for the regression test and
documentation.  A patch should not be checked in until the tests
and documentation are ready.

If you fix a bug, you should write a test case that would have
caught the bug.

If you commit a patch from the SF Patch Manager or fix a bug from
the Jitterbug database, be sure to reference the patch/bug number
in the CVS log message.  Also be sure to change the status in the
patch manager or bug database (if you have access to the bug
database).

It is not acceptable for any checked in code to cause the
regression test to fail.  If a checkin causes a failure, it must
be fixed within 24 hours or it will be backed out.

All contributed C code must be ANSI C.  If possible check it with
two different compilers, e.g. gcc and MSVC.

All contributed Python code must follow Guido's Python style
guide.  http://www.python.org/doc/essays/styleguide.html

It is understood that any code contributed will be released under
an Open Source license.  Do not contribute code if it can't be
released this way.


id: pep-0540.rst
content:
PEP: 540
Title: Add a new UTF-8 Mode
Author: Victor Stinner <vstinner@python.org>
BDFL-Delegate: INADA Naoki
Status: Final
Type: Standards Track
Created: 05-Jan-2016
Python-Version: 3.7
Resolution: https://mail.python.org/pipermail/python-dev/2017-December/151173.html


Abstract
========

Add a new "UTF-8 Mode" to enhance Python's use of UTF-8.  When UTF-8 Mode
is active, Python will:

* use the ``utf-8`` encoding, regardless of the locale currently set by
  the current platform, and
* change the ``stdin`` and ``stdout`` error handlers to
  ``surrogateescape``.

This mode is off by default, but is automatically activated when using
the "POSIX" locale.

Add the ``-X utf8`` command line option and ``PYTHONUTF8`` environment
variable to control UTF-8 Mode.


Rationale
=========

Locale encoding and UTF-8
-------------------------

Python 3.6 uses the locale encoding for filenames, environment
variables, standard streams, etc. The locale encoding is inherited from
the locale; the encoding and the locale are tightly coupled.

Many users inherit the ASCII encoding from the POSIX locale, aka the "C"
locale, but are unable change the locale for various reasons.  This
encoding is very limited in term of Unicode support: any non-ASCII
character is likely to cause trouble.

It isn't always easy to get an accurate locale.  Locales don't get the
exact same name on different Linux distributions, FreeBSD, macOS, etc.
And some locales, like the recent ``C.UTF-8`` locale, are only supported
by a few platforms.  The current locale can even vary on the *same*
platform depending on context; for example, a SSH connection can use a
different encoding than the filesystem or local terminal encoding on the
same machine.

On the flip side, Python 3.6 is already using UTF-8 by default on macOS,
Android and Windows (:pep:`529`) for most functions -- although
``open()`` is a notable exception here. UTF-8 is also the default
encoding of Python scripts, XML and JSON file formats. The Go
programming language
uses UTF-8 for all strings.

UTF-8 support is nearly ubiquitous for data read and written by modern
platforms.  It also has excellent support in Python.  The problem is
simply that the locale is frequently misconfigured.  An obvious solution
suggests itself: ignore the locale encoding and use UTF-8.


Passthrough for undecodable bytes: surrogateescape
--------------------------------------------------

When decoding bytes from UTF-8 using the default ``strict`` error
handler, Python 3 raises a ``UnicodeDecodeError`` on the first
undecodable byte.

Unix command line tools like ``cat`` or ``grep`` and most Python 2
applications simply do not have this class of bugs: they don't decode
data, but process data as a raw bytes sequence.

Python 3 already has a solution to behave like Unix tools and Python 2:
the ``surrogateescape`` error handler (:pep:`383`). It allows processing
data as if it were bytes, but uses Unicode in practice; undecodable
bytes are stored as surrogate characters.

UTF-8 Mode sets the ``surrogateescape`` error handler for ``stdin``
and ``stdout``, since these streams as commonly associated to Unix
command line tools.

However, users have a different expectation on files. Files are expected
to be properly encoded, and Python is expected to fail early when
``open()`` is called with the wrong options, like opening a JPEG picture
in text mode. The ``open()`` default error handler remains ``strict``
for these reasons.


No change by default for best backward compatibility
----------------------------------------------------

While UTF-8 is perfect in most cases, sometimes the locale encoding is
actually the best encoding.

This PEP changes the behaviour for the POSIX locale since this locale is
usually equivalent to the ASCII encoding, whereas UTF-8 is a much better
choice. It does not change the behaviour for other locales to prevent
any risk or regression.

As users are responsible to enable explicitly the new UTF-8 Mode for
these other locales, they are responsible for any potential mojibake
issues caused by UTF-8 Mode.


Proposal
========

Add a new UTF-8 Mode to use the UTF-8 encoding, ignore the locale
encoding, and change ``stdin`` and ``stdout`` error handlers to
``surrogateescape``.

Add the new ``-X utf8`` command line option and ``PYTHONUTF8``
environment variable.  Users can explicitly activate UTF-8 Mode with the
command-line option ``-X utf8`` or by setting the environment variable
``PYTHONUTF8=1``.

This mode is disabled by default and enabled by the POSIX locale.  Users
can explicitly disable UTF-8 Mode with the command-line option ``-X
utf8=0`` or by setting the environment variable ``PYTHONUTF8=0``.

For standard streams, the ``PYTHONIOENCODING`` environment variable has
priority over UTF-8 Mode.

On Windows, the ``PYTHONLEGACYWINDOWSFSENCODING`` environment variable
(:pep:`529`) has the priority over UTF-8 Mode.

Effects of UTF-8 Mode:

* ``sys.getfilesystemencoding()`` returns ``'UTF-8'``.
* ``locale.getpreferredencoding()`` returns ``UTF-8``; its
  *do_setlocale* argument, and the locale encoding, are ignored.
* ``sys.stdin`` and ``sys.stdout`` error handler is set to
  ``surrogateescape``.

Side effects:

* ``open()`` uses the UTF-8 encoding by default.  However, it still
  uses the ``strict`` error handler by default.
* ``os.fsdecode()`` and ``os.fsencode()`` use the UTF-8 encoding.
* Command line arguments, environment variables and filenames use the
  UTF-8 encoding.


Relationship with the locale coercion (PEP 538)
===============================================

The POSIX locale enables the locale coercion (:pep:`538`) and the UTF-8
mode (:pep:`540`). When the locale coercion is enabled, enabling the
UTF-8 mode has no additional effect.

The UTF-8 Mode has the same effect as locale coercion:

* ``sys.getfilesystemencoding()`` returns ``'UTF-8'``,
* ``locale.getpreferredencoding()`` returns ``UTF-8``, and
* the ``sys.stdin`` and ``sys.stdout`` error handlers are set to
  ``surrogateescape``.

These changes only affect Python code. But the locale coercion has
additional effects: the ``LC_CTYPE`` environment variable and the
``LC_CTYPE`` locale are set to a UTF-8 locale like ``C.UTF-8``. One side
effect is that non-Python code is also impacted by the locale coercion.
The two PEPs are complementary.

On platforms like Centos 7 where locale coercion is not supported, the
POSIX locale only enables UTF-8 Mode.  In this case, Python code uses
the UTF-8 encoding and ignores the locale encoding, whereas non-Python
code uses the locale encoding, which is usually ASCII for the POSIX
locale.

While the UTF-8 Mode is supported on all platforms and can be enabled
with any locale, the locale coercion is not supported by all platforms
and is restricted to the POSIX locale.

The UTF-8 Mode has only an impact on Python child processes when the
``PYTHONUTF8`` environment variable is set to ``1``, whereas the locale
coercion sets the ``LC_CTYPE`` environment variables which impacts all
child processes.

The benefit of the locale coercion approach is that it helps ensure that
encoding handling in binary extension modules and child processes is
consistent with Python's encoding handling. The upside of the UTF-8 Mode
approach is that it allows an embedding application to change the
interpreter's behaviour without having to change the process global
locale settings.


Backward Compatibility
======================

The only backward incompatible change is that the POSIX locale now
enables the UTF-8 Mode by default: it will now use the UTF-8 encoding,
ignore the locale encoding, and change ``stdin`` and ``stdout`` error
handlers to ``surrogateescape``.


Annex: Encodings And Error Handlers
===================================

UTF-8 Mode changes the default encoding and error handler used by
``open()``, ``os.fsdecode()``, ``os.fsencode()``, ``sys.stdin``,
``sys.stdout`` and ``sys.stderr``.

Encoding and error handler
--------------------------

=====================


id: pep-0578.rst
content:
PEP: 578
Title: Python Runtime Audit Hooks
Author: Steve Dower <steve.dower@python.org>
BDFL-Delegate: Christian Heimes <christian@python.org>
Status: Final
Type: Standards Track
Created: 16-Jun-2018
Python-Version: 3.8
Post-History: 28-Mar-2019, 07-May-2019

.. canonical-doc:: :ref:`python:audit-events`

Abstract
========

This PEP describes additions to the Python API and specific behaviors
for the CPython implementation that make actions taken by the Python
runtime visible to auditing tools. Visibility into these actions
provides opportunities for test frameworks, logging frameworks, and
security tools to monitor and optionally limit actions taken by the
runtime.

This PEP proposes adding two APIs to provide insights into a running
Python application: one for arbitrary events, and another specific to
the module import system. The APIs are intended to be available in all
Python implementations, though the specific messages and values used
are unspecified here to allow implementations the freedom to determine
how best to provide information to their users. Some examples likely
to be used in CPython are provided for explanatory purposes.

See :pep:`551` for discussion and recommendations on enhancing the
security of a Python runtime making use of these auditing APIs.

Background
==========

Python provides access to a wide range of low-level functionality on
many common operating systems. While this is incredibly useful for
"write-once, run-anywhere" scripting, it also makes monitoring of
software written in Python difficult. Because Python uses native system
APIs directly, existing monitoring tools either suffer from limited
context or auditing bypass.

Limited context occurs when system monitoring can report that an
action occurred, but cannot explain the sequence of events leading to
it. For example, network monitoring at the OS level may be able to
report "listening started on port 5678", but may not be able to
provide the process ID, command line, parent process, or the local
state in the program at the point that triggered the action. Firewall
controls to prevent such an action are similarly limited, typically
to process names or some global state such as the current user, and
in any case rarely provide a useful log file correlated with other
application messages.

Auditing bypass can occur when the typical system tool used for an
action would ordinarily report its use, but accessing the APIs via
Python do not trigger this. For example, invoking "curl" to make HTTP
requests may be specifically monitored in an audited system, but
Python's "urlretrieve" function is not.

Within a long-running Python application, particularly one that
processes user-provided information such as a web app, there is a risk
of unexpected behavior. This may be due to bugs in the code, or
deliberately induced by a malicious user. In both cases, normal
application logging may be bypassed resulting in no indication that
anything out of the ordinary has occurred.

Additionally, and somewhat unique to Python, it is very easy to affect
the code that is run in an application by manipulating either the
import system's search path or placing files earlier on the path than
intended. This is often seen when developers create a script with the
same name as the module they intend to use - for example, a
``random.py`` file that attempts to import the standard library
``random`` module.

This is not sandboxing, as this proposal does not attempt to prevent
malicious behavior (though it enables some new options to do so).
See the `Why Not A Sandbox`_ section below for further discussion.

Overview of Changes
===================

The aim of these changes is to enable both application developers and
system administrators to integrate Python into their existing
monitoring systems without dictating how those systems look or behave.

We propose two API changes to enable this: an Audit Hook and Verified
Open Hook. Both are available from Python and native code, allowing
applications and frameworks written in pure Python code to take
advantage of the extra messages, while also allowing embedders or
system administrators to deploy builds of Python where auditing is
always enabled.

Only CPython is bound to provide the native APIs as described here.
Other implementations should provide the pure Python APIs, and
may provide native versions as appropriate for their underlying
runtimes. Auditing events are likewise considered implementation
specific, but are bound by normal feature compatibility guarantees.

Audit Hook
----------

In order to observe actions taken by the runtime (on behalf of the
caller), an API is required to raise messages from within certain
operations. These operations are typically deep within the Python
runtime or standard library, such as dynamic code compilation, module
imports, DNS resolution, or use of certain modules such as ``ctypes``.

The following new C APIs allow embedders and CPython implementors to
send and receive audit hook messages::

   # Add an auditing hook
   typedef int (*hook_func)(const char *event, PyObject *args,
                            void *userData);
   int PySys_AddAuditHook(hook_func hook, void *userData);

   # Raise an event with all auditing hooks
   int PySys_Audit(const char *event, PyObject *args);

The new Python APIs for receiving and raising audit hooks are::

   # Add an auditing hook
   sys.addaudithook(hook: Callable[[str, tuple]])

   # Raise an event with all auditing hooks
   sys.audit(str, *args)


Hooks are added by calling ``PySys_AddAuditHook()`` from C at any time,
including before ``Py_Initialize()``, or by calling
``sys.addaudithook()`` from Python code. Hooks cannot be removed or
replaced. For CPython, hooks added from C are global, while hooks added
from Python are only for the current interpreter. Global hooks are
executed before interpreter hooks.

When events of interest are occurring, code can either call
``PySys_Audit()`` from C (while the GIL is held) or ``sys.audit()``. The
string argument is the name of the event, and the tuple contains
arguments. A given event name should have a fixed schema for arguments,
which should be considered a public API (for each x.y version release),
and thus should only change between feature releases with updated
documentation. To minimize overhead and simplify handling in native code
hook implementations, named arguments are not supported.

For maximum compatibility, events using the same name as an event in
the reference interpreter CPython should make every attempt to use
compatible arguments. Including the name or an abbreviation of the
implementation in implementation-specific event names will also help
prevent collisions. For example, a ``pypy.jit_invoked`` event is clearly
distinguished from an ``ipy.jit_invoked`` event. Events raised from
Python modules should include their module or package name in the event
name.

While event names may be arbitrary UTF-8 strings, for consistency across
implementations it is recommended to use valid Python dotted names and
avoid encoding specific details in the name. For example, an ``import``
event with the module name ``spam`` as an argument is preferable to a
``spam module imported`` event with no arguments. Avoid using embedded
null characters or you may upset those who implement hooks using C.

When an event is audited, each hook is called in the order it was added
(as much as is possible), passing the event name and arguments. If any
hook returns with an exception set, later hooks are ignored and *in
general* the Python runtime should terminate - exceptions from hooks are
not intended to be handled or treated as expected occurrences. This
allows hook implementations to decide how to respond to any particular
event. The typical responses will be to log the event, abort the
operation with an exception, or to immediately terminate the process with
an operating system exit call.

When an event is audited but no hooks have been set, th


id: pep-0544.rst
content:
PEP: 544
Title: Protocols: Structural subtyping (static duck typing)
Author: Ivan Levkivskyi <levkivskyi@gmail.com>, Jukka Lehtosalo <jukka.lehtosalo@iki.fi>, Åukasz Langa <lukasz@python.org>
BDFL-Delegate: Guido van Rossum <guido@python.org>
Discussions-To: python-dev@python.org
Status: Final
Type: Standards Track
Topic: Typing
Created: 05-Mar-2017
Python-Version: 3.8
Resolution: https://mail.python.org/archives/list/typing-sig@python.org/message/FDO4KFYWYQEP3U2HVVBEBR3SXPHQSHYR/

.. canonical-typing-spec:: :ref:`typing:protocols` and
                           :py:class:`typing.Protocol`


Abstract
========

Type hints introduced in :pep:`484` can be used to specify type metadata
for static type checkers and other third party tools. However, :pep:`484`
only specifies the semantics of *nominal* subtyping. In this PEP we specify
static and runtime semantics of protocol classes that will provide a support
for *structural* subtyping (static duck typing).


.. _PEP 544 rationale:

Rationale and Goals
===================

Currently, :pep:`484` and the ``typing`` module [typing]_ define abstract
base classes for several common Python protocols such as ``Iterable`` and
``Sized``. The problem with them is that a class has to be explicitly marked
to support them, which is unpythonic and unlike what one would
normally do in idiomatic dynamically typed Python code. For example,
this conforms to :pep:`484`::

  from typing import Sized, Iterable, Iterator

  class Bucket(Sized, Iterable[int]):
      ...
      def __len__(self) -> int: ...
      def __iter__(self) -> Iterator[int]: ...

The same problem appears with user-defined ABCs: they must be explicitly
subclassed or registered. This is particularly difficult to do with library
types as the type objects may be hidden deep in the implementation
of the library. Also, extensive use of ABCs might impose additional
runtime costs.

The intention of this PEP is to solve all these problems
by allowing users to write the above code without explicit base classes in
the class definition, allowing ``Bucket`` to be implicitly considered
a subtype of both ``Sized`` and ``Iterable[int]`` by static type checkers
using structural [wiki-structural]_ subtyping::

  from typing import Iterator, Iterable

  class Bucket:
      ...
      def __len__(self) -> int: ...
      def __iter__(self) -> Iterator[int]: ...

  def collect(items: Iterable[int]) -> int: ...
  result: int = collect(Bucket())  # Passes type check

Note that ABCs in ``typing`` module already provide structural behavior
at runtime, ``isinstance(Bucket(), Iterable)`` returns ``True``.
The main goal of this proposal is to support such behavior statically.
The same functionality will be provided for user-defined protocols, as
specified below. The above code with a protocol class matches common Python
conventions much better. It is also automatically extensible and works
with additional, unrelated classes that happen to implement
the required protocol.


Nominal vs structural subtyping
-------------------------------

Structural subtyping is natural for Python programmers since it matches
the runtime semantics of duck typing: an object that has certain properties
is treated independently of its actual runtime class.
However, as discussed in :pep:`483`, both nominal and structural
subtyping have their strengths and weaknesses. Therefore, in this PEP we
*do not propose* to replace the nominal subtyping described by :pep:`484` with
structural subtyping completely. Instead, protocol classes as specified in
this PEP complement normal classes, and users are free to choose
where to apply a particular solution. See section on `rejected
<PEP 544 rejected_>`_ ideas at the end of this PEP for additional motivation.


Non-goals
---------

At runtime, protocol classes will be simple ABCs. There is no intent to
provide sophisticated runtime instance and class checks against protocol
classes. This would be difficult and error-prone and will contradict the logic
of :pep:`484`. As well, following :pep:`484` and :pep:`526` we state that protocols are
**completely optional**:

* No runtime semantics will be imposed for variables or parameters annotated
  with a protocol class.
* Any checks will be performed only by third-party type checkers and
  other tools.
* Programmers are free to not use them even if they use type annotations.
* There is no intent to make protocols non-optional in the future.

To reiterate, providing complex runtime semantics for protocol classes
is not a goal of this PEP, the main goal is to provide a support and standards
for *static* structural subtyping. The possibility to use protocols
in the runtime context as ABCs is rather a minor bonus that exists mostly
to provide a seamless transition for projects that already use ABCs.


Existing Approaches to Structural Subtyping
===========================================

Before describing the actual specification, we review and comment on existing
approaches related to structural subtyping in Python and other languages:

* ``zope.interface`` [zope-interfaces]_ was one of the first widely used
  approaches to structural subtyping in Python. It is implemented by providing
  special classes to distinguish interface classes from normal classes,
  to mark interface attributes, and to explicitly declare implementation.
  For example::

    from zope.interface import Interface, Attribute, implementer

    class IEmployee(Interface):

        name = Attribute("Name of employee")

        def do(work):
            """Do some work"""

    @implementer(IEmployee)
    class Employee:

        name = 'Anonymous'

        def do(self, work):
            return work.start()

  Zope interfaces support various contracts and constraints for interface
  classes. For example::

    from zope.interface import invariant

    def required_contact(obj):
        if not (obj.email or obj.phone):
            raise Exception("At least one contact info is required")

    class IPerson(Interface):

        name = Attribute("Name")
        email = Attribute("Email Address")
        phone = Attribute("Phone Number")

        invariant(required_contact)

  Even more detailed invariants are supported. However, Zope interfaces rely
  entirely on runtime validation. Such focus on runtime properties goes
  beyond the scope of the current proposal, and static support for invariants
  might be difficult to implement. However, the idea of marking an interface
  class with a special base class is reasonable and easy to implement both
  statically and at runtime.

* Python abstract base classes [abstract-classes]_ are the standard
  library tool to provide some functionality similar to structural subtyping.
  The drawback of this approach is the necessity to either subclass
  the abstract class or register an implementation explicitly::

    from abc import ABC

    class MyTuple(ABC):
        pass

    MyTuple.register(tuple)

    assert issubclass(tuple, MyTuple)
    assert isinstance((), MyTuple)

  As mentioned in the `rationale <PEP 544 rationale_>`_,
  we want to avoid such necessity, especially in static context.
  However, in a runtime context, ABCs are good candidates for
  protocol classes and they are already used extensively in
  the ``typing`` module.

* Abstract classes defined in ``collections.abc`` module [collections-abc]_
  are slightly more advanced since they implement a custom
  ``__subclasshook__()`` method that allows runtime structural checks without
  explicit registration::

    from collections.abc import Iterable

    class MyIterable:
        def __iter__(self):
            return []

    assert isinstance(MyIterable(), Iterable)

  Such behavior seems to be a perfect fit for both runtime and static behavior
  of protocols. As discussed in `rationale <PEP 544 rationale_>`_,
  we propose to add static support for such behavior.
  In addition, to allow users to achieve such runtime
  behavior for *user-defined* protocols a sp


id: pep-0222.rst
content:
PEP: 222
Title: Web Library Enhancements
Author: A.M. Kuchling <amk@amk.ca>
Status: Deferred
Type: Standards Track
Created: 18-Aug-2000
Python-Version: 2.1
Post-History: 22-Dec-2000


Abstract
========

This PEP proposes a set of enhancements to the CGI development
facilities in the Python standard library.  Enhancements might be
new features, new modules for tasks such as cookie support, or
removal of obsolete code.

The original intent was to make improvements to Python 2.1.
However, there seemed little interest from the Python community,
and time was lacking, so this PEP has been deferred to some future
Python release.


Open Issues
===========

This section lists changes that have been suggested, but about
which no firm decision has yet been made.  In the final version of
this PEP, this section should be empty, as all the changes should
be classified as accepted or rejected.

cgi.py: We should not be told to create our own subclass just so
we can handle file uploads. As a practical matter, I have yet to
find the time to do this right, so I end up reading cgi.py's temp
file into, at best, another file. Some of our legacy code actually
reads it into a second temp file, then into a final destination!
And even if we did, that would mean creating yet another object
with its ``__init__`` call and associated overhead.

cgi.py: Currently, query data with no ``=`` are ignored.  Even if
``keep_blank_values`` is set, queries like ``...?value=&...`` are
returned with blank values but queries like ``...?value&...`` are
completely lost.  It would be great if such data were made
available through the ``FieldStorage`` interface, either as entries
with ``None`` as values, or in a separate list.

Utility function: build a query string from a list of 2-tuples

Dictionary-related utility classes: ``NoKeyErrors`` (returns an empty
string, never a ``KeyError``), ``PartialStringSubstitution`` (returns
the original key string, never a ``KeyError``)



New Modules
===========

This section lists details about entire new packages or modules
that should be added to the Python standard library.

* fcgi.py : A new module adding support for the FastCGI protocol.
  Robin Dunn's code needs to be ported to Windows, though.


Major Changes to Existing Modules
=================================

This section lists details of major changes to existing modules,
whether in implementation or in interface.  The changes in this
section therefore carry greater degrees of risk, either in
introducing bugs or a backward incompatibility.

The cgi.py module would be deprecated.  (XXX A new module or
package name hasn't been chosen yet: 'web'?  'cgilib'?)


Minor Changes to Existing Modules
=================================

This section lists details of minor changes to existing modules.
These changes should have relatively small implementations, and
have little risk of introducing incompatibilities with previous
versions.


Rejected Changes
================

The changes listed in this section were proposed for Python 2.1,
but were rejected as unsuitable.  For each rejected change, a
rationale is given describing why the change was deemed
inappropriate.

* An HTML generation module is not part of this PEP.  Several such
  modules exist, ranging from HTMLgen's purely programming
  interface to ASP-inspired simple templating to DTML's complex
  templating.  There's no indication of which templating module to
  enshrine in the standard library, and that probably means that
  no module should be so chosen.

* cgi.py: Allowing a combination of query data and POST data.
  This doesn't seem to be standard at all, and therefore is
  dubious practice.


Proposed Interface
==================

XXX open issues: naming convention (studlycaps or
underline-separated?); need to look at the ``cgi.parse*()`` functions
and see if they can be simplified, too.

Parsing functions: carry over most of the ``parse*`` functions from
cgi.py

::

    # The Response class borrows most of its methods from Zope's
    # HTTPResponse class.

    class Response:
        """
        Attributes:
        status: HTTP status code to return
        headers: dictionary of response headers
        body: string containing the body of the HTTP response
        """

        def __init__(self, status=200, headers={}, body=""):
            pass

        def setStatus(self, status, reason=None):
            "Set the numeric HTTP response code"
            pass

        def setHeader(self, name, value):
            "Set an HTTP header"
            pass

        def setBody(self, body):
            "Set the body of the response"
            pass

        def setCookie(self, name, value,
                      path = '/',
                      comment = None,
                      domain = None,
                      max-age = None,
                      expires = None,
                      secure = 0
                      ):
            "Set a cookie"
            pass

        def expireCookie(self, name):
            "Remove a cookie from the user"
            pass

        def redirect(self, url):
            "Redirect the browser to another URL"
            pass

        def __str__(self):
            "Convert entire response to a string"
            pass

        def dump(self):
            "Return a string representation useful for debugging"
            pass

        # XXX methods for specific classes of error:serverError,
        # badRequest, etc.?


    class Request:

        """
        Attributes:

        XXX should these be dictionaries, or dictionary-like objects?
        .headers : dictionary containing HTTP headers
        .cookies : dictionary of cookies
        .fields  : data from the form
        .env     : environment dictionary
        """

        def __init__(self, environ=os.environ, stdin=sys.stdin,
                     keep_blank_values=1, strict_parsing=0):
            """Initialize the request object, using the provided environment
            and standard input."""
            pass

        # Should people just use the dictionaries directly?
        def getHeader(self, name, default=None):
            pass

        def getCookie(self, name, default=None):
            pass

        def getField(self, name, default=None):
            "Return field's value as a string (even if it's an uploaded file)"
            pass

        def getUploadedFile(self, name):
            """Returns a file object that can be read to obtain the contents
            of an uploaded file.  XXX should this report an error if the
            field isn't actually an uploaded file?  Or should it wrap
            a StringIO around simple fields for consistency?
            """

        def getURL(self, n=0, query_string=0):
            """Return the URL of the current request, chopping off 'n' path
            components from the right.  Eg. if the URL is
            "http://foo.com/bar/baz/quux", n=2 would return
            "http://foo.com/bar".  Does not include the query string (if
            any)
            """

        def getBaseURL(self, n=0):
            """Return the base URL of the current request, adding 'n' path
            components to the end to recreate more of the whole URL.

            Eg. if the request URL is
            "http://foo.com/q/bar/baz/qux", n=0 would return
            "http://foo.com/", and n=2 "http://foo.com/q/bar".

            Returned URL does not include the query string, if any.
            """

        def dump(self):
            "String representation suitable for debugging output"
            pass

        # Possibilities?  I don't know if these are worth doing in the
        # basic objects.
        def getBrowser(self):
            "Returns Mozilla/IE/Lynx/Opera/whatever"

        def isSecure(self):
            "Return true if this is an SSLified request"


    # Module-level function
    def wrapper(func, logfile=sys.stderr):
        """
        Calls the function 'func', passing it the ar


id: pep-0236.rst
content:
PEP: 236
Title: Back to the __future__
Author: Tim Peters <tim.peters@gmail.com>
Status: Final
Type: Standards Track
Created: 26-Feb-2001
Python-Version: 2.1
Post-History: 26-Feb-2001


Motivation
==========

From time to time, Python makes an incompatible change to the advertised
semantics of core language constructs, or changes their accidental
(implementation-dependent) behavior in some way.  While this is never done
capriciously, and is always done with the aim of improving the language over
the long term, over the short term it's contentious and disrupting.

:pep:`5`, Guidelines for Language Evolution suggests ways to ease the pain,
and this PEP introduces some machinery in support of that.

:pep:`227`, Statically Nested Scopes is the first application, and will be
used as an example here.


Intent
======

[Note:  This is policy, and so should eventually move into :pep:`5`]

When an incompatible change to core language syntax or semantics is being
made:

1. The release C that introduces the change does not change the syntax or
   semantics by default.

2. A future release R is identified in which the new syntax or semantics will
   be enforced.

3. The mechanisms described in :pep:`230`, Warning Framework are used to
   generate warnings, whenever possible, about constructs or operations whose
   meaning may [1]_ change in release R.

4. The new future_statement (see below) can be explicitly included in a module
   M to request that the code in module M use the new syntax or semantics in
   the current release C.

So old code continues to work by default, for at least one release, although
it may start to generate new warning messages.  Migration to the new syntax or
semantics can proceed during that time, using the future_statement to make
modules containing it act as if the new syntax or semantics were already being
enforced.

Note that there is no need to involve the future_statement machinery in new
features unless they can break existing code; fully backward- compatible
additions can-- and should --be introduced without a corresponding
future_statement.


Syntax
======

A future_statement is simply a from/import statement using the reserved module
name ``__future__``::

    future_statement: "from" "__future__" "import" feature ["as" name]
                      (","feature ["as" name])*

    feature: identifier
    name: identifier

In addition, all future_statements must appear near the top of the module.  The
only lines that can appear before a future_statement are:

+ The module docstring (if any).
+ Comments.
+ Blank lines.
+ Other future_statements.

Example::

    """This is a module docstring."""

    # This is a comment, preceded by a blank line and followed by
    # a future_statement.
    from __future__ import nested_scopes

    from math import sin
    from __future__ import alabaster_weenoblobs  # compile-time error!
    # That was an error because preceded by a non-future_statement.


Semantics
=========

A future_statement is recognized and treated specially at compile time:
changes to the semantics of core constructs are often implemented by
generating different code.  It may even be the case that a new feature
introduces new incompatible syntax (such as a new reserved word), in which
case the compiler may need to parse the module differently.  Such decisions
cannot be pushed off until runtime.

For any given release, the compiler knows which feature names have been
defined, and raises a compile-time error if a future_statement contains a
feature not known to it [2]_.

The direct runtime semantics are the same as for any ``import`` statement:
there is a standard module ``__future__.py``, described later, and it will be
imported in the usual way at the time the future_statement is executed.

The *interesting* runtime semantics depend on the specific feature(s)
"imported" by the future_statement(s) appearing in the module.

Note that there is nothing special about the statement::

    import __future__ [as name]

That is not a future_statement; it's an ordinary import statement, with no
special semantics or syntax restrictions.


Example
=======

Consider this code, in file scope.py::

    x = 42
    def f():
        x = 666
        def g():
            print "x is", x
        g()
    f()

Under 2.0, it prints::

    x is 42

Nested scopes (:pep:`227`) are being introduced in 2.1.  But under 2.1, it still
prints::

    x is 42

and also generates a warning.

In 2.2, and also in 2.1 *if* ``from __future__ import nested_scopes`` is
included at the top of ``scope.py``, it prints::

    x is 666


Standard Module __future__.py
=============================

``Lib/__future__.py`` is a real module, and serves three purposes:

1. To avoid confusing existing tools that analyze import statements and expect
   to find the modules they're importing.

2. To ensure that future_statements run under releases prior to 2.1 at least
   yield runtime exceptions (the import of ``__future__`` will fail, because
   there was no module of that name prior to 2.1).

3. To document when incompatible changes were introduced, and when they will
   be-- or were --made mandatory.  This is a form of executable documentation,
   and can be inspected programmatically via importing ``__future__`` and
   examining its contents.

Each statement in ``__future__.py`` is of the form::

    FeatureName = "_Feature(" OptionalRelease "," MandatoryRelease ")"

where, normally, *OptionalRelease* <  *MandatoryRelease*, and both are
5-tuples of the same form as ``sys.version_info``::

    (PY_MAJOR_VERSION, # the 2 in 2.1.0a3; an int
     PY_MINOR_VERSION, # the 1; an int
     PY_MICRO_VERSION, # the 0; an int
     PY_RELEASE_LEVEL, # "alpha", "beta", "candidate" or "final"; string
     PY_RELEASE_SERIAL # the 3; an int )

*OptionalRelease* records the first release in which::

    from __future__ import FeatureName

was accepted.

In the case of *MandatoryReleases* that have not yet occurred,
*MandatoryRelease* predicts the release in which the feature will become part
of the language.

Else *MandatoryRelease* records when the feature became part of the language;
in releases at or after that, modules no longer need::

    from __future__ import FeatureName

to use the feature in question, but may continue to use such imports.

*MandatoryRelease* may also be ``None``, meaning that a planned feature got
dropped.

Instances of ``class _Feature`` have two corresponding methods,
``.getOptionalRelease()`` and ``.getMandatoryRelease()``.

No feature line will ever be deleted from ``__future__.py``.

Example line::

    nested_scopes = _Feature((2, 1, 0, "beta", 1), (2, 2, 0, "final", 0))

This means that::

    from __future__ import nested_scopes

will work in all releases at or after 2.1b1, and that nested_scopes are
intended to be enforced starting in release 2.2.


Resolved Problem:  Runtime Compilation
======================================

Several Python features can compile code during a module's runtime:

1. The ``exec`` statement.
2. The ``execfile()`` function.
3. The ``compile()`` function.
4. The ``eval()`` function.
5. The ``input()`` function.

Since a module M containing a future_statement naming feature F explicitly
requests that the current release act like a future release with respect to F,
any code compiled dynamically from text passed to one of these from within M
should probably also use the new syntax or semantics associated with F.  The
2.1 release does behave this way.

This isn't always desired, though.  For example, ``doctest.testmod(M)``
compiles examples taken from strings in M, and those examples should use M's
choices, not necessarily the doctest module's choices.  In the 2.1 release,
this isn't possible, and no scheme has yet been suggested for working around
this.  NOTE: :pep:`264` later addressed this in a flexible way, by adding
optional arguments to ``compile()``.

In any case, a future_statement appearing "near the top" (see Syntax above)


id: pep-0550.rst
content:
PEP: 550
Title: Execution Context
Author: Yury Selivanov <yury@edgedb.com>,
        Elvis Pranskevichus <elvis@edgedb.com>
Status: Withdrawn
Type: Standards Track
Created: 11-Aug-2017
Python-Version: 3.7
Post-History: 11-Aug-2017, 15-Aug-2017, 18-Aug-2017, 25-Aug-2017,
              01-Sep-2017


Abstract
========

This PEP adds a new generic mechanism of ensuring consistent access
to non-local state in the context of out-of-order execution, such
as in Python generators and coroutines.

Thread-local storage, such as ``threading.local()``, is inadequate for
programs that execute concurrently in the same OS thread.  This PEP
proposes a solution to this problem.


PEP Status
==========

Due to its breadth and the lack of general consensus on some aspects, this
PEP has been withdrawn and superseded by a simpler :pep:`567`, which has
been accepted and included in Python 3.7.

:pep:`567` implements the same core idea, but limits the ContextVar support
to asynchronous tasks while leaving the generator behavior untouched.
The latter may be revisited in a future PEP.


Rationale
=========

Prior to the advent of asynchronous programming in Python, programs
used OS threads to achieve concurrency.  The need for thread-specific
state was solved by ``threading.local()`` and its C-API equivalent,
``PyThreadState_GetDict()``.

A few examples of where Thread-local storage (TLS) is commonly
relied upon:

* Context managers like decimal contexts, ``numpy.errstate``,
  and ``warnings.catch_warnings``.

* Request-related data, such as security tokens and request
  data in web applications, language context for ``gettext`` etc.

* Profiling, tracing, and logging in large code bases.

Unfortunately, TLS does not work well for programs which execute
concurrently in a single thread.  A Python generator is the simplest
example of a concurrent program.  Consider the following::

    def fractions(precision, x, y):
        with decimal.localcontext() as ctx:
            ctx.prec = precision
            yield Decimal(x) / Decimal(y)
            yield Decimal(x) / Decimal(y ** 2)

    g1 = fractions(precision=2, x=1, y=3)
    g2 = fractions(precision=6, x=2, y=3)

    items = list(zip(g1, g2))

The intuitively expected value of ``items`` is::

    [(Decimal('0.33'), Decimal('0.666667')),
     (Decimal('0.11'), Decimal('0.222222'))]

Rather surprisingly, the actual result is::

    [(Decimal('0.33'), Decimal('0.666667')),
     (Decimal('0.111111'), Decimal('0.222222'))]

This is because implicit Decimal context is stored as a thread-local,
so concurrent iteration of the ``fractions()`` generator would
corrupt the state.  For Decimal, specifically, the only current
workaround is to use explicit context method calls for all arithmetic
operations [28]_.  Arguably, this defeats the usefulness of overloaded
operators and makes even simple formulas hard to read and write.

Coroutines are another class of Python code where TLS unreliability
is a significant issue.

The inadequacy of TLS in asynchronous code has lead to the
proliferation of ad-hoc solutions, which are limited in scope and
do not support all required use cases.

The current status quo is that any library (including the standard
library), which relies on TLS, is likely to be broken when used in
asynchronous code or with generators (see [3]_ as an example issue.)

Some languages, that support coroutines or generators, recommend
passing the context manually as an argument to every function, see
[1]_ for an example.  This approach, however, has limited use for
Python, where there is a large ecosystem that was built to work with
a TLS-like context.  Furthermore, libraries like ``decimal`` or
``numpy`` rely on context implicitly in overloaded operator
implementations.

The .NET runtime, which has support for async/await, has a generic
solution for this problem, called ``ExecutionContext`` (see [2]_).


Goals
=====

The goal of this PEP is to provide a more reliable
``threading.local()`` alternative, which:

* provides the mechanism and the API to fix non-local state issues
  with coroutines and generators;

* implements TLS-like semantics for synchronous code, so that
  users like ``decimal`` and ``numpy`` can switch to the new
  mechanism with minimal risk of breaking backwards compatibility;

* has no or negligible performance impact on the existing code or
  the code that will be using the new mechanism, including
  C extensions.


High-Level Specification
========================

The full specification of this PEP is broken down into three parts:

* High-Level Specification (this section): the description of the
  overall solution.  We show how it applies to generators and
  coroutines in user code, without delving into implementation
  details.

* Detailed Specification: the complete description of new concepts,
  APIs, and related changes to the standard library.

* Implementation Details: the description and analysis of data
  structures and algorithms used to implement this PEP, as well as
  the necessary changes to CPython.

For the purpose of this section, we define *execution context* as an
opaque container of non-local state that allows consistent access to
its contents in the concurrent execution environment.

A *context variable* is an object representing a value in the
execution context.  A call to ``contextvars.ContextVar(name)``
creates a new context variable object.  A context variable object has
three methods:

* ``get()``: returns the value of the variable in the current
  execution context;

* ``set(value)``: sets the value of the variable in the current
  execution context;

* ``delete()``: can be used for restoring variable state, it's
  purpose and semantics are explained in
  `Setting and restoring context variables`_.


Regular Single-threaded Code
----------------------------

In regular, single-threaded code that doesn't involve generators or
coroutines, context variables behave like globals::

    var = contextvars.ContextVar('var')

    def sub():
        assert var.get() == 'main'
        var.set('sub')

    def main():
        var.set('main')
        sub()
        assert var.get() == 'sub'


Multithreaded Code
------------------

In multithreaded code, context variables behave like thread locals::

    var = contextvars.ContextVar('var')

    def sub():
        assert var.get() is None  # The execution context is empty
                                  # for each new thread.
        var.set('sub')

    def main():
        var.set('main')

        thread = threading.Thread(target=sub)
        thread.start()
        thread.join()

        assert var.get() == 'main'


Generators
----------

Unlike regular function calls, generators can cooperatively yield
their control of execution to the caller.  Furthermore, a generator
does not control *where* the execution would continue after it yields.
It may be resumed from an arbitrary code location.

For these reasons, the least surprising behaviour of generators is
as follows:

* changes to context variables are always local and are not visible
  in the outer context, but are visible to the code called by the
  generator;

* once set in the generator, the context variable is guaranteed not
  to change between iterations;

* changes to context variables in outer context (where the generator
  is being iterated) are visible to the generator, unless these
  variables were also modified inside the generator.

Let's review::

    var1 = contextvars.ContextVar('var1')
    var2 = contextvars.ContextVar('var2')

    def gen():
        var1.set('gen')
        assert var1.get() == 'gen'
        assert var2.get() == 'main'
        yield 1

        # Modification to var1 in main() is shielded by
        # gen()'s local modification.
        assert var1.get() == 'gen'

        # But modifications to var2 are visible
        assert var2.get() == 'main modified'
        yield 2

    def main():
        g = gen()

        var1.set('main')
        var2.set('main')
     


id: pep-0587.rst
content:
PEP: 587
Title: Python Initialization Configuration
Author: Victor Stinner <vstinner@python.org>, Alyssa Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: Thomas Wouters <thomas@python.org>
Discussions-To: python-dev@python.org
Status: Final
Type: Standards Track
Created: 27-Mar-2019
Python-Version: 3.8

Abstract
========

Add a new C API to configure the Python Initialization providing finer
control on the whole configuration and better error reporting.

It becomes possible to read the configuration and then override some
computed parameters before it is applied. It also becomes possible to
completely override how Python computes the module search paths
(``sys.path``).

The new `Isolated Configuration`_ provides sane default values to
isolate Python from the system. For example, to embed Python into an
application. Using the environment are now opt-in options, rather than
an opt-out options. For example, environment variables, command line
arguments and global configuration variables are ignored by default.

Building a customized Python which behaves as the regular Python becomes
easier using the new ``Py_RunMain()`` function. Moreover, using the
`Python Configuration`_, ``PyConfig.argv`` arguments are now parsed the
same way the regular Python parses command line arguments, and
``PyConfig.xoptions`` are handled as ``-X opt`` command line options.

This extracts a subset of the API design from the :pep:`432` development
and refactoring work that is now considered sufficiently stable to make
public (allowing 3rd party embedding applications access to the same
configuration APIs that the native CPython CLI is now using).


Rationale
=========

Python is highly configurable but its configuration evolved organically.
The initialization configuration is scattered all around the code using
different ways to set them: global configuration variables (ex:
``Py_IsolatedFlag``), environment variables (ex: ``PYTHONPATH``),
command line arguments (ex: ``-b``), configuration files (ex:
``pyvenv.cfg``), function calls (ex: ``Py_SetProgramName()``). A
straightforward and reliable way to configure Python is needed.

Some configuration parameters are not accessible from the C API, or not
easily. For example, there is no API to override the default values of
``sys.executable``.

Some options like ``PYTHONPATH`` can only be set using an environment
variable which has a side effect on Python child processes if not unset
properly.

Some options also depends on other options: see `Priority and Rules`_.
Python 3.7 API does not provide a consistent view of the overall
configuration.

The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as
input whereas the Python filesystem encoding is set during the
initialization which can lead to mojibake.

Python 3.7 APIs like ``Py_Initialize()`` aborts the process on memory
allocation failure which is not convenient when Python is embedded.
Moreover, ``Py_Main()`` could exit directly the process rather than
returning an exit code. Proposed new API reports the error or exit code
to the caller which can decide how to handle it.

Implementing the :pep:`540` (UTF-8 Mode) and the new ``-X dev`` correctly
was almost impossible in Python 3.6. The code base has been deeply
reworked in Python 3.7 and then in Python 3.8 to read the configuration
into a structure with no side effect. It becomes possible to clear the
configuration (release memory) and read again the configuration if the
encoding changed . It is required to implement properly the UTF-8 which
changes the encoding using ``-X utf8`` command line option. Internally,
bytes ``argv`` strings are decoded from the filesystem encoding. The
``-X dev`` changes the memory allocator (behaves as
``PYTHONMALLOC=debug``), whereas it was not possible to change the
memory allocation *while* parsing the command line arguments. The new
design of the internal implementation not only allowed to implement
properly ``-X utf8`` and ``-X dev``, it also allows to change the Python
behavior way more easily, especially for corner cases like that, and
ensure that the configuration remains consistent: see `Priority and
Rules`_.

This PEP is a partial implementation of :pep:`432` which is the overall
design.  New fields can be added later to ``PyConfig`` structure to
finish the implementation of the :pep:`432` (e.g. by adding a new partial
initialization API which allows to configure Python using Python objects to
finish the full initialization). However, those features are omitted from this
PEP as even the native CPython CLI doesn't work that way - the public API
proposal in this PEP is limited to features which have already been implemented
and adopted as private APIs for us in the native CPython CLI.


Python Initialization C API
===========================

This PEP proposes to add the following new structures and functions.

New structures:

* ``PyConfig``
* ``PyPreConfig``
* ``PyStatus``
* ``PyWideStringList``

New functions:

* ``PyConfig_Clear(config)``
* ``PyConfig_InitIsolatedConfig()``
* ``PyConfig_InitPythonConfig()``
* ``PyConfig_Read(config)``
* ``PyConfig_SetArgv(config, argc, argv)``
* ``PyConfig_SetBytesArgv(config, argc, argv)``
* ``PyConfig_SetBytesString(config, config_str, str)``
* ``PyConfig_SetString(config, config_str, str)``
* ``PyPreConfig_InitIsolatedConfig(preconfig)``
* ``PyPreConfig_InitPythonConfig(preconfig)``
* ``PyStatus_Error(err_msg)``
* ``PyStatus_Exception(status)``
* ``PyStatus_Exit(exitcode)``
* ``PyStatus_IsError(status)``
* ``PyStatus_IsExit(status)``
* ``PyStatus_NoMemory()``
* ``PyStatus_Ok()``
* ``PyWideStringList_Append(list, item)``
* ``PyWideStringList_Insert(list, index, item)``
* ``Py_BytesMain(argc, argv)``
* ``Py_ExitStatusException(status)``
* ``Py_InitializeFromConfig(config)``
* ``Py_PreInitialize(preconfig)``
* ``Py_PreInitializeFromArgs(preconfig, argc, argv)``
* ``Py_PreInitializeFromBytesArgs(preconfig, argc, argv)``
* ``Py_RunMain()``

This PEP also adds ``_PyRuntimeState.preconfig`` (``PyPreConfig`` type)
and ``PyInterpreterState.config`` (``PyConfig`` type) fields to these
internal structures. ``PyInterpreterState.config`` becomes the new
reference configuration, replacing global configuration variables and
other private variables.


PyWideStringList
----------------

``PyWideStringList`` is a list of ``wchar_t*`` strings.

``PyWideStringList`` structure fields:

* ``length`` (``Py_ssize_t``)
* ``items`` (``wchar_t**``)

Methods:

* ``PyStatus PyWideStringList_Append(PyWideStringList *list, const wchar_t *item)``:
  Append *item* to *list*.
* ``PyStatus PyWideStringList_Insert(PyWideStringList *list, Py_ssize_t index, const wchar_t *item)``:
  Insert *item* into *list* at *index*. If *index* is greater than
  *list* length, just append *item* to *list*.

If *length* is non-zero, *items* must be non-NULL and all strings must
be non-NULL.

PyStatus
--------

``PyStatus`` is a structure to store the status of an initialization
function: success, error or exit. For an error, it can store the C
function name which created the error.

Example::

    PyStatus alloc(void **ptr, size_t size)
    {
        *ptr = PyMem_RawMalloc(size);
        if (*ptr == NULL) {
            return PyStatus_NoMemory();
        }
        return PyStatus_Ok();
    }

    int main(int argc, char **argv)
    {
        void *ptr;
        PyStatus status = alloc(&ptr, 16);
        if (PyStatus_Exception(status)) {
            Py_ExitStatusException(status);
        }
        PyMem_Free(ptr);
        return 0;
    }

``PyStatus`` fields:

* ``exitcode`` (``int``):
  Argument passed to ``exit()``.
* ``err_msg`` (``const char*``):
  Error message.
* ``func`` (``const char *``):
  Name of the function which created an error, can be ``NULL``.
* private ``_type`` field: for internal usage only.

Functions to create a status:

* ``PyStatus_Ok()``: Success.
* ``PyStatus_Error(err_msg)``: Initialization error with a message.
* ``PyStatus_NoMemory()``: Memory allocation failure (out of 


id: pep-0593.rst
content:
PEP: 593
Title: Flexible function and variable annotations
Author: Till Varoquaux <till@fb.com>, Konstantin Kashin <kkashin@fb.com>
Sponsor: Ivan Levkivskyi <levkivskyi@gmail.com>
Discussions-To: typing-sig@python.org
Status: Final
Type: Standards Track
Topic: Typing
Created: 26-Apr-2019
Python-Version: 3.9
Post-History: 20-May-2019

.. canonical-typing-spec:: :ref:`typing:annotated` and
                           :py:data:`typing.Annotated`

Abstract
--------

This PEP introduces a mechanism to extend the type annotations from PEP
484 with arbitrary metadata.

Motivation
----------

:pep:`484` provides a standard semantic for the annotations introduced in
:pep:`3107`. :pep:`484` is prescriptive but it is the de facto standard
for most of the consumers of annotations; in many statically checked
code bases, where type annotations are widely used, they have
effectively crowded out any other form of annotation. Some of the use
cases for annotations described in :pep:`3107` (database mapping,
foreign languages bridge) are not currently realistic given the
prevalence of type annotations. Furthermore, the standardisation of type
annotations rules out advanced features only supported by specific type
checkers.

Rationale
---------

This PEP adds an ``Annotated`` type to the typing module to decorate
existing types with context-specific metadata. Specifically, a type
``T`` can be annotated with metadata ``x`` via the typehint
``Annotated[T, x]``. This metadata can be used for either static
analysis or at runtime. If a library (or tool) encounters a typehint
``Annotated[T, x]`` and has no special logic for metadata ``x``, it
should ignore it and simply treat the type as ``T``. Unlike the
``no_type_check`` functionality that currently exists in the ``typing``
module which completely disables typechecking annotations on a function
or a class, the ``Annotated`` type allows for both static typechecking
of ``T`` (e.g., via `mypy <mypy_>`_ or `Pyre <pyre_>`_,
which can safely ignore ``x``)
together with runtime access to ``x`` within a specific application. The
introduction of this type would address a diverse set of use cases of interest
to the broader Python community.

This was originally brought up as `issue 600 <issue-600_>`_ in the typing github
and then discussed in `Python ideas <python-ideas_>`_.

Motivating examples
-------------------

Combining runtime and static uses of annotations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

There's an emerging trend of libraries leveraging the typing annotations at
runtime (e.g.: dataclasses); having the ability to extend the typing annotations
with external data would be a great boon for those libraries.

Here's an example of how a hypothetical module could leverage annotations to
read c structs::

    UnsignedShort = Annotated[int, struct2.ctype('H')]
    SignedChar = Annotated[int, struct2.ctype('b')]

    class Student(struct2.Packed):
        # mypy typechecks 'name' field as 'str'
        name: Annotated[str, struct2.ctype("<10s")]
        serialnum: UnsignedShort
        school: SignedChar

    # 'unpack' only uses the metadata within the type annotations
    Student.unpack(record)
    # Student(name=b'raymond   ', serialnum=4658, school=264)

Lowering barriers to developing new typing constructs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Typically when adding a new type, a developer need to upstream that type to the
typing module and change mypy, `PyCharm <pycharm_>`_, Pyre, `pytype <pytype_>`_,
etc...
This is particularly important when working on open-source code that
makes use of these types, seeing as the code would not be immediately
transportable to other developers' tools without additional logic. As a result,
there is a high cost to developing and trying out new types in a codebase.
Ideally, authors should be able to introduce new types in a manner that allows
for graceful degradation (e.g.: when clients do not have a custom `mypy plugin
<mypy-plugin_>`_), which would lower the barrier to development and ensure some
degree of backward compatibility.

For example, suppose that an author wanted to add support for `tagged unions
<tagged-union_>`_ to Python. One way to accomplish would be to `annotate
<typed-dict_>`_ ``TypedDict`` in Python such that only one field is allowed
to be set::

    Currency = Annotated[
        TypedDict('Currency', {'dollars': float, 'pounds': float}, total=False),
        TaggedUnion,
    ]

This is a somewhat cumbersome syntax but it allows us to iterate on this
proof-of-concept and have people with type checkers (or other tools) that don't
yet support this feature work in a codebase with tagged unions. The author could
easily test this proposal and iron out the kinks before trying to upstream tagged
union to ``typing``, mypy, etc. Moreover, tools that do not have support for
parsing the ``TaggedUnion`` annotation would still be able to treat ``Currency``
as a ``TypedDict``, which is still a close approximation (slightly less strict).

Specification
-------------

Syntax
~~~~~~

``Annotated`` is parameterized with a type and an arbitrary list of
Python values that represent the annotations. Here are the specific
details of the syntax:

* The first argument to ``Annotated`` must be a valid type

* Multiple type annotations are supported (``Annotated`` supports variadic
  arguments)::

    Annotated[int, ValueRange(3, 10), ctype("char")]

* ``Annotated`` must be called with at least two arguments (
  ``Annotated[int]`` is not valid)

* The order of the annotations is preserved and matters for equality
  checks::

    Annotated[int, ValueRange(3, 10), ctype("char")] != Annotated[
        int, ctype("char"), ValueRange(3, 10)
    ]

* Nested ``Annotated`` types are flattened, with metadata ordered
  starting with the innermost annotation::

    Annotated[Annotated[int, ValueRange(3, 10)], ctype("char")] == Annotated[
        int, ValueRange(3, 10), ctype("char")
    ]

* Duplicated annotations are not removed::

    Annotated[int, ValueRange(3, 10)] != Annotated[
        int, ValueRange(3, 10), ValueRange(3, 10)
    ]

* ``Annotated`` can be used with nested and generic aliases::

    Typevar T = ...
    Vec = Annotated[List[Tuple[T, T]], MaxLen(10)]
    V = Vec[int]

    V == Annotated[List[Tuple[int, int]], MaxLen(10)]

Consuming annotations
~~~~~~~~~~~~~~~~~~~~~

Ultimately, the responsibility of how to interpret the annotations (if
at all) is the responsibility of the tool or library encountering the
``Annotated`` type. A tool or library encountering an ``Annotated`` type
can scan through the annotations to determine if they are of interest
(e.g., using ``isinstance()``).

**Unknown annotations:** When a tool or a library does not support
annotations or encounters an unknown annotation it should just ignore it
and treat annotated type as the underlying type. For example, when encountering
an annotation that is not an instance of ``struct2.ctype`` to the annotations
for name (e.g., ``Annotated[str, 'foo', struct2.ctype("<10s")]``), the unpack
method should ignore it.

**Namespacing annotations:** Namespaces are not needed for annotations since
the class used by the annotations acts as a namespace.

**Multiple annotations:** It's up to the tool consuming the annotations
to decide whether the client is allowed to have several annotations on
one type and how to merge those annotations.

Since the ``Annotated`` type allows you to put several annotations of
the same (or different) type(s) on any node, the tools or libraries
consuming those annotations are in charge of dealing with potential
duplicates. For example, if you are doing value range analysis you might
allow this::

    T1 = Annotated[int, ValueRange(-10, 5)]
    T2 = Annotated[T1, ValueRange(-20, 3)]

Flattening nested annotations, this translates to::

    T2 = Annotated[int, ValueRange(-10, 5), ValueRange(-20, 3)]

Interaction with ``get_type_hints()``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``typing


id: pep-0008.rst
content:
PEP: 8
Title: Style Guide for Python Code
Author: Guido van Rossum <guido@python.org>,
        Barry Warsaw <barry@python.org>,
        Alyssa Coghlan <ncoghlan@gmail.com>
Status: Active
Type: Process
Created: 05-Jul-2001
Post-History: 05-Jul-2001, 01-Aug-2013


Introduction
============

This document gives coding conventions for the Python code comprising
the standard library in the main Python distribution.  Please see the
companion informational PEP describing :pep:`style guidelines for the C code
in the C implementation of Python <7>`.

This document and :pep:`257` (Docstring Conventions) were adapted from
Guido's original Python Style Guide essay, with some additions from
Barry's style guide [2]_.

This style guide evolves over time as additional conventions are
identified and past conventions are rendered obsolete by changes in
the language itself.

Many projects have their own coding style guidelines. In the event of any
conflicts, such project-specific guides take precedence for that project.


A Foolish Consistency is the Hobgoblin of Little Minds
======================================================

One of Guido's key insights is that code is read much more often than
it is written.  The guidelines provided here are intended to improve
the readability of code and make it consistent across the wide
spectrum of Python code.  As :pep:`20` says, "Readability counts".

A style guide is about consistency.  Consistency with this style guide
is important.  Consistency within a project is more important.
Consistency within one module or function is the most important.

However, know when to be inconsistent -- sometimes style guide
recommendations just aren't applicable.  When in doubt, use your best
judgment.  Look at other examples and decide what looks best.  And
don't hesitate to ask!

In particular: do not break backwards compatibility just to comply with
this PEP!

Some other good reasons to ignore a particular guideline:

1. When applying the guideline would make the code less readable, even
   for someone who is used to reading code that follows this PEP.

2. To be consistent with surrounding code that also breaks it (maybe
   for historic reasons) -- although this is also an opportunity to
   clean up someone else's mess (in true XP style).

3. Because the code in question predates the introduction of the
   guideline and there is no other reason to be modifying that code.

4. When the code needs to remain compatible with older versions of
   Python that don't support the feature recommended by the style guide.


Code Lay-out
============

Indentation
-----------

Use 4 spaces per indentation level.

Continuation lines should align wrapped elements either vertically
using Python's implicit line joining inside parentheses, brackets and
braces, or using a *hanging indent* [#fn-hi]_.  When using a hanging
indent the following should be considered; there should be no
arguments on the first line and further indentation should be used to
clearly distinguish itself as a continuation line:

.. code-block::
   :class: good

   # Correct:

   # Aligned with opening delimiter.
   foo = long_function_name(var_one, var_two,
                            var_three, var_four)

   # Add 4 spaces (an extra level of indentation) to distinguish arguments from the rest.
   def long_function_name(
           var_one, var_two, var_three,
           var_four):
       print(var_one)

   # Hanging indents should add a level.
   foo = long_function_name(
       var_one, var_two,
       var_three, var_four)

.. code-block::
   :class: bad

   # Wrong:

   # Arguments on first line forbidden when not using vertical alignment.
   foo = long_function_name(var_one, var_two,
       var_three, var_four)

   # Further indentation required as indentation is not distinguishable.
   def long_function_name(
       var_one, var_two, var_three,
       var_four):
       print(var_one)

The 4-space rule is optional for continuation lines.

Optional:

.. code-block::
   :class: good

   # Hanging indents *may* be indented to other than 4 spaces.
   foo = long_function_name(
     var_one, var_two,
     var_three, var_four)

.. _`multiline if-statements`:

When the conditional part of an ``if``-statement is long enough to require
that it be written across multiple lines, it's worth noting that the
combination of a two character keyword (i.e. ``if``), plus a single space,
plus an opening parenthesis creates a natural 4-space indent for the
subsequent lines of the multiline conditional.  This can produce a visual
conflict with the indented suite of code nested inside the ``if``-statement,
which would also naturally be indented to 4 spaces.  This PEP takes no
explicit position on how (or whether) to further visually distinguish such
conditional lines from the nested suite inside the ``if``-statement.
Acceptable options in this situation include, but are not limited to:

.. code-block::
   :class: good

   # No extra indentation.
   if (this_is_one_thing and
       that_is_another_thing):
       do_something()

   # Add a comment, which will provide some distinction in editors
   # supporting syntax highlighting.
   if (this_is_one_thing and
       that_is_another_thing):
       # Since both conditions are true, we can frobnicate.
       do_something()

   # Add some extra indentation on the conditional continuation line.
   if (this_is_one_thing
           and that_is_another_thing):
       do_something()

(Also see the discussion of whether to break before or after binary
operators below.)

The closing brace/bracket/parenthesis on multiline constructs may
either line up under the first non-whitespace character of the last
line of list, as in:

.. code-block::
   :class: good

   my_list = [
       1, 2, 3,
       4, 5, 6,
       ]
   result = some_function_that_takes_arguments(
       'a', 'b', 'c',
       'd', 'e', 'f',
       )

or it may be lined up under the first character of the line that
starts the multiline construct, as in:

.. code-block::
   :class: good

   my_list = [
       1, 2, 3,
       4, 5, 6,
   ]
   result = some_function_that_takes_arguments(
       'a', 'b', 'c',
       'd', 'e', 'f',
   )

Tabs or Spaces?
---------------

Spaces are the preferred indentation method.

Tabs should be used solely to remain consistent with code that is
already indented with tabs.

Python disallows mixing tabs and spaces for indentation.


Maximum Line Length
-------------------

Limit all lines to a maximum of 79 characters.

For flowing long blocks of text with fewer structural restrictions
(docstrings or comments), the line length should be limited to 72
characters.

Limiting the required editor window width makes it possible to have
several files open side by side, and works well when using code
review tools that present the two versions in adjacent columns.

The default wrapping in most tools disrupts the visual structure of the
code, making it more difficult to understand. The limits are chosen to
avoid wrapping in editors with the window width set to 80, even
if the tool places a marker glyph in the final column when wrapping
lines. Some web based tools may not offer dynamic line wrapping at all.

Some teams strongly prefer a longer line length.  For code maintained
exclusively or primarily by a team that can reach agreement on this
issue, it is okay to increase the line length limit up to 99 characters,
provided that comments and docstrings are still wrapped at 72
characters.

The Python standard library is conservative and requires limiting
lines to 79 characters (and docstrings/comments to 72).

The preferred way of wrapping long lines is by using Python's implied
line continuation inside parentheses, brackets and braces.  Long lines
can be broken over multiple lines by wrapping expressions in
parentheses. These should be used in preference to using a backslash
for line continuation.

Backslashes may still be appropriate at times.  For example, long,
multiple ``with``-stat


id: pep-0020.rst
content:
PEP: 20
Title: The Zen of Python
Author: Tim Peters <tim.peters@gmail.com>
Status: Active
Type: Informational
Created: 19-Aug-2004
Post-History: 22-Aug-2004


Abstract
========

Long time Pythoneer Tim Peters succinctly channels the BDFL's guiding
principles for Python's design into 20 aphorisms, only 19 of which
have been written down.


The Zen of Python
=================

.. code-block:: text

    Beautiful is better than ugly.
    Explicit is better than implicit.
    Simple is better than complex.
    Complex is better than complicated.
    Flat is better than nested.
    Sparse is better than dense.
    Readability counts.
    Special cases aren't special enough to break the rules.
    Although practicality beats purity.
    Errors should never pass silently.
    Unless explicitly silenced.
    In the face of ambiguity, refuse the temptation to guess.
    There should be one-- and preferably only one --obvious way to do it.
    Although that way may not be obvious at first unless you're Dutch.
    Now is better than never.
    Although never is often better than *right* now.
    If the implementation is hard to explain, it's a bad idea.
    If the implementation is easy to explain, it may be a good idea.
    Namespaces are one honking great idea -- let's do more of those!


Easter Egg
==========

.. code-block:: pycon

  >>> import this


References
==========

Originally posted to comp.lang.python/python-list@python.org under a
thread called `"The Way of Python"
<https://groups.google.com/d/msg/comp.lang.python/B_VxeTBClM0/L8W9KlsiriUJ>`__


Copyright
=========

This document has been placed in the public domain.


id: pep-0746.rst
content:
PEP: 746
Title: Type checking Annotated metadata
Author: Adrian Garcia Badaracco <adrian@adriangb.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-746-typedmetadata-for-type-checking-of-pep-593-annotated/53834
Status: Draft
Type: Standards Track
Topic: Typing
Created: 20-May-2024
Python-Version: 3.15
Post-History: 20-May-2024

Abstract
========

This PEP proposes a mechanism for type checking metadata that uses
the :py:data:`typing.Annotated` type. Metadata objects that implement
the new ``__supports_annotated_base__`` protocol will be type checked by static
type checkers to ensure that the metadata is valid for the given type.

Motivation
==========

:pep:`593` introduced ``Annotated`` as a way to attach runtime metadata to types.
In general, the metadata is not meant for static type checkers, but even so,
it is often useful to be able to check that the metadata makes sense for the given
type.

Take the first example in :pep:`593`, which uses ``Annotated`` to attach
serialization information to a field::

   class Student(struct2.Packed):
       name: Annotated[str, struct2.ctype("<10s")]

Here, the ``struct2.ctype("<10s")`` metadata is meant to be used by a serialization
library to serialize the field. Such libraries can only serialize a subset of types:
it would not make sense to write, for example, ``Annotated[list[str], struct2.ctype("<10s")]``.
Yet the type system provides no way to enforce this. The metadata are completely
ignored by type checkers.

This use case comes up in libraries like :pypi:`pydantic` and :pypi:`msgspec`, which use
``Annotated`` to attach validation and conversion information to fields or :pypi:`fastapi`,
which uses ``Annotated`` to mark parameters as extracted from headers, query strings or
dependency injection.

Specification
=============
This PEP introduces a protocol that can be used by static and runtime type checkers to validate
the consistency between ``Annotated`` metadata and a given type.
Objects that implement this protocol have an attribute called ``__supports_annotated_base__``
that specifies whether the metadata is valid for a given type::

    class Int64:
        __supports_annotated_base__: int

The attribute may also be marked as a ``ClassVar`` to avoid interaction with dataclasses::

    from dataclasses import dataclass
    from typing import ClassVar

    @dataclass
    class Gt:
        value: int
        __supports_annotated_base__: ClassVar[int]

When a static type checker encounters a type expression of the form ``Annotated[T, M1, M2, ...]``,
it should enforce that for each metadata element in ``M1, M2, ...``, one of the following is true:

* The metadata element evaluates to an object that does not have a ``__supports_annotated_base__`` attribute; or
* The metadata element evaluates to an object ``M`` that has a ``__supports_annotated_base__`` attribute;
  and ``T`` is assignable to the type of ``M.__supports_annotated_base__``.

To support generic ``Gt`` metadata, one might write::

    from typing import Protocol

    class SupportsGt[T](Protocol):
        def __gt__(self, __other: T) -> bool:
            ...

    class Gt[T]:
        __supports_annotated_base__: ClassVar[SupportsGt[T]]

        def __init__(self, value: T) -> None:
            self.value = value

    x1: Annotated[int, Gt(0)] = 1  # OK
    x2: Annotated[str, Gt(0)] = 0  # type checker error: str is not assignable to SupportsGt[int]
    x3: Annotated[int, Gt(1)] = 0  # OK for static type checkers; runtime type checkers may flag this

Backwards Compatibility
=======================

Metadata that does not implement the protocol will be considered valid for all types,
so no breaking changes are introduced for existing code. The new checks only apply
to metadata objects that explicitly implement the protocol specified by this PEP.

Security Implications
=====================

None.

How to Teach This
=================

This protocol is intended mostly for libraries that provide ``Annotated`` metadata;
end users of those libraries are unlikely to need to implement the protocol themselves.
The protocol should be mentioned in the documentation for :py:data:`typing.Annotated` and
in the typing specification.

Reference Implementation
========================

None yet.

Rejected ideas
==============

Introducing a type variable instead of a generic class
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We considered using a special type variable, ``AnnotatedT = TypeVar("AnnotatedT")``,
to represent the type ``T`` of the inner type in ``Annotated``; metadata would be
type checked against this type variable. However, this would require using the old
type variable syntax (before :pep:`695`), which is now a discouraged feature.
In addition, this would use type variables in an unusual way that does not fit well
with the rest of the type system.

Introducing a new type to ``typing.py`` that all metadata objects should subclass
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A previous version of this PEP suggested adding a new generic base class, ``TypedMetadata[U]``,
that metadata objects would subclass. If a metadata object is a subclass of ``TypedMetadata[U]``,
then type checkers would check that the annotation's base type is assignable to ``U``.
However, this mechanism does not integrate as well with the rest of the language; Python
does not generally use marker base classes. In addition, it provides less flexibility than
the current proposal: it would not allow overloads, and it would require metadata objects
to add a new base class, which may make their runtime implementation more complex.

Using a method instead of an attribute for ``__supports_annotated_base__``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We considered using a method instead of an attribute for the protocol, so that this method can be used
at runtime to check the validity of the metadata and to support overloads or returning boolean literals.
However, using a method adds boilerplate to the implementation and the value of the runtime use cases or
more complex scenarios involving overloads and returning boolean literals was not clear.

Acknowledgments
===============

We thank Eric Traut for suggesting the idea of using a protocol and implementing provisional support in Pyright.
Thank you to Jelle Zijlstra for sponsoring this PEP.

Copyright
=========

This document has been placed in the public domain.


id: pep-0752.rst
content:
PEP: 752
Title: Implicit namespaces for package repositories
Author: Ofek Lev <ofekmeister@gmail.com>,
        Jarek Potiuk <potiuk@apache.org>
Sponsor: Barry Warsaw <barry@python.org>
PEP-Delegate: Dustin Ingram <di@python.org>
Discussions-To: https://discuss.python.org/t/63192
Status: Draft
Type: Standards Track
Topic: Packaging
Created: 13-Aug-2024
Post-History: `18-Aug-2024 <https://discuss.python.org/t/61227>`__,
              `07-Sep-2024 <https://discuss.python.org/t/63192>`__,

Abstract
========

This PEP specifies a way for organizations to reserve package name prefixes
for future uploads.

    "Namespaces are one honking great idea -- let's do more of
    those!" - :pep:`20`

Motivation
==========

The current ecosystem lacks a way for projects with many packages to signal a
verified pattern of ownership. Such projects fall into two categories.

The first category is projects [1]_ that want complete control over their
namespace. A few examples:

* Major cloud providers like Amazon, Google and Microsoft have a common prefix
  for each feature's corresponding package [3]_. For example, most of Google's
  packages are prefixed by ``google-cloud-`` e.g. ``google-cloud-compute`` for
  `using virtual machines <https://cloud.google.com/products/compute>`__.
* `OpenTelemetry <https://opentelemetry.io>`__ is an open standard for
  observability with `official packages`__ for the core APIs and SDK with
  `contrib packages`__ to collect data from various sources. All packages
  are prefixed by ``opentelemetry-`` with child prefixes in the form
  ``opentelemetry-<component>-<name>-``. The contrib packages live in a
  central repository and they are the only ones with the ability to publish.
* `Apache Airflow <https://airflow.apache.org>`__ is a platform to programmatically
  author, schedule and monitor workflows. It has providers, where each
  provider package is prefixed by ``apache-airflow-providers-``.

__ https://github.com/open-telemetry/opentelemetry-python
__ https://github.com/open-telemetry/opentelemetry-python-contrib

The second category is projects [2]_ that want to share their namespace such
that some packages are officially maintained and third-party developers are
encouraged to participate by publishing their own. Some examples:

* `Project Jupyter <https://jupyter.org>`__ is devoted to the development of
  tooling for sharing interactive documents. They support `extensions`__
  which in most cases (and in all cases for officially maintained
  extensions) are prefixed by ``jupyter-``.
* `Django <https://www.djangoproject.com>`__ is one of the most widely used web
  frameworks in existence. They have the concept of `reusable apps`__, which
  are commonly installed via
  `third-party packages <https://djangopackages.org>`__ that implement a subset
  of functionality to extend Django-based websites. These packages are by
  convention prefixed by ``django-`` or ``dj-``.

__ https://jupyterlab.readthedocs.io/en/stable/user/extensions.html
__ https://docs.djangoproject.com/en/5.1/intro/reusable-apps/

Such projects are uniquely vulnerable to name-squatting attacks
which can ultimately result in `dependency confusion`__.

__ https://www.activestate.com/resources/quick-reads/dependency-confusion/

For example, say a new product is released for which monitoring would be
valuable. It would be reasonable to assume that
`Datadog <https://www.datadoghq.com>`__ would eventually support it as an
official integration. It takes a nontrivial amount of time to deliver such an
integration due to roadmap prioritization and the time required for
implementation. It would be impossible to reserve the name of every potential
package so in the interim an attacker may create a package that appears
legitimate which would execute malicious code at runtime. Not only are users
more likely to install such packages but doing so taints the perception of the
entire project.

Although :pep:`708` attempts to address this attack vector, it is specifically
about the case of multiple repositories being considered during dependency
resolution and does not offer any protection to the aforementioned use cases.

Namespacing also would drastically reduce the incidence of
`typosquatting <https://en.wikipedia.org/wiki/Typosquatting>`__
because typos would have to be in the prefix itself which is
`normalized <naming_>`_ and likely to be a short, well-known identifier like
``aws-``. In recent years, typosquatting has become a popular attack vector
[4]_.

The `current protection`__ against typosquatting used by PyPI is to normalize
similar characters but that is insufficient for these use cases.

__ https://github.com/pypi/warehouse/blob/8615326918a180eb2652753743eac8e74f96a90b/warehouse/migrations/versions/d18d443f89f0_ultranormalize_name_function.py#L29-L42

Another problem that namespacing would solve is the issue of choosing new names
for packages following the agreed patterns of naming. Often (this is the case
for Apache Airflow for example), there are public discussions that precede
the decision to create a new package. The decision is based on the agreed
name and follow the pattern of the existing packages. If more package names are
considered during the discussion, all the names have to be reserved via a PyPI
interface before the discussion is public, otherwise the names can be taken by
other users. This has happened in the past as explained
in the associated `discussion <https://discuss.python.org/t/pep-752-implicit-namespaces-for-package-repositories/63192/80>`__.

Rationale
=========

Other package ecosystems have generally solved this problem by taking one of
two approaches: either minimizing or maximizing backwards compatibility.

* `NPM <https://www.npmjs.com>`__ has the concept of
  `scoped packages <https://docs.npmjs.com/about-scopes>`__ which were
  `introduced`__ primarily to combat there being a dearth of available good
  package names (whether a real or perceived phenomenon). When a user or
  organization signs up they are given a scope that matches their name. For
  example, the
  `package <https://www.npmjs.com/package/@google-cloud/storage>`__ for using
  Google Cloud Storage is ``@google-cloud/storage`` where ``@google-cloud/`` is
  the scope. Regular user accounts (non-organization) may publish `unscoped`__
  packages for public use.
  This approach has the lowest amount of backwards compatibility because every
  installer and tool has to be modified to account for scopes.
* `NuGet <https://www.nuget.org>`__ has the concept of
  `package ID prefix reservation`__ which was
  `introduced`__ primarily to satisfy users wishing to know where a package
  came from. A package name prefix may be reserved for use by one or more
  owners. Every reserved package has a special indication
  `on its page <https://www.nuget.org/packages/Google.Cloud.Storage.V1>`__ to
  communicate this. After reservation, any upload with a reserved prefix will
  fail if the user is not an owner of the prefix. Existing packages that have a
  prefix that is owned may continue to release as usual. This approach has the
  highest amount of backwards compatibility because only modifications to
  indices like PyPI are required and installers do not need to change.

__ https://blog.npmjs.org/post/116936804365/solving-npms-hard-problem-naming-packages
__ https://docs.npmjs.com/package-scope-access-level-and-visibility
__ https://learn.microsoft.com/en-us/nuget/nuget-org/id-prefix-reservation
__ https://devblogs.microsoft.com/nuget/Package-identity-and-trust/

This PEP specifies the NuGet approach of authorized reservation across a flat
namespace. Any solution that requires new package syntax must be built atop the
existing flat namespace and therefore implicit namespaces acquired via a
reservation mechanism would be a prerequisite to such explicit namespaces.

Although existing packages matching a reserved namespace would be untouched,
preventing future unauthorized uploads and strategically apply


id: pep-0785.rst
content:
PEP: 785
Title: New methods for easier handling of ``ExceptionGroup``\ s
Author: Zac Hatfield-Dodds <zac@zhd.dev>
Sponsor: Gregory P. Smith <greg@krypto.org>
Discussions-To: https://discuss.python.org/t/88244
Status: Draft
Type: Standards Track
Created: 08-Apr-2025
Python-Version: 3.14
Post-History:
  `13-Apr-2025 <https://discuss.python.org/t/88244>`__,

Abstract
========

As :pep:`654` :class:`ExceptionGroup` has come into widespread use across the
Python community, some common but awkward patterns have emerged. We therefore
propose adding two new methods to exception objects:

- :meth:`!BaseExceptionGroup.leaf_exceptions`, returning the 'leaf' exceptions as
  a list, with each traceback composited from any intermediate groups.

- :meth:`!BaseException.preserve_context`, a context manager which
  saves and restores the :attr:`!self.__context__` attribute of ``self``,
  so that re-raising the exception within another handler does not overwrite
  the existing context.

We expect this to enable more concise expression of error handling logic in
many medium-complexity cases. Without them, exception-group handlers will
continue to discard intermediate tracebacks and mis-handle ``__context__``
exceptions, to the detriment of anyone debugging async code.


Motivation
==========

As exception groups come into widespread use, library authors and end users
often write code to process or respond to individual leaf exceptions, for
example when implementing middleware, error logging, or response handlers in
a web framework.

`Searching GitHub`__ found four implementations of :meth:`!leaf_exceptions` by
various names in the first sixty hits, of which none handle
tracebacks.\ [#numbers]_  The same search found thirteen cases where
:meth:`!.leaf_exceptions` could be used.  We therefore believe that providing
a method on the :class:`BaseException` type with proper traceback preservation
will improve error-handling and debugging experiences across the ecosystem.

__ https://github.com/search?q=%2Ffor+%5Cw%2B+in+%5Beg%5D%5Cw*%5C.exceptions%3A%2F+language%3APython&type=code

The rise of exception groups has also made re-raising exceptions caught by an
earlier handler much more common: for example, web-server middleware might
unwrap ``HTTPException`` if that is the sole leaf of a group:

.. code-block:: python

    except* HTTPException as group:
        first, *rest = group.leaf_exceptions()  # get the whole traceback :-)
        if not rest:
            raise first
        raise

However, this innocent-seeming code has a problem: ``raise first`` will do
``first.__context__ = group`` as a side effect. This discards the original
context of the error, which may contain crucial information to understand why
the exception was raised. In many production apps it also causes tracebacks
to balloon from hundreds of lines, to tens or even `hundreds of thousands of
lines`__ - a volume which makes understanding errors far more difficult than
it should be.

__ https://github.com/python-trio/trio/issues/2001#issuecomment-931928509


A new :meth:`!BaseException.preserve_context` method would be a discoverable,
readable, and easy-to-use solution for these cases.


Specification
=============

A new method ``leaf_exceptions()`` will be added to ``BaseExceptionGroup``, with the
following signature:

.. code-block:: python

    def leaf_exceptions(self, *, fix_tracebacks=True) -> list[BaseException]:
        """
        Return a flat list of all 'leaf' exceptions in the group.

        If fix_tracebacks is True, each leaf will have the traceback replaced
        with a composite so that frames attached to intermediate groups are
        still visible when debugging. Pass fix_tracebacks=False to disable
        this modification, e.g. if you expect to raise the group unchanged.
        """

A new method ``preserve_context()`` will be added to ``BaseException``, with the
following signature:

.. code-block:: python

    def preserve_context(self) -> contextlib.AbstractContextManager[Self]:
        """
        Context manager that preserves the exception's __context__ attribute.

        When entering the context, the current values of __context__ is saved.
        When exiting, the saved value is restored, which allows raising an
        exception inside an except block without changing its context chain.
        """

Usage example:

.. code-block:: python

    # We're an async web framework, where user code can raise an HTTPException
    # to return a particular HTTP error code to the client. However, it may
    # (or may not) be raised inside a TaskGroup, so we need to use `except*`;
    # and if there are *multiple* such exceptions we'll treat that as a bug.
    try:
        user_code_here()
    except* HTTPException as group:
        first, *rest = group.leaf_exceptions()
        if rest:
            raise  # handled by internal-server-error middleware
        ... # logging, cache updates, etc.
        with first.preserve_context():
            raise first

Without ``.preserve_context()``, this code would have to either:

* arrange for the exception to be raised *after* the ``except*`` block,
  making code difficult to follow in nontrivial cases, or
* discard the existing ``__context__`` of the ``first`` exception, replacing
  it with an ``ExceptionGroup`` which is simply an implementation detail, or
* use ``try/except`` instead of ``except*``, handling the possibility that the
  group doesn't contain an ``HTTPException`` at all,\ [#catch-raw-group]_ or
* implement the semantics of ``.preserve_context()`` inline; while this is not
  *literally unheard-of*, it remains very rare.


Backwards Compatibility
=======================

Adding new methods to built-in classes, especially those as widely used as
``BaseException``, can have substantial impacts. However, GitHub search shows
no collisions for these method names (`zero hits`__\ [#naming]_ and
`three unrelated hits`__ respectively). If user-defined methods with these
names exist in private code they will shadow those proposed in the PEP,
without changing runtime behavior.

__ https://github.com/search?q=%2F%5C.leaf_exceptions%5C%28%2F+language%3APython&type=code
__ https://github.com/search?q=%2F%5C.preserve_context%5C%28%2F+language%3APython&type=code


How to Teach This
=================

Working with exception groups is an intermediate-to-advanced topic, unlikely
to arise for beginning programmers. We therefore suggest teaching this topic
via documentation, and via just-in-time feedback from static analysis tools.
In intermediate classes, we recommend teaching ``.leaf_exceptions()`` together
with the ``.split()`` and ``.subgroup()`` methods, and mentioning
``.preserve_context()`` as an advanced option to address specific pain points.

Both the API reference and the existing `ExceptionGroup tutorial`__
should be updated to demonstrate and explain the new methods. The tutorial
should include examples of common patterns where ``.leaf_exceptions()`` and
``.preserve_context()`` help simplify error handling logic. Downstream
libraries which often use exception groups could include similar docs.

__ https://docs.python.org/3/tutorial/errors.html#raising-and-handling-multiple-unrelated-exceptions

We have also designed lint rules for inclusion in ``flake8-async`` which will
suggest using ``.leaf_exceptions()`` when iterating over ``group.exceptions``
or re-raising a leaf exception, and suggest using ``.preserve_context()`` when
re-raising a leaf exception inside an ``except*`` block would override any
existing context.


Reference Implementation
========================

While the methods on built-in exceptions will be implemented in C if this PEP
is accepted, we hope that the following Python implementation will be useful
on older versions of Python, and can demonstrate the intended semantics.

We have found these helper functions quite useful when working with
:class:`ExceptionGroup`\ s in a large production codebase.

A ``leaf_except


id: pep-0632.rst
content:
PEP: 632
Title: Deprecate distutils module
Author: Steve Dower <steve.dower@python.org>
Discussions-To: https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134
Status: Final
Type: Standards Track
Created: 03-Sep-2020
Python-Version: 3.10
Post-History: 03-Sep-2020, 22-Jan-2021
Resolution: https://mail.python.org/archives/list/python-dev@python.org/thread/TXU6TVOMBLQU3SV57DMMOA5Y2E67AW7P/


Abstract
========

The distutils module [1]_ has for a long time recommended using the
setuptools package [2]_ instead. Setuptools has recently integrated a
complete copy of distutils and is no longer dependent on the standard
library [3]_. Pip has been silently replacing distutils with
setuptools when installing packages for a long time already, and the
distutils documentation has stated that it is being phased out since
2014 (or earlier). It is time to remove it from the standard library.


Motivation
==========

distutils [1]_ is a largely undocumented and unmaintained collection
of utilities for packaging and distributing Python packages, including
compilation of native extension modules. It defines a configuration
format that describes a Python distribution and provides the tools to
convert a directory of source code into a source distribution, and
some forms of binary distribution. Because of its place in the
standard library, many updates can only be released with a major
release, and users cannot rely on particular fixes being available.

setuptools [2]_ is a better documented and well maintained enhancement
based on distutils. While it provides very similar functionality, it
is much better able to support users on earlier Python releases, and
can respond to bug reports more quickly. A number of platform-specific
enhancements already exist in setuptools that have not been added to
distutils, and there is been a long-standing recommendation in the
distutils documentation to prefer setuptools.

Historically, setuptools has extended distutils using subclassing and
monkeypatching, but has now taken a copy of the underlying code. [3]_
As a result, the second last major dependency on distutils is gone and
there is no need to keep it in the standard library.

The final dependency on distutils is CPython itself, which uses it to
build native extension modules in the standard library (except on
Windows). Because this is a CPython build-time dependency, it is
possible to continue to use distutils for this specific case without
it being part of the standard library.

Deprecation and removal will make it obvious that issues should be
fixed in the setuptools project, and will reduce a source of bug
reports and unnecessary test maintenance. It will also help promote
the development of alternative build backends, which can now be
supported more easily thanks to :pep:`517`.


Specification
=============

In Python 3.10 and 3.11, distutils will be formally marked as
deprecated. All known issues will be closed at this time.
``import distutils`` will raise a deprecation warning. New issues that
would be considered release blocking may still be fixed, but support
for new tools or platforms will not be added.

During Python 3.10 and 3.11, uses of distutils within the standard
library may change to use alternative APIs.

In Python 3.12, distutils will no longer be installed by ``make
install`` or any of the first-party distribution. Third-party
redistributors should no longer include distutils in their bundles or
repositories.

This PEP makes no specification on migrating the parts of the CPython
build process that currently use distutils. Depending on
contributions, this migration may occur at any time.

After Python 3.12 is started and when the CPython build process no
longer depends on distutils being in the standard library, the entire
``Lib/distutils`` directory and ``Lib/test/test_distutils.py`` file
will be removed from the repository.

Other references to distutils will be cleaned up. As of Python 3.9's
initial release, the following modules have references in code or
comments:

* Lib/ctypes/util.py
* Lib/site.py
* Lib/sysconfig.py
* Lib/_aix_support.py
* Lib/_bootsubprocess.py
* Lib/_osx_support.py
* Modules/_decimal/tests/formathelper.py

The following Tools in CPython also refer to distutils. Note that none
of these are installed with CPython:

* PC/layout (references will be removed)
* Tools/msi (references will be removed)
* Tools/peg_generator (will be adapted to a different build tool)
* Tools/test2to3 (example project will be removed)

As the distutils code is already included in setuptools, there is no
need to republish it in any other form. Those who require access to
the functionality should use setuptools or an alternative build
backend.

Backwards Compatibility
=======================

Code that imports distutils will no longer work from Python 3.12.

The suggested migration path is to use the equivalent (though not
identical) imports from setuptools (see [5]_), or to migrate to an
alternative build backend (see :pep:`517`).

Code already exists in setuptools to transparently switch ``setup.py``
files using distutils onto their equivalents, and so most working
build scripts are already known to work with setuptools. Such scripts
may need to update their import statements. Consult the setuptools
documentation for specific migration advice. [5]_

Some projects use alternate sets of patches over distutils, notably,
numpy.distutils. [6]_ Projects that we know are doing this have been
informed.

Many build scripts use custom commands or narrowly scoped patches. As
these packages are already subject to setuptools overriding distutils,
we expect minimal disruption as a result of distutils being removed.
Scripts may still need to be updated to avoid importing distutils.


Reference Implementation
========================

setuptools version 48 includes the complete copy of distutils, and as
such is no longer dependent on the standard library's copy. Most
implementation issues they have faced are due to the continuing
existence of distutils in the standard library, and so removal will
improve the stability of their implementation.

There is not yet a reference implementation for the removal of
distutils from the standard library, nor is there an implementation
for CPython's native module builds without relying on the standard
library copy of distutils.


Migration Advice
================

.. note::
   This section suggests some alternative replacements for popular
   functionality that is being formally deprecated with this PEP. It
   is current at time of writing, but is not kept up to date.

For these modules or types, ``setuptools`` is the best substitute:

* ``distutils.ccompiler``
* ``distutils.cmd.Command``
* ``distutils.command``
* ``distutils.config``
* ``distutils.core.Distribution``
* ``distutils.errors``

For these modules or types, use the standards-defined Python Packaging
Authority packages specified:

* ``distutils.version`` â use the ``packaging`` package

For these modules or functions, use the standard library module shown:

* ``distutils.fancy_getopt`` â use the ``argparse`` module
* ``distutils.spawn.find_executable`` â use the ``shutil.which`` function
* ``distutils.spawn.spawn`` â use the ``subprocess.run`` function
* ``distutils.sysconfig`` â use the ``sysconfig`` module
* ``distutils.util.get_platform`` â use the ``platform`` module

For these functions, and any others not mentioned here, you will need
to reimplement the functionality yourself. The legacy documentation
can be found at https://docs.python.org/3.9/distutils/apiref.html

* ``distutils.dir_util.create_tree``
* ``distutils.util.change_root``
* ``distutils.util.strtobool``


Rejected Ideas
==============

Deprecate but do not delete
---------------------------

The primary concern with this approach is that distutils most
frequently breaks because of platform differences, which means that
without maintenance, it will stop working out-of-sync with any
Python


id: pep-0626.rst
content:
PEP: 626
Title: Precise line numbers for debugging and other tools.
Author: Mark Shannon <mark@hotpy.org>
BDFL-Delegate: Pablo Galindo <pablogsal@python.org>
Status: Final
Type: Standards Track
Created: 15-Jul-2020
Python-Version: 3.10
Post-History: 17-Jul-2020

Abstract
========

Python should guarantee that when tracing is turned on, "line" tracing events are generated for *all* lines of code executed and *only* for lines of
code that are executed.

The ``f_lineno`` attribute of frame objects should always contain the expected line number.
During frame execution, the expected line number is the line number of source code currently being executed.
After a frame has completed, either by returning or by raising an exception,
the expected line number is the line number of the last line of source that was executed.

A side effect of ensuring correct line numbers, is that some bytecodes will need to be marked as artificial, and not have a meaningful line number.
To assist tools, a new ``co_lines`` attribute will be added that describes the mapping from bytecode to source.

Motivation
==========

Users of ``sys.settrace`` and associated tools should be able to rely on tracing events being
generated for all lines of code, and only for actual code.
They should also be able to assume that the line number in ``f_lineno`` is correct.

The current implementation mostly does this, but fails in a few cases.
This requires workarounds in tooling and is a nuisance for alternative Python implementations.

Having this guarantee also benefits implementers of CPython in the long term, as the current behaviour is not obvious and has some odd corner cases.

Rationale
=========

In order to guarantee that line events are generated when expected, the ``co_lnotab`` attribute, in its current form,
can no longer be the source of truth for line number information.

Rather than attempt to fix the ``co_lnotab`` attribute, a new method
``co_lines()`` will be added, which returns an iterator over bytecode offsets and source code lines.

Ensuring that the bytecode is annotated correctly to enable accurate line number information means that
some bytecodes must be marked as artificial, and not have a line number.

Some care must be taken not to break existing tooling.
To minimize breakage, the ``co_lnotab`` attribute will be retained, but lazily generated on demand.

Specification
=============

Line events and the ``f_lineno`` attribute should act as an experienced Python user would expect in *all* cases.

Tracing
'''''''

Tracing generates events for calls, returns, exceptions, lines of source code executed, and, under some circumstances, instructions executed.

Only line events are covered by this PEP.

When tracing is turned on, line events will be generated when:

* A new line of source code is reached.
* A backwards jump occurs, even if it jumps to the same line, as may happen in list comprehensions.

Additionally, line events will *never* be generated for source code lines that are not executed.

What is considered to be code for the purposes of tracing
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''

All expressions and parts of expressions are considered to be executable code.

In general, all statements are also considered to be executable code. However, when a statement is spread over several lines,
we must consider which parts of a statement are considered to be executable code.

Statements are made up of keywords and expressions. Not all keywords have a direct runtime effect, so not all keywords are considered to be executable code.
For example, ``else``, is a necessary part of an ``if`` statement, but there is no runtime effect associated with an ``else``.

For the purposes of tracing, the following keywords will *not* be considered to be executable code:

* ``del`` -- The expression to be deleted is treated as the executable code.
* ``else`` -- No runtime effect
* ``finally`` -- No runtime effect
* ``global`` -- Purely declarative
* ``nonlocal`` -- Purely declarative

All other keywords are considered to be executable code.

Example event sequences
'''''''''''''''''''''''

In the following examples, events are listed as "name", ``f_lineno`` pairs.

The code

::

  1.     global x
  2.     x = a

generates the following event::


  "line" 2

The code

::

  1.     try:
  2.        pass
  3.     finally:
  4.        pass

generates the following events::


  "line" 1
  "line" 2
  "line" 4

The code

::

  1.      for (
  2.          x) in [1]:
  3.          pass
  4.      return

generates the following events::


  "line" 2       # evaluate [1]
  "line" 1       # for
  "line" 2       # store to x
  "line" 3       # pass
  "line" 1       # for
  "line" 4       # return
  "return" 1


The f_lineno attribute
''''''''''''''''''''''

* When a frame object is created, the ``f_lineno`` attribute will be set to the line
  at which the function or class is defined; that is the line on which the ``def`` or ``class`` keyword appears.
  For modules it will be set to zero.
* The ``f_lineno`` attribute will be updated to match the line number about to be executed,
  even if tracing is turned off and no event is generated.

The new co_lines() method of code objects
'''''''''''''''''''''''''''''''''''''''''

The ``co_lines()`` method will return an iterator which yields tuples of values,
each representing the line number of a range of bytecodes. Each tuple will consist of three values:

* ``start`` -- The offset (inclusive) of the start of the bytecode range
* ``end`` -- The offset (exclusive) of the end of the bytecode range
* ``line`` -- The line number, or ``None`` if the bytecodes in the given range do not have a line number.

The sequence generated will have the following properties:

* The first range in the sequence with have a ``start`` of ``0``
* The ``(start, end)`` ranges will be non-decreasing and consecutive.
  That is, for any pair of tuples the ``start`` of the second
  will equal to the ``end`` of the first.
* No range will be backwards, that is ``end >= start`` for all triples.
* The final range in the sequence with have ``end`` equal to the size of the bytecode.
* ``line`` will either be a positive integer, or ``None``

Zero width ranges
-----------------

Zero width range, that is ranges where ``start == end`` are allowed.
Zero width ranges are used for lines that are present in the source code,
but have been eliminated by the bytecode compiler.

The co_linetable attribute
''''''''''''''''''''''''''

The co_linetable attribute will hold the line number information.
The format is opaque, unspecified and may be changed without notice.
The attribute is public only to support creation of new code objects.

The co_lnotab attribute
'''''''''''''''''''''''

Historically the ``co_lnotab`` attribute held a mapping from bytecode offset to line number, but does not support bytecodes without a line number.
For backward compatibility, the ``co_lnotab`` bytes object will be lazily created when needed.
For ranges of bytecodes without a line number, the line number of the previous bytecode range will be used.

Tools that parse the ``co_lnotab`` table should move to using the new ``co_lines()`` method as soon as is practical.


Backwards Compatibility
=======================

The ``co_lnotab`` attribute will be deprecated in 3.10 and removed in 3.12.

Any tools that parse the ``co_lnotab`` attribute of code objects will need to move to using ``co_lines()`` before 3.12 is released.
Tools that use ``sys.settrace`` will be unaffected, except in cases where the "line" events they receive are more accurate.


Examples of code for which the sequence of trace events will change
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

In the following examples, events are listed as "name", ``f_lineno`` pairs.


``pass`` statement in an ``if`` statement.
------------------------------------------

::

  0.  def spam(a):
  1.      if a:
  2.          eggs()
  3.      els


id: pep-0801.rst
content:
PEP: 801
Title: Reserved
Author: Barry Warsaw <barry@python.org>
Status: Active
Type: Informational
Created: 21-Jun-2018


Abstract
========

This PEP is reserved for future use, because
`We are the 801 <https://www.youtube.com/watch?v=FByM0r-VOLc>`_.
Contact the author for details.


Copyright
=========

This document has been placed in the public domain.


id: pep-0418.rst
content:
PEP: 418
Title: Add monotonic time, performance counter, and process time functions
Author: Cameron Simpson <cs@cskk.id.au>,
        Jim J. Jewett <jimjjewett@gmail.com>,
        Stephen J. Turnbull <stephen@xemacs.org>,
        Victor Stinner <vstinner@python.org>
Status: Final
Type: Standards Track
Created: 26-Mar-2012
Python-Version: 3.3


Abstract
========

This PEP proposes to add ``time.get_clock_info(name)``,
``time.monotonic()``, ``time.perf_counter()`` and
``time.process_time()`` functions to Python 3.3.


Rationale
=========

If a program uses the system time to schedule events or to implement
a timeout, it may fail to run events at the right moment or stop the
timeout too early or too late when the system time is changed manually or
adjusted automatically by NTP.  A monotonic clock should be used
instead to not be affected by system time updates:
``time.monotonic()``.

To measure the performance of a function, ``time.clock()`` can be used
but it is very different on Windows and on Unix.  On Windows,
``time.clock()`` includes time elapsed during sleep, whereas it does
not on Unix.  ``time.clock()`` resolution is very good on Windows, but
very bad on Unix.  The new ``time.perf_counter()`` function should be
used instead to always get the most precise performance counter with a
portable behaviour (ex: include time spend during sleep).

Until now, Python did not provide directly a portable
function to measure CPU time.  ``time.clock()`` can be used on Unix,
but it has bad
resolution.  ``resource.getrusage()`` or ``os.times()`` can also be
used on Unix, but they require to compute the sum of time
spent in kernel space and user space.  The new ``time.process_time()``
function acts as a portable counter that always measures CPU time
(excluding time elapsed during sleep) and has the best available
resolution.

Each operating system implements clocks and performance counters
differently, and it is useful to know exactly which function is used
and some properties of the clock like its resolution.  The new
``time.get_clock_info()`` function gives access to all available
information about each Python time function.

New functions:

* ``time.monotonic()``: timeout and scheduling, not affected by system
  clock updates
* ``time.perf_counter()``: benchmarking, most precise clock for short
  period
* ``time.process_time()``: profiling, CPU time of the process

Users of new functions:

* time.monotonic(): concurrent.futures, multiprocessing, queue, subprocess,
  telnet and threading modules to implement timeout
* time.perf_counter(): trace and timeit modules, pybench program
* time.process_time(): profile module
* time.get_clock_info(): pybench program to display information about the
  timer like the resolution

The ``time.clock()`` function is deprecated because it is not
portable: it behaves differently depending on the operating system.
``time.perf_counter()`` or ``time.process_time()`` should be used
instead, depending on your requirements. ``time.clock()`` is marked as
deprecated but is not planned for removal.

Limitations:

* The behaviour of clocks after a system suspend is not defined in the
  documentation of new functions. The behaviour depends on the
  operating system: see the `Monotonic Clocks`_ section below. Some
  recent operating systems provide two clocks, one including time
  elapsed during system suspend, one not including this time. Most
  operating systems only provide one kind of clock.
* time.monotonic() and time.perf_counter() may or may not be adjusted.
  For example, ``CLOCK_MONOTONIC`` is slewed on Linux, whereas
  ``GetTickCount()`` is not adjusted on Windows.
  ``time.get_clock_info('monotonic')['adjustable']`` can be used to check
  if the monotonic clock is adjustable or not.
* No time.thread_time() function is proposed by this PEP because it is
  not needed by Python standard library nor a common asked feature.
  Such function would only be available on Windows and Linux. On
  Linux, it is possible to use
  ``time.clock_gettime(CLOCK_THREAD_CPUTIME_ID)``. On Windows, ctypes or
  another module can be used to call the ``GetThreadTimes()``
  function.


Python functions
================

New Functions
-------------

time.get_clock_info(name)
^^^^^^^^^^^^^^^^^^^^^^^^^

Get information on the specified clock.  Supported clock names:

* ``"clock"``: ``time.clock()``
* ``"monotonic"``: ``time.monotonic()``
* ``"perf_counter"``: ``time.perf_counter()``
* ``"process_time"``: ``time.process_time()``
* ``"time"``: ``time.time()``

Return a ``time.clock_info`` object which has the following attributes:

* ``implementation`` (str): name of the underlying operating system
  function.  Examples: ``"QueryPerformanceCounter()"``,
  ``"clock_gettime(CLOCK_REALTIME)"``.
* ``monotonic`` (bool): True if the clock cannot go backward.
* ``adjustable`` (bool): ``True`` if the clock can be changed automatically
  (e.g. by a NTP daemon) or manually by the system administrator, ``False``
  otherwise
* ``resolution`` (float): resolution in seconds of the clock.


time.monotonic()
^^^^^^^^^^^^^^^^

Monotonic clock, i.e. cannot go backward.  It is not affected by system
clock updates.  The reference point of the returned value is
undefined, so that only the difference between the results of
consecutive calls is valid and is a number of seconds.

On Windows versions older than Vista, ``time.monotonic()`` detects
``GetTickCount()`` integer overflow (32 bits, roll-over after 49.7
days).  It increases an internal epoch (reference time by) 2\
:sup:`32` each time that an overflow is detected.  The epoch is stored
in the process-local state and so
the value of ``time.monotonic()`` may be different in two Python
processes running for more than 49 days. On more recent versions of
Windows and on other operating systems, ``time.monotonic()`` is
system-wide.

Availability: Windows, Mac OS X, Linux, FreeBSD, OpenBSD, Solaris.
Not available on GNU/Hurd.

Pseudo-code [#pseudo]_::

    if os.name == 'nt':
        # GetTickCount64() requires Windows Vista, Server 2008 or later
        if hasattr(_time, 'GetTickCount64'):
            def monotonic():
                return _time.GetTickCount64() * 1e-3
        else:
            def monotonic():
                ticks = _time.GetTickCount()
                if ticks < monotonic.last:
                    # Integer overflow detected
                    monotonic.delta += 2**32
                monotonic.last = ticks
                return (ticks + monotonic.delta) * 1e-3
            monotonic.last = 0
            monotonic.delta = 0

    elif sys.platform == 'darwin':
        def monotonic():
            if monotonic.factor is None:
                factor = _time.mach_timebase_info()
                monotonic.factor = timebase[0] / timebase[1] * 1e-9
            return _time.mach_absolute_time() * monotonic.factor
        monotonic.factor = None

    elif hasattr(time, "clock_gettime") and hasattr(time, "CLOCK_HIGHRES"):
        def monotonic():
            return time.clock_gettime(time.CLOCK_HIGHRES)

    elif hasattr(time, "clock_gettime") and hasattr(time, "CLOCK_MONOTONIC"):
        def monotonic():
            return time.clock_gettime(time.CLOCK_MONOTONIC)


On Windows, ``QueryPerformanceCounter()`` is not used even though it
has a better resolution than ``GetTickCount()``.  It is not reliable
and has too many issues.


time.perf_counter()
^^^^^^^^^^^^^^^^^^^

Performance counter with the highest available resolution to measure a
short duration.  It does include time elapsed during sleep and is
system-wide.  The reference point of the returned value is undefined,
so that only the difference between the results of consecutive calls
is valid and is a number of seconds.

It is available on all platforms.

Pseudo-code::

    if os.name == 'nt':
        def _win_perf_counter():
            if _win_perf_counter.frequency is None:
                _win_perf_counter.frequency = _time.QueryPerformanceFrequency()
            


id: pep-0430.rst
content:
PEP: 430
Title: Migrating to Python 3 as the default online documentation
Author: Alyssa Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: Georg Brandl
Status: Final
Type: Informational
Created: 27-Oct-2012


Abstract
========

This PEP proposes a strategy for migrating the default version of the
Python documentation presented to users of Python when accessing
docs.python.org from 2.7 to Python 3.3.

It proposes a backwards compatible scheme that preserves the meaning of
existing deep links in to the Python 2 documentation, while still
presenting the Python 3 documentation by default, and presenting the
Python 2 and 3 documentation in a way that avoids making the Python 3
documentation look like a second-class citizen.


Background
==========

With the transition of the overall Python ecosystem from Python 2 to Python 3
still in progress, one question which arises periodically [1]_, [2]_ is when
and how to handle the change from providing the Python 2 documentation as
the default version displayed at the docs.python.org root URL to providing
the Python 3 documentation.


Key Concerns
============

There are a couple of key concerns that any migration proposal needs to
address.


Don't Confuse Beginners
-----------------------

Many beginners learn Python through third party resources. These resources,
not all of which are online resources, may reference in to the python.org
online documentation for additional background and details.

Importantly, even when the online documentation is updated, the "version
added" and "version changed" tags usually provide enough information for
users to adjust appropriately for the specific version they are using.

While deep links in to the python.org documentation may occasionally break
within the Python 2 series, this is very rare.

Migrating to Python 3 is a very different matter. Many links would break due
to renames and removals, and the "version added" and "version changed"
information for the Python 2 series is completely absent.


Don't Break Useful Resources
----------------------------

There are many useful Python resources out there, such as the mailing list
archives on python.org and question-and-answer sites like Stack Overflow,
where links are highly unlikely to be updated, no matter how much notice
is provided.

Old posts and answers to questions all currently link to docs.python.org
expecting to get the Python 2 documentation at unqualified URLs. Links from
answers that relate to Python 3 are explicitly qualified with ``/py3k/`` in
the path component.


Proposal
========

This PEP (based on an idea originally put forward back in May [3]_) is to
*not migrate* the Python 2 specific deep links at all, and instead adopt a
scheme where all URLs presented to users on docs.python.org are qualified
appropriately with the relevant release series.

Visitors to the root URL at ``http://docs.python.org`` will be automatically
redirected to ``http://docs.python.org/3/``, but links deeper in
the version-specific hierarchy, such as to
``http://docs.python.org/library/os``, will instead be redirected to
a Python 2 specific link such as ``http://docs.python.org/2/library/os``.

The specific subpaths which will be redirected to explicitly qualified
paths for the Python 2 docs are:

* ``/c-api/``
* ``/distutils/``
* ``/extending/``
* ``/faq/``
* ``/howto/``
* ``/library/``
* ``/reference/``
* ``/tutorial/``
* ``/using/``
* ``/whatsnew/``
* ``/about.html``
* ``/bugs.html``
* ``/contents.html``
* ``/copyright.html``
* ``/license.html``
* ``/genindex.html``
* ``/glossary.html``
* ``/py-modindex.html``
* ``/search.html``

The existing ``/py3k/`` subpath will be redirected to the new ``/3/``
subpath.


Presented URLs
--------------

With this scheme, the following URLs would be presented to users after
resolution of any aliasing and rewriting rules:

* ``http://docs.python.org/x/*``
* ``http://docs.python.org/x.y/*``
* ``http://docs.python.org/dev/*``
* ``http://docs.python.org/release/x.y.z/*``
* ``http://docs.python.org/devguide``

The ``/x/`` URLs mean "give me the latest documentation for a released
version in this release series". It will draw the documentation from the
relevant maintenance branch in source control (this will always be the
2.7 branch for Python 2 and is currently 3.3 for Python 3). Differences
relative to previous versions in the release series will be available
through "version added" and "version changed" markers.

The ``/x.y/`` URLs mean "give me the latest documentation for this release".
It will draw the documentation from the relevant maintenance branch in
source control (or the default branch for the currently in development
version). It differs from the status quo in that the URLs will
actually remain available in the user's browser for easy copy and pasting.
(Currently, references to specific versions that are not the latest in their
release series will resolve to a stable URL for a specific maintenance
version in the "release" hierarchy, while the current latest version in the
release series resolves to the release series URL. This makes it hard to get
a "latest version specific URL", since it is always necessary to construct
them manually).

The ``/dev/`` URL means the documentation for the default branch in source
control.

The ``/release/x.y.x/`` URLs will refer to the documentation of those
releases, exactly as it was at the time of the release.

The developer's guide is not version specific, and thus retains its own
stable ``/devguide/`` URL.

Rationale
=========

There is some desire to switch the unqualified references to mean Python 3
as a sign of confidence in Python 3. Such a move would either break a lot of
things, or else involve an awful lot of work to *avoid* breaking things.

I believe we can get much the same effect without breaking the world by:

1. Deprecating the use of unqualified references to the online
   documentation (while promising to preserve the meaning of such
   references indefinitely)
2. Updating all python.org and python-dev controlled links to use
   qualified references (excluding archived email)
3. Redirecting visitors to the root of ``http://docs.python.org`` to
   ``http://docs.python.org/3.x``

Most importantly, because this scheme doesn't alter the behaviour of any
existing deep links, it could be implemented with a significantly shorter
warning period than would be required for a scheme that risked breaking
deep links, or started to redirect unqualified links to Python 3. The
only part of the scheme which would require any warning at all is the
step of redirecting the "http://docs.python.org/" landing page to the
Python 3.3 documentation.

Namespaces are one honking great idea - let's do more of those.

Note that the approach described in this PEP gives two ways to access the
content of the default branch: as ``/dev/`` or using the appropriate
``/x.y/`` reference. This is deliberate, as the default branch is referenced
for two different purposes:

* to provide additional information when discussing an upcoming feature of
  the next release (a ``/x.y/`` URL is appropriate)
* to provide a stable destination for developers to access the documentation
  of the next feature release, regardless of the version (a ``/dev/`` URL is
  appropriate)


Implementation
==============

The URLs on docs.python.org are controlled by the python.org infrastructure
team rather than through the CPython source repo, so acceptance and
implementation of the ideas in this PEP will be up to the team.


References
==========

.. [1] May 2012 discussion
   (https://mail.python.org/pipermail/python-dev/2012-May/119524.html)

.. [2] October 2012 discussion
   (https://mail.python.org/pipermail/python-ideas/2012-October/017406.html)

.. [3] Using a "/latest/" path prefix
   (https://mail.python.org/pipermail/python-dev/2012-May/119567.html)


Copyright
===========
This document has been placed in the public domain.


id: pep-0356.rst
content:
PEP: 356
Title: Python 2.5 Release Schedule
Author: Neal Norwitz, Guido van Rossum, Anthony Baxter
Status: Final
Type: Informational
Topic: Release
Created: 07-Feb-2006
Python-Version: 2.5
Post-History:


Abstract
========

This document describes the development and release schedule for
Python 2.5.  The schedule primarily concerns itself with PEP-sized
items.  Small features may be added up to and including the first
beta release.  Bugs may be fixed until the final release.

There will be at least two alpha releases, two beta releases, and
one release candidate.  The release date is planned for
12 September 2006.


Release Manager
===============

- Anthony Baxter has volunteered to be Release Manager.
- Martin von Loewis is building the Windows installers,
- Ronald Oussoren is building the Mac installers,
- Fred Drake the doc packages and
- Sean Reifschneider the RPMs.


Release Schedule
================

- alpha 1: April 5, 2006 [completed]
- alpha 2: April 27, 2006 [completed]
- beta 1:  June 20, 2006 [completed]
- beta 2:  July 11, 2006 [completed]
- beta 3:  August 3, 2006 [completed]
- rc 1:    August 17, 2006 [completed]
- rc 2:    September 12, 2006 [completed]
- final:   September 19, 2006 [completed]


Completed features for 2.5
==========================

- :pep:`308`: Conditional Expressions
- :pep:`309`: Partial Function Application
- :pep:`314`: Metadata for Python Software Packages v1.1
- :pep:`328`: Absolute/Relative Imports
- :pep:`338`: Executing Modules as Scripts
- :pep:`341`: Unified try-except/try-finally to try-except-finally
- :pep:`342`: Coroutines via Enhanced Generators
- :pep:`343`: The "with" Statement (still need updates in Doc/ref and for the
  ``contextlib`` module)
- :pep:`352`: Required Superclass for Exceptions
- :pep:`353`: Using ``ssize_t`` as the index type
- :pep:`357`: Allowing Any Object to be Used for Slicing

- ASCII became the default coding

- AST-based compiler

- Access to C AST from Python through new ``_ast`` module

- ``any()``/``all()`` builtin truth functions

New standard library modules:

- ``cProfile`` -- suitable for profiling long running applications
  with minimal overhead

- ``ctypes`` -- optional component of the windows installer

- ``ElementTree`` and ``cElementTree`` -- by Fredrik Lundh

- ``hashlib`` -- adds support for SHA-224, -256, -384, and -512
  (replaces old ``md5`` and ``sha`` modules)

- ``msilib`` -- for creating MSI files and ``bdist_msi`` in distutils.

- ``pysqlite``

- ``uuid``

- ``wsgiref``

Other notable features:

- Added support for reading shadow passwords [1]_

- Added support for the Unicode 4.1 UCD

- Added :pep:`302` ``zipfile``/``__loader__`` support to the following modules:
  ``warnings``, ``linecache``, ``inspect``, ``traceback``, ``site``, and
  ``doctest``

- Added ``pybench`` Python benchmark suite -- by Marc-Andre Lemburg

- Add write support for mailboxes from the code in sandbox/mailbox.
  (Owner: A.M. Kuchling.  It would still be good if another person
  would take a look at the new code.)

- Support for building "fat" Mac binaries (Intel and PPC)

- Add new icons for Windows with the new Python logo?

- New utilities in ``functools`` to help write wrapper functions that
  support naive introspection (e.g. having ``f.__name__`` return
  the original function name).

- Upgrade ``pyexpat`` to use expat 2.0.

- Python core now compiles cleanly with g++


Possible features for 2.5
=========================

Each feature below should implemented prior to beta1 or
will require BDFL approval for inclusion in 2.5.

- Modules under consideration for inclusion:

- Add new icons for MacOS and Unix with the new Python logo?
  (Owner: ???)
  MacOS: http://hcs.harvard.edu/~jrus/python/prettified-py-icons.png

- Check the various bits of code in ``Demo/`` all still work, update or
  remove the ones that don't.
  (Owner: Anthony)

- All modules in ``Modules/`` should be updated to be ssize_t clean.
  (Owner: Neal)


Deferred until 2.6
==================

- ``bdist_deb`` in distutils package [2]_

- ``bdist_egg`` in distutils package

- pure python ``pgen`` module
  (Owner: Guido)

- Remove the ``fpectl`` module?

- Make everything in ``Modules/`` build cleanly with g++


Open issues
===========

- Bugs that need resolving before release, ie, they block release:

  None

- Bugs deferred until 2.5.1 (or later):

  * https://bugs.python.org/issue1544279 - ``Socket`` module is not thread-safe
  * https://bugs.python.org/issue1541420 - tools and demo missing from windows
  * https://bugs.python.org/issue1542451 - crash with continue in nested try/finally
  * https://bugs.python.org/issue1475523 - gettext.py bug (owner: Martin v. Loewis)
  * https://bugs.python.org/issue1467929 - %-formatting and dicts
  * https://bugs.python.org/issue1446043 - ``unicode()`` does not raise ``LookupError``

- The :pep:`302` changes to (at least) ``pkgutil``, ``runpy`` and ``pydoc`` must
  be documented.

- ``test_zipfile64`` takes too long and too much disk space for
  most of the buildbots.  How should this be handled?
  It is currently disabled.

- should C modules listed in "Undocumented modules" be removed too?
  "timing" (listed as obsolete), "cl" (listed as possibly not up-to-date),
  and "sv" (listed as obsolete hardware specific).


References
==========

.. [1] Shadow Password Support Module
       https://bugs.python.org/issue579435

.. [2] Joe Smith, bdist_* to stdlib?
       https://mail.python.org/pipermail/python-dev/2006-February/060926.html

Copyright
=========

This document has been placed in the public domain.


id: pep-0342.rst
content:
PEP: 342
Title: Coroutines via Enhanced Generators
Author: Guido van Rossum, Phillip J. Eby
Status: Final
Type: Standards Track
Created: 10-May-2005
Python-Version: 2.5
Post-History:


Introduction
============

This PEP proposes some enhancements to the API and syntax of generators, to
make them usable as simple coroutines.  It is basically a combination of ideas
from these two PEPs, which may be considered redundant if this PEP is
accepted:

- :pep:`288`, Generators Attributes and Exceptions.  The current PEP covers its
  second half, generator exceptions (in fact the ``throw()`` method name was
  taken from :pep:`288`).  :pep:`342` replaces generator attributes, however, with a
  concept from an earlier revision of :pep:`288`, the *yield expression*.

- :pep:`325`, Resource-Release Support for Generators.  :pep:`342` ties up a few
  loose ends in the :pep:`325` spec, to make it suitable for actual
  implementation.


Motivation
==========

Coroutines are a natural way of expressing many algorithms, such as
simulations, games, asynchronous I/O, and other forms of event-driven
programming or co-operative multitasking.  Python's generator functions are
almost coroutines -- but not quite -- in that they allow pausing execution to
produce a value, but do not provide for values or exceptions to be passed in
when execution resumes.  They also do not allow execution to be paused within
the ``try`` portion of ``try/finally`` blocks, and therefore make it difficult
for an aborted coroutine to clean up after itself.

Also, generators cannot yield control while other functions are executing,
unless those functions are themselves expressed as generators, and the outer
generator is written to yield in response to values yielded by the inner
generator.  This complicates the implementation of even relatively simple use
cases like asynchronous communications, because calling any functions either
requires the generator to *block* (i.e. be unable to yield control), or else a
lot of boilerplate looping code must be added around every needed function
call.

However, if it were possible to pass values or exceptions *into* a generator at
the point where it was suspended, a simple co-routine scheduler or *trampoline
function* would let coroutines *call* each other without blocking -- a
tremendous boon for asynchronous applications.  Such applications could then
write co-routines to do non-blocking socket I/O by yielding control to an I/O
scheduler until data has been sent or becomes available.  Meanwhile, code that
performs the I/O would simply do something like this::

    data = (yield nonblocking_read(my_socket, nbytes))

in order to pause execution until the ``nonblocking_read()`` coroutine produced
a value.

In other words, with a few relatively minor enhancements to the language and to
the implementation of the generator-iterator type, Python will be able to
support performing asynchronous operations without needing to write the entire
application as a series of callbacks, and without requiring the use of
resource-intensive threads for programs that need hundreds or even thousands of
co-operatively multitasking pseudothreads.  Thus, these enhancements will give
standard Python many of the benefits of the Stackless Python fork, without
requiring any significant modification to the CPython core or its APIs.  In
addition, these enhancements should be readily implementable by any Python
implementation (such as Jython) that already supports generators.


Specification Summary
=====================

By adding a few simple methods to the generator-iterator type, and with two
minor syntax adjustments, Python developers will be able to use generator
functions to implement co-routines and other forms of co-operative
multitasking.  These methods and adjustments are:

1. Redefine ``yield`` to be an expression, rather than a statement. The current
   yield statement would become a yield expression whose value is thrown away.
   A yield expression's value is ``None`` whenever the generator is resumed by
   a normal ``next()`` call.

2. Add a new ``send()`` method for generator-iterators, which resumes the
   generator and *sends* a value that becomes the result of the current
   yield-expression.  The ``send()`` method returns the next value yielded by
   the generator, or raises ``StopIteration`` if the generator exits without
   yielding another value.

3. Add a new ``throw()`` method for generator-iterators, which raises an
   exception at the point where the generator was paused, and which returns the
   next value yielded by the generator, raising ``StopIteration`` if the
   generator exits without yielding another value.  (If the generator does not
   catch the passed-in exception, or raises a different exception, then that
   exception propagates to the caller.)

4. Add a ``close()`` method for generator-iterators, which raises
   ``GeneratorExit`` at the point where the generator was paused.  If the
   generator then raises ``StopIteration`` (by exiting normally, or due to
   already being closed) or ``GeneratorExit`` (by not catching the exception),
   ``close()`` returns to its caller.  If the generator yields a value, a
   ``RuntimeError`` is raised.  If the generator raises any other exception, it
   is propagated to the caller. ``close()`` does nothing if the generator has
   already exited due to an exception or normal exit.

5. Add support to ensure that ``close()`` is called when a generator iterator
   is garbage-collected.

6. Allow ``yield`` to be used in ``try/finally`` blocks, since garbage
   collection or an explicit ``close()`` call would now allow the ``finally``
   clause to execute.

A prototype patch implementing all of these changes against the current Python
CVS HEAD is available as SourceForge patch #1223381
(https://bugs.python.org/issue1223381).


Specification: Sending Values into Generators
=============================================

New generator method: ``send(value)``
-------------------------------------

A new method for generator-iterators is proposed, called ``send()``.  It
takes exactly one argument, which is the value that should be *sent in* to
the generator.  Calling ``send(None)`` is exactly equivalent to calling a
generator's ``next()`` method.  Calling ``send()`` with any other value is
the same, except that the value produced by the generator's current
yield expression will be different.

Because generator-iterators begin execution at the top of the generator's
function body, there is no yield expression to receive a value when the
generator has just been created.  Therefore, calling ``send()`` with a
non-``None`` argument is prohibited when the generator iterator has just
started, and a ``TypeError`` is raised if this occurs (presumably due to a
logic error of some kind).  Thus, before you can communicate with a
coroutine you must first call ``next()`` or ``send(None)`` to advance its
execution to the first yield expression.

As with the ``next()`` method, the ``send()`` method returns the next value
yielded by the generator-iterator, or raises ``StopIteration`` if the
generator exits normally, or has already exited.  If the generator raises an
uncaught exception, it is propagated to ``send()``'s caller.

New syntax: Yield Expressions
-----------------------------

The yield-statement will be allowed to be used on the right-hand side of an
assignment; in that case it is referred to as yield-expression.  The value
of this yield-expression is ``None`` unless ``send()`` was called with a
non-``None`` argument; see below.

A yield-expression must always be parenthesized except when it occurs at the
top-level expression on the right-hand side of an assignment.  So

::

    x = yield 42
    x = yield
    x = 12 + (yield 42)
    x = 12 + (yield)
    foo(yield 42)
    foo(yield)

are all legal, but

::

    x = 12 + yield 42
    x = 12 + yield
    foo(yield 42, 12)
    foo(yield, 12)

are all illegal.  (Some of the edge cases are m


id: pep-0424.rst
content:
PEP: 424
Title: A method for exposing a length hint
Author: Alex Gaynor <alex.gaynor@gmail.com>
Status: Final
Type: Standards Track
Created: 14-Jul-2012
Python-Version: 3.4
Post-History: `15-Jul-2012 <https://mail.python.org/pipermail/python-dev/2012-July/120920.html>`__

Abstract
========

CPython currently defines a ``__length_hint__`` method on several
types, such as various iterators.  This method is then used by various
other functions (such as ``list``) to presize lists based on the
estimate returned by ``__length_hint__``.  Types which are not sized,
and thus should not define ``__len__``, can then define
``__length_hint__``, to allow estimating or computing a size (such as
many iterators).

Specification
=============

This PEP formally documents ``__length_hint__`` for other interpreters
and non-standard-library Python modules to implement.

``__length_hint__`` must return an integer (else a ``TypeError`` is
raised) or ``NotImplemented``, and is not required to be accurate.  It
may return a value that is either larger or smaller than the actual
size of the container.  A return value of ``NotImplemented`` indicates
that there is no finite length estimate.  It may not return a negative
value (else a ValueError is raised).

In addition, a new function ``operator.length_hint`` hint is added,
with the following semantics (which define how ``__length_hint__``
should be used)::

    def length_hint(obj, default=0):
        """Return an estimate of the number of items in obj.

        This is useful for presizing containers when building from an
        iterable.

        If the object supports len(), the result will be
        exact. Otherwise, it may over- or under-estimate by an
        arbitrary amount. The result will be an integer >= 0.
        """
        try:
            return len(obj)
        except TypeError:
            try:
                get_hint = type(obj).__length_hint__
            except AttributeError:
                return default
            try:
                hint = get_hint(obj)
            except TypeError:
                return default
            if hint is NotImplemented:
                return default
            if not isinstance(hint, int):
                raise TypeError("Length hint must be an integer, not %r" %
                                type(hint))
            if hint < 0:
                raise ValueError("__length_hint__() should return >= 0")
            return hint


Rationale
=========

Being able to pre-allocate lists based on the expected size, as
estimated by ``__length_hint__``, can be a significant optimization.
CPython has been observed to run some code faster than PyPy, purely
because of this optimization being present.

Copyright
=========

This document has been placed into the public domain.


id: pep-0395.rst
content:
PEP: 395
Title: Qualified Names for Modules
Author: Alyssa Coghlan <ncoghlan@gmail.com>
Status: Withdrawn
Type: Standards Track
Created: 04-Mar-2011
Python-Version: 3.4
Post-History: 05-Mar-2011, 19-Nov-2011


PEP Withdrawal
==============

This PEP was withdrawn by the author in December 2013, as other significant
changes in the time since it was written have rendered several aspects
obsolete. Most notably :pep:`420` namespace packages rendered some of the
proposals related to package detection unworkable and :pep:`451` module
specifications resolved the multiprocessing issues and provide a possible
means to tackle the pickle compatibility issues.

A future PEP to resolve the remaining issues would still be appropriate,
but it's worth starting any such effort as a fresh PEP restating the
remaining problems in an updated context rather than trying to build on
this one directly.


Abstract
========

This PEP proposes new mechanisms that eliminate some longstanding traps for
the unwary when dealing with Python's import system, as well as serialisation
and introspection of functions and classes.

It builds on the "Qualified Name" concept defined in :pep:`3155`.


Relationship with Other PEPs
----------------------------

Most significantly, this PEP is currently deferred as it requires
significant changes in order to be made compatible with the removal
of mandatory __init__.py files in :pep:`420` (which has been implemented
and released in Python 3.3).

This PEP builds on the "qualified name" concept introduced by :pep:`3155`, and
also shares in that PEP's aim of fixing some ugly corner cases when dealing
with serialisation of arbitrary functions and classes.

It also builds on :pep:`366`, which took initial tentative steps towards making
explicit relative imports from the main module work correctly in at least
*some* circumstances.

Finally, :pep:`328` eliminated implicit relative imports from imported modules.
This PEP proposes that the de facto implicit relative imports from main
modules that are provided by the current initialisation behaviour for
``sys.path[0]`` also be eliminated.


What's in a ``__name__``?
=========================

Over time, a module's ``__name__`` attribute has come to be used to handle a
number of different tasks.

The key use cases identified for this module attribute are:

1. Flagging the main module in a program, using the ``if __name__ ==
   "__main__":`` convention.
2. As the starting point for relative imports
3. To identify the location of function and class definitions within the
   running application
4. To identify the location of classes for serialisation into pickle objects
   which may be shared with other interpreter instances


Traps for the Unwary
====================

The overloading of the semantics of ``__name__``, along with some historically
associated behaviour in the initialisation of ``sys.path[0]``, has resulted in
several traps for the unwary. These traps can be quite annoying in practice,
as they are highly unobvious (especially to beginners) and can cause quite
confusing behaviour.


Why are my imports broken?
--------------------------

There's a general principle that applies when modifying ``sys.path``: *never*
put a package directory directly on ``sys.path``. The reason this is
problematic is that every module in that directory is now potentially
accessible under two different names: as a top level module (since the
package directory is on ``sys.path``) and as a submodule of the package (if
the higher level directory containing the package itself is also on
``sys.path``).

As an example, Django (up to and including version 1.3) is guilty of setting
up exactly this situation for site-specific applications - the application
ends up being accessible as both ``app`` and ``site.app`` in the module
namespace, and these are actually two *different* copies of the module. This
is a recipe for confusion if there is any meaningful mutable module level
state, so this behaviour is being eliminated from the default site set up in
version 1.4 (site-specific apps will always be fully qualified with the site
name).

However, it's hard to blame Django for this, when the same part of Python
responsible for setting ``__name__ = "__main__"`` in the main module commits
the exact same error when determining the value for ``sys.path[0]``.

The impact of this can be seen relatively frequently if you follow the
"python" and "import" tags on Stack Overflow. When I had the time to follow
it myself, I regularly encountered people struggling to understand the
behaviour of straightforward package layouts like the following (I actually
use package layouts along these lines in my own projects)::

    project/
        setup.py
        example/
            __init__.py
            foo.py
            tests/
                __init__.py
                test_foo.py

While I would often see it without the ``__init__.py`` files first, that's a
trivial fix to explain. What's hard to explain is that all of the following
ways to invoke ``test_foo.py`` *probably won't work* due to broken imports
(either failing to find ``example`` for absolute imports, complaining
about relative imports in a non-package or beyond the toplevel package for
explicit relative imports, or issuing even more obscure errors if some other
submodule happens to shadow the name of a top-level module, such as an
``example.json`` module that handled serialisation or an
``example.tests.unittest`` test runner)::

    # These commands will most likely *FAIL*, even if the code is correct

    # working directory: project/example/tests
    ./test_foo.py
    python test_foo.py
    python -m package.tests.test_foo
    python -c "from package.tests.test_foo import main; main()"

    # working directory: project/package
    tests/test_foo.py
    python tests/test_foo.py
    python -m package.tests.test_foo
    python -c "from package.tests.test_foo import main; main()"

    # working directory: project
    example/tests/test_foo.py
    python example/tests/test_foo.py

    # working directory: project/..
    project/example/tests/test_foo.py
    python project/example/tests/test_foo.py
    # The -m and -c approaches don't work from here either, but the failure
    # to find 'package' correctly is easier to explain in this case

That's right, that long list is of all the methods of invocation that will
almost certainly *break* if you try them, and the error messages won't make
any sense if you're not already intimately familiar not only with the way
Python's import system works, but also with how it gets initialised.

For a long time, the only way to get ``sys.path`` right with that kind of
setup was to either set it manually in ``test_foo.py`` itself (hardly
something a novice, or even many veteran, Python programmers are going to
know how to do) or else to make sure to import the module instead of
executing it directly::

    # working directory: project
    python -c "from package.tests.test_foo import main; main()"

Since the implementation of :pep:`366` (which defined a mechanism that allows
relative imports to work correctly when a module inside a package is executed
via the ``-m`` switch), the following also works properly::

    # working directory: project
    python -m package.tests.test_foo

The fact that most methods of invoking Python code from the command line
break when that code is inside a package, and the two that do work are highly
sensitive to the current working directory is all thoroughly confusing for a
beginner. I personally believe it is one of the key factors leading
to the perception that Python packages are complicated and hard to get right.

This problem isn't even limited to the command line - if ``test_foo.py`` is
open in Idle and you attempt to run it by pressing F5, or if you try to run
it by clicking on it in a graphical filebrowser, then it will fail in just
the same way it would if run directly from the command line.

There's a reason the ge


