# embeddings

id: pep-0570.rst
content:
PEP: 570
Title: Python Positional-Only Parameters
Author: Larry Hastings <larry@hastings.org>,
        Pablo Galindo <pablogsal@python.org>,
        Mario Corchero <mariocj89@gmail.com>,
        Eric N. Vander Weele <ericvw@gmail.com>
BDFL-Delegate: Guido van Rossum <guido@python.org>
Discussions-To: https://discuss.python.org/t/pep-570-python-positional-only-parameters/1078
Status: Final
Type: Standards Track
Created: 20-Jan-2018
Python-Version: 3.8


========
Abstract
========

This PEP proposes to introduce a new syntax, ``/``, for specifying
positional-only parameters in Python function definitions.

Positional-only parameters have no externally-usable name. When a function
accepting positional-only parameters is called, positional arguments are mapped
to these parameters based solely on their order.

When designing APIs (application programming interfaces), library
authors try to ensure correct and intended usage of an API. Without the ability to
specify which parameters are positional-only, library authors must be careful
when choosing appropriate parameter names. This care must be taken
even for required parameters or when the parameters
have no external semantic meaning for callers of the API.

In this PEP, we discuss:

* Python's history and current semantics for positional-only parameters
* the problems encountered by not having them
* how these problems are handled without language-intrinsic support for
  positional-only parameters
* the benefits of having positional-only parameters

Within context of the motivation, we then:

* discuss why positional-only parameters should be a feature intrinsic to the
  language
* propose the syntax for marking positional-only parameters
* present how to teach this new feature
* note rejected ideas in further detail

==========
Motivation
==========

--------------------------------------------------------
History of Positional-Only Parameter Semantics in Python
--------------------------------------------------------

Python originally supported positional-only parameters. Early versions of the
language lacked the ability to call functions with arguments bound to parameters
by name. Around Python 1.0, parameter semantics changed to be
positional-or-keyword.  Since then, users have been able to provide arguments
to a function either positionally or by the keyword name specified in the
function's definition.

In current versions of Python, many CPython "builtin" and standard library
functions only accept positional-only parameters. The resulting semantics can be
easily observed by calling one of these functions using keyword arguments::

    >>> help(pow)
    ...
    pow(x, y, z=None, /)
    ...

    >>> pow(x=5, y=3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: pow() takes no keyword arguments

``pow()`` expresses that its parameters are positional-only via the
``/`` marker. However, this is only a documentation convention; Python
developers cannot use this syntax in code.

There are functions with other interesting semantics:

* ``range()``, an overloaded function, accepts an optional parameter to the
  *left* of its required parameter. [#RANGE]_

* ``dict()``, whose mapping/iterator parameter is optional and semantically
  must be positional-only. Any externally visible name for this parameter
  would occlude that name going into the ``**kwarg`` keyword variadic parameter
  dict. [#DICT]_

One can emulate these semantics in Python code by accepting
``(*args, **kwargs)`` and parsing the arguments manually. However, this results
in a disconnect between the function definition and what the function
contractually accepts. The function definition does not match the logic of the
argument handling.

Additionally, the ``/`` syntax is used beyond CPython for specifying similar
semantics (i.e., [#numpy-ufuncs]_ [#scipy-gammaln]_); thus, indicating that
these scenarios are not exclusive to CPython and the standard library.

-------------------------------------------
Problems Without Positional-Only Parameters
-------------------------------------------

Without positional-only parameters, there are challenges for library authors
and users of APIs. The following subsections outline the problems
encountered by each entity.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Challenges for Library Authors
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

With positional-or-keyword parameters, the mix of calling conventions is not
always desirable. Authors may want to restrict usage of an API by disallowing
calling the API with keyword arguments, which exposes the name of the parameter when
part of the public API. This approach is especially useful for required function
parameters that already have semantic meaning (e.g,
``namedtuple(typenames, field_names, â¦)`` or when the parameter name has no
true external meaning (e.g., ``arg1``, ``arg2``, â¦, etc for ``min()``). If a
caller of an API starts using a keyword argument, the library author cannot rename
the parameter because it would be a breaking change.

Positional-only parameters can be emulated by extracting arguments from
``*args`` one by one. However, this approach is error-prone and is not
synonymous with the function definition, as previously mentioned. The usage of
the function is ambiguous and forces users to look at ``help()``, the
associated auto-generated documentation, or source code to understand what
parameters the function contractually accepts.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Challenges for Users of an API
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Users may be surprised when first encountering positional-only notation. This
is expected given that it has only recently been documented
[#document-positional-only]_ and it is not possible to use in Python code. For
these reasons, this notation is currently an outlier that appears only in
CPython APIs developed in C. Documenting the notation and making it possible
to use it in Python code would eliminate this disconnect.

Furthermore, the current documentation for positional-only parameters is inconsistent:

* Some functions denote optional groups of positional-only parameters by
  enclosing them in nested square brackets. [#BORDER]_

* Some functions denote optional groups of positional-only parameters by
  presenting multiple prototypes with varying numbers of parameters.
  [#SENDFILE]_

* Some functions use *both* of the above approaches. [#RANGE]_ [#ADDCH]_

Another point the current documentation does not distinguish is
whether a function takes positional-only parameters. ``open()`` accepts keyword
arguments; however, ``ord()`` does not â there is no way of telling just by
reading the existing documentation.

--------------------------------------
Benefits of Positional-Only Parameters
--------------------------------------

Positional-only parameters give more control to library authors to better
express the intended usage of an API and allows the API to evolve in a safe,
backward-compatible way. Additionally, it makes the Python language more
consistent with existing documentation and the behavior of various
"builtin" and standard library functions.

^^^^^^^^^^^^^^^^^^^^^^^^^^
Empowering Library Authors
^^^^^^^^^^^^^^^^^^^^^^^^^^

Library authors would have the flexibility to change the name of
positional-only parameters without breaking callers. This flexibility reduces the
cognitive burden for choosing an appropriate public-facing name for required
parameters or parameters that have no true external semantic meaning.

Positional-only parameters are useful in several situations such as:

* when a function accepts any keyword argument but also can accept a positional one
* when a parameter has no external semantic meaning
* when an API's parameters are required and unambiguous

A key
scenario is when a function accepts any keyword argument but can also accepts a
positional one. Prominent examples are ``Formatter.format`` and
``dict.update``. For instance, ``dict.update`` accepts a dictio


id: pep-0564.rst
content:
PEP: 564
Title: Add new time functions with nanosecond resolution
Author: Victor Stinner <vstinner@python.org>
Status: Final
Type: Standards Track
Created: 16-Oct-2017
Python-Version: 3.7
Resolution: https://mail.python.org/pipermail/python-dev/2017-October/150046.html


Abstract
========

Add six new "nanosecond" variants of existing functions to the ``time``
module: ``clock_gettime_ns()``, ``clock_settime_ns()``,
``monotonic_ns()``, ``perf_counter_ns()``, ``process_time_ns()`` and
``time_ns()``.  While similar to the existing functions without the
``_ns`` suffix, they provide nanosecond resolution: they return a number of
nanoseconds as a Python ``int``.

The ``time.time_ns()`` resolution is 3 times better than the ``time.time()``
resolution on Linux and Windows.


Rationale
=========

Float type limited to 104 days
------------------------------

The clocks resolution of desktop and laptop computers is getting closer
to nanosecond resolution.  More and more clocks have a frequency in MHz,
up to GHz for the CPU TSC clock.

The Python ``time.time()`` function returns the current time as a
floating-point number which is usually a 64-bit binary floating-point
number (in the IEEE 754 format).

The problem is that the ``float`` type starts to lose nanoseconds after 104
days.  Converting from nanoseconds (``int``) to seconds (``float``) and
then back to nanoseconds (``int``) to check if conversions lose
precision::

    # no precision loss
    >>> x = 2 ** 52 + 1; int(float(x * 1e-9) * 1e9) - x
    0
    # precision loss! (1 nanosecond)
    >>> x = 2 ** 53 + 1; int(float(x * 1e-9) * 1e9) - x
    -1
    >>> print(datetime.timedelta(seconds=2 ** 53 / 1e9))
    104 days, 5:59:59.254741

``time.time()`` returns seconds elapsed since the UNIX epoch: January
1st, 1970.  This function hasn't had nanosecond precision since May 1970
(47 years ago)::

    >>> import datetime
    >>> unix_epoch = datetime.datetime(1970, 1, 1)
    >>> print(unix_epoch + datetime.timedelta(seconds=2**53 / 1e9))
    1970-04-15 05:59:59.254741


Previous rejected PEP
---------------------

Five years ago, the :pep:`410` proposed a large and complex change in all
Python functions returning time to support nanosecond resolution using
the ``decimal.Decimal`` type.

The PEP was rejected for different reasons:

* The idea of adding a new optional parameter to change the result type
  was rejected. It's an uncommon (and bad?) programming practice in
  Python.

* It was not clear if hardware clocks really had a resolution of 1
  nanosecond, or if that made sense at the Python level.

* The ``decimal.Decimal`` type is uncommon in Python and so requires
  to adapt code to handle it.


Issues caused by precision loss
-------------------------------

Example 1: measure time delta in long-running process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

A server is running for longer than 104 days.  A clock is read before and
after running a function to measure its performance to detect
performance issues at runtime.  Such benchmark only loses precision
because of the float type used by clocks, not because of the clock
resolution.

On Python microbenchmarks, it is common to see function calls taking
less than 100 ns.  A difference of a few nanoseconds might become
significant.

Example 2: compare times with different resolution
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Two programs "A" and "B" are running on the same system and use the system
clock.  The program A reads the system clock with nanosecond resolution
and writes a timestamp with nanosecond resolution.  The program B reads
the timestamp with nanosecond resolution, but compares it to the system
clock read with a worse resolution.  To simplify the example, let's say
that B reads the clock with second resolution.  If that case, there is a
window of 1 second while the program B can see the timestamp written by A
as "in the future".

Nowadays, more and more databases and filesystems support storing times
with nanosecond resolution.

.. note::
   This issue was already fixed for file modification time by adding the
   ``st_mtime_ns`` field to the ``os.stat()`` result, and by accepting
   nanoseconds in ``os.utime()``. This PEP proposes to generalize the
   fix.


CPython enhancements of the last 5 years
----------------------------------------

Since the :pep:`410` was rejected:

* The ``os.stat_result`` structure got 3 new fields for timestamps as
  nanoseconds (Python ``int``): ``st_atime_ns``, ``st_ctime_ns``
  and ``st_mtime_ns``.

* The :pep:`418` was accepted, Python 3.3 got 3 new clocks:
  ``time.monotonic()``, ``time.perf_counter()`` and
  ``time.process_time()``.

* The CPython private "pytime" C API handling time now uses a new
  ``_PyTime_t`` type: simple 64-bit signed integer (C ``int64_t``).
  The ``_PyTime_t`` unit is an implementation detail and not part of the
  API. The unit is currently ``1 nanosecond``.

Existing Python APIs using nanoseconds as int
---------------------------------------------

The ``os.stat_result`` structure has 3 fields for timestamps as
nanoseconds (``int``): ``st_atime_ns``, ``st_ctime_ns`` and
``st_mtime_ns``.

The ``ns`` parameter of the ``os.utime()`` function accepts a
``(atime_ns: int, mtime_ns: int)`` tuple: nanoseconds.


Changes
=======

New functions
-------------

This PEP adds six new functions to the ``time`` module:

* ``time.clock_gettime_ns(clock_id)``
* ``time.clock_settime_ns(clock_id, time: int)``
* ``time.monotonic_ns()``
* ``time.perf_counter_ns()``
* ``time.process_time_ns()``
* ``time.time_ns()``

These functions are similar to the version without the ``_ns`` suffix,
but return a number of nanoseconds as a Python ``int``.

For example, ``time.monotonic_ns() == int(time.monotonic() * 1e9)`` if
``monotonic()`` value is small enough to not lose precision.

These functions are needed because they may return "large" timestamps,
like ``time.time()`` which uses the UNIX epoch as reference, and so their
``float``-returning variants are likely to lose precision at the nanosecond
resolution.

Unchanged functions
-------------------

Since the ``time.clock()`` function was deprecated in Python 3.3, no
``time.clock_ns()`` is added.

Python has other time-returning functions.  No nanosecond variant is
proposed for these other functions, either because their internal
resolution is greater or equal to 1 us, or because their maximum value
is small enough to not lose precision.  For example, the maximum value of
``time.clock_getres()`` should be 1 second.

Examples of unchanged functions:

* ``os`` module: ``sched_rr_get_interval()``, ``times()``, ``wait3()``
  and ``wait4()``

* ``resource`` module: ``ru_utime`` and ``ru_stime`` fields of
  ``getrusage()``

* ``signal`` module: ``getitimer()``, ``setitimer()``

* ``time`` module: ``clock_getres()``

See also the `Annex: Clocks Resolution in Python`_.

A new nanosecond-returning flavor of these functions may be added later
if an operating system exposes new functions providing better resolution.


Alternatives and discussion
===========================

Sub-nanosecond resolution
-------------------------

``time.time_ns()`` API is not theoretically future-proof: if clock
resolutions continue to increase below the nanosecond level, new Python
functions may be needed.

In practice, the 1 nanosecond resolution is currently enough for all
structures returned by all common operating systems functions.

Hardware clocks with a resolution better than 1 nanosecond already
exist.  For example, the frequency of a CPU TSC clock is the CPU base
frequency: the resolution is around 0.3 ns for a CPU running at 3
GHz.  Users who have access to such hardware and really need
sub-nanosecond resolution can however extend Python for their needs.
Such a rare use case doesn't justify to design the Python standard library
to support sub-nanosecond resolution.

For the CPython implementation, nanosecond resolution is convenient: the
standard and well supp


id: pep-0202.rst
content:
PEP: 202
Title: List Comprehensions
Author: Barry Warsaw <barry@python.org>
Status: Final
Type: Standards Track
Created: 13-Jul-2000
Python-Version: 2.0
Post-History:


Introduction
============

This PEP describes a proposed syntactical extension to Python, list
comprehensions.


The Proposed Solution
=====================

It is proposed to allow conditional construction of list literals using for and
if clauses.  They would nest in the same way for loops and if statements nest
now.


Rationale
=========

List comprehensions provide a more concise way to create lists in situations
where ``map()`` and ``filter()`` and/or nested loops would currently be used.


Examples
========

::

    >>> print [i for i in range(10)]
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

    >>> print [i for i in range(20) if i%2 == 0]
    [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

    >>> nums = [1, 2, 3, 4]
    >>> fruit = ["Apples", "Peaches", "Pears", "Bananas"]
    >>> print [(i, f) for i in nums for f in fruit]
    [(1, 'Apples'), (1, 'Peaches'), (1, 'Pears'), (1, 'Bananas'),
     (2, 'Apples'), (2, 'Peaches'), (2, 'Pears'), (2, 'Bananas'),
     (3, 'Apples'), (3, 'Peaches'), (3, 'Pears'), (3, 'Bananas'),
     (4, 'Apples'), (4, 'Peaches'), (4, 'Pears'), (4, 'Bananas')]
    >>> print [(i, f) for i in nums for f in fruit if f[0] == "P"]
    [(1, 'Peaches'), (1, 'Pears'),
     (2, 'Peaches'), (2, 'Pears'),
     (3, 'Peaches'), (3, 'Pears'),
     (4, 'Peaches'), (4, 'Pears')]
    >>> print [(i, f) for i in nums for f in fruit if f[0] == "P" if i%2 == 1]
    [(1, 'Peaches'), (1, 'Pears'), (3, 'Peaches'), (3, 'Pears')]
    >>> print [i for i in zip(nums, fruit) if i[0]%2==0]
    [(2, 'Peaches'), (4, 'Bananas')]


Reference Implementation
========================

List comprehensions become part of the Python language with release 2.0,
documented in [1]_.


BDFL Pronouncements
===================
* The syntax proposed above is the Right One.

* The form ``[x, y for ...]`` is disallowed; one is required to write
  ``[(x, y) for ...]``.

* The form ``[... for x... for y...]`` nests, with the last index
  varying fastest, just like nested for loops.


References
==========

.. [1] http://docs.python.org/reference/expressions.html#list-displays


id: pep-8104.rst
content:
PEP: 8104
Title: 2023 Term Steering Council election
Author: Ee Durbin <ee@python.org>
Sponsor: Brett Cannon <brett@python.org>
Status: Final
Type: Informational
Topic: Governance
Created: 08-Nov-2022


Abstract
========

This document describes the schedule and other details of the December
2022 election for the Python steering council, as specified in
:pep:`13`. This is the steering council election for the 2023 term
(i.e. Python 3.12).


Election Administration
=======================

The steering council appointed the
`Python Software Foundation <https://www.python.org/psf-landing/>`__
Director of Infrastructure, Ee Durbin, to administer the election.


Schedule
========

There will be a two-week nomination period, followed by a two-week
vote.

The nomination period was: November 14, 2022 through `November 28, 2022 AoE
<https://www.timeanddate.com/worldclock/fixedtime.html?msg=Python+Steering+Council+nominations+close&iso=20221129T00&p1=3399>`_ [#note-aoe]_.

The voting period was: December 1, 2022 through `December 14, 2022 AoE
<https://www.timeanddate.com/worldclock/fixedtime.html?msg=Python+Steering+Council+voting+closes&iso=20221215T00&p1=3399>`_ [#note-aoe]_.


Candidates
==========

Candidates must be nominated by a core team member. If the candidate
is a core team member, they may nominate themselves.

Nominees (in alphabetical order):

- `Brett Cannon <https://discuss.python.org/t/steering-council-nomination-brett-cannon-2023-term/21078>`_
- `Emily Morehouse <https://discuss.python.org/t/steering-council-nomination-emily-morehouse-2023-term/21329>`_
- `Dong-hee Na <https://discuss.python.org/t/steering-council-nomination-dong-hee-na-2023-term/21424/1>`_
- `Pablo Galindo Salgado <https://discuss.python.org/t/steering-council-nomination-pablo-galindo-salgado-2023-term/21307>`_
- `Gregory P. Smith <https://discuss.python.org/t/steering-council-nomination-gregory-p-smith-2023-term/21332/1>`_
- `Victor Stinner <https://discuss.python.org/t/steering-council-nomination-victor-stinner-2023-term/21407/1>`_
- `Petr Viktorin <https://discuss.python.org/t/steering-council-nomination-petr-viktorin-2023-term/21478>`_
- `Thomas Wouters <https://discuss.python.org/t/steering-council-nomination-thomas-wouters-2023-term/21147>`_

Withdrawn nominations:

- None

Voter Roll
==========

All active Python core team members are eligible to vote. Active status
is determined as :pep:`described in PEP 13 <13#membership>`
and implemented via the software at `python/voters <https://github.com/python/voters>`_
[#note-voters]_.

Ballots will be distributed based on the the `Python Voter Roll
<https://github.com/python/voters/blob/master/voter-files/>`_ [#note-voters]_
for this election.

While this file is not public as it contains private email addresses, the
`Complete Voter Roll`_ by name will be made available when the roll is
created.

Election Implementation
=======================

The election will be conducted using the `Helios Voting Service
<https://heliosvoting.org>`__.


Configuration
-------------

Short name: ``2023-python-steering-council``

Name: ``2023 Python Steering Council Election``

Description: ``Election for the Python steering council, as specified in PEP 13. This is steering council election for the 2023 term.``

type: ``Election``

Use voter aliases: ``[X]``

Randomize answer order: ``[X]``

Private: ``[X]``

Help Email Address: ``psf-election@python.org``

Voting starts at: ``December 1, 2022 12:00 UTC``

Voting ends at: ``December 15, 2022 12:00 UTC``

This will create an election in which:

* Voting is not open to the public, only those on the `Voter Roll`_ may
  participate. Ballots will be emailed when voting starts.
* Candidates are presented in random order, to help avoid bias.
* Voter identities and ballots are protected against cryptographic advances.

Questions
---------

Question 1
~~~~~~~~~~

Select between ``0`` and ``- (approval)`` answers. Result Type: ``absolute``

Question: ``Select candidates for the Python Steering Council``

Answer #1 - #N: ``Candidates from Candidates_ Section``



Results
=======

Of 85 eligible voters, 66 cast ballots.

The top five vote-getters are:

* Pablo Galindo Salgado
* Gregory P. Smith
* Emily Morehouse
* Brett Cannon
* Thomas Wouters

No conflict of interest as defined in :pep:`13` were observed.

The full vote counts are as follows:

+-----------------------+----------------+
| Candidate             | Votes Received |
+=======================+================+
| Pablo Galindo Salgado | 61             |
+-----------------------+----------------+
| Gregory P. Smith      | 48             |
+-----------------------+----------------+
| Emily Morehouse       | 47             |
+-----------------------+----------------+
| Brett Cannon          | 42             |
+-----------------------+----------------+
| Thomas Wouters        | 39             |
+-----------------------+----------------+
| Petr Viktorin         | 36             |
+-----------------------+----------------+
| Victor Stinner        | 34             |
+-----------------------+----------------+
| Dong-hee Na           | 29             |
+-----------------------+----------------+

Copyright
=========

This document has been placed in the public domain.


Complete Voter Roll
===================

Active Python core developers
-----------------------------

.. code-block:: text

    Alex Gaynor
    Alex Waygood
    Ammar Askar
    Andrew Svetlov
    Antoine Pitrou
    Barry Warsaw
    Batuhan Taskaya
    Benjamin Peterson
    Berker PeksaÄ
    Brandt Bucher
    Brett Cannon
    Brian Curtin
    Brian Quinlan
    Carol Willing
    Cheryl Sabella
    Chris Jerdonek
    Chris Withers
    Christian Heimes
    Dennis Sweeney
    Dino Viehland
    Dong-hee Na
    Emily Morehouse
    Ãric Araujo
    Eric Snow
    Eric V. Smith
    Erlend Egeberg Aasland
    Ethan Furman
    Ezio Melotti
    Facundo Batista
    Filipe LaÃ­ns
    Fred Drake
    Georg Brandl
    Giampaolo RodolÃ 
    Gregory P. Smith
    Guido van Rossum
    Hugo van Kemenade
    Hynek Schlawack
    Inada Naoki
    Irit Katriel
    Ivan Levkivskyi
    Jason R. Coombs
    Jelle Zijlstra
    Jeremy Kloth
    JesÃºs Cea
    Joannah Nanjekye
    Julien Palard
    Karthikeyan Singaravelan
    Ken Jin
    Kumar Aditya
    Kurt B. Kaiser
    Kushal Das
    Kyle Stanley
    Larry Hastings
    Åukasz Langa
    Lysandros Nikolaou
    Marc-AndrÃ© Lemburg
    Mariatta
    Mark Dickinson
    Mark Shannon
    Nathaniel J. Smith
    Ned Deily
    Neil Schemenauer
    Alyssa Coghlan
    Pablo Galindo
    Paul Ganssle
    Paul Moore
    Petr Viktorin
    R. David Murray
    Raymond Hettinger
    Ronald Oussoren
    Senthil Kumaran
    Serhiy Storchaka
    Stefan Behnel
    StÃ©phane Wirtel
    Steve Dower
    Steven D'Aprano
    Tal Einat
    Terry Jan Reedy
    Thomas Wouters
    Tim Golden
    Tim Peters
    Victor Stinner
    Vinay Sajip
    Yury Selivanov
    Zachary Ware


.. [#note-voters] This repository is private and accessible only to Python Core
   Developers, administrators, and Python Software Foundation Staff as it
   contains personal email addresses.
.. [#note-aoe] AoE: `Anywhere on Earth <https://www.ieee802.org/16/aoe.html>`_.


id: pep-0558.rst
content:
PEP: 558
Title: Defined semantics for locals()
Author: Alyssa Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: Nathaniel J. Smith
Discussions-To: python-dev@python.org
Status: Withdrawn
Type: Standards Track
Created: 08-Sep-2017
Python-Version: 3.13
Post-History: 08-Sep-2017, 22-May-2019, 30-May-2019, 30-Dec-2019, 18-Jul-2021,
              26-Aug-2021

PEP Withdrawal
==============

In December 2021, this PEP and :pep:`667` converged on a common definition of the
proposed changes to the Python level semantics of the ``locals()`` builtin (as
documented in the PEP text below), with the only remaining differences being
in the proposed C API changes and various internal implementation details.

Of those remaining differences, the most significant one was that :pep:`667`
at the time still proposed an immediate backwards compatibility break for the
``PyEval_GetLocals()`` API as soon as the PEP was accepted and implemented.

:pep:`667` has since been changed to propose a generous deprecation period for
the ``PyEval_GetLocals()`` API, continuing to support it in parallel with the
improved semantics offered by the new ``PyEval_GetFrameLocals()`` API.

Any remaining C API design concerns relate to new informational APIs that can be
added at a later date if they are deemed necessary, and any potential concerns
about the exact performance characteristics of the frame locals view implementation
are outweighed by the availability of a viable reference implementation.

Accordingly, this PEP has been withdrawn in favour of proceeding with :pep:`667`.

Note: while implementing :pep:`667` it became apparent that the rationale for and impact
of ``locals()`` being updated to return independent snapshots in
:term:`optimized scopes <py3.13:optimized scope>` was not entirely clear in either PEP.
The Motivation and Rationale sections in this PEP have been updated accordingly (since those
aspects are equally applicable to the accepted :pep:`667`).

Abstract
========

The semantics of the ``locals()`` builtin have historically been underspecified
and hence implementation dependent.

This PEP proposes formally standardising on the behaviour of the CPython 3.10
reference implementation for most execution scopes, with some adjustments to the
behaviour at function scope to make it more predictable and independent of the
presence or absence of tracing functions.

In addition, it proposes that the following functions be added to the stable
Python C API/ABI:

.. code-block:: c

    typedef enum {
      PyLocals_UNDEFINED = -1,
      PyLocals_DIRECT_REFERENCE = 0,
      PyLocals_SHALLOW_COPY = 1,
      _PyLocals_ENSURE_32BIT_ENUM = 2147483647
    } PyLocals_Kind;

    PyLocals_Kind PyLocals_GetKind();
    PyObject * PyLocals_Get();
    PyObject * PyLocals_GetCopy();

It also proposes the addition of several supporting functions and type
definitions to the CPython C API.

.. _pep-558-motivation:

Motivation
==========

While the precise semantics of the ``locals()`` builtin are nominally undefined,
in practice, many Python programs depend on it behaving exactly as it behaves in
CPython (at least when no tracing functions are installed).

Other implementations such as PyPy are currently replicating that behaviour,
up to and including replication of local variable mutation bugs that
can arise when a trace hook is installed [1]_.

While this PEP considers CPython's current behaviour when no trace hooks are
installed to be largely acceptable, it considers the current
behaviour when trace hooks are installed to be problematic, as it causes bugs
like [1]_ *without* even reliably enabling the desired functionality of allowing
debuggers like ``pdb`` to mutate local variables [3]_.

Review of the initial PEP and the draft implementation then identified an
opportunity for simplification of both the documentation and implementation
of the function level ``locals()`` behaviour by updating it to return an
independent snapshot of the function locals and closure variables on each
call, rather than continuing to return the semi-dynamic intermittently updated
shared copy that it has historically returned in CPython.

Specifically, the proposal in this PEP eliminates the historical behaviour where
adding a new local variable can change the behaviour of code executed with
``exec()`` in function scopes, even if that code runs *before* the local variable
is defined.

For example::

    def f():
        exec("x = 1")
        print(locals().get("x"))
    f()

prints ``1``, but::

    def f():
        exec("x = 1")
        print(locals().get("x"))
        x = 0
    f()

prints ``None`` (the default value from the ``.get()`` call).

With this PEP both examples would print ``None``, as the call to
``exec()`` and the subsequent call to ``locals()`` would use
independent dictionary snapshots of the local variables rather
than using the same shared dictionary cached on the frame object.

Proposal
========

The expected semantics of the ``locals()`` builtin change based on the current
execution scope. For this purpose, the defined scopes of execution are:

* module scope: top-level module code, as well as any other code executed using
  ``exec()`` or ``eval()`` with a single namespace
* class scope: code in the body of a ``class`` statement, as well as any other
  code executed using ``exec()`` or ``eval()`` with separate local and global
  namespaces
* function scope: code in the body of a ``def`` or ``async def`` statement,
  or any other construct that creates an optimized code block in CPython (e.g.
  comprehensions, lambda functions)

This PEP proposes elevating most of the current behaviour of the CPython
reference implementation to become part of the language specification, *except*
that each call to ``locals()`` at function scope will create a new dictionary
object, rather than caching a common dict instance in the frame object that
each invocation will update and return.

This PEP also proposes to largely eliminate the concept of a separate "tracing"
mode from the CPython reference implementation. In releases up to and including
Python 3.10, the CPython interpreter behaves differently when a trace hook has
been registered in one or more threads via an implementation dependent mechanism
like ``sys.settrace`` ([4]_) in CPython's ``sys`` module or
``PyEval_SetTrace`` ([5]_) in CPython's C API. If this PEP is accepted, then
the only remaining behavioural difference when a trace hook is installed is that
some optimisations in the interpreter eval loop are disabled when the tracing
logic needs to run after each opcode.

This PEP proposes changes to CPython's behaviour at function scope that make
the ``locals()`` builtin semantics when a trace hook is registered identical to
those used when no trace hook is registered, while also making the related frame
API semantics clearer and easier for interactive debuggers to rely on.

The proposed elimination of tracing mode affects the semantics of frame object
references obtained through other means, such as via a traceback, or via the
``sys._getframe()`` API, as the write-through semantics needed for trace hook
support are always provided by the ``f_locals`` attribute on frame objects,
rather than being runtime state dependent.


New ``locals()`` documentation
------------------------------

The heart of this proposal is to revise the documentation for the ``locals()``
builtin to read as follows:

    Return a mapping object representing the current local symbol table, with
    variable names as the keys, and their currently bound references as the
    values.

    At module scope, as well as when using ``exec()`` or ``eval()`` with a
    single namespace, this function returns the same namespace as ``globals()``.

    At class scope, it returns the namespace that will be passed to the
    metaclass constructor.

    When using ``exec()`` or ``eval()`` with separate local and global
    namespaces, it returns the local namespace passed in to the function call.

    In al


id: pep-0228.rst
content:
PEP: 228
Title: Reworking Python's Numeric Model
Author: Moshe Zadka <moshez@zadka.site.co.il>, Guido van Rossum <guido@python.org>
Status: Withdrawn
Type: Standards Track
Created: 04-Nov-2000
Post-History:


Withdrawal
==========

This PEP has been withdrawn in favor of :pep:`3141`.


Abstract
========

Today, Python's numerical model is similar to the C numeric model:
there are several unrelated numerical types, and when operations
between numerical types are requested, coercions happen.  While
the C rationale for the numerical model is that it is very similar
to what happens at the hardware level, that rationale does not
apply to Python.  So, while it is acceptable to C programmers that
``2/3 == 0``, it is surprising to many Python programmers.

NOTE: in the light of recent discussions in the newsgroup, the
motivation in this PEP (and details) need to be extended.


Rationale
=========

In usability studies, one of the least usable aspect of Python was
the fact that integer division returns the floor of the division.
This makes it hard to program correctly, requiring casts to
``float()`` in various parts through the code.  Python's numerical
model stems from C, while a model that might be easier to work with
can be based on the mathematical understanding of numbers.


Other Numerical Models
======================

Perl's numerical model is that there is one type of numbers --
floating point numbers.  While it is consistent and superficially
non-surprising, it tends to have subtle gotchas.  One of these is
that printing numbers is very tricky, and requires correct
rounding.  In Perl, there is also a mode where all numbers are
integers.  This mode also has its share of problems, which arise
from the fact that there is not even an approximate way of
dividing numbers and getting meaningful answers.


Suggested Interface For Python's Numerical Model
================================================

While coercion rules will remain for add-on types and classes, the
built in type system will have exactly one Python type -- a
number.  There are several things which can be considered "number
methods":

1. ``isnatural()``
2. ``isintegral()``
3. ``isrational()``
4. ``isreal()``
5. ``iscomplex()``
6. ``isexact()``

Obviously, a number which answers true to a question from 1 to 5, will
also answer true to any following question. If ``isexact()`` is not true,
then any answer might be wrong.
(But not horribly wrong: it's close to the truth.)

Now, there is two thing the models promises for the field operations
(``+``, ``-``, ``/``, ``*``):

- If both operands satisfy ``isexact()``, the result satisfies
  ``isexact()``.

- All field rules are true, except that for not-``isexact()`` numbers,
  they might be only approximately true.

One consequence of these two rules is that all exact calculations
are done as (complex) rationals: since the field laws must hold,
then ::

    (a/b)*b == a

must hold.

There is built-in function, ``inexact()`` which takes a number
and returns an inexact number which is a good approximation.
Inexact numbers must be as least as accurate as if they were
using IEEE-754.

Several of the classical Python functions will return exact numbers
even when given inexact numbers: e.g, ``int()``.


Coercion
========

The number type does not define ``nb_coerce``
Any numeric operation slot, when receiving something other then ``PyNumber``,
refuses to implement it.


Inexact Operations
==================

The functions in the ``math`` module will be allowed to return
inexact results for exact values.  However, they will never return
a non-real number.  The functions in the ``cmath`` module are also
allowed to return an inexact result for an exact argument, and are
furthermore allowed to return a complex result for a real
argument.


Numerical Python Issues
=======================

People who use Numerical Python do so for high-performance vector
operations.  Therefore, NumPy should keep its hardware based
numeric model.


Unresolved Issues
=================

Which number literals will be exact, and which inexact?

How do we deal with IEEE 754 operations? (probably, isnan/isinf should
be methods)

On 64-bit machines, comparisons between ints and floats may be
broken when the comparison involves conversion to float.  Ditto
for comparisons between longs and floats.  This can be dealt with
by avoiding the conversion to float.  (Due to Andrew Koenig.)


Copyright
=========

This document has been placed in the public domain.


id: pep-8106.rst
content:
PEP: 8106
Title: 2025 Term Steering Council election
Author: Ee Durbin <ee@python.org>
Sponsor: Thomas Wouters <thomas@python.org>
Status: Final
Type: Informational
Topic: Governance
Created: 21-Oct-2024


Abstract
========

This document describes the schedule and other details of the
2024 election for the Python steering council, as specified in
:pep:`13`. This is the steering council election for the 2025 term
(i.e. Python 3.14).


Election Administration
=======================

The steering council appointed the
`Python Software Foundation <https://www.python.org/psf-landing/>`__
Director of Infrastructure, Ee Durbin, to administer the election.


Schedule
========

There was a two-week nomination period, followed by a two-week
vote.

The nomination period was: November 7, 2024 through `November 21, 2024 AoE
<https://www.timeanddate.com/worldclock/fixedtime.html?msg=Python+Steering+Council+nominations+close&iso=20241122T00&p1=3399>`_ [#note-aoe]_.

The voting period was: November 25, 2024 through `December 9, 2024 AoE
<https://www.timeanddate.com/worldclock/fixedtime.html?msg=Python+Steering+Council+voting+closes&iso=20241210T00&p1=3399>`_ [#note-aoe]_.


Candidates
==========

Candidates must be nominated by a core team member. If the candidate
is a core team member, they may nominate themselves.

Nominees (in alphabetical order by first name):

- `Barry Warsaw <https://discuss.python.org/t/steering-council-nominations-barry-warsaw-2025-term/71732/>`_
- `Donghee Na <https://discuss.python.org/t/steering-council-nomination-donghee-na-2025-term/71731>`_
- `Emily Morehouse <https://discuss.python.org/t/steering-council-nomination-emily-morehouse-2025-term/71824>`_
- `Ethan Furman <https://discuss.python.org/t/steering-council-nomination-ethan-furman-2025-term/71899>`_
- `Gregory P. Smith <https://discuss.python.org/t/steering-council-nomination-gregory-p-smith-2025-term/71947>`_
- `Mariatta <https://discuss.python.org/t/steering-council-nomination-mariatta-2025-term/71950>`_
- `Pablo Galindo Salgado <https://discuss.python.org/t/steering-council-nomination-pablo-galindo-salgado-2025-term/71915>`_
- `Thomas Wouters <https://discuss.python.org/t/steering-council-nomination-thomas-wouters-2025-term/71792>`_

Withdrawn nominations:

- None

Voter Roll
==========

All active Python core team members are eligible to vote. Active status
is determined as :pep:`described in PEP 13 <13#membership>`
and implemented via the software at `python/voters <https://github.com/python/voters>`_
[#note-voters]_.

Ballots were distributed based on the the `Python Voter Roll
<https://github.com/python/voters/blob/master/voter-files/>`_ [#note-voters]_
for this election.

While this file is not public as it contains private email addresses, the
`Complete Voter Roll`_ is available with a list of all eligible voters by name.

Election Implementation
=======================

The election was conducted using the `Helios Voting Service
<https://heliosvoting.org>`__.


Configuration
-------------

Short name: ``2025-python-steering-council``

Name: ``2025 Python Steering Council Election``

Description: ``Election for the Python steering council, as specified in PEP 13. This is steering council election for the 2025 term.``

type: ``Election``

Use voter aliases: ``[X]``

Randomize answer order: ``[X]``

Private: ``[X]``

Help Email Address: ``psf-election@python.org``

Voting starts at: ``November 25, 2024 12:00 UTC``

Voting ends at: ``December 10, 2024 12:00 UTC``

This will create an election in which:

* Voting is not open to the public, only those on the `Voter Roll`_ may
  participate. Ballots will be emailed when voting starts.
* Candidates are presented in random order, to help avoid bias.
* Voter identities and ballots are protected against cryptographic advances.

Questions
---------

Question 1
~~~~~~~~~~

Select between ``0`` and ``- (approval)`` answers. Result Type: ``absolute``

Question: ``Select candidates for the Python Steering Council``

Answer #1 - #N: ``Candidates from Candidates_ Section``



Results
=======

Of 100 eligible voters, 76 cast ballots.

The top five vote-getters are:

* Barry Warsaw
* Donghee Na
* Emily Morehouse
* Gregory P. Smith
* Pablo Galindo Salgado

No conflict of interest as defined in :pep:`13` were observed.

The full vote counts are as follows:

+-----------------------+----------------+
| Candidate             | Votes Received |
+=======================+================+
| Barry Warsaw          | 58             |
+-----------------------+----------------+
| Donghee Na            | 48             |
+-----------------------+----------------+
| Emily Morehouse       | 52             |
+-----------------------+----------------+
| Ethan Furman          | 31             |
+-----------------------+----------------+
| Gregory P. Smith      | 50             |
+-----------------------+----------------+
| Mariatta              | 23             |
+-----------------------+----------------+
| Pablo Galindo Salgado | 63             |
+-----------------------+----------------+
| Thomas Wouters        | 38             |
+-----------------------+----------------+


Copyright
=========

This document has been placed in the public domain.


Complete Voter Roll
===================

Active Python core developers
-----------------------------

.. code-block:: text

   Adam Turner
   Alex Gaynor
   Alex Waygood
   Alexander Belopolsky
   Alyssa Coghlan
   Ammar Askar
   Andrew Svetlov
   Antoine Pitrou
   Barney Gale
   Barry Warsaw
   Batuhan Taskaya
   Benjamin Peterson
   Berker PeksaÄ
   Brandt Bucher
   Brett Cannon
   Brian Curtin
   C.A.M. Gerlach
   CF Bolz-Tereick
   Carl Meyer
   Carol Willing
   Cheryl Sabella
   Chris Withers
   Christian Heimes
   Dennis Sweeney
   Dino Viehland
   Donghee Na
   Emily Morehouse
   Ãric Araujo
   Eric Snow
   Eric V. Smith
   Erlend Egeberg Aasland
   Ethan Furman
   Ezio Melotti
   Facundo Batista
   Filipe LaÃ­ns
   Fred Drake
   Georg Brandl
   Giampaolo RodolÃ 
   Gregory P. Smith
   Guido van Rossum
   Hugo van Kemenade
   Hynek Schlawack
   Inada Naoki
   Irit Katriel
   Ivan Levkivskyi
   Jack Jansen
   Jason R. Coombs
   Jelle Zijlstra
   Jeremy Hylton
   Jeremy Kloth
   JesÃºs Cea
   Joannah Nanjekye
   Julien Palard
   Karthikeyan Singaravelan
   Ken Jin
   Kirill Podoprigora
   Kumar Aditya
   Kurt B. Kaiser
   Kushal Das
   Kyle Stanley
   Larry Hastings
   Åukasz Langa
   Lysandros Nikolaou
   Marc-AndrÃ© Lemburg
   Mariatta
   Mark Shannon
   Matt Page
   Michael Droettboom
   Nathaniel J. Smith
   Ned Batchelder
   Ned Deily
   Neil Schemenauer
   Nikita Sobolev
   Pablo Galindo
   Paul Ganssle
   Paul Moore
   Petr Viktorin
   Pradyun Gedam
   R. David Murray
   Raymond Hettinger
   Ronald Oussoren
   Russell Keith-Magee
   Sam Gross
   Savannah Ostrowski
   Senthil Kumaran
   Serhiy Storchaka
   Shantanu Jain
   Stefan Behnel
   Steve Dower
   StÃ©phane Wirtel
   Tal Einat
   Terry Jan Reedy
   Thomas Wouters
   Tian Gao
   Tim Golden
   Tim Peters
   Victor Stinner
   Vinay Sajip
   Yury Selivanov
   Zachary Ware


.. [#note-voters] This repository is private and accessible only to Python Core
   Developers, administrators, and Python Software Foundation Staff as it
   contains personal email addresses.
.. [#note-aoe] AoE: `Anywhere on Earth <https://www.ieee802.org/16/aoe.html>`_.


id: pep-0572.rst
content:
PEP: 572
Title: Assignment Expressions
Author: Chris Angelico <rosuav@gmail.com>, Tim Peters <tim.peters@gmail.com>,
        Guido van Rossum <guido@python.org>
Status: Final
Type: Standards Track
Created: 28-Feb-2018
Python-Version: 3.8
Post-History: 28-Feb-2018, 02-Mar-2018, 23-Mar-2018, 04-Apr-2018, 17-Apr-2018,
              25-Apr-2018, 09-Jul-2018, 05-Aug-2019
Resolution: https://mail.python.org/pipermail/python-dev/2018-July/154601.html


Abstract
========

This is a proposal for creating a way to assign to variables within an
expression using the notation ``NAME := expr``.

As part of this change, there is also an update to dictionary comprehension
evaluation order to ensure key expressions are executed before value
expressions (allowing the key to be bound to a name and then re-used as part of
calculating the corresponding value).

During discussion of this PEP, the operator became informally known as
"the walrus operator". The construct's formal name is "Assignment Expressions"
(as per the PEP title), but they may also be referred to as "Named Expressions"
(e.g. the CPython reference implementation uses that name internally).


Rationale
=========

Naming the result of an expression is an important part of programming,
allowing a descriptive name to be used in place of a longer expression,
and permitting reuse.  Currently, this feature is available only in
statement form, making it unavailable in list comprehensions and other
expression contexts.

Additionally, naming sub-parts of a large expression can assist an interactive
debugger, providing useful display hooks and partial results. Without a way to
capture sub-expressions inline, this would require refactoring of the original
code; with assignment expressions, this merely requires the insertion of a few
``name :=`` markers. Removing the need to refactor reduces the likelihood that
the code be inadvertently changed as part of debugging (a common cause of
Heisenbugs), and is easier to dictate to another programmer.

The importance of real code
---------------------------

During the development of this PEP many people (supporters and critics
both) have had a tendency to focus on toy examples on the one hand,
and on overly complex examples on the other.

The danger of toy examples is twofold: they are often too abstract to
make anyone go "ooh, that's compelling", and they are easily refuted
with "I would never write it that way anyway".

The danger of overly complex examples is that they provide a
convenient strawman for critics of the proposal to shoot down ("that's
obfuscated").

Yet there is some use for both extremely simple and extremely complex
examples: they are helpful to clarify the intended semantics.
Therefore, there will be some of each below.

However, in order to be *compelling*, examples should be rooted in
real code, i.e. code that was written without any thought of this PEP,
as part of a useful application, however large or small.  Tim Peters
has been extremely helpful by going over his own personal code
repository and picking examples of code he had written that (in his
view) would have been *clearer* if rewritten with (sparing) use of
assignment expressions.  His conclusion: the current proposal would
have allowed a modest but clear improvement in quite a few bits of
code.

Another use of real code is to observe indirectly how much value
programmers place on compactness.  Guido van Rossum searched through a
Dropbox code base and discovered some evidence that programmers value
writing fewer lines over shorter lines.

Case in point: Guido found several examples where a programmer
repeated a subexpression, slowing down the program, in order to save
one line of code, e.g. instead of writing::

    match = re.match(data)
    group = match.group(1) if match else None

they would write::

    group = re.match(data).group(1) if re.match(data) else None

Another example illustrates that programmers sometimes do more work to
save an extra level of indentation::

    match1 = pattern1.match(data)
    match2 = pattern2.match(data)
    if match1:
        result = match1.group(1)
    elif match2:
        result = match2.group(2)
    else:
        result = None

This code tries to match ``pattern2`` even if ``pattern1`` has a match
(in which case the match on ``pattern2`` is never used).  The more
efficient rewrite would have been::

    match1 = pattern1.match(data)
    if match1:
        result = match1.group(1)
    else:
        match2 = pattern2.match(data)
        if match2:
            result = match2.group(2)
        else:
            result = None


Syntax and semantics
====================

In most contexts where arbitrary Python expressions can be used, a
**named expression** can appear.  This is of the form ``NAME := expr``
where ``expr`` is any valid Python expression other than an
unparenthesized tuple, and ``NAME`` is an identifier.

The value of such a named expression is the same as the incorporated
expression, with the additional side-effect that the target is assigned
that value::

    # Handle a matched regex
    if (match := pattern.search(data)) is not None:
        # Do something with match

    # A loop that can't be trivially rewritten using 2-arg iter()
    while chunk := file.read(8192):
       process(chunk)

    # Reuse a value that's expensive to compute
    [y := f(x), y**2, y**3]

    # Share a subexpression between a comprehension filter clause and its output
    filtered_data = [y for x in data if (y := f(x)) is not None]

Exceptional cases
-----------------

There are a few places where assignment expressions are not allowed,
in order to avoid ambiguities or user confusion:

- Unparenthesized assignment expressions are prohibited at the top
  level of an expression statement.  Example::

    y := f(x)  # INVALID
    (y := f(x))  # Valid, though not recommended

  This rule is included to simplify the choice for the user between an
  assignment statement and an assignment expression -- there is no
  syntactic position where both are valid.

- Unparenthesized assignment expressions are prohibited at the top
  level of the right hand side of an assignment statement.  Example::

    y0 = y1 := f(x)  # INVALID
    y0 = (y1 := f(x))  # Valid, though discouraged

  Again, this rule is included to avoid two visually similar ways of
  saying the same thing.

- Unparenthesized assignment expressions are prohibited for the value
  of a keyword argument in a call.  Example::

    foo(x = y := f(x))  # INVALID
    foo(x=(y := f(x)))  # Valid, though probably confusing

  This rule is included to disallow excessively confusing code, and
  because parsing keyword arguments is complex enough already.

- Unparenthesized assignment expressions are prohibited at the top
  level of a function default value.  Example::

    def foo(answer = p := 42):  # INVALID
        ...
    def foo(answer=(p := 42)):  # Valid, though not great style
        ...

  This rule is included to discourage side effects in a position whose
  exact semantics are already confusing to many users (cf. the common
  style recommendation against mutable default values), and also to
  echo the similar prohibition in calls (the previous bullet).

- Unparenthesized assignment expressions are prohibited as annotations
  for arguments, return values and assignments.  Example::

    def foo(answer: p := 42 = 5):  # INVALID
        ...
    def foo(answer: (p := 42) = 5):  # Valid, but probably never useful
        ...

  The reasoning here is similar to the two previous cases; this
  ungrouped assortment of symbols and operators composed of ``:`` and
  ``=`` is hard to read correctly.

- Unparenthesized assignment expressions are prohibited in lambda functions.
  Example::

    (lambda: x := 1) # INVALID
    lambda: (x := 1) # Valid, but unlikely to be useful
    (x := lambda: 1) # Valid
    lambda line: (m := re.match(pattern, line)) and m.group(1) # Valid

  This allows ``lambda`` to always bind less tig


id: pep-0214.rst
content:
PEP: 214
Title: Extended Print Statement
Author: Barry Warsaw <barry@python.org>
Status: Final
Type: Standards Track
Created: 24-Jul-2000
Python-Version: 2.0
Post-History: 16-Aug-2000


Introduction
============

This PEP describes a syntax to extend the standard 'print'
statement so that it can be used to print to any file-like object,
instead of the default ``sys.stdout``.  This PEP tracks the status and
ownership of this feature.  It contains a description of the
feature and outlines changes necessary to support the feature.
This PEP summarizes discussions held in mailing list forums, and
provides URLs for further information, where appropriate.  The CVS
revision history of this file contains the definitive historical
record.


Proposal
========

This proposal introduces a syntax extension to the print
statement, which allows the programmer to optionally specify the
output file target.  An example usage is as follows::

    print >> mylogfile, 'this message goes to my log file'

Formally, the syntax of the extended print statement is::

    print_stmt: ... | '>>' test [ (',' test)+ [','] ] )

where the ellipsis indicates the original print_stmt syntax
unchanged.  In the extended form, the expression just after >>
must yield an object with a ``write()`` method (i.e. a file-like
object).  Thus these two statements are equivalent::

    print 'hello world'
    print >> sys.stdout, 'hello world'

As are these two statements::

    print
    print >> sys.stdout

These two statements are syntax errors::

    print ,
    print >> sys.stdout,


Justification
=============

'print' is a Python keyword and introduces the print statement as
described in section 6.6 of the language reference manual [1]_.
The print statement has a number of features:

- it auto-converts the items to strings
- it inserts spaces between items automatically
- it appends a newline unless the statement ends in a comma

The formatting that the print statement performs is limited; for
more control over the output, a combination of ``sys.stdout.write()``,
and string interpolation can be used.

The print statement by definition outputs to ``sys.stdout``.  More
specifically, ``sys.stdout`` must be a file-like object with a ``write()``
method, but it can be rebound to redirect output to files other
than specifically standard output.  A typical idiom is::

    save_stdout = sys.stdout
    try:
        sys.stdout = mylogfile
        print 'this message goes to my log file'
    finally:
        sys.stdout = save_stdout

The problem with this approach is that the binding is global, and
so affects every statement inside the try: clause.  For example,
if we added a call to a function that actually did want to print
to stdout, this output too would get redirected to the logfile.

This approach is also very inconvenient for interleaving prints to
various output streams, and complicates coding in the face of
legitimate try/except or try/finally clauses.


Reference Implementation
========================

A reference implementation, in the form of a patch against the
Python 2.0 source tree, is available on SourceForge's patch
manager [2]_.  This approach adds two new opcodes, ``PRINT_ITEM_TO`` and
``PRINT_NEWLINE_TO``, which simply pop the file like object off the
top of the stack and use it instead of ``sys.stdout`` as the output
stream.

(This reference implementation has been adopted in Python 2.0.)


Alternative Approaches
======================

An alternative to this syntax change has been proposed (originally
by Moshe Zadka) which requires no syntax changes to Python.  A
``writeln()`` function could be provided (possibly as a builtin), that
would act much like extended print, with a few additional
features::

    def writeln(*args, **kws):
        import sys
        file = sys.stdout
        sep = ' '
        end = '\n'
        if kws.has_key('file'):
            file = kws['file']
            del kws['file']
        if kws.has_key('nl'):
            if not kws['nl']:
                end = ' '
            del kws['nl']
        if kws.has_key('sep'):
            sep = kws['sep']
            del kws['sep']
        if kws:
            raise TypeError('unexpected keywords')
        file.write(sep.join(map(str, args)) + end)

``writeln()`` takes a three optional keyword arguments.  In the
context of this proposal, the relevant argument is 'file' which
can be set to a file-like object with a ``write()`` method.  Thus::

    print >> mylogfile, 'this goes to my log file'

would be written as::

    writeln('this goes to my log file', file=mylogfile)

``writeln()`` has the additional functionality that the keyword
argument 'nl' is a flag specifying whether to append a newline or
not, and an argument 'sep' which specifies the separator to output
in between each item.


More Justification by the BDFL
==============================

The proposal has been challenged on the newsgroup.  One series of
challenges doesn't like '>>' and would rather see some other
symbol.

* Challenge: Why not one of these?

  ::

    print in stderr items,....
    print + stderr items,.......
    print[stderr] items,.....
    print to stderr items,.....

  Response: If we want to use a special symbol (``print <symbol>``
  expression), the Python parser requires that it is not already a
  symbol that can start an expression -- otherwise it can't decide
  which form of print statement is used.  (The Python parser is a
  simple LL(1) or recursive descent parser.)

  This means that we can't use the "keyword only in context trick"
  that was used for "import as", because an identifier can start an
  expression.  This rules out +stderr, \[sterr\], and to stderr.  It
  leaves us with binary operator symbols and other miscellaneous
  symbols that are currently illegal here, such as 'import'.

  If I had to choose between 'print in file' and 'print >> file' I
  would definitely choose '>>'.  In part because 'in' would be a new
  invention (I know of no other language that uses it, while '>>' is
  used in sh, awk, Perl, and C++), in part because '>>', being
  non-alphabetic, stands out more so is more likely to catch the
  reader's attention.

* Challenge: Why does there have to be a comma between the file and
  the rest?

  Response: The comma separating the file from the following expression is
  necessary!  Of course you want the file to be an arbitrary
  expression, not just a single word.  (You definitely want to be
  able to write ``print >>sys.stderr``.)  Without the expression the
  parser wouldn't be able to distinguish where that expression ends
  and where the next one begins, e.g.

  ::

      print >>i +1, 2
      print >>a [1], 2
      print >>f (1), 2

* Challenge: Why do you need a syntax extension?  Why not
  writeln(file, item, ...)?

  Response: First of all, this is lacking a feature of the print
  statement: the trailing comma to print which suppresses the final
  newline.  Note that 'print a,' still isn't equivalent to
  'sys.stdout.write(a)' -- print inserts a space between items, and
  takes arbitrary objects as arguments; ``write()`` doesn't insert a
  space and requires a single string.

  When you are considering an extension for the print statement,
  it's not right to add a function or method that adds a new feature
  in one dimension (where the output goes) but takes away in another
  dimension (spaces between items, and the choice of trailing
  newline or not).  We could add a whole slew of methods or
  functions to deal with the various cases but that seems to add
  more confusion than necessary, and would only make sense if we
  were to deprecate the print statement altogether.

  I feel that this debate is really about whether print should have
  been a function or method rather than a statement.  If you are in
  the function camp, of course adding special syntax to the existing
  print statement is not something you like.  I suspect the
  objection to the new syntax comes mostly from people who 


id: pep-0200.rst
content:
PEP: 200
Title: Python 2.0 Release Schedule
Author: Jeremy Hylton <jeremy@alum.mit.edu>
Status: Final
Type: Informational
Topic: Release
Created: 12-Jul-2000
Python-Version: 2.0
Post-History:



Introduction
============

This PEP describes the Python 2.0 release schedule, tracking the
status and ownership of the major new features, summarizes discussions
held in mailing list forums, and provides URLs for further
information, patches, and other outstanding issues.  The CVS revision
history of this file contains the definitive historical record.

Release Schedule
================

[revised 5 Oct 2000]


* 26-Sep-2000: 2.0 beta 2
* 9-Oct-2000: 2.0 release candidate 1 (2.0c1)
* 16-Oct-2000: 2.0 final

Previous milestones
===================

* 14-Aug-2000: All 2.0 PEPs finished / feature freeze
* 5-Sep-2000: 2.0 beta 1

What is release candidate 1?
============================

We believe that release candidate 1 will fix all known bugs that we
intend to fix for the 2.0 final release.  This release should be a bit
more stable than the previous betas.  We would like to see even more
widespread testing before the final release, so we are producing this
release candidate.  The final release will be exactly the same unless
any show-stopping (or brown bag) bugs are found by testers of the
release candidate.

Guidelines for submitting patches and making changes
====================================================

Use good sense when committing changes.  You should know what we mean
by good sense or we wouldn't have given you commit privileges <0.5
wink>.  Some specific examples of good sense include:

* Do whatever the dictator tells you.

* Discuss any controversial changes on python-dev first.  If you get
  a lot of +1 votes and no -1 votes, make the change.  If you get a
  some -1 votes, think twice; consider asking Guido what he thinks.

* If the change is to code you contributed, it probably makes sense
  for you to fix it.

* If the change affects code someone else wrote, it probably makes
  sense to ask him or her first.

* You can use the SF Patch Manager to submit a patch and assign it to
  someone for review.

Any significant new feature must be described in a PEP and approved
before it is checked in.

Any significant code addition, such as a new module or large patch,
must include test cases for the regression test and documentation.  A
patch should not be checked in until the tests and documentation are
ready.

If you fix a bug, you should write a test case that would have caught
the bug.

If you commit a patch from the SF Patch Manager or fix a bug from the
Jitterbug database, be sure to reference the patch/bug number in the
CVS log message.  Also be sure to change the status in the patch
manager or bug database (if you have access to the bug database).

It is not acceptable for any checked in code to cause the regression
test to fail.  If a checkin causes a failure, it must be fixed within
24 hours or it will be backed out.

All contributed C code must be ANSI C.  If possible check it with two
different compilers, e.g. gcc and MSVC.

All contributed Python code must follow Guido's Python style guide.
http://www.python.org/doc/essays/styleguide.html

It is understood that any code contributed will be released under an
Open Source license.  Do not contribute code if it can't be released
this way.


Failing test cases need to get fixed
====================================

We need to resolve errors in the regression test suite quickly.
Changes should not be committed to the CVS tree unless the regression
test runs cleanly with the changes applied.  If it fails, there may be
bugs lurking in the code.  (There may be bugs anyway, but that's
another matter.)  If the test cases are known to fail, they serve no
useful purpose.

::

    test case         platform    date reported
    ---------         --------    -------------
    test_mmap         Win ME      03-Sep-2000       Windows 2b1p2 prelease
        [04-Sep-2000 tim
         reported by Audun S. Runde mailto:audun@mindspring.com
         the mmap constructor fails w/
            WindowsError: [Errno 6] The handle is invalid
         since there are no reports of this failing on other
         flavors of Windows, this looks like to be an ME bug
        ]

Open items -- Need to be resolved before 2.0 final release
==========================================================

Decide whether cycle-gc should be enabled by default.

Resolve compatibility issues between core xml package and the XML-SIG
XML package.

Update Tools/compiler so that it is compatible with list
comprehensions, import as, and any other new language features.

Improve code coverage of test suite.

Finish writing the PEPs for the features that went out with 2.0b1(!
sad, but realistic -- we'll get better with practice).

Major effort to whittle the bug database down to size.  I've (tim)
seen this before: if you can keep all the open bugs fitting on one
screen, people will generally keep it that way.  But let it slobber
over a screen for a month, & it just goes to hell (no "visible
progress" indeed!).

Accepted and in progress
========================

* Currently none left. [4-Sep-2000 guido]

Open: proposed but not accepted or rejected
===========================================

* There are a number of open patches again.  We need to clear these
  out soon.

Previously failing test cases
=============================

If you find a test bouncing between this section and the previous one,
the code it's testing is in trouble!

::

    test case         platform    date reported
    ---------         --------    -------------
    test_fork1        Linux       26-Jul-2000
        [28-aug-2000 fixed by cgw; solution is to create copies of
        lock in child process]
        [19-Aug-2000 tim
         Charles Waldman whipped up a patch to give child processes a new
         "global lock":
         http://sourceforge.net/patch/?func=detailpatch&patch_id=101226&group_id=5470
         While this doesn't appear to address the symptoms we *saw*, it
         *does* so far appear to be fixing the failing cases anyway
        ]

    test_parser       all         22-Aug-2000
    test_posixpath    all         22-Aug-2000

    test_popen2       Win32       26-Jul-2000
        [31-Aug-2000 tim
         This died again, but for an entirely different reason:  it uses a
         dict to map file pointers to process handles, and calls a dict
         access function during popen.close().  But .close releases threads,
         which left the internal popen code accessing the dict without a
         valid thread state.  The dict implementation changed so that's no
         longer accepted.  Fixed by creating a temporary thread state in the
         guts of popen's close routine, and grabbing the global lock with
         it for the duration]
        [20-Aug-2000 tim
         changed the popen2.py _test function to use the "more" cmd
         when os.name == "nt".  This makes test_popen2 pass under
         Win98SE.
         HOWEVER, the Win98 "more" invents a leading newline out
         of thin air, and I'm not sure that the other Windows flavors
         of "more" also do that.
         So, somebody please try under other Windows flavors!
        ]
        [still fails 15-Aug-2000 for me, on Win98 - tim
             test test_popen2 crashed -- exceptions.AssertionError :
         The problem is that the test uses "cat", but there is
         no such thing under Windows (unless you install it).
         So it's the test that's broken here, not (necessarily)
         the code.
        ]

    test_winreg        Win32      26-Jul-2000
        [works 15-Aug-2000 for me, on Win98 - tim]

    test_mmap          Win32      26-Jul-2000
        [believe that was fixed by Mark H.]
        [works 15-Aug-2000 for me, on Win98 - tim]

    test_longexp      Win98+?     15-Aug-2000
        [fails in release build,
         passes in release build under verbose m


id: pep-0566.rst
content:
PEP: 566
Title: Metadata for Python Software Packages 2.1
Author: Dustin Ingram <di@python.org>
BDFL-Delegate: Daniel Holth
Discussions-To: distutils-sig@python.org
Status: Final
Type: Standards Track
Topic: Packaging
Created: 01-Dec-2017
Python-Version: 3.x
Post-History:
Replaces: 345
Resolution: https://mail.python.org/pipermail/distutils-sig/2018-February/032014.html


.. canonical-pypa-spec:: :ref:`packaging:core-metadata`


Abstract
========

This PEP describes the changes between versions 1.2 and 2.1 of the core
metadata specification for Python packages. Version 1.2 is specified in
:pep:`345`.

It also changes to the canonical source for field specifications to the `Core
Metadata Specification`_ reference document, which includes specifics of the
field names, and their semantics and usage.

Fields
======

The canonical source for the names and semantics of each of the supported
metadata fields is the `Core Metadata Specification`_ document.

Fields marked with "(Multiple use)" may be specified multiple times in a single
PKG-INFO file.  Other fields may only occur once in a PKG-INFO file.  Fields
marked with "(optional)" are not required to appear in a valid PKG-INFO file;
all other fields must be present.

New in Version 2.1
------------------

Description-Content-Type (optional)
:::::::::::::::::::::::::::::::::::

A string stating the markup syntax (if any) used in the distribution's
description, so that tools can intelligently render the description.

Historically, tools like PyPI assume that a package's description is formatted
in `reStructuredText (reST)
<http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html>`_, and
fall back on plain text if the description is not valid reST.

The introduction of this field allows PyPI to support additional types of
markup syntax, and not need to make this assumption.

The full specification for this field is defined in the `Core Metadata
Specification`_.


Provides-Extra (optional, multiple use)
:::::::::::::::::::::::::::::::::::::::

A string containing the name of an optional feature. Must be a valid Python
identifier. May be used to make a dependency conditional on whether the
optional feature has been requested.

This introduction of this field allows package installation tools (such as
``pip``) to determine which extras are provided by a given package, and so that
package publication tools (such as ``twine``) can check for issues with
environment markers which use extras.

The full specification for this field is defined in the `Core Metadata
Specification`_.

Changed in Version 2.1
----------------------

Name
::::

The specification for the format of this field is now identical to the
distribution name specification defined in :pep:`508`.

Description
:::::::::::

In addition to the ``Description`` header field, the distribution's
description may instead be provided in the message body (i.e., after a
completely blank line following the headers, with no indentation or other
special formatting necessary).

Version Specifiers
==================

Version numbering requirements and the semantics for specifying comparisons
between versions are defined in :pep:`440`. Direct references as defined in
:pep:`440` are also permitted as an alternative to version specifiers.

Following :pep:`508`, version specifiers no longer need to be surrounded by
parentheses in the fields Requires-Dist, Provides-Dist, Obsoletes-Dist or
Requires-External, so e.g. ``requests >= 2.8.1`` is now a valid value.
The recommended format is without parentheses, but tools parsing metadata should
also be able to handle version specifiers in parentheses. Further, public index
servers MAY prohibit strict version matching clauses or direct references in
these fields.

Usage of version specifiers is otherwise unchanged from :pep:`345`.

Environment markers
===================

An **environment marker** is a marker that can be added at the end of a
field after a semi-colon (";"), to add a condition about the execution
environment.

The environment marker format used to declare such a condition is defined in
the environment markers section of :pep:`508`.

Usage of environment markers is otherwise unchanged from :pep:`345`.

JSON-compatible Metadata
========================

It may be necessary to store metadata in a data structure which does not
allow for multiple repeated keys, such as JSON.

The canonical method to transform metadata fields into such a data structure is
as follows:

#. The original key-value format should be read with
   ``email.parser.HeaderParser``;
#. All transformed keys should be reduced to lower case. Hyphens should be
   replaced with underscores, but otherwise should retain all other characters;
#. The transformed value for any field marked with "(Multiple-use") should be a
   single list containing all the original values for the given key;
#. The ``Keywords`` field should be converted to a list by splitting the
   original value on commas;
#. The message body, if present, should be set to the value of the
   ``description`` key.
#. The result should be stored as a string-keyed dictionary.

Summary of Differences From PEP 345
===================================

* Metadata-Version is now 2.1.

* Fields are now specified via the `Core Metadata Specification`_.

* Added two new fields: ``Description-Content-Type`` and ``Provides-Extra``

* Acceptable values for the ``Name`` field are now specified as per :pep:`508`.

* Added canonical method of transformation into JSON-compatible data structure.

References
==========

This document specifies version 2.1 of the metadata format.
Version 1.0 is specified in :pep:`241`.
Version 1.1 is specified in :pep:`314`.
Version 1.2 is specified in :pep:`345`.
Version 2.0, while not formally accepted, was specified in :pep:`426`.

.. _`Core Metadata Specification`:
   https://packaging.python.org/specifications/core-metadata/

Copyright
=========

This document has been placed in the public domain.

Acknowledgements
================

Thanks to Alyssa Coghlan and Thomas Kluyver for contributing to this PEP.


id: pep-0599.rst
content:
PEP: 599
Title: The manylinux2014 Platform Tag
Author: Dustin Ingram <di@python.org>
Sponsor: Paul Moore <p.f.moore@gmail.com>
BDFL-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/the-next-manylinux-specification/1043
Status: Superseded
Type: Informational
Topic: Packaging
Created: 29-Apr-2019
Post-History: 29-Apr-2019
Superseded-By: 600
Resolution: https://discuss.python.org/t/the-next-manylinux-specification/1043/199


Abstract
========

This PEP proposes the creation of a ``manylinux2014`` platform tag to
succeed the ``manylinux2010`` tag introduced by :pep:`513`. It also
proposes that PyPI and ``pip`` both be updated to support uploading,
downloading, and installing ``manylinux2014`` distributions on
compatible platforms.

Rationale
=========

CentOS 6 is now the oldest supported CentOS release, and will receive
maintenance updates through November 30th, 2020, [1]_ at which point
it will reach end-of-life, and no further updates such as security
patches will be made available. All wheels built under the
``manylinux2010`` images will remain at obsolete versions after that
point.

Therefore, we propose the continuation of the existing manylinux
standard, and that a new :pep:`425`-style platform tag called
``manylinux2014`` be derived from CentOS 7 and that the ``manylinux``
toolchain, PyPI, and ``pip`` be updated to support it.

Similar to how :pep:`571` and :pep:`513` drew allowed shared
libraries and their symbol versions from CentOS 5.11 and CentOS 6,
respectively, a ``manylinux2014`` platform tag will draw its libraries
and symbol versions from CentOS 7, which will reach end-of-life on
June 30th, 2024. [1]_

The ``manylinuxYYYY`` pattern has a number of advantages that motivate
continuing with the current status quo:

- Well-defined Docker images with clearly specified compatible
  libraries;
- No need to survey for compatibility issues across multiple releases;
- A single build image and ``auditwheel`` profile per architecture.

There are also some disadvantages:

- Requires drafting a new PEP for every new standard;
- Requires adding the new platform tag to installers (e.g., ``pip``);
- Installers are unable to install a platform tag which predates a
  given release.

There are also challenges which would exist for any proposal,
including the time and effort it takes to define, prepare and release
the Docker images and corresponding ``auditwheel`` profiles. These
challenges were experienced in the long rollout period for
``manylinux2010``, which took approximately 1 year from PEP acceptance
to compatible build environment published. [3]_

However, if this PEP can be an indicator, the process is now
well-defined and easily repeatable, which should increase the timeline
for rollout of a newer, updated platform tag.

The ``manylinux2014`` policy
============================

The following criteria determine a ``linux`` wheel's eligibility for
the ``manylinux2014`` tag:

1. The wheel may only contain binary executables and shared objects
   compiled for one of the following architectures supported by CentOS
   7, or a CentOS 7 compatible base image (such as ubi7): [4]_ ::

       x86_64
       i686
       aarch64
       armv7l
       ppc64
       ppc64le
       s390x

   This list adds support for ARMv7 (armv7l), ARMv8 (aarch64) and PowerPC
   (ppc64, ppc64le) architectures supported by the CentOS Alternative
   Architecture Special Interest Group, as well as the IBM Z (s390x)
   architecture. [5]_

2. The wheel's binary executables or shared objects may not link
   against externally-provided libraries except those in the following
   list: ::

       libgcc_s.so.1
       libstdc++.so.6
       libm.so.6
       libdl.so.2
       librt.so.1
       libc.so.6
       libnsl.so.1
       libutil.so.1
       libpthread.so.0
       libresolv.so.2
       libX11.so.6
       libXext.so.6
       libXrender.so.1
       libICE.so.6
       libSM.so.6
       libGL.so.1
       libgobject-2.0.so.0
       libgthread-2.0.so.0
       libglib-2.0.so.0

   This list is identical to the externally-provided libraries
   originally allowed for ``manylinux2010``, with one exception:
   ``libcrypt.so.1`` was removed due to being deprecated in Fedora 30.
   ``libpythonX.Y`` remains ineligible for inclusion for the same
   reasons outlined in :pep:`513`.

   On Debian-based systems, these libraries are provided by the
   packages:

   ============  =======================================================
   Package       Libraries
   ============  =======================================================
   libc6         libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6,
                 libpthread.so.0, libm.so.6, libutil.so.1, libnsl.so.1
   libgcc1       libgcc_s.so.1
   libgl1        libGL.so.1
   libglib2.0-0  libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0
   libice6       libICE.so.6
   libsm6        libSM.so.6
   libstdc++6    libstdc++.so.6
   libx11-6      libX11.so.6
   libxext6      libXext.so.6
   libxrender1   libXrender.so.1
   ============  =======================================================

   On RPM-based systems, they are provided by these packages:

   ============  =======================================================
   Package       Libraries
   ============  =======================================================
   glib2         libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0
   glibc         libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1,
                 libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6
   libICE        libICE.so.6
   libX11        libX11.so.6
   libXext:      libXext.so.6
   libXrender    libXrender.so.1
   libgcc:       libgcc_s.so.1
   libstdc++     libstdc++.so.6
   mesa          libGL.so.1
   ============  =======================================================

3. If the wheel contains binary executables or shared objects linked
   against any allowed libraries that also export versioned symbols,
   they may only depend on the following maximum versions::

       GLIBC_2.17
       CXXABI_1.3.7, CXXABI_TM_1 is also allowed
       GLIBCXX_3.4.19
       GCC_4.8.0

   As an example, ``manylinux2014`` wheels may include binary
   artifacts that require ``glibc`` symbols at version ``GLIBC_2.12``,
   because this an earlier version than the maximum of ``GLIBC_2.17``.
4. If a wheel is built for any version of CPython 2 or CPython
   versions 3.0 up to and including 3.2, it *must* include a CPython
   ABI tag indicating its Unicode ABI.  A ``manylinux2014`` wheel
   built against Python 2, then, must include either the ``cpy27mu``
   tag indicating it was built against an interpreter with the UCS-4
   ABI or the ``cpy27m`` tag indicating an interpreter with the UCS-2
   ABI. (:pep:`3149` [7]_)
5. A wheel *must not* require the ``PyFPE_jbuf`` symbol.  This is
   achieved by building it against a Python compiled *without* the
   ``--with-fpectl`` ``configure`` flag.

Compilation of Compliant Wheels
===============================

Like ``manylinux1``, the ``auditwheel`` tool adds ``manylinux2014``
platform tags to ``linux`` wheels built by ``pip wheel`` or
``bdist_wheel`` in a ``manylinux2014`` Docker container.

Docker Images
-------------

A ``manylinux2014`` Docker image based on CentOS 7 x86_64 should be
provided for building binary ``linux`` wheels that can reliably be
converted to ``manylinux2014`` wheels. This image will come with a
full compiler suite installed (``gcc``, ``g++``, and ``gfortran``
4.8.5) as well as the latest releases of Python and ``pip``.

Auditwheel
----------

The ``auditwheel`` tool will also be updated to produce
``manylinux2014`` wheels. [8]_ Its behavior and purpose will be
otherwise unchanged from :pep:`513`.

Platform Detection for Installers
=================================

Platforms may define a ``manylinux2014_compatible`` boolean attribute
on the ``_manylinux`` module described in :pep:`513`. A platform is
considered incomp


id: pep-0758.rst
content:
PEP: 758
Title: Allow ``except`` and ``except*`` expressions without parentheses
Author: Pablo Galindo <pablogsal@python.org>, Brett Cannon <brett@python.org>
Status: Accepted
Type: Standards Track
Created: 30-Sep-2024
Python-Version: 3.14
Post-History: `02-Oct-2024 <https://discuss.python.org/t/66453>`__
Resolution: `14-Mar-2025 <https://discuss.python.org/t/66453/63>`__

Abstract
========

This PEP [1]_ proposes to allow unparenthesized ``except`` and ``except*``
blocks in Python's exception handling syntax only when not using the ``as``
clause. Currently, when catching multiple exceptions, parentheses are required
around the exception types. This was a Python 2 remnant. This PEP suggests allowing
the omission of these parentheses, simplifying the syntax, making it more consistent
with other parts of the syntax that make parentheses optional, and improving readability
in certain cases.

Motivation
==========

The current syntax for catching multiple exceptions requires parentheses in the
``except`` expression (equivalently for the ``except*`` expression). For
example:

.. code-block:: python

    try:
        ...
    except (ExceptionA, ExceptionB, ExceptionC):
        ...

While this syntax is clear and unambiguous, it can be seen as unnecessarily
verbose in some cases, especially when catching a large number of exceptions. By
allowing the omission of parentheses, we can simplify the syntax:

.. code-block:: python

    try:
        ...
    except ExceptionA, ExceptionB, ExceptionC:
        ...

This change would bring the syntax more in line with other comma-separated lists
in Python, such as function arguments, generator expressions inside of a
function call, and tuple literals, where parentheses are optional.

The same change would apply to ``except*`` expressions. For example:

.. code-block:: python

    try:
        ...
    except* ExceptionA, ExceptionB, ExceptionC:
        ...

When using the ``as`` clause to capture the exception instance parentheses must
be used as before. Some users have expressed that they would find it confusing not
to require parentheses as it would be unclear what exactly is being assigned to
the target since in other parts of the language multiple ``as`` clauses can be used
in similar situations (like in imports and context managers). This means that if
an ``as`` clause its being added to the previous example it must be done as:

.. code-block:: python

    try:
        ...
    except (ExceptionA, ExceptionB, ExceptionC) as e:
        ...


Rationale
=========

The decision to allow unparenthesized ``except`` blocks is based on the
following considerations:

1. Simplicity: Removing the requirement for parentheses simplifies the syntax,
   making it more consistent with other parts of the language.

2. Readability: In cases where many exceptions are being caught, the removal of
   parentheses can improve readability by reducing visual clutter.

3. Consistency: This change makes the ``except`` clause more consistent with
   other parts of Python where unambiguous, comma-separated lists don't require
   parentheses.

Specification
=============

The syntax for the except clause will be modified to allow an unparenthesized
list of exception types. The grammar will be updated as follows:

.. code-block:: peg

    except_block:
        | 'except' expressions ':' block
        | 'except' expression 'as' NAME ':' block
        | 'except' ':' block

    except_star_block
        | 'except' '*' expressions ':' block
        | 'except' '*' expression 'as' NAME ':' block

This allows both the current parenthesized syntax and the new unparenthesized
syntax while requiring parentheses when the ``as`` keyword is used:

.. code-block:: python

    try:
        ...
    except (ExceptionA, ExceptionB):  # Still valid
        ...
    except ExceptionC, ExceptionD:    # New syntax
        ...
    except (ExceptionE, ExceptionF) as e: # Parentheses still required
        ...

The semantics of exception handling remain unchanged. The interpreter will catch
any of the listed exceptions, regardless of whether they are parenthesized or
not.


Backwards Compatibility
=======================

This change is fully backwards compatible. All existing code using parenthesized
``except`` and ``except*`` blocks will continue to work without modification.
The new syntax is purely additive and does not break any existing code.

It's worth noting that in Python 2 the unparenthesized syntax was allowed with
two elements, but had different semantics, in which the first element of the
list was used as the exception type and the second element as the capture
variable. This change does not reintroduce the Python 2 semantics, and the
unparenthesized syntax will behave identically to the parenthesized version.


Security Implications
=====================

There are no known security implications for this change. The semantics of
exception handling remain the same, and this is purely a syntactic change.


How to Teach This
=================

For new Python users, the unparenthesized syntax can be taught as the standard
way to catch multiple exceptions:

.. code-block:: python

    try:
        risky_operation()
    except ValueError, TypeError, OSError:
        handle_errors()

For experienced users, it can be introduced as a new, optional syntax that can
be used interchangeably with the parenthesized version. Documentation should
note that both forms are equivalent:

.. code-block:: python

    # These are equivalent:
    except (ValueError, TypeError):
        ...

    except ValueError, TypeError:
        ...

It should be emphasized that this is purely a syntactic change and does not
affect the behaviour of exception handling.


Reference Implementation
========================

A proof-of-concept implementation is available at
https://github.com/pablogsal/cpython/commits/notuples/. This implementation
modifies the Python parser to accept the new syntax and ensures that it behaves
identically to the parenthesized version.


Rejected Ideas
==============

1. Allowing mixed parenthesized and unparenthesized syntax:

   .. code-block:: python

      try:
          ...
      except (ValueError, TypeError), OSError:
         ...

   This was rejected due to the potential for confusion and to maintain a clear
   distinction between the two styles.

Deferred Ideas
==============

1. Allowing unparenthesized expressions when the ``as`` keyword is used. The reason
   we have decided to defer this particular form given that there isn't clear
   consensus either way and there are reasonable arguments for both positions, the safest
   approach is keeping the parentheses requirement since it can be removed later if users
   find the disconnect too acute, while taking it out and users deciding it was a bad idea
   doesnât make it easy to put back.

Footnotes
=========

.. [1] Originally named "Parenthetically Speaking, We Don't Need 'Em"

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0770.rst
content:
PEP: 770
Title: Improving measurability of Python packages with Software Bill-of-Materials
Author: Seth Larson <seth@python.org>
Sponsor: Brett Cannon <brett@python.org>
PEP-Delegate: Brett Cannon <brett@python.org>
Discussions-To: https://discuss.python.org/t/76308
Status: Accepted
Type: Standards Track
Topic: Packaging
Created: 02-Jan-2025
Post-History:
  `05-Nov-2024 <https://discuss.python.org/t/70261>`__,
  `06-Jan-2025 <https://discuss.python.org/t/76308>`__,
Resolution: `11-Apr-2025 <https://discuss.python.org/t/76308/112>`__

.. canonical-pypa-spec:: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#the-dist-info-sboms-directory

Abstract
========

Almost all Python packages today are accurately measurable by software
composition analysis (SCA) tools. For projects that are not accurately
measurable, there is no existing mechanism to annotate a Python package
with composition data to improve measurability.

Software Bill-of-Materials (SBOM) is a technology-and-ecosystem-agnostic
method for describing software composition, provenance, heritage, and more.
SBOMs are used as inputs for SCA tools, such as scanners for vulnerabilities and
licenses, and have been gaining traction in global software regulations and
frameworks.

This PEP proposes using SBOM documents included in Python packages as a
means to improve automated software measurability for Python packages.

Motivation
==========

Measurability and Phantom Dependencies
--------------------------------------

Python packages are particularly affected by the "`phantom dependency`_"
problem, where software components that aren't written in Python are included
in Python packages for many reasons, such as ease of installation and
compatibility with standards:

* Python serves scientific, data, web, and machine-learning use-cases which
  use compiled or non-Python languages like Rust, C, C++, Fortran, JavaScript,
  and others.
* The Python wheel format is preferred by users due to the ease-of-installation.
  No code is executed during the installation step, only extracting the archive.
* The Python wheel format requires bundling shared compiled libraries without
  a method to encode metadata about these libraries.
* Packages related to Python packaging sometimes need to solve the
  "bootstrapping" problem, so include pure Python projects inside their
  source code.

These software components can't be described using Python package metadata and
thus are likely to be missed by software composition analysis (SCA) software
which can mean vulnerable software components aren't reported accurately.

`For example <https://sethmlarson.dev/early-promising-results-with-sboms-and-python-packages>`__,
the Python package Pillow includes 16 shared object libraries in the wheel that
were bundled by auditwheel as a part of the build. None of those shared object
libraries are detected when using common SCA tools like Syft and Grype.
If an SBOM document is included annotating all the included shared libraries
then SCA tools can identify the included software reliably.

Build Tools, Environment, and Reproducibility
---------------------------------------------

Going beyond the runtime dependencies of a package: SBOMs can also record the
tools and environments used to build a package. Recording the exact tools
and versions used to build a package is often required to establish
`build reproducibility <https://reproducible-builds.org>`__.
Build reproducibility is a property of software that can be used to detect
incorrectly or maliciously modified software components when compared to their
upstream sources. Without a recorded list of build tools and versions it can
become difficult to impossible for a third-party to verify build reproducibility.

Regulations
-----------

SBOMs are required by recent software security regulations, like the
`Secure Software Development Framework`_ (SSDF) and the
`Cyber Resilience Act`_ (CRA). Due to their inclusion in these regulations,
the demand for SBOM documents of open source projects is expected to be high.
One goal is to minimize the demands on open source project maintainers by
enabling open source users that need SBOMs to self-serve using existing
tooling.

Another goal is to enable contributions from users who need SBOMs to annotate
projects they depend on with SBOM information. Today there is no mechanism to
propagate the results of those contributions for a Python package so there is
no incentive for users to contribute this type of work.

.. _Cyber Resilience Act: https://digital-strategy.ec.europa.eu/en/policies/cyber-resilience-act
.. _Secure Software Development Framework: https://csrc.nist.gov/Projects/ssdf

Rationale
=========

Using SBOM standards instead of Core Metadata fields
----------------------------------------------------

Attempting to add every field offered by SBOM standards into Python package
Core Metadata would result in an explosion of new Core Metadata fields,
including the need to keep up-to-date as SBOM standards continue to evolve
to suit new needs in that space.

Instead, this proposal delegates SBOM-specific metadata to SBOM documents that
are included in Python packages into a named directory under dist-info.

This standard also doesn't aim to replace Core Metadata with SBOMs,
instead focusing on the SBOM information being supplemental to Core Metadata.
Included SBOMs only contain information about dependencies included in the
package archive or information about the top-level software in the package that
can't be encoded into Core Metadata but is relevant for the SBOM use-case
("software identifiers", "purpose", "support level", etc).

Zero-or-more opaque SBOM documents
----------------------------------

Rather than requiring at most one included SBOM document per Python package,
this PEP proposes that one or more SBOM documents may be included in a Python
package. This means that code attempting to annotate a Python package with SBOM
data may do so without being concerned about corrupting data already contained
within other SBOM documents.

Additionally, this PEP treats SBOM document data opaquely instead relying on
final end-users of the SBOM data to process the contained SBOM data.
This choice acknowledges that SBOM standards are an active area of development
where there is not yet (and may never be) a single definitive SBOM standard
and that SBOM standards can continue to evolve independent of Python packaging
standards. Already tools that consume SBOM documents support a multitude of
SBOM standards to handle this reality.

These decisions mean this PEP is capable of supporting any SBOM standard
and does not favor one over the other, instead deferring the decision to
producing projects and tools and consuming user tooling.

Adding data to Python packages without new metadata versions
------------------------------------------------------------

The rollout of a new metadata version and field requires that many different
projects and teams need to adopt the metadata version in sequence to avoid
widespread breakage. This effect usually means a substantial delay in how
quickly users and tools can start using new packaging features.

For example, a single metadata version bump requires
updates to PyPI, various ``pyproject.toml`` parsing and schema projects,
the ``packaging`` library, wait for releases, then ``pip`` and other installers
need to bundle the changes to ``packaging`` and release, then build backends can
begin emitting the new metadata version, again wait for releases, and only then
can projects begin using the new features. Even with this careful approach it's
not guaranteed that tools won't break on new metadata versions and fields.

To avoid this delay, simplify overall how to include SBOMs, and to give
flexibility to build backends and tools, this PEP proposes using a subdirectory
under ``.dist-info`` to safely add data to a Python package while avoiding the
need for new metadata fields and vers


id: pep-0764.rst
content:
PEP: 764
Title: Inline typed dictionaries
Author: Victorien Plot <contact@vctrn.dev>
Sponsor: Eric Traut <erictr at microsoft.com>
Discussions-To: https://discuss.python.org/t/78779
Status: Draft
Type: Standards Track
Topic: Typing
Created: 25-Oct-2024
Python-Version: 3.15
Post-History: `29-Jan-2025 <https://discuss.python.org/t/78779>`__


Abstract
========

:pep:`589` defines a :ref:`class-based <typing:typeddict-class-based-syntax>`
and a :ref:`functional syntax <typing:typeddict-functional-syntax>` to create
typed dictionaries. In both scenarios, it requires defining a class or
assigning to a value. In some situations, this can add unnecessary
boilerplate, especially if the typed dictionary is only used once.

This PEP proposes the addition of a new inline syntax, by subscripting the
:class:`~typing.TypedDict` type::

    from typing import TypedDict

    def get_movie() -> TypedDict[{'name': str, 'year': int}]:
        return {
            'name': 'Blade Runner',
            'year': 1982,
        }

Motivation
==========

Python dictionaries are an essential data structure of the language. Many
times, it is used to return or accept structured data in functions. However,
it can get tedious to define :class:`~typing.TypedDict` classes:

* A typed dictionary requires a name, which might not be relevant.
* Nested dictionaries require more than one class definition.

Taking a simple function returning some nested structured data as an example::

    from typing import TypedDict

    class ProductionCompany(TypedDict):
        name: str
        location: str

    class Movie(TypedDict):
        name: str
        year: int
        production: ProductionCompany


    def get_movie() -> Movie:
        return {
            'name': 'Blade Runner',
            'year': 1982,
            'production': {
                'name': 'Warner Bros.',
                'location': 'California',
            }
        }


Rationale
=========

The new inline syntax can be used to resolve these problems::

    def get_movie() -> TypedDict[{'name': str, 'year': int, 'production': TypedDict[{'name': str, 'location': str}]}]:
        ...

While less useful (as the functional or even the class-based syntax can be
used), inline typed dictionaries can be assigned to a variable, as an alias::

    InlineTD = TypedDict[{'name': str}]

    def get_movie() -> InlineTD:
        ...


Specification
=============

The :class:`~typing.TypedDict` special form is made subscriptable, and accepts
a single type argument which must be a :class:`dict`, following the same
semantics as the :ref:`functional syntax <typing:typeddict-functional-syntax>`
(the dictionary keys are strings representing the field names, and values are
valid :ref:`annotation expressions <typing:annotation-expression>`). Only the
comma-separated list of ``key: value`` pairs within braces constructor
(``{k: <type>}``) is allowed, and should be specified directly as the type
argument (i.e. it is not allowed to use a variable which was previously
assigned a :class:`dict` instance).

Inline typed dictionaries can be referred to as *anonymous*, meaning they
don't have a specific name (see the `runtime behavior <Runtime behavior>`_
section).

It is possible to define a nested inline dictionary::

    Movie = TypedDict[{'name': str, 'production': TypedDict[{'location': str}]}]

    # Note that the following is invalid as per the updated `type_expression` grammar:
    Movie = TypedDict[{'name': str, 'production': {'location': str}}]

Although it is not possible to specify any class arguments such as ``total``,
any :term:`typing:type qualifier` can be used for individual fields::

    Movie = TypedDict[{'name': NotRequired[str], 'year': ReadOnly[int]}]

Inline typed dictionaries are implicitly *total*, meaning all keys must be
present. Using the :data:`~typing.Required` type qualifier is thus redundant.

Type variables are allowed in inline typed dictionaries, provided that they
are bound to some outer scope::

    class C[T]:
        inline_td: TypedDict[{'name': T}]  # OK, `T` is scoped to the class `C`.

    reveal_type(C[int]().inline_td['name'])  # Revealed type is 'int'


    def fn[T](arg: T) -> TypedDict[{'name': T}]: ...  # OK: `T` is scoped to the function `fn`.

    reveal_type(fn('a')['name'])  # Revealed type is 'str'


    type InlineTD[T] = TypedDict[{'name': T}]  # OK, `T` is scoped to the type alias.


    T = TypeVar('T')

    InlineTD = TypedDict[{'name': T}]  # OK, same as the previous type alias, but using the old-style syntax.


    def func():
        InlineTD = TypedDict[{'name': T}]  # Not OK: `T` refers to a type variable that is not bound to the scope of `func`.


Inline typed dictionaries can be extended::

    InlineTD = TypedDict[{'a': int}]

    class SubTD(InlineTD):
        pass

Typing specification changes
----------------------------

The inline typed dictionary adds a new kind of
:term:`typing:type expression`. As such, the
:external+typing:token:`~expression-grammar:type_expression` production will
be updated to include the inline syntax:

.. productionlist:: inline-typed-dictionaries-grammar
    new-type_expression: `~expression-grammar:type_expression`
                       : | <TypedDict> '[' '{' (string: ':' `~expression-grammar:annotation_expression` ',')* '}' ']'
                       :       (where string is any string literal)

Runtime behavior
----------------

Creating an inline typed dictionary results in a new class, so ``T1`` and
``T2`` are of the same type::

    from typing import TypedDict

    T1 = TypedDict('T1', {'a': int})
    T2 = TypedDict[{'a': int}]

As inline typed dictionaries are meant to be *anonymous*, their
:attr:`~type.__name__` attribute will be set to the ``<inline TypedDict>``
string literal. In the future, an explicit class attribute could be added
to make them distinguishable from named classes.

Although :class:`~typing.TypedDict` is documented as a class, the way it is
defined is an implementation detail. The implementation will have to be tweaked
so that :class:`~typing.TypedDict` can be made subscriptable.


Backwards Compatibility
=======================

This PEP does not bring any backwards incompatible changes.


Security Implications
=====================

There are no known security consequences arising from this PEP.


How to Teach This
=================

The new inline syntax will be documented both in the :mod:`typing` module
documentation and the :ref:`typing specification <typing:typed-dictionaries>`.

When complex dictionary structures are used, having everything defined on a
single line can hurt readability. Code formatters can help by formatting the
inline type dictionary across multiple lines::

    def edit_movie(
        movie: TypedDict[{
            'name': str,
            'year': int,
            'production': TypedDict[{
                'location': str,
            }],
        }],
    ) -> None:
        ...


Reference Implementation
========================

Mypy supports a similar syntax as an :option:`experimental feature <mypy:mypy.--enable-incomplete-feature>`::

    def test_values() -> {"int": int, "str": str}:
        return {"int": 42, "str": "test"}

Support for this PEP is added in `this pull request <https://github.com/python/mypy/pull/18889>`__.

Pyright added support for the new syntax in version `1.1.387`_.

.. _1.1.387: https://github.com/microsoft/pyright/releases/tag/1.1.387

Runtime implementation
----------------------

The necessary changes were first implemented in
`typing_extensions <https://typing-extensions.readthedocs.io/en/latest/>`_
in `this pull request <https://github.com/python/typing_extensions/pull/580>`__.


Rejected Ideas
==============

Using the functional syntax in annotations
------------------------------------------

The alternative functional syntax could be used as an annotation directly::

    def get_movie() -> TypedDict('Movie', {'title': str}): ...

However, call expressions are curr


id: pep-0002.rst
content:
PEP: 2
Title: Procedure for Adding New Modules
Author: Brett Cannon <brett@python.org>,
        Martijn Faassen <faassen@infrae.com>
Status: Active
Type: Process
Created: 07-Jul-2001
Post-History: 07-Jul-2001, 09-Mar-2002


Introduction
============

The Python Standard Library contributes significantly to Python's
success.  The language comes with "batteries included", so it is easy
for people to become productive with just the standard library alone.
It is therefore important that the usefulness of the standard library
be maintained.

Due to the visibility and importance of the standard library, it must
be maintained thoughtfully. As such, any code within it must be
maintained by Python's development team which leads to a perpetual
cost to each addition made. There is also added cognitive load for
users in familiarizing themselves with what is in the standard
library to be considered.

New functionality is commonly added to the library in the form of new
modules. This PEP will describe the procedure for the *addition* of
new modules.  :pep:`4` deals with procedures for deprecation of modules;
the *removal* of old and unused modules from the standard library.


Acceptance Procedure
====================

For top-level modules/packages, a PEP is required. The procedure for
writing a PEP is covered in :pep:`1`.

For submodules of a preexisting package in the standard library,
additions are at the discretion of the general Python development team
and its members.

General guidance on what modules typically are accepted into the
standard library, the overall process, etc. are covered in the
`developer's guide <https://devguide.python.org/stdlibchanges/>`_.


Maintenance Procedure
=====================

Anything accepted into the standard library is expected to be
primarily maintained there, within Python's development infrastructure.
While some members of the development team may choose to maintain a
backport of a module outside of the standard library, it is up to them
to keep their external code in sync as appropriate.


Copyright
=========

This document has been placed in the public domain.


id: pep-0638.rst
content:
PEP: 638
Title: Syntactic Macros
Author: Mark Shannon <mark@hotpy.org>
Discussions-To: https://mail.python.org/archives/list/python-dev@python.org/thread/U4C4XHNRC4SHS3TPZWCTY4SN4QU3TT6V/
Status: Draft
Type: Standards Track
Created: 24-Sep-2020
Post-History: 26-Sep-2020

Abstract
========

This PEP adds support for syntactic macros to Python.
A macro is a compile-time function that transforms
a part of the program to allow functionality that cannot be
expressed cleanly in normal library code.

The term "syntactic" means that this sort of macro operates on the program's
syntax tree. This reduces the chance of mistranslation that can happen
with text-based substitution macros, and allows the implementation
of `hygienic macros`__.

__ https://en.wikipedia.org/wiki/Hygienic_macro

Syntactic macros allow libraries to modify the abstract syntax tree during compilation,
providing the ability to extend the language for specific domains without
adding to complexity to the language as a whole.

Motivation
==========

New language features can be controversial, disruptive and sometimes divisive.
Python is now sufficiently powerful and complex, that many proposed additions
are a net loss for the language due to the additional complexity.

Although a language change may make certain patterns easy to express,
it will have a cost. Each new feature makes the language larger,
harder to learn and harder to understand.
Python was once described as `Python Fits Your Brain`__,
but that becomes less and less true as more and more features are added.

Because of the high cost of adding a new feature,
it is very difficult or impossible to add a feature that would benefit only
some users, regardless of how many users, or how beneficial that feature would
be to them.

The use of Python in data science and machine learning has grown very rapidly
over the last few years.
However, most of the core developers of Python do not have a background in
data science or machine learning.
This makes it extremely difficult for the core developers to determine whether a
language extension for machine learning is worthwhile.

By allowing language extensions to be modular and distributable, like libraries,
domain-specific extensions can be implemented without negatively impacting
users outside of that domain.
A web developer is likely to want a very different set of extensions from
a data scientist.
We need to let the community develop their own extensions.

Without some form of user-defined language extensions,
there will be a constant battle between those wanting to keep the
language compact and fitting their brains, and those wanting a new feature
that suits their domain or programming style.

__ https://www.linuxjournal.com/article/4731


Improving the expressiveness of libraries for specific domains
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Many domains see repeated patterns that are difficult or impossible
to express as a library.
Macros can express those patterns in a more concise and less error-prone way.

Trialing new language features
''''''''''''''''''''''''''''''

It is possible to demonstrate potential language extensions using macros.
For example, macros would have enabled the ``with`` statement and
``yield from`` expression to have been trialed.
Doing so might well have lead to a higher quality implementation
at first release, by allowing more testing
before those features were included in the language.

It is nearly impossible to make sure that a new feature is completely reliable
before it is released; bugs relating to the ``with`` and  ``yield from``
features were still being fixed many years after they were released.

Long term stability for the bytecode interpreter
''''''''''''''''''''''''''''''''''''''''''''''''

Historically, new language features have been implemented by naive compilation
of the AST into new, complex bytecode instructions.
Those bytecodes have often had their own internal flow-control, performing
operations that could, and should, have been done in the compiler.

For example,
until recently flow control within the ``try``-``finally`` and ``with``
statements was managed by complicated bytecodes with context-dependent semantics.
The control flow within those statements is now implemented in the compiler, making
the interpreter simpler and faster.

By implementing new features as AST transformations, the existing compiler can
generate the bytecode for a feature without having to modify the interpreter.

A stable interpreter is necessary if we are to improve the performance and
portability of the CPython VM.

Rationale
=========

Python is both expressive and easy to learn;
it is widely recognized as the easiest-to-learn, widely used programming language.
However, it is not the most flexible. That title belongs to lisp.

Because lisp is homoiconic, meaning that lisp programs are lisp data structures,
lisp programs can be manipulated by lisp programs.
Thus much of the language can be defined in itself.

We would like that ability in Python,
without the many parentheses that characterize lisp.
Fortunately, homoiconicity is not needed for a language to be able to
manipulate itself, all that is needed is the ability to manipulate programs
after parsing, but before translation to an executable form.

Python already has the components needed.
The syntax tree of Python is available through the ``ast`` module.
All that is needed is a marker to tell the compiler that a macro is present,
and the ability for the compiler to callback into user code to manipulate the AST.

Specification
=============

Syntax
''''''

Lexical analysis
~~~~~~~~~~~~~~~~

Any sequence of identifier characters followed by an exclamation point
(exclamation mark, UK English) will be tokenized as a ``MACRO_NAME``.

Statement form
~~~~~~~~~~~~~~

::

    macro_stmt = MACRO_NAME testlist [ "import" NAME ] [ "as"  NAME ] [ ":" NEWLINE suite ]

Expression form
~~~~~~~~~~~~~~~

::

    macro_expr = MACRO_NAME "(" testlist ")"

Resolving ambiguity
~~~~~~~~~~~~~~~~~~~

The statement form of a macro takes precedence, so that the code
``macro_name!(x)`` will be parsed as a macro statement,
not as an expression statement containing a macro expression.

Semantics
'''''''''

Compilation
~~~~~~~~~~~

Upon encountering a ``macro`` during translation to bytecode,
the code generator will look up the macro processor registered for the macro,
and pass the AST rooted at the macro to the processor function.
The returned AST will then be substituted for the original tree.

For macros with multiple names,
several trees will be passed to the macro processor,
but only one will be returned and substituted,
shorting the enclosing block of statements.

This process can be repeated,
to enable macros to return AST nodes including other macros.

The compiler will not look up a macro processor until that macro is reached,
so that inner macros do not need to have processors registered.
For example, in a ``switch`` macro, the ``case`` and ``default`` macros wouldn't
need processors registered as they would be eliminated by the ``switch`` processor.

To enable definition of macros to be imported,
the macros ``import!`` and ``from!`` are predefined.
They support the following syntax:

::

    "import!" dotted_name "as" name

    "from!" dotted_name "import" name [ "as" name ]

The ``import!`` macro performs a compile-time import of ``dotted_name``
to find the macro processor, then registers it under ``name``
for the scope currently being compiled.

The ``from!`` macro performs a compile-time import of ``dotted_name.name``
to find the macro processor, then registers it under ``name``
(using the ``name`` following "as", if present)
for the scope currently being compiled.

Note that, since ``import!`` and ``from!`` only define the macro for the
scope in which the import is present, all uses of a macro must be preceded by
an explicit ``import!`` or ``from!`` to improve clarity.

For example, to import the macr


id: pep-0604.rst
content:
PEP: 604
Title: Allow writing union types as ``X | Y``
Author: Philippe PRADOS <python@prados.fr>, Maggie Moss <maggiebmoss@gmail.com>
Sponsor: Chris Angelico <rosuav@gmail.com>
BDFL-Delegate: Guido van Rossum <guido@python.org>
Discussions-To: typing-sig@python.org
Status: Final
Type: Standards Track
Topic: Typing
Created: 28-Aug-2019
Python-Version: 3.10
Post-History: 28-Aug-2019, 05-Aug-2020

.. canonical-doc:: :ref:`python:types-union`

Abstract
========

This PEP proposes overloading the ``|`` operator on types to allow
writing ``Union[X, Y]`` as ``X | Y``, and allows it to appear in
``isinstance`` and ``issubclass`` calls.


Motivation
==========

:pep:`484` and :pep:`526` propose a generic syntax to add typing to variables,
parameters and function returns. :pep:`585` proposes to :pep:`expose
parameters to generics at runtime
<585#parameters-to-generics-are-available-at-runtime>`.
Mypy [1]_ accepts a syntax which looks like::

    annotation: name_type
    name_type: NAME (args)?
    args: '[' paramslist ']'
    paramslist: annotation (',' annotation)* [',']

- To describe a disjunction (union type), the user must use ``Union[X, Y]``.

The verbosity of this syntax does not help with type adoption.


Proposal
========

Inspired by Scala [2]_ and Pike [3]_, this proposal adds operator
``type.__or__()``.  With this new operator, it is possible to write
``int | str`` instead of ``Union[int, str]``. In addition to
annotations, the result of this expression would then be valid in
``isinstance()`` and ``issubclass()``::

    isinstance(5, int | str)
    issubclass(bool, int | float)

We will also be able to write ``t | None`` or ``None | t`` instead of
``Optional[t]``::

    isinstance(None, int | None)
    isinstance(42, None | int)

Specification
=============

The new union syntax should be accepted for function, variable and parameter annotations.

Simplified Syntax
-----------------
::

    # Instead of
    # def f(list: List[Union[int, str]], param: Optional[int]) -> Union[float, str]
    def f(list: List[int | str], param: int | None) -> float | str:
        pass

    f([1, "abc"], None)

    # Instead of typing.List[typing.Union[str, int]]
    typing.List[str | int]
    list[str | int]

    # Instead of typing.Dict[str, typing.Union[int, float]]
    typing.Dict[str, int | float]
    dict[str, int | float]

The existing ``typing.Union`` and ``|`` syntax should be equivalent.

::

  int | str == typing.Union[int, str]

  typing.Union[int, int] == int
  int | int == int

The order of the items in the Union should not matter for equality.

::

  (int | str) == (str | int)
  (int | str | float) == typing.Union[str, float, int]


Optional values should be equivalent to the new union syntax

::

  None | t == typing.Optional[t]

A new Union.__repr__() method should be implemented.

::

  str(int | list[str])
  # int | list[str]

  str(int | int)
  # int

isinstance and issubclass
-------------------------

The new syntax should be accepted for calls to ``isinstance`` and ``issubclass`` as long as the Union items are valid arguments to ``isinstance`` and ``issubclass`` themselves.

::

    # valid
    isinstance("", int | str)

    # invalid
    isinstance(2, list[int]) # TypeError: isinstance() argument 2 cannot be a parameterized generic
    isinstance(1, int | list[int])

    # valid
    issubclass(bool, int | float)

    # invalid
    issubclass(bool, bool | list[int])



Incompatible changes
====================

In some situations, some exceptions will not be raised as expected.

If a metaclass implements the ``__or__`` operator, it will override this::

    >>> class M(type):
    ...     def __or__(self, other): return "Hello"
    ...
    >>> class C(metaclass=M): pass
    ...
    >>> C | int
    'Hello'
    >>> int | C
    typing.Union[int, __main__.C]
    >>> Union[C, int]
    typing.Union[__main__.C, int]


Objections and responses
========================

For more details about discussions, see links below:

- `Discussion in python-ideas <https://mail.python.org/archives/list/python-ideas@python.org/thread/FCTXGDT2NNKRJQ6CDEPWUXHVG2AAQZZY/>`_
- `Discussion in typing-sig <https://mail.python.org/archives/list/typing-sig@python.org/thread/D5HCB4NT4S3WSK33WI26WZSFEXCEMNHN/>`_

1. Add a new operator for ``Union[type1, type2]``?
--------------------------------------------------

PROS:

- This syntax can be more readable, and is similar to other languages (Scala, ...)
- At runtime, ``int|str`` might return a simple object in 3.10, rather than everything that
  you'd need to grab from importing ``typing``


CONS:

- Adding this operator introduces a dependency between ``typing`` and ``builtins``
- Breaks the backport (in that ``typing`` can easily be backported but core ``types`` can't)
- If Python itself doesn't have to be changed, we'd still need to implement it in mypy, Pyre, PyCharm,
  Pytype, and who knows what else (it's a minor change see "Reference Implementation")


2. Change only PEP 484 (Type hints) to accept the syntax ``type1 | type2`` ?
----------------------------------------------------------------------------

:pep:`563` (Postponed Evaluation of Annotations) is enough to accept this proposition,
if we accept to not be compatible with the dynamic evaluation of annotations (``eval()``).

::

    >>> from __future__ import annotations
    >>> def foo() -> int | str: pass
    ...
    >>> eval(foo.__annotations__['return'])
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<string>", line 1, in <module>
    TypeError: unsupported operand type(s) for |: 'type' and 'type'

3. Extend ``isinstance()`` and ``issubclass()`` to accept ``Union`` ?
---------------------------------------------------------------------

::

    isinstance(x, str | int) ==> "is x an instance of str or int"

PROS:

- If they were permitted, then instance checking could use an extremely clean-looking notation

CONS:

- Must migrate all of the ``typing`` module in ``builtin``


Reference Implementation
========================

A new built-in ``Union`` type must be implemented to hold the return
value of ``t1 | t2``, and it must be supported by ``isinstance()`` and
``issubclass()``. This type can be placed in the ``types`` module.
Interoperability between ``types.Union`` and ``typing.Union`` must be
provided.

Once the Python language is extended, mypy [1]_ and other type checkers will
need to be updated to accept this new syntax.


- A proposed implementation for `cpython is here
  <https://github.com/python/cpython/pull/21515>`_.
- A proposed implementation for `mypy is here
  <https://github.com/pprados/mypy/tree/PEP604>`_.


References
==========

.. [1] mypy
   http://mypy-lang.org/
.. [2] Scala Union Types
   https://dotty.epfl.ch/docs/reference/new-types/union-types.html
.. [3] Pike
   http://pike.lysator.liu.se/docs/man/chapter_3.html#3.5


Copyright
=========

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.


id: pep-0610.rst
content:
PEP: 610
Title: Recording the Direct URL Origin of installed distributions
Author: StÃ©phane Bidoul <stephane.bidoul@gmail.com>, Chris Jerdonek <chris.jerdonek@gmail.com>
Sponsor: Alyssa Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: Pradyun Gedam <pradyunsg@gmail.com>
Discussions-To: https://discuss.python.org/t/recording-the-source-url-of-an-installed-distribution/1535
Status: Final
Type: Standards Track
Topic: Packaging
Created: 21-Apr-2019
Post-History:
Resolution: https://discuss.python.org/t/1535/56


.. canonical-pypa-spec:: :ref:`packaging:direct-url`


Abstract
========

Following :pep:`440`, a distribution can be identified by a name and either a
version, or a direct URL reference (see :pep:`PEP440 Direct References <440#direct-references>`).
After installation, the name and version are captured in the project metadata,
but currently there is no way to obtain details of the URL used when the
distribution was identified by a direct URL reference.

This proposal defines
additional metadata, to be added to the installed distribution by the
installation front end, which records the Direct URL Origin for use by
consumers which introspect the database of installed packages (see :pep:`376`).


Motivation
==========

The original motivation of this PEP was to permit tools with a "freeze"
operation allowing a Python environment to be recreated to work in a broader
range of situations.

Specifically, the PEP originated from the desire to address `pip issue #609`_:
i.e. improving the behavior of ``pip freeze`` in the presence of distributions
installed from direct URL references. It follows a
`thread on discuss.python.org`_ about the best course of action to implement
it.

Installation from direct URL references
---------------------------------------

Python installers such as pip are capable of downloading and installing
distributions from package indexes. They are also capable of downloading
and installing source code from requirements specifying arbitrary URLs of
source archives and Version Control Systems (VCS) repositories,
as standardized in :pep:`PEP440 Direct References <440#direct-references>`.

In other words, two relevant installation modes exist.

1. the package to install is specified as a name and version specifier:

  In this case, the installer looks in a package index (or optionally
  using ``--find-links`` in the case of pip) to find the distribution to install.

2. The package to install is specified as a direct URL reference:

  In this case, the installer downloads whatever is specified by the URL
  (typically a wheel, a source archive or a VCS repository) and installs it.

  In this mode, installers typically download the source code in a
  temporary directory, invoke the :pep:`517` build backend to produce a wheel
  if needed, install the wheel, and delete the temporary directory.

  After installation, no trace of the URL the user requested to download the
  package is left on the user system.

Freezing an environment
-----------------------

Pip also sports a command named ``pip freeze`` which examines the Database of
Installed Python Distributions to generate a list of requirements. The main
goal of this command is to help users generating a list of requirements that
will later allow the re-installation the same environment with the highest
possible fidelity.

As of pip version 19.3, the ``pip freeze`` command outputs a ``name==version``
line for each installed
distribution (except for editable installs). To achieve the goal of
reinstalling the same environment, this requires the (name, version)
tuple to refer to an immutable version of the
distribution. The immutability is guaranteed by package indexes
such as Warehouse. The package index to use is typically known from
environmental or command line parameters of the installer.

This freeze mechanism therefore works fine for installation mode 1 (i.e.
when the package to install was specified as a name plus version specifier).

For installation mode 2, i.e. when the package to install was specified as a
direct URL reference, the ``name==version`` tuple is obviously not sufficient
to reinstall the same distribution and users of the freeze command expect it
to output the URL that was originally requested.

The reasoning above is equally applicable to tools, other than ``pip freeze``,
that would attempt to generate a ``Pipfile.lock`` or any other similar format
from the Database of Installed Python Distributions. Unless specified
otherwise, "freeze" is used in this document as a generic term for such
an operation.

The importance of installing from (VCS) URLs for application integrators
------------------------------------------------------------------------

For an application integrator, it is important to be able to reliably install
and freeze unreleased version of python distributions.
For instance when a developer needs to deploy an unreleased patched version
of a dependency, it is common to install the dependency directly from a VCS
branch that has the patch, while waiting for the maintainer to release an
updated version.

In such cases, it is important for "freeze" to pin the exact VCS
reference (commit-hash if available) that was installed, in order to create
reproducible builds with the highest possible fidelity.

Additional origin metadata available for VCS URLs
-------------------------------------------------

For VCS URLs, there is additional origin information available only at
install time useful for introspection and certain workflows. For example,
when installing a revision from a VCS URL, a tool can determine if the
revision corresponds to a branch, tag or (in the case of Git) a ref. This
information can be used when introspecting the Database of Installed Distributions
to communicate to users more information about what version was installed
(e.g. whether a branch or tag was installed and, if so, the name of the
branch or tag). This also permits one to know whether a :pep:`440` direct
reference URL can be constructed using the tag form, as only tags have the
semantics of immutability.

In cases where the revision is mutable (e.g. branches and Git refs), knowing
this information enables workflows where users can e.g. update to the latest
version of a branch they are tracking, or update to the latest version of a
pull request they are reviewing locally. In contrast, when the revision is a
tag, tools can know in advance (e.g. without network calls) that no update is
needed.

As with the URL itself, if this information isn't recorded at install time
when the VCS repository is available, it would otherwise be lost.

Note about "editable" installs
------------------------------

The editable installation mode of pip roughly lets a user insert a
local directory in sys.path for development purpose. This mode is somewhat
abused to work around the fact that a non-editable install from a VCS URL
loses track of the origin after installation.
Indeed, editable installs implicitly record the VCS origin in the checkout
directory, so the information can be recovered when running "freeze".

The use of this workaround, although useful, is fragile, creates confusion
about the purpose of the editable mode, and works only when the distribution
can be installed with setuptools (i.e. it is not usable with other :pep:`517`
build backends).

When this PEP is implemented, it will not be necessary anymore to use
editable installs for the purpose of making pip freeze work correctly with
VCS references.

Rationale
=========

This PEP specifies a new ``direct_url.json`` metadata file in the
``.dist-info`` directory of an installed distribution.

The fields specified are sufficient to reproduce the source archive and `VCS
URLs supported by pip`_. They are also sufficient to reproduce
:pep:`PEP440 Direct References <440#direct-references>`,
as well as `Pipfile and Pipfile.lock`_ entries. Finally, they
are sufficient to record the branch, tag, and/or Git ref origin of the
installed version that


id: pep-0348.rst
content:
PEP: 348
Title: Exception Reorganization for Python 3.0
Author: Brett Cannon <brett@python.org>
Status: Rejected
Type: Standards Track
Created: 28-Jul-2005
Post-History:

.. |2.x| replace:: 2.5

.. note:: This PEP has been rejected [#rejected]_.


Abstract
========

Python, as of version 2.4, has 38 exceptions (including warnings) in
the built-in namespace in a rather shallow hierarchy.  These
classes have come about over the years without a chance to learn from
experience.  This PEP proposes doing a reorganization of the hierarchy
for Python 3.0 when backwards-compatibility is not as much of an
issue.

Along with this reorganization, adding a requirement that all
objects passed to a ``raise`` statement must inherit from a specific
superclass is proposed.  This is to have guarantees about the basic
interface of exceptions and to further enhance the natural hierarchy
of exceptions.

Lastly, bare ``except`` clauses will be changed to be semantically
equivalent to ``except Exception``.  Most people currently use bare
``except`` clause for this purpose and with the exception hierarchy
reorganization becomes a viable default.


Rationale For Wanting Change
============================

Exceptions are a critical part of Python.  While exceptions are
traditionally used to signal errors in a program, they have also grown
to be used for flow control for things such as iterators.

While their importance is great, there is a lack of structure to them.
This stems from the fact that any object can be raised as an
exception.  Because of this you have no guarantee in terms of what
kind of object will be raised, destroying any possible hierarchy
raised objects might adhere to.

But exceptions do have a hierarchy, showing the severity of the
exception.  The hierarchy also groups related exceptions together to
simplify catching them in ``except`` clauses.  To allow people to
be able to rely on this hierarchy, a common superclass that all
raise objects must inherit from is being proposed.  It also allows
guarantees about the interface to raised objects to be made (see
:pep:`344`).  A discussion about all of this has occurred
before on python-dev [#Summary2004-08-01]_.

As bare ``except`` clauses stand now, they catch *all* exceptions.
While this can be handy, it is rather overreaching for the common
case.  Thanks to having a required superclass, catching all
exceptions is as easy as catching just one specific exception.
This allows bare ``except`` clauses to be used for a more useful
purpose.
Once again, this has been discussed on python-dev [#python-dev3]_.

Finally, slight changes to the exception hierarchy will make it much
more reasonable in terms of structure.  By minor rearranging
exceptions
that should not typically be caught can be allowed to propagate to the
top of the execution stack, terminating the interpreter as intended.


Philosophy of Reorganization
============================

For the reorganization of the hierarchy, there was a general
philosophy followed that developed from discussion of earlier drafts
of this PEP [#python-dev-thread1]_, [#python-dev-thread2]_,
[#python-dev-thread3]_, [#python-dev-thread4]_,
[#python-dev-thread5]_, [#python-dev-thread6]_.
First and foremost was to not break anything
that works.  This meant that renaming exceptions was out of the
question unless the name was deemed severely bad.  This
also meant no removal of exceptions unless they were viewed as
truly misplaced. The introduction of new exceptions were only done in
situations where there might be a use for catching a superclass of a
category of exceptions.  Lastly, existing exceptions would have their
inheritance tree changed only if it was felt they were truly
misplaced to begin with.

For all new exceptions, the proper suffix had to be chosen.  For
those that signal an error, "Error" is to be used.  If the exception
is a warning, then "Warning".  "Exception" is to be used when none
of the other suffixes are proper to use and no specific suffix is
a better fit.

After that it came down to choosing which exceptions should and
should not inherit from Exception.  This was for the purpose of
making bare ``except`` clauses more useful.

Lastly, the entire existing hierarchy had to inherit from the new
exception meant to act as the required superclass for all exceptions
to inherit from.


New Hierarchy
=============

.. Note:: Exceptions flagged with "stricter inheritance" will no
   longer inherit from a certain class.  A "broader inheritance" flag
   means a class has been added to the exception's inheritance tree.
   All comparisons are against the Python 2.4 exception hierarchy.

.. parsed-literal::

   +-- BaseException (new; broader inheritance for subclasses)
       +-- Exception
           +-- GeneratorExit (defined in :pep:`342`)
           +-- StandardError
               +-- ArithmeticError
                   +-- DivideByZeroError
                   +-- FloatingPointError
                   +-- OverflowError
               +-- AssertionError
               +-- AttributeError
               +-- EnvironmentError
                   +-- IOError
                   +-- EOFError
                   +-- OSError
               +-- ImportError
               +-- LookupError
                   +-- IndexError
                   +-- KeyError
               +-- MemoryError
               +-- NameError
                   +-- UnboundLocalError
               +-- NotImplementedError (stricter inheritance)
               +-- SyntaxError
                   +-- IndentationError
                       +-- TabError
               +-- TypeError
               +-- RuntimeError
               +-- UnicodeError
                   +-- UnicodeDecodeError
                   +-- UnicodeEncodeError
                   +-- UnicodeTranslateError
               +-- ValueError
               +-- ReferenceError
           +-- StopIteration
           +-- SystemError
           +-- Warning
               +-- DeprecationWarning
               +-- FutureWarning
               +-- PendingDeprecationWarning
               +-- RuntimeWarning
               +-- SyntaxWarning
               +-- UserWarning
           + -- WindowsError
       +-- KeyboardInterrupt (stricter inheritance)
       +-- SystemExit (stricter inheritance)


Differences Compared to Python 2.4
==================================

A more thorough explanation of terms is needed when discussing
inheritance changes.  Inheritance changes result in either broader or
more restrictive inheritance.  "Broader" is when a class has an
inheritance tree like ``cls, A`` and then becomes ``cls, B, A``.
"Stricter" is the reverse.


BaseException
-------------

The superclass that all exceptions must inherit from.  It's name was
chosen to reflect that it is at the base of the exception hierarchy
while being an exception itself.  "Raisable" was considered as a name,
it was passed on because its name did not properly reflect the fact
that it is an exception itself.

Direct inheritance of BaseException is not expected, and will
be discouraged for the general case.  Most user-defined
exceptions should inherit from Exception instead.  This allows
catching Exception to continue to work in the common case of catching
all exceptions that should be caught.  Direct inheritance of
BaseException should only be done in cases where an entirely new
category of exception is desired.

But, for cases where all
exceptions should be caught blindly, ``except BaseException`` will
work.


KeyboardInterrupt and SystemExit
--------------------------------

Both exceptions are no longer under Exception.  This is to allow bare
``except`` clauses to act as a more viable default case by catching
exceptions that inherit from Exception.  With both KeyboardInterrupt
and SystemExit acting as signals that the interpreter is expected to
exit, catching them in the common case is the wrong semantics.


NotImplementedError
-------------------

Inherits from Exception instead of from Runti


id: pep-0406.rst
content:
PEP: 406
Title: Improved Encapsulation of Import State
Author: Alyssa Coghlan <ncoghlan@gmail.com>, Greg Slodkowicz <jergosh@gmail.com>
Status: Withdrawn
Type: Standards Track
Created: 04-Jul-2011
Python-Version: 3.4
Post-History: 31-Jul-2011, 13-Nov-2011, 04-Dec-2011

Abstract
========

This PEP proposes the introduction of a new 'ImportEngine' class as part of
``importlib`` which would encapsulate all state related to importing modules
into a single object. Creating new instances of this object would then provide
an alternative to completely replacing the built-in implementation of the
import statement, by overriding the ``__import__()`` function. To work with
the builtin import functionality and importing via import engine objects,
this PEP proposes a context management based approach to temporarily replacing
the global import state.

The PEP also proposes inclusion of a ``GlobalImportEngine`` subclass and a
globally accessible instance of that class, which "writes through" to the
process global state. This provides a backwards compatible bridge between the
proposed encapsulated API and the legacy process global state, and allows
straightforward support for related state updates (e.g. selectively
invalidating path cache entries when ``sys.path`` is modified).


PEP Withdrawal
==============

The import system has seen substantial changes since this PEP was originally
written, as part of :pep:`420` in Python 3.3 and :pep:`451` in Python 3.4.

While providing an encapsulation of the import state is still highly
desirable, it is better tackled in a new PEP using :pep:`451` as a foundation,
and permitting only the use of :pep:`451` compatible finders and loaders (as
those avoid many of the issues of direct manipulation of global state
associated with the previous loader API).


Rationale
=========

Currently, most state related to the import system is stored as module level
attributes in the ``sys`` module. The one exception is the import lock, which
is not accessible directly, but only via the related functions in the ``imp``
module. The current process global import state comprises:

* sys.modules
* sys.path
* sys.path_hooks
* sys.meta_path
* sys.path_importer_cache
* the import lock (imp.lock_held()/acquire_lock()/release_lock())

Isolating this state would allow multiple import states to be
conveniently stored within a process. Placing the import functionality
in a self-contained object would also allow subclassing to add additional
features (e.g. module import notifications or fine-grained control
over which modules can be imported). The engine would also be
subclassed to make it possible to use the import engine API to
interact with the existing process-global state.

The namespace PEPs (especially :pep:`402`) raise a potential need for
*additional* process global state, in order to correctly update package paths
as ``sys.path`` is modified.

Finally, providing a coherent object for all this state makes it feasible to
also provide context management features that allow the import state to be
temporarily substituted.


Proposal
========

We propose introducing an ImportEngine class to encapsulate import
functionality. This includes an ``__import__()`` method which can
be used as an alternative to the built-in ``__import__()`` when
desired and also an ``import_module()`` method, equivalent to
``importlib.import_module()`` [3]_.

Since there are global import state invariants that are assumed and should be
maintained, we introduce a ``GlobalImportState`` class with an interface
identical to ``ImportEngine`` but directly accessing the current global import
state. This can be easily implemented using class properties.


Specification
=============

ImportEngine API
~~~~~~~~~~~~~~~~

The proposed extension consists of the following objects:

``importlib.engine.ImportEngine``

    ``from_engine(self, other)``

        Create a new import object from another ImportEngine instance. The
        new object is initialised with a copy of the state in ``other``. When
        called on ``importlib engine.sysengine``, ``from_engine()`` can be
        used to create an ``ImportEngine`` object with a **copy** of the
        global import state.

    ``__import__(self, name, globals={}, locals={}, fromlist=[], level=0)``

        Reimplementation of the builtin ``__import__()`` function. The
        import of a module will proceed using the state stored in the
        ImportEngine instance rather than the global import state. For full
        documentation of ``__import__`` functionality, see [2]_ .
        ``__import__()`` from ``ImportEngine`` and its subclasses can be used
        to customise the behaviour of the ``import`` statement by replacing
        ``__builtin__.__import__`` with ``ImportEngine().__import__``.

    ``import_module(name, package=None)``

        A reimplementation of ``importlib.import_module()`` which uses the
        import state stored in the ImportEngine instance. See [3]_ for a full
        reference.

    ``modules, path, path_hooks, meta_path, path_importer_cache``

        Instance-specific versions of their process global ``sys`` equivalents


``importlib.engine.GlobalImportEngine(ImportEngine)``

    Convenience class to provide engine-like access to the global state.
    Provides ``__import__()``, ``import_module()`` and ``from_engine()``
    methods like ``ImportEngine`` but writes through to the global state
    in ``sys``.

To support various namespace package mechanisms, when ``sys.path`` is altered,
tools like ``pkgutil.extend_path`` should be used to also modify other parts
of the import state (in this case, package ``__path__`` attributes). The path
importer cache should also be invalidated when a variety of changes are made.

The ``ImportEngine`` API will provide convenience methods that automatically
make related import state updates as part of a single operation.


Global variables
~~~~~~~~~~~~~~~~

``importlib.engine.sysengine``

    A precreated instance of ``GlobalImportEngine``. Intended for use by
    importers and loaders that have been updated to accept optional ``engine``
    parameters and with ``ImportEngine.from_engine(sysengine)`` to start with
    a copy of the process global import state.


No changes to finder/loader interfaces
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Rather than attempting to update the :pep:`302` APIs to accept additional state,
this PEP proposes that ``ImportEngine`` support the content management
protocol (similar to the context substitution mechanisms in the ``decimal``
module).

The context management mechanism for ``ImportEngine`` would:

* On entry:
  * Acquire the import lock
  * Substitute the global import state with the import engine's own state
* On exit:
  * Restore the previous global import state
  * Release the import lock

The precise API for this is TBD (but will probably use a distinct context
management object, along the lines of that created by
``decimal.localcontext``).


Open Issues
===========


API design for falling back to global import state
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The current proposal relies on the ``from_engine()`` API to fall back to the
global import state. It may be desirable to offer a variant that instead falls
back to the global import state dynamically.

However, one big advantage of starting with an "as isolated as possible"
design is that it becomes possible to experiment with subclasses that blur
the boundaries between the engine instance state and the process global state
in various ways.


Builtin and extension modules must be process global
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Due to platform limitations, only one copy of each builtin and extension
module can readily exist in each process. Accordingly, it is impossible for
each ``ImportEngine`` instance to load such modules independently.

The simplest solution is for ``ImportEngine`` to refuse to load such modules,
raising ``ImportError``. ``Globa


id: pep-0360.rst
content:
PEP: 360
Title: Externally Maintained Packages
Author: Brett Cannon <brett@python.org>
Status: Final
Type: Process
Created: 30-May-2006
Post-History:


.. warning:: No new modules are to be added to this PEP.  It has been
             deemed dangerous to codify external maintenance of any
             code checked into Python's code repository.  Code
             contributors should expect Python's development
             methodology to be used for any and all code checked into
             Python's code repository.

Abstract
========

There are many great pieces of Python software developed outside of
the Python standard library (a.k.a., the "stdlib").  Sometimes it
makes sense to incorporate these externally maintained packages into
the stdlib in order to fill a gap in the tools provided by Python.

But by having the packages maintained externally it means Python's
developers do not have direct control over the packages' evolution and
maintenance.  Some package developers prefer to have bug reports and
patches go through them first instead of being directly applied to
Python's repository.

This PEP is meant to record details of packages in the stdlib that are
maintained outside of Python's repository.  Specifically, it is meant
to keep track of any specific maintenance needs for each package.  It
should be mentioned that changes needed in order to fix bugs and keep
the code running on all of Python's supported platforms will be done
directly in Python's repository without worrying about going through
the contact developer.  This is so that Python itself is not held up
by a single bug and allows the whole process to scale as needed.

It also is meant to allow people to know which version of a package is
released with which version of Python.


Externally Maintained Packages
==============================

The section title is the name of the package as it is known outside of
the Python standard library.  The "standard library name" is what the
package is named within Python.  The "contact person" is the Python
developer in charge of maintaining the package.  The "synchronisation
history" lists what external version of the package was included in
each version of Python (if different from the previous Python
release).


ElementTree
-----------

:Web site:
    http://effbot.org/zone/element-index.htm
:Standard library name:
    xml.etree
:Contact person:
    Fredrik Lundh

Fredrik has ceded ElementTree maintenance to the core Python development
team [#element-tree]_.

Expat XML parser
----------------

:Web site:
    http://www.libexpat.org/
:Standard library name:
    N/A (this refers to the parser itself, and not the Python
    bindings)
:Contact person:
    None


Optik
-----

:Web site:
    http://optik.sourceforge.net/
:Standard library name:
    optparse
:Contact person:
    Greg Ward

External development seems to have ceased. For new applications, optparse
itself has been largely superseded by argparse.


wsgiref
-------
:Web site:
    None
:Standard library name:
    wsgiref
:Contact Person:
    Phillip J. Eby

This module is maintained in the standard library, but significant bug
reports and patches should pass through the Web-SIG mailing list
[#web-sig]_ for discussion.


References
==========

.. [#element-tree] Fredrik's handing over of ElementTree
   (https://mail.python.org/pipermail/python-dev/2012-February/116389.html)

.. [#web-sig] Web-SIG mailing list
   (https://mail.python.org/mailman/listinfo/web-sig)


Copyright
=========

This document has been placed in the public domain.


id: pep-0374.rst
content:
PEP: 374
Title: Choosing a distributed VCS for the Python project
Author: Brett Cannon <brett@python.org>,
        Stephen J. Turnbull <stephen@xemacs.org>,
        Alexandre Vassalotti <alexandre@peadrop.com>,
        Barry Warsaw <barry@python.org>,
        Dirkjan Ochtman <dirkjan@ochtman.nl>
Status: Final
Type: Process
Created: 07-Nov-2008
Post-History: 07-Nov-2008,
              22-Jan-2009


Rationale
=========

Python has been using a centralized version control system (VCS;
first CVS, now Subversion) for years to great effect. Having a master
copy of the official version of Python provides people with a single
place to always get the official Python source code. It has also
allowed for the storage of the history of the language, mostly for
help with development, but also for posterity. And of course the V in
VCS is very helpful when developing.

But a centralized version control system has its drawbacks. First and
foremost, in order to have the benefits of version control with
Python in a seamless fashion, one must be a "core developer" (i.e.
someone with commit privileges on the master copy of Python). People
who are not core developers but who wish to work with Python's
revision tree, e.g. anyone writing a patch for Python or creating a
custom version, do not have direct tool support for revisions. This
can be quite a limitation, since these non-core developers cannot
easily do basic tasks such as reverting changes to a previously
saved state, creating branches, publishing one's changes with full
revision history, etc. For non-core developers, the last safe tree
state is one the Python developers happen to set, and this prevents
safe development. This second-class citizenship is a hindrance to
people who wish to contribute to Python with a patch of any
complexity and want a way to incrementally save their progress to
make their development lives easier.

There is also the issue of having to be online to be able to commit
one's work. Because centralized VCSs keep a central copy that stores
all revisions, one must have Internet access in order for their
revisions to be stored; no Net, no commit. This can be annoying if
you happen to be traveling and lack any Internet. There is also the
situation of someone wishing to contribute to Python but having a
bad Internet connection where committing is time-consuming and
expensive and it might work out better to do it in a single step.

Another drawback to a centralized VCS is that a common use case is
for a developer to revise patches in response to review comments.
This is more difficult with a centralized model because there's no
place to contain intermediate work. It's either all checked in or
none of it is checked in. In the centralized VCS, it's also very
difficult to track changes to the trunk as they are committed, while
you're working on your feature or bug fix branch. This increases
the risk that such branches will grow stale, out-dated, or that
merging them into the trunk will generate too may conflicts to be
easily resolved.

Lastly, there is the issue of maintenance of Python. At any one time
there is at least one major version of Python under development (at
the time of this writing there are two). For each major version of
Python under development there is at least the maintenance version
of the last minor version and the in-development minor version (e.g.
with 2.6 just released, that means that both 2.6 and 2.7 are being
worked on). Once a release is done, a branch is created between the
code bases where changes in one version do not (but could) belong in
the other version. As of right now there is no natural support for
this branch in time in central VCSs; you must use tools that
simulate the branching. Tracking merges is similarly painful for
developers, as revisions often need to be merged between four active
branches (e.g. 2.6 maintenance, 3.0 maintenance, 2.7 development,
3.1 development). In this case, VCSs such as Subversion only handle
this through arcane third party tools.

Distributed VCSs (DVCSs) solve all of these problems. While one can
keep a master copy of a revision tree, anyone is free to copy that
tree for their own use. This gives everyone the power to commit
changes to their copy, online or offline. It also more naturally
ties into the idea of branching in the history of a revision tree
for maintenance and the development of new features bound for
Python. DVCSs also provide a great many additional features that
centralized VCSs don't or can't provide.

This PEP explores the possibility of changing Python's use of Subversion
to any of the currently popular  DVCSs, in order to gain
the benefits outlined above. This PEP does not guarantee that a switch
to a DVCS will occur at the conclusion of this PEP. It is quite
possible that no clear winner will be found and that svn will continue
to be used. If this happens, this PEP will be revisited and revised in
the future as the state of DVCSs evolves.


Terminology
===========

Agreeing on a common terminology is surprisingly difficult,
primarily because each VCS uses these terms when describing subtly
different tasks, objects, and concepts. Where possible, we try to
provide a generic definition of the concepts, but you should consult
the individual system's glossaries for details. Here are some basic
references for terminology, from some of the standard web-based
references on each VCS. You can also refer to glossaries for each
DVCS:

* Subversion : http://svnbook.red-bean.com/en/1.5/svn.basic.html
* Bazaar : http://bazaar-vcs.org/BzrGlossary
* Mercurial : http://www.selenic.com/mercurial/wiki/index.cgi/UnderstandingMercurial
* git : http://book.git-scm.com/1_the_git_object_model.html


branch
    A line of development; a collection of revisions, ordered by
    time.

checkout/working copy/working tree
    A tree of code the developer can edit, linked to a branch.

index
    A "staging area" where a revision is built (unique to git).

repository
    A collection of revisions, organized into branches.

clone
    A complete copy of a branch or repository.

commit
    To record a revision in a repository.

merge
    Applying all the changes and history from one branch/repository
    to another.

pull
    To update a checkout/clone from the original branch/repository,
    which can be remote or local

push/publish
    To copy a revision, and all revisions it depends on, from a one
    repository to another.

cherry-pick
    To merge one or more specific revisions from one branch to
    another, possibly in a different repository, possibly without its
    dependent revisions.

rebase
    To "detach" a branch, and move it to a new branch point; move
    commits to the beginning of a branch instead of where they
    happened in time.


Typical Workflow
================

At the moment, the typical workflow for a Python core developer is:


* Edit code in a checkout until it is stable enough to commit/push.
* Commit to the master repository.

It is a rather simple workflow, but it has drawbacks. For one,
because any work that involves the repository takes time thanks to
the network, commits/pushes tend to not necessarily be as atomic as
possible. There is also the drawback of there not being a
necessarily cheap way to create new checkouts beyond a recursive
copy of the checkout directory.

A DVCS would lead to a workflow more like this:

* Branch off of a local clone of the master repository.
* Edit code, committing in atomic pieces.
* Merge the branch into the mainline, and
* Push all commits to the master repository.

While there are more possible steps, the workflow is much more
independent of the master repository than is currently possible. By
being able to commit locally at the speed of your disk, a core
developer is able to do atomic commits much more frequently,
minimizing having commits that do multiple things to the code. Also
by using a branch, the changes are isolated (if desired) from other
changes being made by other


id: pep-0412.rst
content:
PEP: 412
Title: Key-Sharing Dictionary
Author: Mark Shannon <mark@hotpy.org>
Status: Final
Type: Standards Track
Created: 08-Feb-2012
Python-Version: 3.3
Post-History: 08-Feb-2012


Abstract
========

This PEP proposes a change in the implementation of the builtin
dictionary type ``dict``.  The new implementation allows dictionaries
which are used as attribute dictionaries (the ``__dict__`` attribute
of an object) to share keys with other attribute dictionaries of
instances of the same class.

Motivation
==========

The current dictionary implementation uses more memory than is
necessary when used as a container for object attributes as the keys
are replicated for each instance rather than being shared across many
instances of the same class.  Despite this, the current dictionary
implementation is finely tuned and performs very well as a
general-purpose mapping object.

By separating the keys (and hashes) from the values it is possible to
share the keys between multiple dictionaries and improve memory use.
By ensuring that keys are separated from the values only when
beneficial, it is possible to retain the high-performance of the
current dictionary implementation when used as a general-purpose
mapping object.

Behaviour
=========

The new dictionary behaves in the same way as the old implementation.
It fully conforms to the Python API, the C API and the ABI.

Performance
===========

Memory Usage
------------

Reduction in memory use is directly related to the number of
dictionaries with shared keys in existence at any time.  These
dictionaries are typically half the size of the current dictionary
implementation.

Benchmarking shows that memory use is reduced by 10% to 20% for
object-oriented programs with no significant change in memory use for
other programs.

Speed
-----

The performance of the new implementation is dominated by memory
locality effects.  When keys are not shared (for example in module
dictionaries and dictionary explicitly created by ``dict()`` or
``{}``) then performance is unchanged (within a percent or two) from
the current implementation.

For the shared keys case, the new implementation tends to separate
keys from values, but reduces total memory usage.  This will improve
performance in many cases as the effects of reduced memory usage
outweigh the loss of locality, but some programs may show a small slow
down.

Benchmarking shows no significant change of speed for most benchmarks.
Object-oriented benchmarks show small speed ups when they create large
numbers of objects of the same class (the gcbench benchmark shows a
10% speed up; this is likely to be an upper limit).

Implementation
==============

Both the old and new dictionaries consist of a fixed-sized dict struct
and a re-sizeable table.  In the new dictionary the table can be
further split into a keys table and values array.  The keys table
holds the keys and hashes and (for non-split tables) the values as
well.  It differs only from the original implementation in that it
contains a number of fields that were previously in the dict struct.
If a table is split the values in the keys table are ignored, instead
the values are held in a separate array.

Split-Table dictionaries
------------------------

When dictionaries are created to fill the __dict__ slot of an object,
they are created in split form.  The keys table is cached in the type,
potentially allowing all attribute dictionaries of instances of one
class to share keys.  In the event of the keys of these dictionaries
starting to diverge, individual dictionaries will lazily convert to
the combined-table form.  This ensures good memory use in the common
case, and correctness in all cases.

When resizing a split dictionary it is converted to a combined table.
If resizing is as a result of storing an instance attribute, and there
is only instance of a class, then the dictionary will be re-split
immediately.  Since most OO code will set attributes in the __init__
method, all attributes will be set before a second instance is created
and no more resizing will be necessary as all further instance
dictionaries will have the correct size.  For more complex use
patterns, it is impossible to know what is the best approach, so the
implementation allows extra insertions up to the point of a resize
when it reverts to the combined table (non-shared keys).

A deletion from a split dictionary does not change the keys table, it
simply removes the value from the values array.

Combined-Table dictionaries
---------------------------

Explicit dictionaries (``dict()`` or ``{}``), module dictionaries and
most other dictionaries are created as combined-table dictionaries.  A
combined-table dictionary never becomes a split-table dictionary.
Combined tables are laid out in much the same way as the tables in the
old dictionary, resulting in very similar performance.

Implementation
==============

The new dictionary implementation is available at [1]_.

Pros and Cons
=============

Pros
----

Significant memory savings for object-oriented applications.  Small
improvement to speed for programs which create lots of similar
objects.

Cons
----

Change to data structures: Third party modules which meddle with the
internals of the dictionary implementation will break.

Changes to repr() output and iteration order: For most cases, this
will be unchanged.  However, for some split-table dictionaries the
iteration order will change.

Neither of these cons should be a problem.  Modules which meddle with
the internals of the dictionary implementation are already broken and
should be fixed to use the API.  The iteration order of dictionaries
was never defined and has always been arbitrary; it is different for
Jython and PyPy.

Alternative Implementation
--------------------------

An alternative implementation for split tables, which could save even
more memory, is to store an index in the value field of the keys table
(instead of ignoring the value field).  This index would explicitly
state where in the value array to look.  The value array would then
only require 1 field for each usable slot in the key table, rather
than each slot in the key table.

This "indexed" version would reduce the size of value array by about
one third. The keys table would need an extra "values_size" field,
increasing the size of combined dicts by one word.  The extra
indirection adds more complexity to the code, potentially reducing
performance a little.

The "indexed" version will not be included in this implementation, but
should be considered deferred rather than rejected, pending further
experimentation.

References
==========

.. [1] Reference Implementation:
   https://bitbucket.org/markshannon/cpython_new_dict

Copyright
=========

This document has been placed in the public domain.


id: pep-0375.rst
content:
PEP: 375
Title: Python 3.1 Release Schedule
Author: Benjamin Peterson <benjamin@python.org>
Status: Final
Type: Informational
Topic: Release
Created: 08-Feb-2009
Python-Version: 3.1


Abstract
========

This document describes the development and release schedule for Python 3.1.
The schedule primarily concerns itself with PEP-sized items.  Small features may
be added up to and including the first beta release.  Bugs may be fixed until
the final release.


Release Manager and Crew
========================

=================== ==================
Position            Name
=================== ==================
3.1 Release Manager Benjamin Peterson
Windows installers  Martin v. Loewis
Mac installers      Ronald Oussoren
=================== ==================


Release Schedule
================

- 3.1a1 March 7, 2009
- 3.1a2 April 4, 2009
- 3.1b1 May 6, 2009
- 3.1rc1 May 30, 2009
- 3.1rc2 June 13, 2009
- 3.1 final June 27, 2009

Maintenance Releases
====================

3.1 is no longer maintained. 3.1 received security fixes until June
2012.

Previous maintenance releases are:

- v3.1.1rc1 2009-08-13
- v3.1.1 2009-08-16
- v3.1.2rc1 2010-03-06
- v3.1.2 2010-03-20
- v3.1.3rc1 2010-11-13
- v3.1.3 2010-11-27
- v3.1.4rc1 2011-05-29
- v3.1.4 2011-06-11
- v3.1.5rc1 2012-02-23
- v3.1.5rc2 2012-03-15
- v3.1.5 2012-04-06

Features for 3.1
================

- importlib
- io in C
- Update simplejson to the latest external version [#simplejson]_.
- Ordered dictionary for collections (:pep:`372`).
- auto-numbered replacement fields in str.format() strings [#strformat]_
- Nested with-statements in one with statement


Footnotes
=========

.. [#simplejson]
   http://bugs.python.org/issue4136

.. [#strformat]
   http://bugs.python.org/issue5237



Copyright
=========

This document has been placed in the public domain.


id: pep-0413.rst
content:
PEP: 413
Title: Faster evolution of the Python Standard Library
Author: Alyssa Coghlan <ncoghlan@gmail.com>
Status: Withdrawn
Type: Process
Created: 24-Feb-2012
Post-History: 24-Feb-2012, 25-Feb-2012


PEP Withdrawal
==============

With the acceptance of :pep:`453` meaning that ``pip`` will be available to
most new Python users by default, this will hopefully reduce the pressure
to add new modules to the standard library before they are sufficiently
mature.

The last couple of years have also seen increased usage of the model where
a standard library package also has an equivalent available from the Python
Package Index that also supports older versions of Python.

Given these two developments and the level of engagement throughout the
Python 3.4 release cycle, the PEP author no longer feels it would be
appropriate to make such a fundamental change to the standard library
development process.

Abstract
========

This PEP proposes the adoption of a separate versioning scheme for the
standard library (distinct from, but coupled to, the existing language
versioning scheme) that allows accelerated releases of the Python standard
library, while maintaining (or even slowing down) the current rate of
change in the core language definition.

Like :pep:`407`, it aims to adjust the current balance between measured
change that allows the broader community time to adapt and being able to
keep pace with external influences that evolve more rapidly than the current
release cycle can handle (this problem is particularly notable for
standard library elements that relate to web technologies).

However, it's more conservative in its aims than :pep:`407`, seeking to
restrict the increased pace of development to builtin and standard library
interfaces, without affecting the rate of change for other elements such
as the language syntax and version numbering as well as the CPython
binary API and bytecode format.


Rationale
=========

To quote the :pep:`407` abstract:

    Finding a release cycle for an open-source project is a delicate exercise
    in managing mutually contradicting constraints: developer manpower,
    availability of release management volunteers, ease of maintenance for
    users and third-party packagers, quick availability of new features (and
    behavioural changes), availability of bug fixes without pulling in new
    features or behavioural changes.

    The current release cycle errs on the conservative side. It is adequate
    for people who value stability over reactivity. This PEP is an attempt to
    keep the stability that has become a Python trademark, while offering a
    more fluid release of features, by introducing the notion of long-term
    support versions.

I agree with the :pep:`407` authors that the current release cycle of the
*standard library* is too slow to effectively cope with the pace of change
in some key programming areas (specifically, web protocols and related
technologies, including databases, templating and serialisation formats).

However, I have written this competing PEP because I believe that the
approach proposed in :pep:`407` of offering full, potentially binary
incompatible releases of CPython every 6 months places too great a burden
on the wider Python ecosystem.

Under the current CPython release cycle, distributors of key binary
extensions will often support Python releases even after the CPython branches
enter "security fix only" mode (for example, Twisted currently ships binaries
for 2.5, 2.6 and 2.7, NumPy and SciPy support those 3 along with 3.1 and 3.2,
PyGame adds a 2.4 binary release, wxPython provides both 32-bit and 64-bit
binaries for 2.6 and 2.7, etc).

If CPython were to triple (or more) its rate of releases, the developers of
those libraries (many of which are even more resource starved than CPython)
would face an unpalatable choice: either adopt the faster release cycle
themselves (up to 18 simultaneous binary releases for PyGame!), drop
older Python versions more quickly, or else tell their users to stick to the
CPython LTS releases (thus defeating the entire point of speeding up the
CPython release cycle in the first place).

Similarly, many support tools for Python (e.g. syntax highlighters) can take
quite some time to catch up with language level changes.

At a cultural level, the Python community is also accustomed to a certain
meaning for Python version numbers - they're linked to deprecation periods,
support periods, all sorts of things. :pep:`407` proposes that collective
knowledge all be swept aside, without offering a compelling rationale for why
such a course of action is actually *necessary* (aside from, perhaps, making
the lives of the CPython core developers a little easier at the expense of
everyone else).

However, if we go back to the primary rationale for increasing the pace of
change (i.e. more timely support for web protocols and related technologies),
we can note that those only require *standard library* changes. That means
many (perhaps even most) of the negative effects on the wider community can
be avoided by explicitly limiting which parts of CPython are affected by the
new release cycle, and allowing other parts to evolve at their current, more
sedate, pace.


Proposal
========

This PEP proposes the introduction of a new kind of CPython release:
"standard library releases". As with :pep:`407`, this will give CPython 3 kinds
of release:

* Language release: "x.y.0"
* Maintenance release: "x.y.z" (where z > 0)
* Standard library release: "x.y (xy.z)" (where z > 0)

Under this scheme, an unqualified version reference (such as "3.3") would
always refer to the most recent corresponding language or maintenance
release. It will never be used without qualification to refer to a standard
library release (at least, not by python-dev - obviously, we can only set an
example, not force the rest of the Python ecosystem to go along with it).

Language releases will continue as they are now, as new versions of the
Python language definition, along with a new version of the CPython
interpreter and the Python standard library. Accordingly, a language
release may contain any and all of the following changes:

* new language syntax
* new standard library changes (see below)
* new deprecation warnings
* removal of previously deprecated features
* changes to the emitted bytecode
* changes to the AST
* any other significant changes to the compilation toolchain
* changes to the core interpreter eval loop
* binary incompatible changes to the C ABI (although the :pep:`384` stable ABI
  must still be preserved)
* bug fixes

Maintenance releases will also continue as they do today, being strictly
limited to bug fixes for the corresponding language release. No new features
or radical internal changes are permitted.

The new standard library releases will occur in parallel with each
maintenance release and will be qualified with a new version identifier
documenting the standard library version. Standard library releases may
include the following changes:

* new features in pure Python modules
* new features in C extension modules (subject to :pep:`399` compatibility
  requirements)
* new features in language builtins (provided the C ABI remains unaffected)
* bug fixes from the corresponding maintenance release

Standard library version identifiers are constructed by combining the major
and minor version numbers for the Python language release into a single two
digit number and then appending a sequential standard library version
identifier.


Release Cycle
-------------

When maintenance releases are created, *two* new versions of Python would
actually be published on python.org (using the first 3.3 maintenance release,
planned for February 2013 as an example)::

    3.3.1       # Maintenance release
    3.3 (33.1)  # Standard library release

A further 6 months later, the next 3.3 maintenance release would again be
accompanied by a new standard library release::

    3.3.2       # Maintenanc


id: pep-0407.rst
content:
PEP: 407
Title: New release cycle and introducing long-term support versions
Author: Antoine Pitrou <solipsis@pitrou.net>,
        Georg Brandl <georg@python.org>,
        Barry Warsaw <barry@python.org>
Status: Deferred
Type: Process
Created: 12-Jan-2012
Post-History: 17-Jan-2012


Abstract
========

Finding a release cycle for an open-source project is a delicate
exercise in managing mutually contradicting constraints: developer
manpower, availability of release management volunteers, ease of
maintenance for users and third-party packagers, quick availability of
new features (and behavioural changes), availability of bug fixes
without pulling in new features or behavioural changes.

The current release cycle errs on the conservative side.  It is
adequate for people who value stability over reactivity.  This PEP is
an attempt to keep the stability that has become a Python trademark,
while offering a more fluid release of features, by introducing the
notion of long-term support versions.


Scope
=====

This PEP doesn't try to change the maintenance period or release
scheme for the 2.7 branch.  Only 3.x versions are considered.


Proposal
========

Under the proposed scheme, there would be two kinds of feature
versions (sometimes dubbed "minor versions", for example 3.2 or 3.3):
normal feature versions and long-term support (LTS) versions.

Normal feature versions would get either zero or at most one bugfix
release; the latter only if needed to fix critical issues.  Security
fix handling for these branches needs to be decided.

LTS versions would get regular bugfix releases until the next LTS
version is out.  They then would go into security fixes mode, up to a
termination date at the release manager's discretion.

Periodicity
-----------

A new feature version would be released every X months.  We
tentatively propose X = 6 months.

LTS versions would be one out of N feature versions.  We tentatively
propose N = 4.

With these figures, a new LTS version would be out every 24 months,
and remain supported until the next LTS version 24 months later.  This
is mildly similar to today's 18 months bugfix cycle for every feature
version.

Pre-release versions
--------------------

More frequent feature releases imply a smaller number of disruptive
changes per release.  Therefore, the number of pre-release builds
(alphas and betas) can be brought down considerably.  Two alpha builds
and a single beta build would probably be enough in the regular case.
The number of release candidates depends, as usual, on the number of
last-minute fixes before final release.


Effects
=======

Effect on development cycle
---------------------------

More feature releases might mean more stress on the development and
release management teams.  This is quantitatively alleviated by the
smaller number of pre-release versions; and qualitatively by the
lesser amount of disruptive changes (meaning less potential for
breakage).  The shorter feature freeze period (after the first beta
build until the final release) is easier to accept.  The rush for
adding features just before feature freeze should also be much
smaller.

Effect on bugfix cycle
----------------------

The effect on fixing bugs should be minimal with the proposed figures.
The same number of branches would be simultaneously open for bugfix
maintenance (two until 2.x is terminated, then one).

Effect on workflow
------------------

The workflow for new features would be the same: developers would only
commit them on the ``default`` branch.

The workflow for bug fixes would be slightly updated: developers would
commit bug fixes to the current LTS branch (for example ``3.3``) and
then merge them into ``default``.

If some critical fixes are needed to a non-LTS version, they can be
grafted from the current LTS branch to the non-LTS branch, just like
fixes are ported from 3.x to 2.7 today.

Effect on the community
-----------------------

People who value stability can just synchronize on the LTS releases
which, with the proposed figures, would give a similar support cycle
(both in duration and in stability).

People who value reactivity and access to new features (without taking
the risk to install alpha versions or Mercurial snapshots) would get
much more value from the new release cycle than currently.

People who want to contribute new features or improvements would be
more motivated to do so, knowing that their contributions will be more
quickly available to normal users.  Also, a smaller feature freeze
period makes it less cumbersome to interact with contributors of
features.


Discussion
==========

These are open issues that should be worked out during discussion:

* Decide on X (months between feature releases) and N (feature releases
  per LTS release) as defined above.

* For given values of X and N, is the no-bugfix-releases policy for
  non-LTS versions feasible?

* What is the policy for security fixes?

* Restrict new syntax and similar changes (i.e. everything that was
  prohibited by :pep:`3003`) to LTS versions?

* What is the effect on packagers such as Linux distributions?

* How will release version numbers or other identifying and marketing
  material make it clear to users which versions are normal feature
  releases and which are LTS releases?  How do we manage user
  expectations?

* Does the faster release cycle mean we could some day reach 3.10 and
  above?  Some people expressed a tacit expectation that version numbers
  always fit in one decimal digit.

A community poll or survey to collect opinions from the greater Python
community would be valuable before making a final decision.


Copyright
=========

This document has been placed in the public domain.


id: pep-0361.rst
content:
PEP: 361
Title: Python 2.6 and 3.0 Release Schedule
Author: Neal Norwitz, Barry Warsaw
Status: Final
Type: Informational
Topic: Release
Created: 29-Jun-2006
Python-Version: 2.6, 3.0
Post-History: 17-Mar-2008


Abstract
========

This document describes the development and release schedule for
Python 2.6 and 3.0.  The schedule primarily concerns itself with
PEP-sized items.  Small features may be added up to and including
the first beta release.  Bugs may be fixed until the final
release.

There will be at least two alpha releases, two beta releases, and
one release candidate.  The releases are planned for October 2008.

Python 2.6 is not only the next advancement in the Python 2
series, it is also a transitional release, helping developers
begin to prepare their code for Python 3.0.  As such, many
features are being backported from Python 3.0 to 2.6.  Thus, it
makes sense to release both versions in at the same time.  The
precedence for this was set with the Python 1.6 and 2.0 release.

Until rc, we will be releasing Python 2.6 and 3.0 in lockstep, on
a monthly release cycle.  The releases will happen on the first
Wednesday of every month through the beta testing cycle.  Because
Python 2.6 is ready sooner, and because we have outside deadlines
we'd like to meet, we've decided to split the rc releases.  Thus
Python 2.6 final is currently planned to come out two weeks before
Python 3.0 final.


Release Manager and Crew
========================

- 2.6/3.0 Release Manager: Barry Warsaw
- Windows installers: Martin v. Loewis
- Mac installers: Ronald Oussoren
- Documentation: Georg Brandl
- RPMs: Sean Reifschneider


Release Lifespan
================

Python 3.0 is no longer being maintained for any purpose.

Python 2.6.9 is the final security-only source-only maintenance
release of the Python 2.6 series.  With its release on October 29,
2013, all official support for Python 2.6 has ended.  Python 2.6
is no longer being maintained for any purpose.


Release Schedule
================

- Feb 29 2008: Python 2.6a1 and 3.0a3 are released
- Apr 02 2008: Python 2.6a2 and 3.0a4 are released
- May 08 2008: Python 2.6a3 and 3.0a5 are released
- Jun 18 2008: Python 2.6b1 and 3.0b1 are released
- Jul 17 2008: Python 2.6b2 and 3.0b2 are released
- Aug 20 2008: Python 2.6b3 and 3.0b3 are released
- Sep 12 2008: Python 2.6rc1 is released
- Sep 17 2008: Python 2.6rc2 and 3.0rc1 released
- Oct 01 2008: Python 2.6 final released
- Nov 06 2008: Python 3.0rc2 released
- Nov 21 2008: Python 3.0rc3 released
- Dec 03 2008: Python 3.0 final released
- Dec 04 2008: Python 2.6.1 final released
- Apr 14 2009: Python 2.6.2 final released
- Oct 02 2009: Python 2.6.3 final released
- Oct 25 2009: Python 2.6.4 final released
- Mar 19 2010: Python 2.6.5 final released
- Aug 24 2010: Python 2.6.6 final released
- Jun 03 2011: Python 2.6.7 final released (security-only)
- Apr 10 2012: Python 2.6.8 final released (security-only)
- Oct 29 2013: Python 2.6.9 final released (security-only)


Completed features for 3.0
==========================

See :pep:`3000` and :pep:`3100` for details on the
Python 3.0 project.


Completed features for 2.6
==========================

PEPs:

- :pep:`352`: Raising a string exception now triggers a ``TypeError``.
  Attempting to catch a string exception raises ``DeprecationWarning``.
  ``BaseException.message`` has been deprecated.
- :pep:`358`: The "bytes" Object
- :pep:`366`: Main module explicit relative imports
- :pep:`370`: Per user site-packages directory
- :pep:`3112`: Bytes literals in Python 3000
- :pep:`3127`: Integer Literal Support and Syntax
- :pep:`371`: Addition of the multiprocessing package

New modules in the standard library:

- ``json``
- new enhanced ``turtle`` module
- ``ast``

Deprecated modules and functions in the standard library:

- ``buildtools``
- ``cfmfile``
- ``commands.getstatus()``
- ``macostools.touched()``
- ``md5``
- ``MimeWriter``
- ``mimify``
- ``popen2``, ``os.popen[234]()``
- ``posixfile``
- ``sets``
- ``sha``

Modules removed from the standard library:

- ``gopherlib``
- ``rgbimg``
- ``macfs``

Warnings for features removed in Py3k:

- builtins: ``apply``, ``callable``, ``coerce``, ``dict.has_key``, ``execfile``,
  ``reduce``, ``reload``
- backticks and ``<>``
- float args to ``xrange``
- ``coerce`` and all its friends
- comparing by default comparison
- ``{}.has_key()``
- ``file.xreadlines``
- softspace removal for ``print()`` function
- removal of modules because of :pep:`4`/:pep:`3100`/:pep:`3108`

Other major features:

- ``with``/``as`` will be keywords
- a ``__dir__()`` special method to control ``dir()`` was added [1]_
- AtheOS support stopped.
- ``warnings`` module implemented in C
- ``compile()`` takes an AST and can convert to byte code


Possible features for 2.6
=========================

New features *should* be implemented prior to alpha2, particularly
any C modifications or behavioral changes.  New features *must* be
implemented prior to beta1 or will require Release Manager approval.

The following PEPs are being worked on for inclusion in 2.6: None.

Each non-trivial feature listed here that is not a PEP must be
discussed on python-dev.  Other enhancements include:

- ``distutils`` replacement (requires a PEP)

New modules in the standard library:

- ``winerror``
  https://bugs.python.org/issue1505257
  (Patch rejected, module should be written in C)

- ``setuptools``
  BDFL pronouncement for inclusion in 2.5:
  https://mail.python.org/pipermail/python-dev/2006-April/063964.html

PJE's withdrawal from 2.5 for inclusion in 2.6:
https://mail.python.org/pipermail/python-dev/2006-April/064145.html

Modules to gain a DeprecationWarning (as specified for Python 2.6
or through negligence):

- ``rfc822``
- ``mimetools``
- ``multifile``
- ``compiler`` package (or a Py3K warning instead?)

- Convert ``Parser/*.c`` to use the C ``warnings`` module rather than ``printf``

- Add warnings for Py3k features removed:

  * ``__getslice__``/``__setslice__``/``__delslice__``

  * float args to ``PyArgs_ParseTuple``

  * ``__cmp__``?

  * other comparison changes?

  * int division?

  * All ``PendingDeprecationWarnings`` (e.g. exceptions)

  * using ``zip()`` result as a list

  * the ``exec`` statement (use function syntax)

  * function attributes that start with ``func_*`` (should use ``__*__``)

  * the ``L`` suffix for long literals

  * renaming of ``__nonzero__`` to ``__bool__``

  * multiple inheritance with classic classes? (MRO might change)

  * properties and classic classes? (instance attrs shadow property)

- use ``__bool__`` method if available and there's no ``__nonzero__``

- Check the various bits of code in ``Demo/`` and ``Tools/`` all still work,
  update or remove the ones that don't.

- All modules in ``Modules/`` should be updated to be ``ssize_t`` clean.

- All of Python (including ``Modules/``) should compile cleanly with g++

- Start removing deprecated features and generally moving towards Py3k

- Replace all old style tests (operate on import) with ``unittest`` or ``docttest``

- Add tests for all untested modules

- Document undocumented modules/features

- ``bdist_deb`` in ``distutils`` package
  https://mail.python.org/pipermail/python-dev/2006-February/060926.html

- ``bdist_egg`` in ``distutils`` package

- pure python ``pgen`` module
  (Owner: Guido)
  Deferral to 2.6:
  https://mail.python.org/pipermail/python-dev/2006-April/064528.html

- Remove the ``fpectl`` module?


Deferred until 2.7
==================

None


Open issues
===========

How should import warnings be handled?

- https://mail.python.org/pipermail/python-dev/2006-June/066345.html
- https://bugs.python.org/issue1515609
- https://bugs.python.org/issue1515361


References
==========

.. [1] Adding a __dir__() magic method
       https://mail.python.org/pipermail/python-dev/2006-July/067139.html

.. _Google calendar: http://www.google.com/calendar/ical/b6v58qvojllt0i6ql654r1vh00%40group.calendar.google.com/public/basic.


id: pep-0349.rst
content:
PEP: 349
Title: Allow str() to return unicode strings
Author: Neil Schemenauer <nas@arctrix.com>
Status: Rejected
Type: Standards Track
Created: 02-Aug-2005
Python-Version: 2.5
Post-History: 06-Aug-2005
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/M2Y3PUFLAE23NPRJPVBYF6P5LW5LVN6F/


Abstract
========

This PEP proposes to change the ``str()`` built-in function so that it
can return unicode strings.  This change would make it easier to
write code that works with either string type and would also make
some existing code handle unicode strings.  The C function
``PyObject_Str()`` would remain unchanged and the function
``PyString_New()`` would be added instead.


Rationale
=========

Python has had a Unicode string type for some time now but use of
it is not yet widespread.  There is a large amount of Python code
that assumes that string data is represented as str instances.
The long-term plan for Python is to phase out the str type and use
unicode for all string data.  Clearly, a smooth migration path
must be provided.

We need to upgrade existing libraries, written for str instances,
to be made capable of operating in an all-unicode string world.
We can't change to an all-unicode world until all essential
libraries are made capable for it.  Upgrading the libraries in one
shot does not seem feasible.  A more realistic strategy is to
individually make the libraries capable of operating on unicode
strings while preserving their current all-str environment
behaviour.

First, we need to be able to write code that can accept unicode
instances without attempting to coerce them to str instances.  Let
us label such code as Unicode-safe.  Unicode-safe libraries can be
used in an all-unicode world.

Second, we need to be able to write code that, when provided only
str instances, will not create unicode results.  Let us label such
code as str-stable.  Libraries that are str-stable can be used by
libraries and applications that are not yet Unicode-safe.

Sometimes it is simple to write code that is both str-stable and
Unicode-safe.  For example, the following function just works::

    def appendx(s):
        return s + 'x'

That's not too surprising since the unicode type is designed to
make the task easier.  The principle is that when str and unicode
instances meet, the result is a unicode instance.  One notable
difficulty arises when code requires a string representation of an
object; an operation traditionally accomplished by using the ``str()``
built-in function.

Using the current ``str()`` function makes the code not Unicode-safe.
Replacing a ``str()`` call with a ``unicode()`` call makes the code not
str-stable.  Changing ``str()`` so that it could return unicode
instances would solve this problem.  As a further benefit, some code
that is currently not Unicode-safe because it uses ``str()`` would
become Unicode-safe.


Specification
=============

A Python implementation of the ``str()`` built-in follows::

    def str(s):
        """Return a nice string representation of the object.  The
        return value is a str or unicode instance.
        """
        if type(s) is str or type(s) is unicode:
            return s
        r = s.__str__()
        if not isinstance(r, (str, unicode)):
            raise TypeError('__str__ returned non-string')
        return r

The following function would be added to the C API and would be the
equivalent to the ``str()`` built-in (ideally it be called ``PyObject_Str``,
but changing that function could cause a massive number of
compatibility problems)::

    PyObject *PyString_New(PyObject *);

A reference implementation is available on Sourceforge [1]_ as a
patch.


Backwards Compatibility
=======================

Some code may require that ``str()`` returns a str instance.  In the
standard library, only one such case has been found so far.  The
function ``email.header_decode()`` requires a str instance and the
``email.Header.decode_header()`` function tries to ensure this by
calling ``str()`` on its argument.  The code was fixed by changing
the line "header = str(header)" to::

    if isinstance(header, unicode):
        header = header.encode('ascii')

Whether this is truly a bug is questionable since ``decode_header()``
really operates on byte strings, not character strings.  Code that
passes it a unicode instance could itself be considered buggy.


Alternative Solutions
=====================

A new built-in function could be added instead of changing ``str()``.
Doing so would introduce virtually no backwards compatibility
problems.  However, since the compatibility problems are expected to
rare, changing ``str()`` seems preferable to adding a new built-in.

The basestring type could be changed to have the proposed behaviour,
rather than changing ``str()``.  However, that would be confusing
behaviour for an abstract base type.


References
==========

.. [1] https://bugs.python.org/issue1266570


Copyright
=========

This document has been placed in the public domain.


id: pep-0611.rst
content:
PEP: 611
Title: The one million limit
Author: Mark Shannon <mark@hotpy.org>
Status: Withdrawn
Type: Standards Track
Created: 05-Dec-2019
Post-History:


Abstract
========
This PR proposes a soft limit of one million (1 000 000), and a larger hard limit
for various aspects of Python code and its implementation.

The Python language does not specify limits for many of its features.
Not having any limit to these values seems to enhance programmer freedom,
at least superficially, but in practice the CPython VM and other Python virtual
machines have implicit limits or are forced to assume that the limits are
astronomical, which is expensive.

This PR lists a number of features which are to have a limit of one million.

For CPython the hard limit will be eight million (8 000 000).

Motivation
==========

There are many values that need to be represented in a virtual machine.
If no limit is specified for these values,
then the representation must either be inefficient or vulnerable to overflow.
The CPython virtual machine represents values like line numbers,
stack offsets and instruction offsets by 32 bit values. This is inefficient, and potentially unsafe.

It is inefficient as actual values rarely need more than a dozen or so bits to represent them.

It is unsafe as malicious or poorly generated code could cause values to exceed 2\ :sup:`32`.

For example, line numbers are represented by 32 bit values internally.
This is inefficient, given that modules almost never exceed a few thousand lines.
Despite being inefficient, it is still vulnerable to overflow as
it is easy for an attacker to created a module with billions of newline characters.

Memory access is usually a limiting factor in the performance of modern CPUs.
Better packing of data structures enhances locality and reduces memory bandwidth,
at a modest increase in ALU usage (for shifting and masking).
Being able to safely store important values in 20 bits would allow memory savings
in several data structures including, but not limited to:

* Frame objects
* Object headers
* Code objects

There is also the potential for a more efficient instruction format, speeding up interpreter dispatch.

Is this a worthwhile trade off?
-------------------------------

The downside of any form of limit is that it might potentially make someone's job harder,
for example, it may be harder to write a code generator that keeps the size of modules to one million lines.
However, it is the author's opinion, having written many code generators,
that such a limit is extremely unlikely to be a problem in practice.

The upside of these limits is the freedom it grants implementers of runtimes, whether CPython,
PyPy, or any other implementation, to improve performance.
It is the author's belief, that the potential value of even a 0.1% reduction in the cost
of running Python programs globally will hugely exceed the cost of modifying a handful of code generators.

Rationale
=========

Imposing a limit on values such as lines of code in a module, and the number of local variables,
has significant advantages for ease of implementation and efficiency of virtual machines.
If the limit is sufficiently large, there is no adverse effect on users of the language.

By selecting a fixed but large limit for these values,
it is possible to have both safety and efficiency whilst causing no inconvenience to human programmers
and only very rare problems for code generators.

One million
-----------

The value "one million" is very easy to remember.

The one million limit is mostly a limit on human generated code, not runtime sizes.

One million lines in a single module is a ridiculous concentration of code;
the entire Python standard library is about 2/3rd of a million lines, spread over 1600 files.

The Java Virtual Machine (JVM) [1]_ specifies a limit of 2\ :sup:`16`-1 (65535) for many program
elements similar to those covered here.
This limit enables limited values to fit in 16 bits, which is a very efficient machine representation.
However, this limit is quite easily exceeded in practice by code generators and
the author is aware of existing Python code that already exceeds 2\ :sup:`16` lines of code.

The hard limit of eight million fits into 23 bits which, although not as convenient for machine representation,
is still reasonably compact.
A limit of eight million is small enough for efficiency advantages (only 23 bits),
but large enough not to impact users (no one has ever written a module that large).

While it is possible that generated code could exceed the limit,
it is easy for a code generator to modify its output to conform.
The author has hit the 64K limit in the JVM on at least two occasions when generating Java code.
The workarounds were relatively straightforward and wouldn't
have been necessary with a limit of one million bytecodes or lines of code.

Where necessary, the soft limit can increased for those programs that exceed the one million limit.

Having a soft limit of one million provides a warning of problematic code, without causing an error and forcing an immediate fix.
It also allows dynamic optimizers to use more compact formats without inline checks.

Specification
=============

This PR proposes that the following language features and runtime values have a soft limit of one million.

* The number of source code lines in a module
* The number of bytecode instructions in a code object.
* The sum of local variables and stack usage for a code object.
* The number of classes in a running interpreter.
* The recursion depth of Python code.

It is likely that memory constraints would be a limiting factor before the number of classes reaches one million.

Recursion depth
---------------

The recursion depth limit only applies to pure Python code. Code written in a foreign language, such as C,
may consume hardware stack and thus be limited to a recursion depth of a few thousand.
It is expected that implementations will raise an exception should the hardware stack get close to its limit.
For code that mixes Python and C calls, it is most likely that the hardware limit will apply first.
The size of the hardware recursion may vary at runtime and will not be visible.

Soft and hard limits
====================

Implementations should emit a warning whenever a soft limit is exceeded, unless the hard limit has the same value as the soft limit.
When a hard limit is exceeded, then an exception should be raised.

Depending on the implementation, different hard limits might apply. In some cases the hard limit might be below the soft limit.
For example, many micropython ports are unlikely to be able to support such large limits.


Introspecting and modifying the limits
--------------------------------------

One or more functions will be provided in the ``sys`` module to introspect or modify the soft limits at runtime,
but the limits may not be raised above the hard limit.

Inferred limits
---------------

These limits are not part of the specification, but a limit of less than one million
can be inferred from the limit on the number of bytecode instructions in a code object.
Because there would be insufficient instructions to load more than
one million constants or use more than one million names.

* The number of distinct names in a code object.
* The number of constants in a code object.

The advantages for CPython of imposing these limits:
----------------------------------------------------

Line of code in a module and code object restrictions.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When compiling source code to bytecode or modifying bytecode for profiling or debugging,
an intermediate form is required. By limiting operands to 23 bits,
instructions can be represented in a compact 64 bit form allowing
very fast passes over the instruction sequence.

Having 23 bit operands (24 bits for relative branches) allows instructions
to fit into 32 bits without needing additional ``EXTENDED_ARG`` instructions.
This improves


id: pep-0605.rst
content:
PEP: 605
Title: A rolling feature release stream for CPython
Author: Steve Dower <steve.dower@python.org>, Alyssa Coghlan <ncoghlan@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-605-a-rolling-feature-release-stream-for-cpython/2418
Status: Rejected
Type: Informational
Created: 20-Sep-2019
Python-Version: 3.9
Post-History: 01-Oct-2019, 06-Oct-2019, 20-Oct-2019

Rejection Notice
================

This PEP was rejected in favour of :pep:`602`. The potential alpha/beta alternation
was deemed too confusing and the two year cadence between releases deemed too
long.


Abstract
========

For a long time, CPython's nominal feature release cadence has been "every 18-24
months", and in recent years, has been pretty consistently on the "18 month"
end of that window. :pep:`607` provides some common background on the problems
that arise due to that choice of cadence, as well as some of the risks that
need to be accounted for when proposing to change it.

The proposal in this PEP aims to allow CPython's user base to self-select into
two distinct but overlapping groups:

* users of stable feature releases (and their associated maintenance release
  streams) that are published every 24 months; and
* early adopters of a new rolling release stream that takes the place of the
  previous CPython pre-release process

As part of this proposal, the usage guidance given for beta releases would
become "suitable for production use only in environments with sufficiently
robust compatibility testing and operational monitoring capabilities", rather
than the currently unqualified "not for production use".

Similarly, the guidance given for alpha releases would be amended to state
"intended for library compatibility testing and the creation of ABI compatible
binary artifacts", rather than simply saying "not for production use".

The PEP authors believe these outcomes can be achieved by amending CPython's
pre-release management process as described in the Proposal section below.

This PEP also proposes that the frequency of X.Y.0 releases be adjusted to
begin each new release series in August every two years (starting in 2021,
around two years after the release of Python 3.8.0).

Example Future Release Schedules
================================

Under this proposal, Python 3.9.0a1 would be released in December 2019, two
months after the Python 3.8.0 baseline feature release in October 2019.

Assuming no further breaking changes were made to the full CPython ABI, the
3.9.0b2 release would then follow 2 months later in February 2020, continuing
through to 3.9.0b9 in April 2021.

Any time a breaking change to the full CPython ABI was introduced, the first
pre-release that included it would be marked as an alpha release.

3.9.0rc1 would be published in June 2021, 3.9.0rc2 in July 2021, and then
the full release published as 3.9.0 in August 2021.

The cycle would start over again in October 2021, with the publication
of 3.10.0a1 (4 months after the creation of the 3.9.x maintenance branch).

The exact schedule of maintenance releases would be up to the release team,
but assuming maintenance releases of 3.9.x were also to occur every other month
(offset from the 3.10.0 beta releases), the overall release timeline
would look like:

* 2019-12: 3.9.0a1
* 2020-02: 3.9.0b2
* ... beta (or alpha) releases every other month
* 2021-04: 3.9.0b9
* 2021-06: 3.9.0rc1 (feature freeze, ABI freeze, pyc format freeze)
* 2021-07: 3.9.0rc2
* 2021-08: 3.9.0
* 2021-09: 3.9.1, 3.8.x (final 3.8.x binary maintenance release)
* 2021-10: 3.10.0a1
* 2021-11: 3.9.2
* 2021-12: 3.10.0b2
* ... beta (or alpha) and maintenance releases continue in alternate months
* 2023-04: 3.10.0b10
* 2023-05: 3.9.11
* 2023-06: 3.10.0rc1 (feature freeze, ABI freeze, pyc format freeze)
* 2023-07: 3.10.0rc2, 3.9.12
* 2023-08: 3.10.0
* 2023-09: 3.10.1, 3.9.13 (final 3.9.x binary maintenance release)
* 2023-10: 3.11.0a1
* 2023-12: 3.11.0b2
* ... etc

If we assume two additional pre-releases were made that introduced breaking
changes to the full CPython ABI in the 3.9.0a5 and 3.9.0a7 releases, then the
overall calendar would look like:

.. figure:: pep-0605-example-release-calendar.png
   :align: center
   :width: 100%

   Figure 1. Impact of the pre-release process changes on the calendar.

There are always two or three active maintenance branches in this model,
which preserves the status quo in that respect. The major difference is that
we would start encouraging publishers to provide pre-built binaries for the
pre-freeze rolling releases in addition to providing them for the stable
maintenance branches.

.. figure:: pep-0605-overlapping-support-matrix.png
   :align: center
   :width: 50%

   Figure 2. Testing matrix in the 18-month cadence vs. the 24-month


Package publishers targeting the full CPython ABI that choose to provide
pre-built binaries for the rolling pre-freeze releases would at least need
to build new wheel archives following the 3.9.0a1 release. Whether they needed
to publish updated binaries after subsequent alpha releases (e.g. 3.9.0a5 or
3.9.0a7 releases in the example timeline) would depend on whether or not they
were actually affected by the ABI breaks in those later releases.

As with the status quo, all package publishers wishing to provide pre-built
binaries for the final release will need to build new wheel archives following
the ABI freeze date. Unlike the status quo, this date will be clearly marked
by the publication of the first release candidate, and it will occur early
enough to give publishers a couple of months to get ready for the final release.


Example Future Release Announcements
====================================

If this PEP is accepted, the primary channels used to communicate the updated
pre-release management process to end users would be the Python 3.9 What's New
document, and the release announcements themselves.

This section provides initial drafts of text that could be used for those
purposes.


Suggested "What's New in Python 3.9" Entry
------------------------------------------

The following subsection would be added to the Python 3.9 What's New document,
and then linked from each of the Python 3.9 alpha and beta announcements.

PEP 605: Changes to the pre-release management process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

As detailed in :pep:`605`, the pre-release management process has been updated to
produce a rolling series of beta releases that are considered suitable for
production use in environments with sufficiently robust integration testing and
operational monitoring capabilities.

Under this new rolling model, the alpha and beta releases are intermingled as
part of a combined "pre-freeze" period, with alpha releases indicating breaks
in the full CPython ABI that may require recompilation of extension modules or
embedding applications, and beta releases indicating full binary compatibility
with the immediately preceding pre-release.

Unlike previous releases, publishing pre-built binaries for 3.9.0 alpha and beta
releases is actively encouraged, as a new pre-release ABI flag ("p") is now
set when building and loading extension modules prior to the full CPython ABI
freeze, ensuring that all such pre-freeze extension module builds will be
ignored by post-freeze interpreter builds.

The full CPython ABI will be frozen, and the pre-release flag dropped from the
ABI flags, in 3.9.0rc1, which is expected to occur 2 months prior to the final
3.9.0 release (refer to the release schedule in :pep:`596` for exact target dates).

For application developers, migrating to the rolling release stream provides
the opportunity to be actively involved in the design and development of
enhancements to the standard library and reference interpreter prior to the
next stable release. It also provides the opportunity to benefit from
interpreter performance enhancements up to a year or more before they become
available in a stable release.

For library developers that p


id: pep-0639.rst
content:
PEP: 639
Title: Improving License Clarity with Better Package Metadata
Author: Philippe Ombredanne <pombredanne@nexb.com>,
        C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>,
        Karolina Surma <karolina.surma@gazeta.pl>,
PEP-Delegate: Brett Cannon <brett@python.org>
Discussions-To: https://discuss.python.org/t/53020
Status: Final
Type: Standards Track
Topic: Packaging
Created: 15-Aug-2019
Post-History: `15-Aug-2019 <https://discuss.python.org/t/2154>`__,
              `17-Dec-2021 <https://discuss.python.org/t/12622>`__,
              `10-May-2024 <https://discuss.python.org/t/53020>`__,
Resolution: https://discuss.python.org/t/53020/106


.. canonical-pypa-spec:: :ref:`core-metadata`


.. _639-abstract:

Abstract
========

This PEP defines a specification how licenses are documented in the Python
projects.

To achieve that, it:

- Adopts the `SPDX license expression syntax <639-spdx_>`__ as a
  means of expressing the license for a Python project.

- Defines how to include license files within the projects, source and built
  distributions.

- Specifies the necessary changes to :term:`Core Metadata` and
  the corresponding :term:`Pyproject Metadata key`\s

- Describes the necessary changes to
  the `source distribution (sdist) <sdistspec_>`__,
  `built distribution (wheel) <wheelspec_>`__ and
  `installed project <installedspec_>`__ standards.

This will make license declaration simpler and less ambiguous for
package authors to create, end users to understand,
and tools to programmatically process.

The changes will update the
`Core Metadata specification <coremetadataspec_>`__ to version 2.4.

.. _639-goals:

Goals
=====

This PEP's scope is limited to covering new mechanisms for documenting
the license of a :term:`distribution package`, specifically defining:

- A means of specifying a SPDX :term:`license expression`.
- A method of including license texts in :term:`distribution package`\s
  and installed :term:`Project`\s.

The changes that this PEP requires have been designed to minimize impact and
maximize backward compatibility.


.. _639-non-goals:

Non-Goals
=========

This PEP doesn't recommend any particular license to be chosen by any
particular package author.

If projects decide not to use the new fields, no additional restrictions are
imposed by this PEP when uploading to PyPI.

This PEP also is not about license documentation for individual files,
though this is a :ref:`surveyed topic <639-license-doc-source-files>`
in an appendix, nor does it intend to cover cases where the
:term:`source distribution <Source Distribution (or "sdist")>` and
:term:`binary distribution` packages don't have
:ref:`the same licenses <639-rejected-ideas-difference-license-source-binary>`.


.. _639-motivation:

Motivation
==========

Software must be licensed in order for anyone other than its creator to
download, use, share and modify it.
Today, there are multiple fields where licenses
are documented in :term:`Core Metadata`,
and there are limitations to what can be expressed in each of them.
This often leads to confusion both for package authors
and end users, including distribution re-packagers.

This has triggered a number of license-related discussions and issues,
including on `outdated and ambiguous PyPI classifiers <classifierissue_>`__,
`license interoperability with other ecosystems <interopissue_>`__,
`too many confusing license metadata options <packagingissue_>`__,
`limited support for license files in the Wheel project <wheelfiles_>`__, and
`the lack of precise license metadata <pepissue_>`__.

As a result, on average, Python packages tend to have more ambiguous and
missing license information than other common ecosystems. This is supported by
the `statistics page <cdstats_>`__ of the
`ClearlyDefined project <clearlydefined_>`__, an
`Open Source Initiative <osi_>`__ effort to help
improve licensing clarity of other FOSS projects, covering all packages
from PyPI, Maven, npm and Rubygems.

The current license classifiers could be extended to include the full range of
the SPDX identifiers while deprecating the ambiguous classifiers
(such as ``License :: OSI Approved :: BSD License``).

However, there are multiple arguments against such an approach:

- It requires a great effort to duplicate the SPDX license list and keep it in
  sync.

- It is a hard break in backward compatibility, forcing package authors
  to update to new classifiers immediately when PyPI deprecates the old ones.

- It only covers packages under a single license;
  it doesn't address projects that vendor dependencies (e.g. Setuptools),
  offer a choice of licenses (e.g. Packaging) or were relicensed,
  adapt code from other projects or contain fonts, images,
  examples, binaries or other assets under other licenses.

- It requires both authors and tools understand and implement the PyPI-specific
  classifier system.

- It does not provide as clear an indicator that a package
  has adopted the new system, and should be treated accordingly.


.. _639-rationale:

Rationale
=========

A survey was conducted to map the existing license metadata
definitions in the :ref:`Python ecosystem <639-license-doc-python>` and a
:ref:`variety of other packaging systems, Linux distributions,
language ecosystems and applications <639-license-doc-other-projects>`.

The takeaways from the survey have guided the recommendations of this PEP:

- SPDX and SPDX-like syntaxes are the most popular :term:`license expression`\s
  in many modern package systems.

- Most Free and Open Source Software licenses require package authors to
  include their full text in a :term:`Distribution Package`.

Therefore, this PEP introduces two new Core Metadata fields:

- :ref:`License-Expression <639-spec-field-license-expression>` that
  provides an unambiguous way to express the license of a package
  using SPDX license expressions.

- :ref:`License-File <639-spec-field-license-file>` that
  offers a standardized way to include the full text of the license(s)
  with the package when distributed,
  and allows other tools consuming the :term:`Core Metadata`
  to locate a :term:`distribution archive`'s license files.

Furthermore, this specification builds upon
existing practice in the `Setuptools <setuptoolsfiles_>`__ and
`Wheel <wheelfiles_>`__ projects.
An up-to-date version of the current draft of this PEP is
`implemented <hatchimplementation_>`__ in the
`Hatch <hatch_>`__ packaging tool, and an earlier draft of the
:ref:`license files portion <639-spec-field-license-file>`
is `implemented in Setuptools <setuptoolspep639_>`__.


.. _639-terminology:

Terminology
===========

The keywords "MUST", "MUST NOT", "REQUIRED",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL"
in this document are to be interpreted as described in :rfc:`2119`.


.. _639-terminology-license:

License terms
-------------

The license-related terminology draws heavily from the `SPDX Project <spdx_>`__,
particularly :term:`license identifier` and :term:`license expression`.

.. glossary::

    license classifier
        A `PyPI Trove classifier <classifiers_>`__
        (as :ref:`described <packaging:core-metadata-classifier>`
        in the :term:`Core Metadata` specification)
        which begins with ``License ::``.

    license expression
    SPDX expression
        A string with valid `SPDX license expression syntax <spdxpression_>`__
        including one or more SPDX :term:`license identifier`\(s),
        which describes a :term:`Project`'s license(s)
        and how they inter-relate.
        Examples:
        ``GPL-3.0-or-later``,
        ``MIT AND (Apache-2.0 OR BSD-2-clause)``

    license identifier
    SPDX identifier
        A valid `SPDX short-form license identifier <spdxid_>`__,
        as described in the
        :ref:`639-spec-field-license-expression` section of this PEP.
        This includes all valid SPDX identifiers and
        the custom ``LicenseRef-[idstring]`` strings conforming to the
        `SPDX spec


id: pep-0765.rst
content:
PEP: 765
Title: Disallow return/break/continue that exit a finally block
Author: Irit Katriel <irit@python.org>, Alyssa Coghlan <ncoghlan@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-765-disallow-return-break-continue-that-exit-a-finally-block/71348
Status: Final
Type: Standards Track
Created: 15-Nov-2024
Python-Version: 3.14
Post-History: `09-Nov-2024 <https://discuss.python.org/t/an-analysis-of-return-in-finally-in-the-wild/70633>`__,
              `16-Nov-2024 <https://discuss.python.org/t/pep-765-disallow-return-break-continue-that-exit-a-finally-block/71348>`__,
Replaces: 601
Resolution: https://discuss.python.org/t/71348/111

.. canonical-doc:: :external+py3.14:ref:`finally`

Abstract
========

This PEP proposes to withdraw support for ``return``, ``break`` and
``continue`` statements that break out of a ``finally`` block.
This was proposed in the past by :pep:`601`. The current PEP
is based on empirical evidence regarding the cost/benefit of
this change, which did not exist at the time that :pep:`601`
was rejected. It also proposes a slightly different solution
than that which was proposed by :pep:`601`.

Motivation
==========

The semantics of ``return``, ``break`` and ``continue`` in a
finally block are surprising for many developers.
The :ref:`documentation <python:tut-cleanup>` mentions that:

- If the ``finally`` clause executes a ``break``, ``continue``
  or ``return`` statement, exceptions are not re-raised.

- If a ``finally`` clause includes a ``return`` statement, the
  returned value will be the one from the ``finally`` clauseâs
  ``return`` statement, not the value from the ``try`` clauseâs
  ``return`` statement.

Both of these behaviours cause confusion, but the first is
particularly dangerous because a swallowed exception is more
likely to slip through testing, than an incorrect return value.

In 2019, :pep:`601` proposed to change Python to emit a
``SyntaxWarning`` for a few releases and then turn it into a
``SyntaxError``. It was rejected in favour of viewing this
as a programming style issue, to be handled by linters and :pep:`8`.
Indeed, :pep:`8` now recommends not to use control flow statements
in a ``finally`` block, and linters such as
`Pylint <https://pylint.readthedocs.io/en/stable/>`__,
`Ruff <https://docs.astral.sh/ruff/>`__ and
`flake8-bugbear <https://github.com/PyCQA/flake8-bugbear>`__
flag them as a problem.

Rationale
=========

A recent
`analysis of real world code <https://github.com/iritkatriel/finally/blob/main/README.md>`__ shows that:

- These features are rare (2 per million LOC in the top 8,000 PyPI
  packages, 4 per million LOC in a random selection of packages).
  This could be thanks to the linters that flag this pattern.
- Most of the usages are incorrect, and introduce unintended
  exception-swallowing bugs.
- Code owners are typically receptive to fixing the bugs, and
  find that easy to do.

See `the appendix <#appendix>`__ for more details.

This new data indicates that it would benefit Python's users if
Python itself moved them away from this harmful feature.

One of the arguments brought up in
`the PEP 601 discussion <https://discuss.python.org/t/pep-601-forbid-return-break-continue-breaking-out-of-finally/2239/24>`__
was that language features should be orthogonal, and combine without
context-based restrictions. However, in the meantime :pep:`654` has
been implemented, and it forbids ``return``, ``break`` and ``continue``
in an ``except*`` clause because the semantics of that would violate
the property that ``except*`` clauses operate *in parallel*, so the
code of one clause should not suppress the invocation of another.
In that case we accepted that a combination of features can be
harmful enough that it makes sense to disallow it.


Specification
=============

The change is to specify as part of the language spec that
Python's compiler may emit a ``SyntaxWarning`` or ``SyntaxError``
when a ``return``, ``break`` or ``continue`` would transfer
control flow from within a ``finally`` block to a location outside
of it.

These examples may emit a ``SyntaxWarning`` or ``SyntaxError``:

.. code-block::
   :class: bad

    def f():
        try:
            ...
        finally:
            return 42

    for x in o:
        try:
            ...
        finally:
            break  # (or continue)

These examples would not emit the warning or error:

.. code-block::
   :class: good

    try:
        ...
    finally:
        def f():
            return 42

    try:
        ...
    finally:
        for x in o:
            break  # (or continue)


CPython will emit a ``SyntaxWarning`` in version 3.14, and we leave
it open whether, and when, this will become a ``SyntaxError``.
However, we specify here that a ``SyntaxError`` is permitted by
the language spec, so that other Python implementations can choose
to implement that.

The CPython implementation will emit the ``SyntaxWarning`` during
``AST`` construction, to ensure that the warning will show up during
static anlaysis and compilation, but not during execution of
pre-compiled code.  We expect that the warning will be seen by a
project maintainer (when they run static analysis, or CI which
does not have precompiled files). However, end users of a project
will only see a warning if they skip precompilation at installation
time, check installation time warnings, or run static analysis over
their dependencies.

Backwards Compatibility
=======================

For backwards compatibility reasons, we are proposing that CPython
emit only a ``SyntaxWarning``, with no concrete plan to upgrade that
to an error. Code running with ``-We`` may stop working once this
is introduced.

Security Implications
=====================

The warning/error will help programmers avoid some hard to find bugs,
so will have a security benefit. We are not aware of security issues
related to raising a new ``SyntaxWarning`` or ``SyntaxError``.

How to Teach This
=================

The change will be documented in the language spec and in the
What's New documentation. The ``SyntaxWarning`` will alert users
that their code needs to change. The `empirical evidence <#appendix>`__
shows that the changes necessary are typically quite
straightforward.

Rejected Ideas
==============

Emit ``SyntaxError`` in CPython
-------------------------------

:pep:`601` proposed that CPython would emit ``SyntaxWarning`` for a couple of
releases and ``SyntaxError`` afterwards. We are leaving it open whether, and
when, this will become a ``SyntaxError`` in CPython, because we believe that a
``SyntaxWarning`` would provide most of the benefit with less risk.

Change Semantics
----------------

It `was suggested <https://discuss.python.org/t/pep-765-disallow-return-break-continue-that-exit-a-finally-block/71348/32>`__
to change the semantics of control flow instructions in ``finally`` such that an
in-flight exception takes precedence over them.  In other words, a ``return``,
``break`` or ``continue`` would be permitted, and would exit the ``finally``
block, but the exception would still be raised.

This was rejected for two reasons. First, it would change the semantics of
working code in a way that can be hard to debug: a ``finally`` that was written
with the intention of swallowing all exceptions (correctly using the documented
semantics) would now allow the exception to propagate on. This may happen only
in rare edge cases at runtime, and is not guaranteed to be detected in testing.
Even if the code is wrong, and has an exception swallowing bug, it could be
hard for users to understand why a program started raising exceptions in 3.14,
while it did not in 3.13.
In contrast, a ``SyntaxWarning`` is likely to be seen during testing, it would
point to the precise location of the problem in the code, and it would not
prevent the program from running.

The second objection was about the proposed semantics. The motivation for
allowing control flow statements is not that this would be useful, bu


id: pep-0003.rst
content:
PEP: 3
Title: Guidelines for Handling Bug Reports
Author: Jeremy Hylton <jeremy@alum.mit.edu>
Status: Withdrawn
Type: Process
Created: 25-Sep-2000
Post-History:

.. withdrawn::

   This PEP contained guidelines for handling bug reports in the Python
   bug tracker.  It has been replaced by the
   `Developer's Guide description of issue triaging
   <https://devguide.python.org/triage/triaging/index.html>`_.
   Guidelines for people submitting Python bugs are `in the Python docs
   <http://docs.python.org/bugs.html>`__.

Original Guidelines
===================

1. Make sure the bug category and bug group are correct.  If they are
   correct, it is easier for someone interested in helping to find
   out, say, what all the open Tkinter bugs are.

2. If it's a minor feature request that you don't plan to address
   right away, add it to :pep:`42` or ask the owner to add it for you.
   If you add the bug to :pep:`42`, mark the bug as "feature request",
   "later", and "closed"; and add a comment to the bug saying that
   this is the case (mentioning the PEP explicitly).

    ::

        XXX do we prefer the tracker or PEP 42?

3. Assign the bug a reasonable priority.  We don't yet have a clear
   sense of what each priority should mean.  One rule, however, is
   that bugs with priority "urgent" or higher must be fixed before
   the next release.

4. If a bug report doesn't have enough information to allow you to
   reproduce or diagnose it, ask the original submitter for more
   information.  If the original report is really thin and your email
   doesn't get a response after a reasonable waiting period, you can
   close the bug.

5. If you fix a bug, mark the status as "Fixed" and close it.  In the
   comments, include the SVN revision numbers of the commit(s). In
   the SVN checkin message, include the issue number **and** a
   normal description of the change, mentioning the contributor if a
   patch was applied.

6. If you are assigned a bug that you are unable to deal with, assign
   it to someone else if you think they will be able to deal with it,
   otherwise it's probably best to unassign it.


References
==========

* https://bugs.python.org


id: pep-0771.rst
content:
PEP: 771
Title: Default Extras for Python Software Packages
Author: Thomas Robitaille <thomas.robitaille@gmail.com>, Jonathan Dekhtiar <jonathan@dekhtiar.com>
Sponsor: Pradyun Gedam <pradyunsg@gmail.com>
Discussions-To: https://discuss.python.org/t/79706/
Status: Draft
Type: Standards Track
Topic: Packaging
Created: 13-Jan-2025
Post-History:
  `15-Jan-2025 <https://discuss.python.org/t/77892/>`__,
  `06-Feb-2025 <https://discuss.python.org/t/79706/>`__,

Abstract
========

:pep:`508` specifies a mini-language for
declaring package dependencies. One feature of this language is the ability to
specify *extras*, which are optional components of a distribution that, when
used, install additional dependencies. This PEP proposes a mechanism to allow
one or more extras to be installed by default if none are provided explicitly.

Motivation
==========

Various use cases for default extras and possible solutions in this PEP were discussed
extensively on `this DPO thread <https://discuss.python.org/t/adding-a-default-extra-require-environment/4898>`__.
These fall into two broad cases that that provide the
motivation for the present PEP.

Recommended but not required dependencies
-----------------------------------------

Package maintainers often use extras to declare optional dependencies that
extend the functionality or performance of a package. In some cases, it can be
difficult to determine which dependencies should be required and which should be
categorized as extras. A balance must be struck between the needs of typical
users (who may prefer most features to be available 'by default') and users who
want minimal installations without large, optional dependencies. One solution
with existing Python packaging infrastructure is for package maintainers to
define an extra called, for example, ``recommended``, which
includes all non-essential but suggested dependencies. Users are then instructed to
install the package using ``package[recommended]``, while those who prefer more
control can use ``package``. However, in practice, many users are unaware
of the ``[recommended]`` syntax, placing the burden on them to know this for a
typical installation. Having a way to have recommended dependencies be installed
by default while providing a way for users to request a more minimal installation
would satisfy this use case, and this PEP describes a solution to this.

Examples of packages that demonstrate this pattern by encouraging users to
include extra dependencies by default include:

* `astropy <https://www.astropy.org/>`_: ``astropy[recommended]``
* `fastapi <https://fastapi.tiangolo.com/>`_: ``fastapi[standard]``
* `tensorflow <https://www.tensorflow.org>`_: ``tensorflow[and-cuda]``

Packages supporting multiple backends or frontends
--------------------------------------------------

Another common use case for using extras is to define different backends or
frontends and dependencies that need to be installed for each backend or
frontend. A package might need at least one backend or frontend to be installed
in order to be functional, but may be flexible on which backend or frontend this
is. Concrete examples of such frontends or backends include:

* The Qt frontend library, which can be provided by `PyQt5 <https://pypi.org/project/PyQt5/>`_, `PyQt6
  <https://pypi.org/project/PyQt6/>`_, `PySide2
  <https://pypi.org/project/PySide2/>`_, or `PySide6
  <https://pypi.org/project/PySide6/>`_
* BLAS/LAPACK, which have different possible implementations (e.g. OpenBLAS, and MKL)
* FFT libraries, which also have different implementations (e.g. ``scipy.fft`` and pyFFTW)

With current packaging standards, maintainers have to either
require one of the backends or frontends, or require users
to always specify extras, e.g. ``package[backend]`` and therefore risk users
having an unusable installation if they only install ``package``. Having a
way to specify one or more default backend or frontend and providing a way to
override these defaults would provide a much better experience for users, and
the approach described in this PEP will allow this.

Note that this PEP does not aim to address the issue of disallowing conflicting
or incompatible extras - for example if a package requires exactly one frontend
or backend package. There is currently no mechanism in Python packaging
infrastructure to disallow conflicting or incompatible extras to be installed,
and this PEP does not change that.

Examples of packages that require at least one backend or frontend to work and
recommend a default extra to install a backend or frontend include:

* `kivy <https://kivy.org/>`_: ``kivy[base]``

* `napari <https://napari.org>`_: ``napari[all]``

* `glueviz <https://www.glueviz.org>`_: ``glueviz[qt]``

In all three cases, installing the package without any extras results in a
broken installation (and this is a commonly reported issue for some of these
packages).

Rationale
=========

A number of possible solutions have been extensively and vigorously discussed by
the community for several years, including in `this DPO thread
<https://discuss.python.org/t/adding-a-default-extra-require-environment/4898>`__
as well as in numerous issues and pull requests. The solution that is
presented below:

* is an opt-in solution which means that package maintainers can choose whether or not to use it
* is flexible enough to accommodate both of the major use cases described in `Motivation`_
* re-uses the syntax from :pep:`508`

It is the only solution out of all those discussed that meets all three criteria.

Specification
=============

``Default-Extra`` Metadata Field
---------------------------------

A new multiple-use metadata field, ``Default-Extra``, will be added to the `core package
metadata <https://packaging.python.org/en/latest/specifications/core-metadata/#core-metadata>`_.
For this field, each entry must be a string specifying an extra that will be
automatically included when the package is installed without any extras specified explicitly.

Only entries already specified in a `Provides-Extra
<https://packaging.python.org/en/latest/specifications/core-metadata/#provides-extra-multiple-use>`_
entry can be used in a ``Default-Extra`` entry.

Examples::

    Default-Extra: recommended
    Default-Extra: backend1
    Default-Extra: backend2
    Default-Extra: backend3

Since this introduces a new field in the core package metadata, this will require
`Metadata-Version <https://packaging.python.org/en/latest/specifications/core-metadata/#metadata-version>`_
to be bumped to the next minor version (2.5 at the time of writing).

New key in ``[project]`` metadata table
---------------------------------------

A new key will be added to the ``[project]`` table in project metadata as
originally defined in :pep:`621` and now defined in the `PyPA specifications
<https://packaging.python.org/en/latest/specifications/pyproject-toml/>`_. This key will be named
``default-optional-dependency-keys`` with the following description:

* `TOML <https://toml.io/>`_ type: Array of strings
* Corresponding core metadata field: ``Default-Extra``

Each string in ``default-optional-dependency-keys`` must be the name of an extra
defined in `optional-dependencies
<https://packaging.python.org/en/latest/specifications/pyproject-toml/#dependencies-optional-dependencies>`_,
and each extra in this array will be converted to a matching ``Default-Extra``
entry in the core package metadata. Examples of valid usage which would
produce the example ``Default-Extra`` entries presented in the previous section are:

.. code-block:: toml

    [project]
    default-optional-dependency-keys = [
        "recommended",
    ]

and:

.. code-block:: toml

    [project]
    default-optional-dependency-keys = [
        "backend1",
        "backend2",
        "backend3"
    ]


Overriding default extras
-------------------------

If extras are explicitly given in a dependency specification, the default
extras are ignored. Otherwise, the default extr


id: pep-0759.rst
content:
PEP: 759
Title: External Wheel Hosting
Author: Barry Warsaw <barry@python.org>,
        Emma Harper Smith <emma@python.org>
PEP-Delegate: Donald Stufft <donald@python.org>
Discussions-To: https://discuss.python.org/t/pep-759-external-wheel-hosting/66458
Status: Withdrawn
Type: Standards Track
Topic: Packaging
Created: 01-Oct-2024
Post-History: 10-Oct-2024, 31-Jan-2025
Resolution: `31-Jan-2025 <https://discuss.python.org/t/pep-759-external-wheel-hosting/66458/48>`__

Abstract
========

This PEP proposes a mechanism by which projects hosted on `pypi.org
<https://pypi.org>`__ can safely host wheel artifacts on external sites other
than PyPI. This PEP explicitly does *not* propose external hosting of
projects, packages, or their metadata. That functionality is already available
by externally hosting independent package indexes. Because this PEP only
provides a mechanism for projects to customize the download URL for specific
released wheel artifacts, dependency resolution as already implemented by
common installer tools such as `pip <https://pip.pypa.io/en/stable/>`__ and
`uv <https://docs.astral.sh/uv/>`__ does not need to change.

This PEP defines what it means to be "safe" in this context, along with a new
package upload file format called a ``.rim`` file. It defines how ``.rim``
files affect the metadata returned for a package's :ref:`Simple Repository API
<packaging:simple-repository-api>`
in both HTML and JSON formats, and how traditional wheels can easily be turned
into ``.rim`` files.

PEP withdrawn
=============

This PEP was withdrawn by the authors on 2025-01-31.  Our reading of the sentiment in the discussion
thread is that, while the problems this PEP attempts to solve are valid, most folks would prefer a
different approach.  Specifically, our read is that most users would prefer more control over the
ability to specify multiple indexes, how those indexes interoperate, and the priority and trust
assertions for those indexes.  For example, solutions such as :pep:`766` may provide a better way
forward.  Existing stop gap measures (e.g. PyPI limit increase requests and the `"wheel-stub"
<https://pypi.org/project/wheel-stub/>`_ approach) are sufficient -- if not ideal -- in the
meantime.  The authors wish to thank everyone who contributed to the constructive discussion, and
especially those who showed their support for this PEP, both in public and private.

Rationale
=========

The Python Package Index, hosted at https://pypi.org, imposes `default limits
<https://pypi.org/help/>`__ on upload artifact file size (100 MiB) and total project size
(10 GiB). Most projects can comfortably fit within these limits during the lifetime of the
project, through years of uploads. A few projects have encountered these limits, and have
been granted both file size and project size exceptions, allowing them to continue
uploading new releases without having to take more drastic measures, such as removing
files which may potentially still be in use by consumers (e.g. through version pins).

A related workaround is the `"wheel stub" <https://github.com/wheel-next/wheel-stub>`__
approach, which provides an indirect link between PyPI and an external third party package
index, where such limitations can be avoided. Wheel stubs are :ref:`source distributions
<packaging:source-distribution-format>` (a.k.a. "sdists") which utilize a :pep:`517` build
backend that, instead of turning source code into a binary wheel, performs some logic to
calculate the URL for an existing, externally hosted wheel to download and install.  This
approach works, but it obscures the connection between PyPI, the sdist, and the externally
hosted wheel, since there is no way to present this information to ``pip`` or other such
tools.

Historical context
------------------

In 2013, :pep:`438` proposed a "backward-compatible two-phase transition
process" to modify several aspects of release file hosting on PyPI. As this
PEP describes, PyPI originally supported only project and release
*registration* without also allowing for artifact file hosting. As such, most
projects hosted release file artifacts elsewhere. Artifact hosting was later
added, but the mix of externally and PyPI-hosted files led to a wide range of
usability and potential security-related problems. PEP 438 was an attempt to
provide several facilities to allow external hosting while promoting a
PyPI-first hosting preference.

PEP 438 was complex, with three different "hosting modes", ``rel`` metadata in
the simple HTML index pages to signify hosting locations, and a two-phase
transition plan affecting PyPI and installer tools.  PEP 438 was ultimately
retracted in 2015 by :pep:`470`, which acknowledges that PEP 438 did succeed
in...

   bringing about more people to utilize PyPI's repository features, an
   altogether good thing given the global CDN powering PyPI providing speed
   ups for a lot of people[...]

Instead of external hosting, PEP 470 promoted the use of explicit multiple
repositories, providing full package indexing and artifact hosting, and
enabled through installer tool support, such as ``pip install
--extra-index-url`` allowing ``pip`` to essentially treat multiple
repositories as `one single global repository
<https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-extra-index-url>`__
for package installation resolution. Because this has been the blessed norm
for so many years, all Python package installation tools support querying
multiple indexes for dependency resolution.

The problem with multiple indexes
---------------------------------

Why then does this PEP propose to allow a more limited form of external
hosting, and how does this proposal avoid the problems documented in PEP 470?

One well-known problem that consolidating multiple indexes enables is
`dependency confusion attacks
<https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610>`__, to
which Python *can* be particularly vulnerable, due to the algorithm that ``pip
install`` uses for resolving package dependencies and preferred versions. The
``uv`` tool addresses this by supporting an additional `index strategy
<https://docs.astral.sh/uv/reference/settings/#index-strategy>`__ option,
whereby users can select between, e.g. a ``pip``-compatible strategy, and a
more limited strategy that prevents such dependency confusion attacks.

:pep:`708` provides additional background about dependency confusion attacks,
and takes a different approach to preventing them. At its core, PEP 708 allows
repository owners to indicate that projects track across different
repositories, which allows installers to determine how to treat the global
package namespace when combined across multiple repositories. PEP 708 has been
provisionally accepted, pending several required conditions as outlined in PEP
708, some of which may have an indeterminate future. As PEP 708 itself says,
this won't by itself solve dependency confusion attacks, but is one way to
provide enough information to installers to help minimize these attacks.

While there can still be valid use cases for standing up a totally independent
package index (such as providing richer platform support for GPUs until a
fully formed `variant proposal
<https://discuss.python.org/t/selecting-variant-wheels-according-to-a-semi-static-specification/53446>`__
is accepted), this PEP takes a different, simpler approach and doesn't replace
any of the existing, proposed, or approved package index cooperation
specifications.

This PEP also preserves the core purpose of PyPI, and allows it to
remain the traditional, canonical, centralized index of all Python
packages.

Addressing PyPI limits
----------------------

This proposal also addresses the problem of size limits imposed by PyPI, where there is a
`default artifact size limit <https://pypi.org/help/#file-size-limit>`__ of 100 MiB and a
default overall `project size limit <https://pypi.org/help/#project-size-limit>`__ of 10
GiB. Most packages and artifacts can easily


id: pep-0598.rst
content:
PEP: 598
Title: Introducing incremental feature releases
Author: Alyssa Coghlan <ncoghlan@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-596-python-3-9-release-schedule-doubling-the-release-cadence/1828
Status: Withdrawn
Type: Informational
Created: 15-Jun-2019
Python-Version: 3.9


Abstract
========

:pep:`602` proposes reducing the feature delivery latency for the Python
standard library and CPython reference interpreter by increasing the frequency
of CPython feature releases from every 18-24 months to instead occur every 9-12
months.

This PEP proposes to instead *reduce* the frequency of new baseline feature
releases (with the associated filesystem layout changes, bytecode format
changes, and C ABI compatibility breaks) to occur only every other year (2020,
2022, 2024, etc), but to combine that change with a new policy and approach that
allows the introduction of backwards compatible features in the initial set of
point releases within a given release series.


PEP Withdrawal
==============

This PEP has been withdrawn in favour of the rolling beta release stream
proposal in :pep:`605`.

However, the concerns raised in this PEP are likely to apply to any other
"Long Term Support branch" proposals that allow feature backports to improve
the developer experience of supporting such releases (such as the EL Python
draft at [3]_), so the ideas presented here may provide useful design
suggestions for such proposals.


Summary
=======

The proposal to keep the current CPython release compatibility management
process, but go through it more often has significant practical downsides,
as a CPython feature release carries certain expectations (most notably, a 5-year
maintenance lifecycle, support for parallel installation with the previous
feature release, and the possibility of breaking changes to the CPython-specific
ABI, requiring recompilation of all extension modules) that mean faster feature
releases in their current form have the potential to significantly increase the
burden of maintaining 3rd party Python libraries and applications across all
actively supported CPython releases.

It's also arguable whether such an approach would noticeably reduce the typical
feature delivery latency in practice for most end users, as the adoption cycle
for new feature releases is typically measured in months or years, so more
frequent releases may just lead to end users updating to every 3rd or 4th
feature release, rather than every 2nd or 3rd feature release (as often happens
today).

This PEP presents a competing proposal to instead *slow down* the frequency of
parallel installable feature releases that change the filesystem layout
and CPython ABI to a consistent 24-month cycle, but to compensate for this by
introducing the notion of build compatible incremental feature releases, and
then deferring the full feature freeze of a given feature release series from
the initial baseline X.Y.0 release to a subsequent X.Y.Z feature complete
release that occurs ~12 months after the initial baseline feature release.

A new ``feature_complete`` attribute on the ``sys.version_info`` structure will
provide a programmatic indicator as to whether or not a release series remains
open to further incremental feature releases. Alternate implementations of
Python would also be free to clear this flag to indicate that their support for
their nominal Python version may still be a work in progress.

For compatibility testing purposes, and to maintain pickle compatibility in
mixed version environments, a new ``sys.feature_limit`` attribute (and
corresponding CPython CLI parameter, ``--feature-limit X.Y.Z``, and environment
variable, ``PYTHONFEATURELIMIT``) will provide a way to limit the runtime
availability of features added in incremental feature releases.

The existing cycle and the new cycle would be synchronised on their feature
freeze release dates, so the full Python 3.9.x feature freeze would occur in
October 2021, 24 months after the Python 3.8.0 feature release, with the initial
Python 3.9.0 release taking place in October 2020.


Example Future Release Schedules
================================

Under this proposal, Python 3.9.0a1 would be released in November 2019, shortly
after the Python 3.8.0 feature complete release in October 2019.

The 3.9.0b1 release would then follow 6 months later in May 2020, with 3.9.0
itself being released in October 2020.

Assuming micro releases of 3.9.x were to occur quarterly, then the overall
release timeline would look like:

* 2019-11: 3.9.0a1
* ... additional alpha releases as determined by the release manager
* 2020-05: 3.9.0b1
* ... additional beta releases as determined by the release manager
* 2020-08: 3.9.0bX (final beta release that locks ABI compatibility)
* 2020-09: 3.9.0rc1
* ... additional release candidates as determined by the release manager
* 2020-10: 3.9.0 (BFR - baseline feature release)
* 2021-01: 3.9.1 (IFR - incremental feature release)
* 2021-04: 3.9.2 (IFR)
* 2021-07: 3.9.3 (IFR)
* 2021-10: 3.9.4 (feature complete release)
* 2022-01: 3.9.5
* 2022-04: 3.9.6
* 2022-07: 3.9.7
* 2022-10: 3.9.8 (final regular maintenance release)
* ... additional security fix only releases as needed
* 2025-10: 3.9.x branch closed

Feature complete release numbers would typically be written without any
qualifier (as they are today), while the baseline and incremental feature
releases would be expected to have a qualifier appended indicating that they
aren't a traditional CPython release (``3.9.0 (BFR)``, ``3.9.1 (IFR)``, etc).

The Python 3.10 release series would start being published the month after the
first Python 3.9 feature complete release, in parallel with the final 12 months
of routine Python 3.9 maintenance releases:

* 2021-11: 3.10.0a1
* ... additional alpha releases as determined by the release manager
* 2022-05: 3.10.0b1
* ... additional beta releases as determined by the release manager
* 2022-08: 3.10.0bX (final beta release that locks ABI compatibility)
* 2022-09: 3.10.0rc1
* ... additional release candidates as determined by the release manager
* 2022-10: 3.10.0 (BFR)
* 2023-01: 3.10.1 (IFR)
* 2023-04: 3.10.2 (IFR)
* 2023-07: 3.10.3 (IFR)
* 2023-10: 3.10.4
* 2024-01: 3.10.5
* 2024-04: 3.10.6
* 2024-07: 3.10.7
* 2024-10: 3.10.8 (final regular maintenance release)
* ... additional security fix only releases as needed
* 2027-10: 3.10.x branch closed

In this model, there are always two or three active branches:

* 2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x maintenance
* 2019-10 -> 2020-05: 3.9.0 pre-beta, 3.8.x maintenance
* 2020-05 -> 2020-10: 3.10.0 pre-alpha, 3.9.0 pre-release, 3.8.x maintenance
* 2020-10 -> 2021-10: 3.10.0 pre-alpha, 3.9.x feature releases, 3.8.x maintenance
* 2021-10 -> 2022-05: 3.10.0 pre-beta, 3.9.x maintenance
* 2022-05 -> 2022-10: 3.11.0 pre-alpha, 3.10.0 pre-release, 3.9.x maintenance
* 2022-10 -> 2023-10: 3.11.0 pre-alpha, 3.10.x feature releases, 3.9.x maintenance
* 2023-10 -> 2024-05: 3.11.0 pre-beta, 3.10.x maintenance
* 2024-05 -> 2024-10: 3.12.0 pre-alpha, 3.11.0 pre-release, 3.10.x maintenance
* ... etc

(Pre-alpha and pre-beta development occurs on the main git branch, all other
development occurs on a release specific branch with changes typically
backported from the main git branch)

TODO: this really needs a diagram to help explain it, so I'll add a picture
once I have one to add.

This is quite similar to the status quo, but with a more consistent cadence,
alternating between baseline feature release years (2020, 2022, etc) that focus
on the alpha and beta cycle for a new baseline feature release (while continuing
to publish maintenance releases for the previous feature release series), and
feature complete release years (2021, 2023, etc), that focus on making
smaller improvements to the current feature release series (while making plans
for the next feature release series the following year).


Proposal
========

Excluding alpha and beta releases, CPyth


id: pep-0201.rst
content:
PEP: 201
Title: Lockstep Iteration
Author: Barry Warsaw <barry@python.org>
Status: Final
Type: Standards Track
Created: 13-Jul-2000
Python-Version: 2.0
Post-History: 27-Jul-2000



Introduction
============

This PEP describes the 'lockstep iteration' proposal.  This PEP tracks
the status and ownership of this feature, slated for introduction in
Python 2.0.  It contains a description of the feature and outlines
changes necessary to support the feature. This PEP summarizes
discussions held in mailing list forums, and provides URLs for further
information, where appropriate.  The CVS revision history of this file
contains the definitive historical record.


Motivation
==========

Standard for-loops in Python iterate over every element in a sequence
until the sequence is exhausted [1]_.  However, for-loops iterate over
only a single sequence, and it is often desirable to loop over more
than one sequence in a lock-step fashion.  In other words, in a way
such that the i-th iteration through the loop returns an object
containing the i-th element from each sequence.

The common idioms used to accomplish this are unintuitive.  This PEP
proposes a standard way of performing such iterations by introducing a
new builtin function called ``zip``.

While the primary motivation for zip() comes from lock-step iteration,
by implementing zip() as a built-in function, it has additional
utility in contexts other than for-loops.

Lockstep For-Loops
==================

Lockstep for-loops are non-nested iterations over two or more
sequences, such that at each pass through the loop, one element from
each sequence is taken to compose the target.  This behavior can
already be accomplished in Python through the use of the map() built-in
function::

    >>> a = (1, 2, 3)
    >>> b = (4, 5, 6)
    >>> for i in map(None, a, b): print i
    ...
    (1, 4)
    (2, 5)
    (3, 6)
    >>> map(None, a, b)
    [(1, 4), (2, 5), (3, 6)]

The for-loop simply iterates over this list as normal.

While the map() idiom is a common one in Python, it has several
disadvantages:

* It is non-obvious to programmers without a functional programming
  background.

* The use of the magic ``None`` first argument is non-obvious.

* It has arbitrary, often unintended, and inflexible semantics when
  the lists are not of the same length: the shorter sequences are
  padded with ``None``::

    >>> c = (4, 5, 6, 7)
    >>> map(None, a, c)
    [(1, 4), (2, 5), (3, 6), (None, 7)]

For these reasons, several proposals were floated in the Python 2.0
beta time frame for syntactic support of lockstep for-loops. Here are
two suggestions::

  for x in seq1, y in seq2:
    # stuff

::

  for x, y in seq1, seq2:
    # stuff

Neither of these forms would work, since they both already mean
something in Python and changing the meanings would break existing
code.  All other suggestions for new syntax suffered the same problem,
or were in conflict with other another proposed feature called 'list
comprehensions' (see :pep:`202`).

The Proposed Solution
=====================

The proposed solution is to introduce a new built-in sequence
generator function, available in the ``__builtin__`` module.  This
function is to be called ``zip`` and has the following signature::

    zip(seqa, [seqb, [...]])

``zip()`` takes one or more sequences and weaves their elements
together, just as ``map(None, ...)`` does with sequences of equal
length.  The weaving stops when the shortest sequence is exhausted.


Return Value
============

``zip()`` returns a real Python list, the same way ``map()`` does.


Examples
========

Here are some examples, based on the reference implementation below::

    >>> a = (1, 2, 3, 4)
    >>> b = (5, 6, 7, 8)
    >>> c = (9, 10, 11)
    >>> d = (12, 13)

    >>> zip(a, b)
    [(1, 5), (2, 6), (3, 7), (4, 8)]

    >>> zip(a, d)
    [(1, 12), (2, 13)]

    >>> zip(a, b, c, d)
    [(1, 5, 9, 12), (2, 6, 10, 13)]

Note that when the sequences are of the same length, ``zip()`` is
reversible::

    >>> a = (1, 2, 3)
    >>> b = (4, 5, 6)
    >>> x = zip(a, b)
    >>> y = zip(*x) # alternatively, apply(zip, x)
    >>> z = zip(*y) # alternatively, apply(zip, y)
    >>> x
    [(1, 4), (2, 5), (3, 6)]
    >>> y
    [(1, 2, 3), (4, 5, 6)]
    >>> z
    [(1, 4), (2, 5), (3, 6)]
    >>> x == z
    1

It is not possible to reverse zip this way when the sequences are not
all the same length.


Reference Implementation
========================

Here is a reference implementation, in Python of the zip() built-in
function.  This will be replaced with a C implementation after final
approval::

    def zip(*args):
        if not args:
            raise TypeError('zip() expects one or more sequence arguments')
        ret = []
        i = 0
        try:
            while 1:
                item = []
                for s in args:
                    item.append(s[i])
                ret.append(tuple(item))
                i = i + 1
        except IndexError:
            return ret



BDFL Pronouncements
===================

Note: the BDFL refers to Guido van Rossum, Python's Benevolent
Dictator For Life.

* The function's name.  An earlier version of this PEP included an
  open issue listing 20+ proposed alternative names to ``zip()``.  In
  the face of no overwhelmingly better choice, the BDFL strongly
  prefers ``zip()`` due to its Haskell [2]_ heritage.  See version 1.7
  of this PEP for the list of alternatives.

* ``zip()`` shall be a built-in function.

* Optional padding.  An earlier version of this PEP proposed an
  optional ``pad`` keyword argument, which would be used when the
  argument sequences were not the same length.  This is similar
  behavior to the ``map(None, ...)`` semantics except that the user
  would be able to specify pad object.  This has been rejected by the
  BDFL in favor of always truncating to the shortest sequence, because
  of the KISS principle.  If there's a true need, it is easier to add
  later.  If it is not needed, it would still be impossible to delete
  it in the future.

* Lazy evaluation.  An earlier version of this PEP proposed that
  ``zip()`` return a built-in object that performed lazy evaluation
  using ``__getitem__()`` protocol.  This has been strongly rejected
  by the BDFL in favor of returning a real Python list.  If lazy
  evaluation is desired in the future, the BDFL suggests an ``xzip()``
  function be added.

* ``zip()`` with no arguments.  the BDFL strongly prefers this raise a
  TypeError exception.

* ``zip()`` with one argument.  the BDFL strongly prefers that this
  return a list of 1-tuples.

* Inner and outer container control.  An earlier version of this PEP
  contains a rather lengthy discussion on a feature that some people
  wanted, namely the ability to control what the inner and outer
  container types were (they are tuples and list respectively in this
  version of the PEP).  Given the simplified API and implementation,
  this elaboration is rejected.  For a more detailed analysis, see
  version 1.7 of this PEP.

Subsequent Change to ``zip()``
==============================

In Python 2.4, zip() with no arguments was modified to return an empty
list rather than raising a TypeError exception.  The rationale for the
original behavior was that the absence of arguments was thought to
indicate a programming error.  However, that thinking did not
anticipate the use of zip() with the ``*`` operator for unpacking
variable length argument lists.  For example, the inverse of zip could
be defined as:  ``unzip = lambda s: zip(*s)``.  That transformation
also defines a matrix transpose or an equivalent row/column swap for
tables defined as lists of tuples.  The latter transformation is
commonly used when reading data files with records as rows and fields
as columns.  For example, the code::

    date, rain, high, low = zip(*csv.reader(file("weather.csv")))

rearranges columnar data so that each field is collected into
individual tuples for straightforward looping and s


id: pep-0567.rst
content:
PEP: 567
Title: Context Variables
Author: Yury Selivanov <yury@edgedb.com>
Status: Final
Type: Standards Track
Created: 12-Dec-2017
Python-Version: 3.7
Post-History: 12-Dec-2017, 28-Dec-2017, 16-Jan-2018


Abstract
========

This PEP proposes a new ``contextvars`` module and a set of new
CPython C APIs to support context variables.  This concept is
similar to thread-local storage (TLS), but, unlike TLS, it also allows
correctly keeping track of values per asynchronous task, e.g.
``asyncio.Task``.

This proposal is a simplified version of :pep:`550`.  The key
difference is that this PEP is concerned only with solving the case
for asynchronous tasks, not for generators.  There are no proposed
modifications to any built-in types or to the interpreter.

This proposal is not strictly related to Python Context Managers.
Although it does provide a mechanism that can be used by Context
Managers to store their state.


API Design and Implementation Revisions
=======================================

In **Python 3.7.1** the signatures of all context variables
C APIs were **changed** to use ``PyObject *`` pointers instead
of ``PyContext *``, ``PyContextVar *``, and ``PyContextToken *``,
e.g.::

    // in 3.7.0:
    PyContext *PyContext_New(void);

    // in 3.7.1+:
    PyObject *PyContext_New(void);

See [6]_ for more details.  The `C API`_ section of this PEP was
updated to reflect the change.


Rationale
=========

Thread-local variables are insufficient for asynchronous tasks that
execute concurrently in the same OS thread.  Any context manager that
saves and restores a context value using ``threading.local()`` will
have its context values bleed to other code unexpectedly when used
in async/await code.

A few examples where having a working context local storage for
asynchronous code is desirable:

* Context managers like ``decimal`` contexts and ``numpy.errstate``.

* Request-related data, such as security tokens and request
  data in web applications, language context for ``gettext``, etc.

* Profiling, tracing, and logging in large code bases.


Introduction
============

The PEP proposes a new mechanism for managing context variables.
The key classes involved in this mechanism are ``contextvars.Context``
and ``contextvars.ContextVar``.  The PEP also proposes some policies
for using the mechanism around asynchronous tasks.

The proposed mechanism for accessing context variables uses the
``ContextVar`` class.  A module (such as ``decimal``) that wishes to
use the new mechanism should:

* declare a module-global variable holding a ``ContextVar`` to
  serve as a key;

* access the current value via the ``get()`` method on the
  key variable;

* modify the current value via the ``set()`` method on the
  key variable.

The notion of "current value" deserves special consideration:
different asynchronous tasks that exist and execute concurrently
may have different values for the same key.  This idea is well known
from thread-local storage but in this case the locality of the value is
not necessarily bound to a thread.  Instead, there is the notion of the
"current ``Context``" which is stored in thread-local storage.
Manipulation of the current context is the responsibility of the
task framework, e.g. asyncio.

A ``Context`` is a mapping of ``ContextVar`` objects to their values.
The ``Context`` itself exposes the ``abc.Mapping`` interface
(not ``abc.MutableMapping``!), so it cannot be modified directly.
To set a new value for a context variable in a ``Context`` object,
the user needs to:

* make the ``Context`` object "current" using the ``Context.run()``
  method;

* use ``ContextVar.set()`` to set a new value for the context
  variable.

The ``ContextVar.get()`` method looks for the variable in the current
``Context`` object using ``self`` as a key.

It is not possible to get a direct reference to the current ``Context``
object, but it is possible to obtain a shallow copy of it using the
``contextvars.copy_context()`` function.  This ensures that the
*caller* of ``Context.run()`` is the sole owner of its ``Context``
object.


Specification
=============

A new standard library module ``contextvars`` is added with the
following APIs:

1. The ``copy_context() -> Context`` function is used to get a copy of
   the current ``Context`` object for the current OS thread.

2. The ``ContextVar`` class to declare and access context variables.

3. The ``Context`` class encapsulates context state.  Every OS thread
   stores a reference to its current ``Context`` instance.
   It is not possible to control that reference directly.
   Instead, the ``Context.run(callable, *args, **kwargs)`` method is
   used to run Python code in another context.


contextvars.ContextVar
----------------------

The ``ContextVar`` class has the following constructor signature:
``ContextVar(name, *, default=_NO_DEFAULT)``.  The ``name`` parameter
is used for introspection and debug purposes, and is exposed
as a read-only ``ContextVar.name`` attribute.  The ``default``
parameter is optional.  Example::

    # Declare a context variable 'var' with the default value 42.
    var = ContextVar('var', default=42)

(The ``_NO_DEFAULT`` is an internal sentinel object used to
detect if the default value was provided.)

``ContextVar.get(default=_NO_DEFAULT)`` returns a value for
the context variable for the current ``Context``::

    # Get the value of `var`.
    var.get()

If there is no value for the variable in the current context,
``ContextVar.get()`` will:

* return the value of the *default* argument of the ``get()`` method,
  if provided; or

* return the default value for the context variable, if provided; or

* raise a ``LookupError``.

``ContextVar.set(value) -> Token`` is used to set a new value for
the context variable in the current ``Context``::

    # Set the variable 'var' to 1 in the current context.
    var.set(1)

``ContextVar.reset(token)`` is used to reset the variable in the
current context to the value it had before the ``set()`` operation
that created the ``token`` (or to remove the variable if it was
not set)::

    # Assume: var.get(None) is None

    # Set 'var' to 1:
    token = var.set(1)
    try:
        # var.get() == 1
    finally:
        var.reset(token)

    # After reset: var.get(None) is None,
    # i.e. 'var' was removed from the current context.

The ``ContextVar.reset()`` method raises:

* a ``ValueError`` if it is called with a token object created
  by another variable;

* a ``ValueError`` if the current ``Context`` object does not match
  the one where the token object was created;

* a ``RuntimeError`` if the token object has already been used once
  to reset the variable.


contextvars.Token
-----------------

``contextvars.Token`` is an opaque object that should be used to
restore the ``ContextVar`` to its previous value, or to remove it from
the context if the variable was not set before.  It can be created
only by calling ``ContextVar.set()``.

For debug and introspection purposes it has:

* a read-only attribute ``Token.var`` pointing to the variable
  that created the token;

* a read-only attribute ``Token.old_value`` set to the value the
  variable had before the ``set()`` call, or to ``Token.MISSING``
  if the variable wasn't set before.


contextvars.Context
-------------------

``Context`` object is a mapping of context variables to values.

``Context()`` creates an empty context.  To get a copy of the current
``Context`` for the current OS thread, use the
``contextvars.copy_context()`` method::

    ctx = contextvars.copy_context()

To run Python code in some ``Context``, use ``Context.run()``
method::

    ctx.run(function)

Any changes to any context variables that ``function`` causes will
be contained in the ``ctx`` context::

    var = ContextVar('var')
    var.set('spam')

    def main():
        # 'var' was set to 'spam' before
        # calling 'copy_context()' and 'ctx.run(main)', so:
        # var.get() == ctx[var] == 'spam'

        var.set


id: pep-0573.rst
content:
PEP: 573
Title: Module State Access from C Extension Methods
Author: Petr Viktorin <encukou@gmail.com>,
        Alyssa Coghlan <ncoghlan@gmail.com>,
        Eric Snow <ericsnowcurrently@gmail.com>,
        Marcel Plch <gmarcel.plch@gmail.com>
BDFL-Delegate: Stefan Behnel
Discussions-To: import-sig@python.org
Status: Final
Type: Standards Track
Created: 02-Jun-2016
Python-Version: 3.9
Post-History:


Abstract
========

This PEP proposes to add a way for CPython extension methods to access context,
such as the state of the modules they are defined in.

This will allow extension methods to use direct pointer dereferences
rather than PyState_FindModule for looking up module state, reducing or
eliminating the performance cost of using module-scoped state over process
global state.

This fixes one of the remaining roadblocks for adoption of :pep:`3121` (Extension
module initialization and finalization) and :pep:`489`
(Multi-phase extension module initialization).

While this PEP takes an additional step towards fully solving the problems that
:pep:`3121` and :pep:`489` started tackling, it does not attempt to resolve *all*
remaining concerns. In particular, access to the module state
from slot methods (``nb_add``, etc) is not solved.


Terminology
===========

Process-Global State
--------------------

C-level static variables. Since this is very low-level
memory storage, it must be managed carefully.

Per-module State
----------------

State local to a module object, allocated dynamically as part of a
module object's initialization. This isolates the state from other
instances of the module (including those in other subinterpreters).

Accessed by ``PyModule_GetState()``.


Static Type
-----------

A type object defined as a C-level static variable, i.e. a compiled-in type
object.

A static type needs to be shared between module instances and has no
information of what module it belongs to.
Static types do not have ``__dict__`` (although their instances might).


Heap Type
---------

A type object created at run time.


Defining Class
--------------

The defining class of a method (either bound or unbound) is the class on which
the method was defined.
A class that merely inherits the method from its base is not the defining class.

For example, ``int`` is the defining class of ``True.to_bytes``,
``True.__floor__`` and ``int.__repr__``.

In C, the defining class is the one defined with the corresponding
``tp_methods`` or "tp slots" [#tp-slots]_ entry.
For methods defined in Python, the defining class is saved in the
``__class__`` closure cell.


C-API
-----

The "Python/C API" as described in Python documentation.
CPython implements the C-API, but other implementations exist.


Rationale
=========

:pep:`489` introduced a new way to initialize extension modules, which brings
several advantages to extensions that implement it:

* The extension modules behave more like their Python counterparts.
* The extension modules can easily support loading into pre-existing
  module objects, which paves the way for extension module support for
  ``runpy`` or for systems that enable extension module reloading.
* Loading multiple modules from the same extension is possible, which
  makes it possible to test module isolation (a key feature for proper
  sub-interpreter support) from a single interpreter.

The biggest hurdle for adoption of :pep:`489` is allowing access to module state
from methods of extension types.
Currently, the way to access this state from extension methods is by looking up
the module via ``PyState_FindModule`` (in contrast to module level functions in
extension modules, which receive a module reference as an argument).
However, ``PyState_FindModule`` queries the thread-local state, making it
relatively costly compared to C level process global access and consequently
deterring module authors from using it.

Also, ``PyState_FindModule`` relies on the assumption that in each
subinterpreter, there is at most one module corresponding to
a given ``PyModuleDef``.  This assumption does not hold for modules that use
:pep:`489`'s multi-phase initialization, so ``PyState_FindModule`` is unavailable
for these modules.

A faster, safer way of accessing module-level state from extension methods
is needed.


Background
===========

The implementation of a Python method may need access to one or more of
the following pieces of information:

* The instance it is called on (``self``)
* The underlying function
* The *defining class*, i. e. the class the method was defined in
* The corresponding module
* The module state

In Python code, the Python-level equivalents may be retrieved as::

    import sys

    class Foo:
        def meth(self):
            instance = self
            module_globals = globals()
            module_object = sys.modules[__name__]  # (1)
            underlying_function = Foo.meth         # (1)
            defining_class = Foo                   # (1)
            defining_class = __class__             # (2)

.. note::

    The defining class is not ``type(self)``, since ``type(self)`` might
    be a subclass of ``Foo``.

The statements marked (1) implicitly rely on name-based lookup via the
function's ``__globals__``: either the ``Foo`` attribute to access the defining
class and Python function object, or ``__name__`` to find the module object in
``sys.modules``.

In Python code, this is feasible, as ``__globals__`` is set appropriately when
the function definition is executed, and even if the namespace has been
manipulated to return a different object, at worst an exception will be raised.

The ``__class__`` closure, (2), is a safer way to get the defining class, but it
still relies on ``__closure__`` being set appropriately.

By contrast, extension methods are typically implemented as normal C functions.
This means that they only have access to their arguments and C level thread-local
and process-global states. Traditionally, many extension modules have stored
their shared state in C-level process globals, causing problems when:

* running multiple initialize/finalize cycles in the same process
* reloading modules (e.g. to test conditional imports)
* loading extension modules in subinterpreters

:pep:`3121` attempted to resolve this by offering the ``PyState_FindModule`` API,
but this still has significant problems when it comes to extension methods
(rather than module level functions):

* it is markedly slower than directly accessing C-level process-global state
* there is still some inherent reliance on process global state that means it
  still doesn't reliably handle module reloading

It's also the case that when looking up a C-level struct such as module state,
supplying an unexpected object layout can crash the interpreter, so it's
significantly more important to ensure that extension methods receive the kind
of object they expect.


Proposal
========

Currently, a bound extension method (``PyCFunction`` or
``PyCFunctionWithKeywords``) receives only ``self``, and (if applicable) the
supplied positional and keyword arguments.

While module-level extension functions already receive access to the defining
module object via their ``self`` argument, methods of extension types don't have
that luxury: they receive the bound instance via ``self``, and hence have no
direct access to the defining class or the module level state.

The additional module level context described above can be made available with
two changes.
Both additions are optional; extension authors need to opt in to start
using them:

* Add a pointer to the module to heap type objects.

* Pass the defining class to the underlying C function.

  In CPython, the defining class is readily available at the time the built-in
  method object (``PyCFunctionObject``) is created, so it can be stored
  in a new struct that extends ``PyCFunctionObject``.

The module state can then be retrieved from the module object via
``PyModule_GetState``.

Note that this proposal implies that any type


id: pep-0215.rst
content:
PEP: 215
Title: String Interpolation
Author: Ka-Ping Yee <ping@zesty.ca>
Status: Superseded
Type: Standards Track
Created: 24-Jul-2000
Python-Version: 2.1
Post-History:
Superseded-By: 292

.. superseded:: 292

Abstract
========

This document proposes a string interpolation feature for Python
to allow easier string formatting.  The suggested syntax change
is the introduction of a '$' prefix that triggers the special
interpretation of the '$' character within a string, in a manner
reminiscent to the variable interpolation found in Unix shells,
awk, Perl, or Tcl.


Copyright
=========

This document is in the public domain.


Specification
=============

Strings may be preceded with a '$' prefix that comes before the
leading single or double quotation mark (or triplet) and before
any of the other string prefixes ('r' or 'u').  Such a string is
processed for interpolation after the normal interpretation of
backslash-escapes in its contents.  The processing occurs just
before the string is pushed onto the value stack, each time the
string is pushed.  In short, Python behaves exactly as if '$'
were a unary operator applied to the string.  The operation
performed is as follows:

The string is scanned from start to end for the '$' character
(``\x24`` in 8-bit strings or ``\u0024`` in Unicode strings).  If there
are no '$' characters present, the string is returned unchanged.

Any '$' found in the string, followed by one of the two kinds of
expressions described below, is replaced with the value of the
expression as evaluated in the current namespaces.  The value is
converted with ``str()`` if the containing string is an 8-bit string,
or with ``unicode()`` if it is a Unicode string.

1.  A Python identifier optionally followed by any number of
    trailers, where a trailer consists of:
    - a dot and an identifier,
    - an expression enclosed in square brackets, or
    - an argument list enclosed in parentheses
    (This is exactly the pattern expressed in the Python grammar
    by "``NAME trailer*``", using the definitions in ``Grammar/Grammar``.)

2.  Any complete Python expression enclosed in curly braces.

Two dollar-signs ("$$") are replaced with a single "$".


Examples
========

Here is an example of an interactive session exhibiting the
expected behaviour of this feature. ::

   >>> a, b = 5, 6
   >>> print $'a = $a, b = $b'
   a = 5, b = 6
   >>> $u'uni${a}ode'
   u'uni5ode'
   >>> print $'\$a'
   5
   >>> print $r'\$a'
   \5
   >>> print $'$$$a.$b'
   $5.6
   >>> print $'a + b = ${a + b}'
   a + b = 11
   >>> import sys
   >>> print $'References to $a: $sys.getrefcount(a)'
   References to 5: 15
   >>> print $"sys = $sys, sys = $sys.modules['sys']"
   sys = <module 'sys' (built-in)>, sys = <module 'sys' (built-in)>
   >>> print $'BDFL = $sys.copyright.split()[4].upper()'
   BDFL = GUIDO


Discussion
==========

'$' is chosen as the interpolation character within the
string for the sake of familiarity, since it is already used
for this purpose in many other languages and contexts.

It is then natural to choose '$' as a prefix, since it is a
mnemonic for the interpolation character.

Trailers are permitted to give this interpolation mechanism
even more power than the interpolation available in most other
languages, while the expression to be interpolated remains
clearly visible and free of curly braces.

'$' works like an operator and could be implemented as an
operator, but that prevents the compile-time optimization
and presents security issues.  So, it is only allowed as a
string prefix.


Security Issues
===============

"$" has the power to eval, but only to eval a literal.  As
described here (a string prefix rather than an operator), it
introduces no new security issues since the expressions to be
evaluated must be literally present in the code.


Implementation
==============

The ``Itpl`` module at [1]_ provides a
prototype of this feature.  It uses the ``tokenize`` module to find
the end of an expression to be interpolated, then calls ``eval()``
on the expression each time a value is needed.  In the prototype,
the expression is parsed and compiled again each time it is
evaluated.

As an optimization, interpolated strings could be compiled
directly into the corresponding bytecode; that is, ::

   $'a = $a, b = $b'

could be compiled as though it were the expression ::

   ('a = ' + str(a) + ', b = ' + str(b))

so that it only needs to be compiled once.


References
==========

.. [1] http://www.lfw.org/python/Itpl.py


id: pep-0229.rst
content:
PEP: 229
Title: Using Distutils to Build Python
Author: A.M. Kuchling <amk@amk.ca>
Status: Final
Type: Standards Track
Created: 16-Nov-2000
Python-Version: 2.1
Post-History:


Introduction
============

The ``Modules/Setup`` mechanism has some flaws:

* People have to remember to uncomment bits of ``Modules/Setup`` in
  order to get all the possible modules.

* Moving ``Setup`` to a new version of Python is tedious; new modules
  have been added, so you can't just copy the older version, but
  have to reconcile the two versions.

* Users have to figure out where the needed libraries, such as
  ``zlib``, are installed.


Proposal
========

Use the Distutils to build the modules that come with Python.

The changes can be broken up into several pieces:

1. The Distutils needs some Python modules to be able to build
   modules.  Currently I believe the minimal list is ``posix``, ``_sre``,
   and ``string``.

   These modules will have to be built before the Distutils can be
   used, so they'll simply be hardwired into ``Modules/Makefile`` and
   be automatically built.

2. A top-level setup.py script will be written that checks the
   libraries installed on the system and compiles as many modules
   as possible.

3. ``Modules/Setup`` will be kept and settings in it will override
   ``setup.py``'s usual behavior, so you can disable a module known
   to be buggy, or specify particular compilation or linker flags.
   However, in the common case where ``setup.py`` works correctly,
   everything in ``Setup`` will remain commented out.  The other
   ``Setup.*`` become unnecessary, since nothing will be generating
   ``Setup`` automatically.

The patch was checked in for Python 2.1, and has been subsequently
modified.


Implementation
==============

Patch #102588 on SourceForge contains the proposed patch.
Currently the patch tries to be conservative and to change as few
files as possible, in order to simplify backing out the patch.
For example, no attempt is made to rip out the existing build
mechanisms.  Such simplifications can wait for later in the beta
cycle, when we're certain the patch will be left in, or they can
wait for Python 2.2.

The patch makes the following changes:

* Makes some required changes to distutils/sysconfig (these will
  be checked in separately)

* In the top-level ``Makefile.in``, the "sharedmods" target simply
  runs ``"./python setup.py build"``, and "sharedinstall" runs
  ``"./python setup.py install"``.  The "clobber" target also deletes
  the ``build/`` subdirectory where Distutils puts its output.

* ``Modules/Setup.config.in`` only contains entries for the ``gc`` and ``thread``
  modules; the ``readline``, ``curses``, and ``db`` modules are removed because
  it's now ``setup.py``'s job to handle them.

* ``Modules/Setup.dist`` now contains entries for only 3 modules --
  ``_sre``, ``posix``, and ``strop``.

* The ``configure`` script builds ``setup.cfg`` from ``setup.cfg.in``.  This
  is needed for two reasons: to make building in subdirectories
  work, and to get the configured installation prefix.

* Adds ``setup.py`` to the top directory of the source tree.  ``setup.py``
  is the largest piece of the puzzle, though not the most
  complicated.  ``setup.py`` contains a subclass of the ``BuildExt``
  class, and extends it with a ``detect_modules()`` method that does
  the work of figuring out when modules can be compiled, and adding
  them to the 'exts' list.


Unresolved Issues
=================

Do we need to make it possible to disable the 3 hard-wired modules
without manually hacking the Makefiles?  [Answer: No.]

The Distutils always compile modules as shared libraries.  How do
we support compiling them statically into the resulting Python
binary?

[Answer: building a Python binary with the Distutils should be
feasible, though no one has implemented it yet.  This should be
done someday, but isn't a pressing priority as messing around with
the top-level ``Makefile.pre.in`` is good enough.]


Copyright
=========

This document has been placed in the public domain.


id: pep-0514.rst
content:
PEP: 514
Title: Python registration in the Windows registry
Author: Steve Dower <steve.dower@python.org>
BDFL-Delegate: Paul Moore <p.f.moore@gmail.com>
Status: Active
Type: Informational
Created: 02-Feb-2016
Post-History: 02-Feb-2016, 01-Mar-2016, 18-Jul-2016
Resolution: https://mail.python.org/pipermail/python-dev/2016-July/145697.html

Abstract
========

This PEP defines a schema for the Python registry key to allow third-party
installers to register their installation, and to allow tools and applications
to detect and correctly display all Python environments on a user's machine. No
implementation changes to Python are proposed with this PEP.

Python environments are not required to be registered unless they want to be
automatically discoverable by external tools. As this relates to Windows only,
these tools are expected to be predominantly GUI applications. However, console
applications may also make use of the registered information. This PEP covers
the information that may be made available, but the actual presentation and use
of this information is left to the tool designers.

The schema matches the registry values that have been used by the official
installer since at least Python 2.5, and the resolution behaviour matches the
behaviour of the official Python releases. Some backwards compatibility rules
are provided to ensure tools can correctly detect versions of CPython that do
not register full information.

Motivation
==========

When installed on Windows, the official Python installer creates a registry key
for discovery and detection by other applications. This allows tools such as
installers or IDEs to automatically detect and display a user's Python
installations. For example, the :pep:`397` ``py.exe`` launcher and editors such as
PyCharm and Visual Studio already make use of this information.

Third-party installers, such as those used by distributions, typically create
identical keys for the same purpose. Most tools that use the registry to detect
Python installations only inspect the keys used by the official installer. As a
result, third-party installations that wish to be discoverable will overwrite
these values, often causing users to "lose" their original Python installation.

By describing a layout for registry keys that allows third-party installations
to register themselves uniquely, as well as providing tool developers guidance
for discovering all available Python installations, these collisions should be
prevented. We also take the opportunity to add some well-known metadata so that
more information can be presented to users.

Definitions
===========

A "registry key" is the equivalent of a file-system path into the registry. Each
key may contain "subkeys" (keys nested within keys) and "values" (named and
typed attributes attached to a key). These are used on Windows to store settings
in much the same way that directories containing configuration files would work.

``HKEY_CURRENT_USER`` is the root of settings for the currently logged-in user,
and this user can generally read and write all settings under this root.

``HKEY_LOCAL_MACHINE`` is the root of settings for all users. Generally, any
user can read these settings but only administrators can modify them. It is
typical for values under ``HKEY_CURRENT_USER`` to take precedence over those in
``HKEY_LOCAL_MACHINE``.

On 64-bit Windows, ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` is a special key
that 32-bit processes transparently read and write to rather than accessing the
``Software`` key directly.

Further documentation regarding registry redirection on Windows is available
from the MSDN Library [1]_.

Structure
=========

We consider there to be a single collection of Python environments on a machine,
where the collection may be different for each user of the machine. There are
three potential registry locations where the collection may be stored based on
the installation options of each environment::

    HKEY_CURRENT_USER\Software\Python\<Company>\<Tag>
    HKEY_LOCAL_MACHINE\Software\Python\<Company>\<Tag>
    HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\<Company>\<Tag>

Official Python releases use ``PythonCore`` for Company, and the value of
``sys.winver`` for Tag. The Company ``PyLauncher`` is reserved. Other registered
environments may use any values for Company and Tag. Recommendations are made
later in this document.

Company-Tag pairs are case-insensitive, and uniquely identify each environment.
Depending on the purpose and intended use of a tool, there are two suggested
approaches for resolving conflicts between Company-Tag pairs.

Tools that list every installed environment may choose to include those
even where the Company-Tag pairs match. They should ensure users can easily
identify whether the registration was per-user or per-machine, and which
registration has the higher priority.

Tools that aim to select a single installed environment from all registered
environments based on the Company-Tag pair, such as the ``py.exe`` launcher,
should always select the environment registered in ``HKEY_CURRENT_USER`` when
than the matching one in ``HKEY_LOCAL_MACHINE``.

Conflicts between ``HKEY_LOCAL_MACHINE\Software\Python`` and
``HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python`` should only occur when both
64-bit and 32-bit versions of an interpreter have the same Tag. In this case,
the tool should select whichever is more appropriate for its use.

If a tool is able to determine from the provided information (or lack thereof)
that it cannot use a registered environment, there is no obligation to present
it to users.

Except as discussed in the section on backwards compatibility, Company and Tag
values are considered opaque to tools, and no information about the interpreter
should be inferred from the text. However, some tools may display the Company
and Tag values to users, so ideally the Tag will be able to help users identify
the associated environment.

Python environments are not required to register themselves unless they want to
be automatically discoverable by external tools.

Backwards Compatibility
-----------------------

Python 3.4 and earlier did not distinguish between 32-bit and 64-bit builds in
``sys.winver``. As a result, it is not possible to have valid side-by-side
installations of both 32-bit and 64-bit interpreters under this scheme since it
would result in duplicate Tags.

To ensure backwards compatibility, applications should treat environments listed
under the following two registry keys as distinct, even when the Tag matches::

    HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<Tag>
    HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\<Tag>

Environments listed under ``HKEY_CURRENT_USER`` may be treated as distinct from
both of the above keys, potentially resulting in three environments discovered
using the same Tag. Alternatively, a tool may determine whether the per-user
environment is 64-bit or 32-bit and give it priority over the per-machine
environment, resulting in a maximum of two discovered environments.

It is not possible to detect side-by-side installations of both 64-bit and
32-bit versions of Python prior to 3.5 when they have been installed for the
current user. Python 3.5 and later always uses different Tags for 64-bit and
32-bit versions.

The following section describe user-visible information that may be registered.
For Python 3.5 and earlier, none of this information is available, but
alternative defaults are specified for the ``PythonCore`` key.

Environments registered under other Company names have no backward compatibility
requirements and must use distinct Tags to support side-by-side installations.
Tools consuming these registrations are not required to disambiguate tags other
than by preferring the user's setting.

Company
-------

The Company part of the key is intended to group related environments and to
ensure that Tags are namespaced appropriately. The key name should be
alphanumeric without spac


id: pep-0272.rst
content:
PEP: 272
Title: API for Block Encryption Algorithms v1.0
Author: A.M. Kuchling <amk@amk.ca>
Status: Final
Type: Informational
Created: 18-Sep-2001
Post-History: 17-Apr-2002, 29-May-2002


Abstract
========

This document specifies a standard API for secret-key block
encryption algorithms such as DES or Rijndael, making it easier to
switch between different algorithms and implementations.


Introduction
============

Encryption algorithms transform their input data (called
plaintext) in some way that is dependent on a variable key,
producing ciphertext.  The transformation can easily be reversed
if and only if one knows the key.  The key is a sequence of bits
chosen from some very large space of possible keys.  There are two
classes of encryption algorithms: block ciphers and stream ciphers.

Block ciphers encrypt multibyte inputs of a fixed size (frequently
8 or 16 bytes long), and can be operated in various feedback
modes.  The feedback modes supported in this specification are:

======  ========  =====================
Number  Constant  Description
======  ========  =====================
1       MODE_ECB  Electronic Code Book
2       MODE_CBC  Cipher Block Chaining
3       MODE_CFB  Cipher Feedback
5       MODE_OFB  Output Feedback
6       MODE_CTR  Counter
======  ========  =====================

These modes are to be implemented as described in NIST publication
SP 800-38A [1]_.  Descriptions of the first three feedback modes can
also be found in Bruce Schneier's book *Applied Cryptography* [2]_.

(The numeric value 4 is reserved for MODE_PGP, a variant of CFB
described in :rfc:`2440`: "OpenPGP Message Format". This mode
isn't considered important enough to make it worth requiring it
for all block encryption ciphers, though supporting it is a nice
extra feature.)

In a strict formal sense, stream ciphers encrypt data bit-by-bit;
practically, stream ciphers work on a character-by-character
basis.  This PEP only aims at specifying an interface for block
ciphers, though stream ciphers can support the interface described
here by fixing 'block_size' to 1.  Feedback modes also don't make
sense for stream ciphers, so the only reasonable feedback mode
would be ECB mode.


Specification
=============

Encryption modules can add additional functions, methods, and
attributes beyond those described in this PEP, but all of the
features described in this PEP must be present for a module to
claim compliance with it.

Secret-key encryption modules should define one function::

    new(key, mode, [IV], **kwargs)

Returns a ciphering object, using the secret key contained in the
string 'key', and using the feedback mode 'mode', which must be
one of the constants from the table above.

If 'mode' is MODE_CBC or MODE_CFB, 'IV' must be provided and must
be a string of the same length as the block size.  Not providing a
value of 'IV' will result in a ``ValueError`` exception being raised.

Depending on the algorithm, a module may support additional
keyword arguments to this function.  Some keyword arguments are
specified by this PEP, and modules are free to add additional
keyword arguments.  If a value isn't provided for a given keyword,
a secure default value should be used.  For example, if an
algorithm has a selectable number of rounds between 1 and 16, and
1-round encryption is insecure and 8-round encryption is believed
secure, the default value for 'rounds' should be 8 or more.
(Module implementors can choose a very slow but secure value, too,
such as 16 in this example.  This decision is left up to the
implementor.)

The following table lists keyword arguments defined by this PEP:

============  ============================================
Keyword       Meaning
============  ============================================
counter       Callable object that returns counter blocks
              (see below; CTR mode only)

rounds        Number of rounds of encryption to use

segment_size  Size of data and ciphertext segments,
              measured in bits (see below; CFB mode only)
============  ============================================

The Counter feedback mode requires a sequence of input blocks,
called counters, that are used to produce the output.  When 'mode'
is MODE_CTR, the 'counter' keyword argument must be provided, and
its value must be a callable object, such as a function or method.
Successive calls to this callable object must return a sequence of
strings that are of the length 'block_size' and that never
repeats.  (Appendix B of the NIST publication gives a way to
generate such a sequence, but that's beyond the scope of this
PEP.)

The CFB mode operates on segments of the plaintext and ciphertext
that are 'segment_size' bits long.  Therefore, when using this
mode, the input and output strings must be a multiple of
'segment_size' bits in length.  'segment_size' must be an integer
between 1 and block_size\*8, inclusive.  (The factor of 8 comes
from 'block_size' being measured in bytes and not in bits).  The
default value for this parameter should be block_size\*8.
Implementors are allowed to constrain 'segment_size' to be a
multiple of 8 for simplicity, but they're encouraged to support
arbitrary values for generality.

Secret-key encryption modules should define two variables:

- block_size

  An integer value; the size of the blocks encrypted by this
  module, measured in bytes.  For all feedback modes, the length
  of strings passed to the encrypt() and decrypt() must be a
  multiple of the block size.

- key_size

  An integer value; the size of the keys required by this
  module, measured in bytes.  If key_size is None, then the
  algorithm accepts variable-length keys.  This may mean the
  module accepts keys of any random length, or that there are a
  few different possible lengths, e.g. 16, 24, or 32 bytes.  You
  cannot pass a key of length 0 (that is, the null string '') as
  a variable-length key.


Cipher objects should have two attributes:

- block_size

  An integer value equal to the size of the blocks encrypted by
  this object.  For algorithms with a variable block size, this
  value is equal to the block size selected for this object.

- IV

  Contains the initial value which will be used to start a
  cipher feedback mode; it will always be a string exactly one
  block in length.  After encrypting or decrypting a string,
  this value is updated to reflect the modified feedback text.
  It is read-only, and cannot be assigned a new value.


Cipher objects require the following methods:

- decrypt(string)

  Decrypts 'string', using the key-dependent data in the object
  and with the appropriate feedback mode.  The string's length
  must be an exact multiple of the algorithm's block size or, in
  CFB mode, of the segment size.  Returns a string containing
  the plaintext.

- encrypt(string)

  Encrypts a non-empty string, using the key-dependent data in
  the object, and with the appropriate feedback mode.  The
  string's length must be an exact multiple of the algorithm's
  block size or, in CFB mode, of the segment size.  Returns a
  string containing the ciphertext.

Here's an example, using a module named 'DES'::

    >>> import DES
    >>> obj = DES.new('abcdefgh', DES.MODE_ECB)
    >>> plaintext = "Guido van Rossum is a space alien."
    >>> len(plaintext)
    34
    >>> obj.encrypt(plaintext)
    Traceback (innermost last):
      File "<stdin>", line 1, in ?
    ValueError: Strings for DES must be a multiple of 8 in length
    >>> ciphertext = obj.encrypt(plain+'XXXXXX')   # Add padding
    >>> ciphertext
    '\021,\343Nq\214DY\337T\342pA\372\255\311s\210\363,\300j\330\250\312\347\342I\3215w\03561\303dgb/\006'
    >>> obj.decrypt(ciphertext)
    'Guido van Rossum is a space alien.XXXXXX'


References
==========

.. [1] NIST publication SP 800-38A, "Recommendation for Block Cipher
       Modes of Operation" (http://csrc.nist.gov/encryption/modes/)

.. [2] Applied Cryptography


Changes
=======

2002-04: Removed references to strea


id: pep-0266.rst
content:
PEP: 266
Title: Optimizing Global Variable/Attribute Access
Author: Skip Montanaro <skip@pobox.com>
Status: Withdrawn
Type: Standards Track
Created: 13-Aug-2001
Python-Version: 2.3
Post-History:


Abstract
========

The bindings for most global variables and attributes of other modules
typically never change during the execution of a Python program, but because
of Python's dynamic nature, code which accesses such global objects must run
through a full lookup each time the object is needed.  This PEP proposes a
mechanism that allows code that accesses most global objects to treat them as
local objects and places the burden of updating references on the code that
changes the name bindings of such objects.


Introduction
============

Consider the workhorse function ``sre_compile._compile``.  It is the internal
compilation function for the ``sre`` module.  It consists almost entirely of a
loop over the elements of the pattern being compiled, comparing opcodes with
known constant values and appending tokens to an output list.  Most of the
comparisons are with constants imported from the ``sre_constants`` module.
This means there are lots of ``LOAD_GLOBAL`` bytecodes in the compiled output
of this module.  Just by reading the code it's apparent that the author
intended ``LITERAL``, ``NOT_LITERAL``, ``OPCODES`` and many other symbols to
be constants.  Still, each time they are involved in an expression, they must
be looked up anew.

Most global accesses are actually to objects that are "almost constants".
This includes global variables in the current module as well as the attributes
of other imported modules.  Since they rarely change, it seems reasonable to
place the burden of updating references to such objects on the code that
changes the name bindings.  If ``sre_constants.LITERAL`` is changed to refer
to another object, perhaps it would be worthwhile for the code that modifies
the ``sre_constants`` module dict to correct any active references to that
object.  By doing so, in many cases global variables and the attributes of
many objects could be cached as local variables.  If the bindings between the
names given to the objects and the objects themselves changes rarely, the cost
of keeping track of such objects should be low and the potential payoff fairly
large.

In an attempt to gauge the effect of this proposal, I modified the Pystone
benchmark program included in the Python distribution to cache global
functions.  Its main function, ``Proc0``, makes calls to ten different
functions inside its ``for`` loop.  In addition, ``Func2`` calls ``Func1``
repeatedly inside a loop.  If local copies of these 11 global identifiers are
made before the functions' loops are entered, performance on this particular
benchmark improves by about two percent (from 5561 pystones to 5685 on my
laptop).  It gives some indication that performance would be improved by
caching most global variable access.  Note also that the pystone benchmark
makes essentially no accesses of global module attributes, an anticipated area
of improvement for this PEP.


Proposed Change
===============

I propose that the Python virtual machine be modified to include
``TRACK_OBJECT`` and ``UNTRACK_OBJECT`` opcodes.  ``TRACK_OBJECT`` would
associate a global name or attribute of a global name with a slot in the local
variable array and perform an initial lookup of the associated object to fill
in the slot with a valid value.  The association it creates would be noted by
the code responsible for changing the name-to-object binding to cause the
associated local variable to be updated.  The ``UNTRACK_OBJECT`` opcode would
delete any association between the name and the local variable slot.


Threads
=======

Operation of this code in threaded programs will be no different than in
unthreaded programs.  If you need to lock an object to access it, you would
have had to do that before ``TRACK_OBJECT`` would have been executed and
retain that lock until after you stop using it.

FIXME: I suspect I need more here.


Rationale
=========

Global variables and attributes rarely change.  For example, once a function
imports the math module, the binding between the name *math* and the
module it refers to aren't likely to change.  Similarly, if the function that
uses the ``math`` module refers to its *sin* attribute, it's unlikely to
change.  Still, every time the module wants to call the ``math.sin`` function,
it must first execute a pair of instructions::

    LOAD_GLOBAL     math
    LOAD_ATTR       sin

If the client module always assumed that ``math.sin`` was a local constant and
it was the responsibility of "external forces" outside the function to keep
the reference correct, we might have code like this::

    TRACK_OBJECT       math.sin
    ...
    LOAD_FAST          math.sin
    ...
    UNTRACK_OBJECT     math.sin

If the ``LOAD_FAST`` was in a loop the payoff in reduced global loads and
attribute lookups could be significant.

This technique could, in theory, be applied to any global variable access or
attribute lookup.  Consider this code::

    l = []
    for i in range(10):
        l.append(math.sin(i))
    return l

Even though *l* is a local variable, you still pay the cost of loading
``l.append`` ten times in the loop.  The compiler (or an optimizer) could
recognize that both ``math.sin`` and ``l.append`` are being called in the loop
and decide to generate the tracked local code, avoiding it for the builtin
``range()`` function because it's only called once during loop setup.
Performance issues related to accessing local variables make tracking
``l.append`` less attractive than tracking globals such as ``math.sin``.

According to a post to python-dev by Marc-Andre Lemburg [1]_, ``LOAD_GLOBAL``
opcodes account for over 7% of all instructions executed by the Python virtual
machine.  This can be a very expensive instruction, at least relative to a
``LOAD_FAST`` instruction, which is a simple array index and requires no extra
function calls by the virtual machine.  I believe many ``LOAD_GLOBAL``
instructions and ``LOAD_GLOBAL/LOAD_ATTR`` pairs could be converted to
``LOAD_FAST`` instructions.

Code that uses global variables heavily often resorts to various tricks to
avoid global variable and attribute lookup.  The aforementioned
``sre_compile._compile`` function caches the ``append`` method of the growing
output list.  Many people commonly abuse functions' default argument feature
to cache global variable lookups.  Both of these schemes are hackish and
rarely address all the available opportunities for optimization.  (For
example, ``sre_compile._compile`` does not cache the two globals that it uses
most frequently: the builtin ``len`` function and the global ``OPCODES`` array
that it imports from ``sre_constants.py``.


Questions
=========

What about threads?  What if ``math.sin`` changes while in cache?
-----------------------------------------------------------------

I believe the global interpreter lock will protect values from being
corrupted.  In any case, the situation would be no worse than it is today.
If one thread modified ``math.sin`` after another thread had already executed
``LOAD_GLOBAL math``, but before it executed ``LOAD_ATTR sin``, the client
thread would see the old value of ``math.sin``.

The idea is this.  I use a multi-attribute load below as an example, not
because it would happen very often, but because by demonstrating the recursive
nature with an extra call hopefully it will become clearer what I have in
mind.  Suppose a function defined in module ``foo`` wants to access
``spam.eggs.ham`` and that ``spam`` is a module imported at the module level
in ``foo``::

    import spam
    ...
    def somefunc():
    ...
    x = spam.eggs.ham

Upon entry to ``somefunc``, a ``TRACK_GLOBAL`` instruction will be executed::

    TRACK_GLOBAL spam.eggs.ham n

*spam.eggs.ham* is a string literal stored in the function's constants
array.  *n* is a fastlocals index.  ``&fastlocals[n]`` is


id: pep-0500.rst
content:
PEP: 500
Title: A protocol for delegating datetime methods to their tzinfo implementations
Author: Alexander Belopolsky <alexander.belopolsky@gmail.com>, Tim Peters <tim.peters@gmail.com>
Discussions-To: datetime-sig@python.org
Status: Rejected
Type: Standards Track
Requires: 495
Created: 08-Aug-2015
Resolution: https://mail.python.org/pipermail/datetime-sig/2015-August/000354.html

Abstract
========

This PEP specifies a new protocol (PDDM - "A Protocol for Delegating
Datetime Methods") that can be used by concrete implementations of the
``datetime.tzinfo`` interface to override aware datetime arithmetics,
formatting and parsing.  We describe changes to the
``datetime.datetime`` class to support the new protocol and propose a
new abstract class ``datetime.tzstrict`` that implements parts of this
protocol necessary to make aware datetime instances to follow "strict"
arithmetic rules.


Rationale
=========

As of Python 3.5, aware datetime instances that share a ``tzinfo``
object follow the rules of arithmetics that are induced by a simple
bijection between (year, month, day, hour, minute, second,
microsecond) 7-tuples and large integers.  In this arithmetics, the
difference between YEAR-11-02T12:00 and YEAR-11-01T12:00 is always 24
hours, even though in the US/Eastern timezone, for example, there are
25 hours between 2014-11-01T12:00 and 2014-11-02T12:00 because the
local clocks were rolled back one hour at 2014-11-02T02:00,
introducing an extra hour in the night between 2014-11-01 and
2014-11-02.

Many business applications require the use of Python's simplified view
of local dates.  No self-respecting car rental company will charge its
customers more for a week that straddles the end of DST than for any
other week or require that they return the car an hour early.
Therefore, changing the current rules for aware datetime arithmetics
will not only create a backward compatibility nightmare, it will
eliminate support for legitimate and common use cases.

Since it is impossible to choose universal rules for local time
arithmetics, we propose to delegate implementation of those rules to
the classes that implement ``datetime.tzinfo`` interface.  With such
delegation in place, users will be able to choose between different
arithmetics by simply picking instances of different classes for the
value of ``tzinfo``.


Protocol
========

Subtraction of datetime
-----------------------

A ``tzinfo`` subclass supporting the PDDM, may define a method called
``__datetime_diff__`` that should take two ``datetime.datetime``
instances and return a ``datetime.timedelta`` instance representing
the time elapsed from the time represented by the first datetime
instance to another.


Addition
--------

A ``tzinfo`` subclass supporting the PDDM, may define a method called
``__datetime_add__`` that should take two arguments--a datetime and a
timedelta instances--and return a datetime instance.


Subtraction of timedelta
------------------------

A ``tzinfo`` subclass supporting the PDDM, may define a method called
``__datetime_sub__`` that should take two arguments--a datetime and a
timedelta instances--and return a datetime instance.


Formatting
----------

A ``tzinfo`` subclass supporting the PDDM, may define methods called
``__datetime_isoformat__`` and ``__datetime_strftime__``.

The ``__datetime_isoformat__`` method should take a datetime instance
and an optional separator and produce a string representation of the
given datetime instance.

The ``__datetime_strftime__`` method should take a datetime instance
and a format string and produce a string representation of the given
datetime instance formatted according to the given format.


Parsing
-------

A ``tzinfo`` subclass supporting the PDDM, may define a class method
called ``__datetime_strptime__`` and register the "canonical" names of
the timezones that it implements with a registry. **TODO** Describe a
registry.


Changes to datetime methods
===========================

Subtraction
-----------

::

   class datetime:
       def __sub__(self, other):
           if isinstance(other, datetime):
               try:
                   self_diff = self.tzinfo.__datetime_diff__
               except AttributeError:
                   self_diff = None
               try:
                   other_diff = self.tzinfo.__datetime_diff__
               except AttributeError:
                   other_diff = None
               if self_diff is not None:
                   if self_diff is not other_diff and self_diff.__func__ is not other_diff.__func__:
                       raise ValueError("Cannot find difference of two datetimes with "
                                        "different tzinfo.__datetime_diff__ implementations.")
                   return self_diff(self, other)
           elif isinstance(other, timedelta):
               try:
                   sub = self.tzinfo.__datetime_sub__
               except AttributeError:
                   pass
               else:
                   return sub(self, other)
               return self + -other
           else:
               return NotImplemented
           # current implementation


Addition
--------

Addition of a timedelta to a datetime instance will be delegated to the
``self.tzinfo.__datetime_add__`` method whenever it is defined.


Strict arithmetics
==================

A new abstract subclass of ``datetime.tzinfo`` class called  ``datetime.tzstrict``
will be added to the ``datetime`` module.  This subclass will not implement the
``utcoffset()``, ``tzname()`` or ``dst()`` methods, but will implement some of the
methods of the PDDM.

The PDDM methods implemented by ``tzstrict`` will be equivalent to the following::

  class tzstrict(tzinfo):
      def __datetime_diff__(self, dt1, dt2):
          utc_dt1 = dt1.astimezone(timezone.utc)
          utc_dt2 = dt2.astimezone(timezone.utc)
          return utc_dt2 - utc_dt1

      def __datetime_add__(self, dt, delta):
          utc_dt = dt.astimezone(timezone.utc)
          return (utc_dt + delta).astimezone(self)

      def __datetime_sub__(self, dt, delta):
          utc_dt = dt.astimezone(timezone.utc)
          return (utc_dt - delta).astimezone(self)


Parsing and formatting
----------------------

Datetime methods ``strftime`` and ``isoformat`` will delegate to the namesake
methods of their ``tzinfo`` members whenever those methods are defined.

When the ``datetime.strptime`` method is given a format string that
contains a ``%Z`` instruction, it will lookup the ``tzinfo``
implementation in the registry by the given timezone name and call its
``__datetime_strptime__`` method.

Applications
============

This PEP will enable third party implementation of many different
timekeeping schemes including:

* Julian / Microsoft Excel calendar.
* "Right" timezones with the leap second support.
* French revolutionary calendar (with a lot of work).

Copyright
=========

This document has been placed in the public domain.


id: pep-0528.rst
content:
PEP: 528
Title: Change Windows console encoding to UTF-8
Author: Steve Dower <steve.dower@python.org>
Status: Final
Type: Standards Track
Created: 27-Aug-2016
Python-Version: 3.6
Post-History: 01-Sep-2016, 04-Sep-2016
Resolution: https://mail.python.org/pipermail/python-dev/2016-September/146278.html

Abstract
========

Historically, Python uses the ANSI APIs for interacting with the Windows
operating system, often via C Runtime functions. However, these have been long
discouraged in favor of the UTF-16 APIs. Within the operating system, all text
is represented as UTF-16, and the ANSI APIs perform encoding and decoding using
the active code page.

This PEP proposes changing the default standard stream implementation on Windows
to use the Unicode APIs. This will allow users to print and input the full range
of Unicode characters at the default Windows console. This also requires a
subtle change to how the tokenizer parses text from readline hooks.

Specific Changes
================

Add _io.WindowsConsoleIO
------------------------

Currently an instance of ``_io.FileIO`` is used to wrap the file descriptors
representing standard input, output and error. We add a new class (implemented
in C) ``_io.WindowsConsoleIO`` that acts as a raw IO object using the Windows
console functions, specifically, ``ReadConsoleW`` and ``WriteConsoleW``.

This class will be used when the legacy-mode flag is not in effect, when opening
a standard stream by file descriptor and the stream is a console buffer rather
than a redirected file. Otherwise, ``_io.FileIO`` will be used as it is today.

This is a raw (bytes) IO class that requires text to be passed encoded with
utf-8, which will be decoded to utf-16-le and passed to the Windows APIs.
Similarly, bytes read from the class will be provided by the operating system as
utf-16-le and converted into utf-8 when returned to Python.

The use of an ASCII compatible encoding is required to maintain compatibility
with code that bypasses the ``TextIOWrapper`` and directly writes ASCII bytes to
the standard streams (for example, `Twisted's process_stdinreader.py`_). Code that assumes
a particular encoding for the standard streams other than ASCII will likely
break.

Add _PyOS_WindowsConsoleReadline
--------------------------------

To allow Unicode entry at the interactive prompt, a new readline hook is
required. The existing ``PyOS_StdioReadline`` function will delegate to the new
``_PyOS_WindowsConsoleReadline`` function when reading from a file descriptor
that is a console buffer and the legacy-mode flag is not in effect (the logic
should be identical to above).

Since the readline interface is required to return an 8-bit encoded string with
no embedded nulls, the ``_PyOS_WindowsConsoleReadline`` function transcodes from
utf-16-le as read from the operating system into utf-8.

The function ``PyRun_InteractiveOneObject`` which currently obtains the encoding
from ``sys.stdin`` will select utf-8 unless the legacy-mode flag is in effect.
This may require readline hooks to change their encodings to utf-8, or to
require legacy-mode for correct behaviour.

Add legacy mode
---------------

Launching Python with the environment variable ``PYTHONLEGACYWINDOWSSTDIO`` set
will enable the legacy-mode flag, which completely restores the previous
behaviour.

Alternative Approaches
======================

The `win_unicode_console package`_ is a pure-Python alternative to changing the
default behaviour of the console. It implements essentially the same
modifications as described here using pure Python code.

Code that may break
===================

The following code patterns may break or see different behaviour as a result of
this change. All of these code samples require explicitly choosing to use a raw
file object in place of a more convenient wrapper that would prevent any visible
change.

Assuming stdin/stdout encoding
------------------------------

Code that assumes that the encoding required by ``sys.stdin.buffer`` or
``sys.stdout.buffer`` is ``'mbcs'`` or a more specific encoding may currently be
working by chance, but could encounter issues under this change. For example::

    >>> sys.stdout.buffer.write(text.encode('mbcs'))
    >>> r = sys.stdin.buffer.read(16).decode('cp437')

To correct this code, the encoding specified on the ``TextIOWrapper`` should be
used, either implicitly or explicitly::

    >>> # Fix 1: Use wrapper correctly
    >>> sys.stdout.write(text)
    >>> r = sys.stdin.read(16)

    >>> # Fix 2: Use encoding explicitly
    >>> sys.stdout.buffer.write(text.encode(sys.stdout.encoding))
    >>> r = sys.stdin.buffer.read(16).decode(sys.stdin.encoding)

Incorrectly using the raw object
--------------------------------

Code that uses the raw IO object and does not correctly handle partial reads and
writes may be affected. This is particularly important for reads, where the
number of characters read will never exceed one-fourth of the number of bytes
allowed, as there is no feasible way to prevent input from encoding as much
longer utf-8 strings::

    >>> raw_stdin = sys.stdin.buffer.raw
    >>> data = raw_stdin.read(15)
    abcdefghijklm
    b'abc'
    # data contains at most 3 characters, and never more than 12 bytes
    # error, as "defghijklm\r\n" is passed to the interactive prompt

To correct this code, the buffered reader/writer should be used, or the caller
should continue reading until its buffer is full::

    >>> # Fix 1: Use the buffered reader/writer
    >>> stdin = sys.stdin.buffer
    >>> data = stdin.read(15)
    abcedfghijklm
    b'abcdefghijklm\r\n'

    >>> # Fix 2: Loop until enough bytes have been read
    >>> raw_stdin = sys.stdin.buffer.raw
    >>> b = b''
    >>> while len(b) < 15:
    ...     b += raw_stdin.read(15)
    abcedfghijklm
    b'abcdefghijklm\r\n'

Using the raw object with small buffers
---------------------------------------

Code that uses the raw IO object and attempts to read less than four characters
will now receive an error. Because it's possible that any single character may
require up to four bytes when represented in utf-8, requests must fail::

    >>> raw_stdin = sys.stdin.buffer.raw
    >>> data = raw_stdin.read(3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: must read at least 4 bytes

The only workaround is to pass a larger buffer::

    >>> # Fix: Request at least four bytes
    >>> raw_stdin = sys.stdin.buffer.raw
    >>> data = raw_stdin.read(4)
    a
    b'a'
    >>> >>>

(The extra ``>>>`` is due to the newline remaining in the input buffer and is
expected in this situation.)

Copyright
=========

This document has been placed in the public domain.

References
==========

.. _Twisted's process_stdinreader.py: https://github.com/twisted/twisted/blob/trunk/src/twisted/test/process_stdinreader.py
.. _win_unicode_console package: https://pypi.org/project/win_unicode_console/


id: pep-0299.rst
content:
PEP: 299
Title: Special __main__() function in modules
Author: Jeff Epler <jepler@unpythonic.net>
Status: Rejected
Type: Standards Track
Created: 12-Aug-2002
Python-Version: 2.3
Post-History: 29-Mar-2006



Abstract
========

Many Python modules are also intended to be callable as standalone
scripts.  This PEP proposes that a special function called ``__main__()``
should serve this purpose.


Motivation
==========

There should be one simple and universal idiom for invoking a module
as a standalone script.

The semi-standard idiom::

    if __name__ == '__main__':
        perform "standalone" functionality

is unclear to programmers of languages like C and C++.  It also does
not permit invocation of the standalone function when the module is
imported.  The variant::

    if __name__ == '__main__':
        main_function()

is sometimes seen, but there exists no standard name for the function,
and because arguments are taken from sys.argv it is not possible to
pass specific arguments without changing the argument list seen by all
other modules.  (Imagine a threaded Python program, with two threads
wishing to invoke the standalone functionality of different modules
with different argument lists)


Proposal
========

The standard name of the 'main function' should be ``__main__``. When a
module is invoked on the command line, such as::

    python mymodule.py

then the module behaves as though the following lines existed at the
end of the module (except that the attribute __sys may not be used or
assumed to exist elsewhere in the script)::

    if globals().has_key("__main__"):
        import sys as __sys
        __sys.exit(__main__(__sys.argv))

Other modules may execute::

    import mymodule mymodule.__main__(['mymodule', ...])

It is up to ``mymodule`` to document thread-safety issues or other
issues which might restrict use of ``__main__``.  (Other issues might
include use of mutually exclusive GUI modules, non-sharable resources
like hardware devices, reassignment of ``sys.stdin``/``stdout``, etc)


Implementation
==============

In ``modules/main.c``, the block near line 385 (after the
``PyRun_AnyFileExFlags`` call) will be changed so that the above code
(or its C equivalent) is executed.


Open Issues
===========

* Should the return value from ``__main__`` be treated as the exit value?

  Yes.  Many ``__main__`` will naturally return ``None``, which
  ``sys.exit`` translates into a "success" return code.  In those that
  return a numeric result, it behaves just like the argument to
  ``sys.exit()`` or the return value from C's main().

* Should the argument list to ``__main__`` include ``argv[0]``, or just the
  "real" arguments ``argv[1:]``?

  ``argv[0]`` is included for symmetry with ``sys.argv`` and easy
  transition to the new standard idiom.


Rejection
=========

In a short discussion on python-dev [1]_, two major backwards
compatibility problems were brought up and Guido pronounced that he
doesn't like the idea anyway as it's "not worth the change (in docs,
user habits, etc.) and there's nothing particularly broken."


References
==========

.. [1] Georg Brandl, "What about PEP 299",
    https://mail.python.org/pipermail/python-dev/2006-March/062951.html


Copyright
=========

This document has been placed in the public domain.


id: pep-0702.rst
content:
PEP: 702
Title: Marking deprecations using the type system
Author: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-702-marking-deprecations-using-the-type-system/23036
Status: Final
Type: Standards Track
Topic: Typing
Created: 30-Dec-2022
Python-Version: 3.13
Post-History: `01-Jan-2023 <https://mail.python.org/archives/list/typing-sig@python.org/thread/AKTFUYW3WDT7R7PGRIJQZMYHMDJNE4QH/>`__,
              `22-Jan-2023 <https://discuss.python.org/t/pep-702-marking-deprecations-using-the-type-system/23036>`__
Resolution: `07-Nov-2023 <https://discuss.python.org/t/pep-702-marking-deprecations-using-the-type-system/23036/61>`__

.. canonical-typing-spec:: :ref:`typing:deprecated` and
                           :external+py3.13:func:`@warnings.deprecated<warnings.deprecated>`

Abstract
========

This PEP adds an ``@warnings.deprecated()`` decorator that marks a class or function
as deprecated, enabling static checkers to warn when it is used. By default, this
decorator will also raise a runtime ``DeprecationWarning``.

Motivation
==========

As software evolves, new functionality is added and old functionality becomes
obsolete. Library developers want to work towards removing obsolete code while
giving their users time to migrate to new APIs. Python provides a mechanism for
achieving these goals: the :exc:`DeprecationWarning` warning class, which is
used to show warnings when deprecated functionality is used. This mechanism is
widely used: as of the writing of this PEP, the CPython main branch contains
about 150 distinct code paths that raise :exc:`!DeprecationWarning`. Many
third-party libraries also use :exc:`!DeprecationWarning` to mark deprecations.
In the `top 5000 PyPI packages <https://dev.to/hugovk/how-to-search-5000-python-projects-31gk>`__,
there are:

- 1911 matches for the regex ``warnings\.warn.*\bDeprecationWarning\b``,
  indicating use of :exc:`!DeprecationWarning` (not including cases where the
  warning is split over multiple lines);
- 1661 matches for the regex ``^\s*@deprecated``, indicating use of some sort of
  deprecation decorator.

However, the current mechanism is often insufficient to ensure that users
of deprecated functionality update their code in time. For example, the
removal of various long-deprecated :mod:`unittest` features had to be
`reverted <https://github.com/python/cpython/commit/b50322d20337ca468f2070eedb051a16ee1eba94>`__
from Python 3.11 to give users more time to update their code.
Users may run their test suite with warnings disabled for practical reasons,
or deprecations may be triggered in code paths that are not covered by tests.

Providing more ways for users to find out about deprecated functionality
can speed up the migration process. This PEP proposes to leverage static type
checkers to communicate deprecations to users. Such checkers have a thorough
semantic understanding of user code, enabling them to detect and report
deprecations that a single ``grep`` invocation could not find. In addition, many type
checkers integrate with IDEs, enabling users to see deprecation warnings
right in their editors.

Rationale
=========

At first glance, deprecations may not seem like a topic that type checkers should
touch. After all, type checkers are concerned with checking whether code will
work as is, not with potential future changes. However, the analysis that type
checkers perform on code to find type errors is very similar to the analysis
that would be needed to detect usage of many deprecations. Therefore, type
checkers are well placed to find and report deprecations.

Other languages already have similar functionality:

* GCC supports a ``deprecated`` `attribute <https://gcc.gnu.org/onlinedocs/gcc-3.1.1/gcc/Type-Attributes.html>`__
  on function declarations. This powers CPython's ``Py_DEPRECATED`` macro.
* GraphQL `supports <https://spec.graphql.org/June2018/#sec-Field-Deprecation>`__
  marking fields as ``@deprecated``.
* Kotlin `supports <https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-deprecated/>`__
  a ``Deprecated`` annotation.
* Scala `supports <https://www.scala-lang.org/api/2.12.5/scala/deprecated.html>`__
  an ``@deprecated`` annotation.
* Swift `supports <https://docs.swift.org/swift-book/ReferenceManual/Attributes.html>`__
  using the ``@available`` attribute to mark APIs as deprecated.
* TypeScript `uses <https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html#deprecated>`__
  the ``@deprecated`` JSDoc tag to issue a hint marking use of
  deprecated functionality.

Several users have requested support for such a feature:

* `typing-sig thread <https://mail.python.org/archives/list/typing-sig@python.org/thread/E24WTMQUTGKPFKEXVCGGEFFMG7LDF3WT/>`__
* `Pyright feature request <https://github.com/microsoft/pyright/discussions/2300>`__
* `mypy feature request <https://github.com/python/mypy/issues/11439>`__

There are similar existing third-party tools:

* `Deprecated <https://pypi.org/project/Deprecated/>`__ provides a decorator to
  mark classes, functions, or methods as deprecated. Access to decorated objects
  raises a runtime warning, but is not detected by type checkers.
* `flake8-deprecated <https://pypi.org/project/flake8-deprecated/>`__ is a linter
  plugin that warns about use of deprecated features. However, it is limited to
  a short, hard-coded list of deprecations.

Specification
=============

A new decorator ``@deprecated()`` is added to the :mod:`warnings` module. This
decorator can be used on a class, function or method to mark it as deprecated.
This includes :class:`typing.TypedDict` and :class:`typing.NamedTuple` definitions.
With overloaded functions, the decorator may be applied to individual overloads,
indicating that the particular overload is deprecated. The decorator may also be
applied to the overload implementation function, indicating that the entire function
is deprecated.

The decorator takes the following arguments:

* A required positional-only argument representing the deprecation message.
* Two keyword-only arguments, ``category`` and ``stacklevel``, controlling
  runtime behavior (see under "Runtime behavior" below).

The positional-only argument is of type ``str`` and contains a message that should
be shown by the type checker when it encounters a usage of the decorated object.
Tools may clean up the deprecation message for display, for example
by using :func:`inspect.cleandoc` or equivalent logic.
The message must be a string literal.
The content of deprecation messages is up to the user, but it may include the version
in which the deprecated object is to be removed, and information about suggested
replacement APIs.

Type checkers should produce a diagnostic whenever they encounter a usage of an
object marked as deprecated. For deprecated overloads, this includes all calls
that resolve to the deprecated overload.
For deprecated classes and functions, this includes:

* References through module, class, or instance attributes (``module.deprecated_object``,
  ``module.SomeClass.deprecated_method``, ``module.SomeClass().deprecated_method``)
* Any usage of deprecated objects in their defining module
  (``x = deprecated_object()`` in ``module.py``)
* If ``import *`` is used, usage of deprecated objects from the
  module (``from module import *; x = deprecated_object()``)
* ``from`` imports (``from module import deprecated_object``)
* Any syntax that indirectly triggers a call to the function. For example,
  if the ``__add__`` method of a class ``C`` is deprecated, then
  the code ``C() + C()`` should trigger a diagnostic. Similarly, if the
  setter of a property is marked deprecated, attempts to set the property
  should trigger a diagnostic.

If a method is marked with the :func:`typing.override` decorator from :pep:`698`
and the base class method it overrides is deprecated, the type checker should
produce a diagnostic.

There are additional scenarios where deprecations could come into play.
For example, 


id: pep-0662.rst
content:
PEP: 662
Title: Editable installs via virtual wheels
Author: BernÃ¡t GÃ¡bor <gaborjbernat@gmail.com>
Sponsor: Brett Cannon <brett@python.org>
Discussions-To: https://discuss.python.org/t/discuss-tbd-editable-installs-by-gaborbernat/9071
Status: Rejected
Type: Standards Track
Topic: Packaging
Created: 28-May-2021
Post-History:
Resolution: https://discuss.python.org/t/pronouncement-on-peps-660-and-662-editable-installs/9450


Abstract
========

This document describes extensions to the build backend and frontend
communication (as introduced by :pep:`517`) to allow projects to be installed in
editable mode by introducing virtual wheels.

Motivation
==========

During development, many Python users prefer to install their libraries so that
changes to the underlying source code and resources are automatically reflected
in subsequent interpreter invocations without an additional installation step.
This mode is usually called "development mode" or "editable installs".
Currently, there is no standardized way to accomplish this, as it was explicitly
left out of :pep:`517` due to the complexity of the actual observed behaviors.

At the moment, users to get this behaviour perform one of the following:

-  For just Python code by adding the relevant source directories to
   ``sys.path`` (configurable from the command line interface via the
   ``PYTHONPATH`` environment variable). Note in this case, the users have to
   install the project dependencies themselves, and entry points or project
   metadata are not generated.

-  setuptools_ provides the `setup.py develop`_ mechanism: that installs a
   ``pth`` file that injects the project root onto the ``sys.path`` at
   interpreter startup time, generates the project metadata, and also installs
   project dependencies. pip_ exposes calling this mechanism via the
   `pip install -e <project_directory>`_ command-line interface.

-  flit_ provides the `flit install --symlink`_ command that symlinks the
   project files into the interpreters ``purelib`` folder, generates the
   project metadata, and also installs dependencies. Note, this allows
   supporting resource files too.

As these examples shows an editable install can be achieved in multiple ways
and at the moment there's no standard way of doing it. Furthermore, it's not
clear whose responsibility it is to achieve and define what an editable
installation is:

1. allow the build backend to define and materialize it,
2. allow the build frontend to define and materialize it,
3. explicitly define and standardize one method from the possible options.

The author of this PEP believes there's no one size fits all solution here,
each method of achieving editable effect has its pros and cons. Therefore
this PEP rejects option three as it's unlikely for the community to agree on a
single solution. Furthermore, question remains as to whether the frontend or the
build backend should own this responsibility. :pep:`660` proposes the build
backend to own this, while the current PEP proposes primarily the frontend,
but still allows the backend to take take control if it wants to do so.

Rationale
=========

:pep:`517` deferred "Editable installs" because this would have delayed further
its adoption, and there wasn't an agreement on how editable installs should be
achieved. Due to the popularity of the setuptools_ and pip_ projects, the status
quo prevailed, and the backend could achieve editable mode by providing a
``setup.py develop`` implementation, which the user could trigger via `pip
install -e <project_directory>`_. By defining an editable interface between the
build backend and frontend, we can eliminate the ``setup.py`` file and their
current communication method.

Terminology and goals
=====================

This PEP aims to delineate the frontend and the backend roles clearly and give
the developers of each the maximum ability to provide valuable features to
their users. In this proposal, the backend's role is to prepare the project for
an editable installation, and then provide enough information to the frontend
so that the frontend can manifest and enforce the editable installation.

The information the backend provides to the frontend is a wheel that follows
the existing specification within :pep:`427`. The wheel metadata about the
archive itself (``{distribution}-{version}.dist-info/WHEEL``) must also contain
the key ``Editable`` with value of ``true``.

However, instead of providing the project files within the wheel, it must
provide an ``editable.json`` file (at the root level of the wheel) that defines
the files to be exposed by the frontend. The content of this file is formulated
as a mapping of absolute source tree paths to relative target interpreter
destination paths within a scheme mapping.

A wheel that satisfies the previous two paragraphs is a virtual wheel. The
frontend's role is to take the virtual wheel and install the project in
editable mode. The way it achieves this is entirely up to the frontend and is
considered implementation detail.

The editable installation mode implies that the source code of the project
being installed is available in a local directory. Once the project is
installed in editable mode, some changes to the project code in the local
source tree will become effective without the need for a new installation step.
At a minimum, changes to the text of non-generated files that existed at the
installation time should be reflected upon the subsequent import of the
package.

Some kinds of changes, such as adding or modifying entry points or new
dependencies, require a new installation step to become effective. These changes
are typically made in build backend configuration files (such as
``pyproject.toml``). This requirement is consistent with the general user
expectation that such modifications will only become effective after
re-installation.

While users expect editable installations to behave identically to standard
installations, this may not always be possible and may be in tension with other
user expectations. Depending on how a frontend implements the editable mode,
some differences may be visible, such as the presence of additional files
(compared to a typical installation), either in the source tree or the
interpreter's installation path.

Frontends should seek to minimize differences between the behavior of editable
and standard installations and document known differences.

For reference, a non-editable installation works as follows:

#. The **developer** is using a tool, we'll call it here the **frontend**, to
   drive the project development (e.g., pip_). When the user wants to trigger a
   package build and installation of a project, they'll communicate with the
   **frontend**.

#. The frontend uses a **build frontend** to trigger the build of a wheel (e.g.,
   build_). The build frontend uses :pep:`517` to communicate with the **build
   backend** (e.g. setuptools_) - with the build backend installed into a
   :pep:`518` environment. Once invoked, the backend returns a wheel.

#. The frontend takes the wheel and feeds it to an **installer**
   (e.g., installer_) to install the wheel into the target Python interpreter.

The Mechanism
=============

This PEP adds two optional hooks to the :pep:`517` backend interface. One of the
hooks is used to specify the build dependencies of an editable install. The
other hook returns the necessary information via the build frontend the frontend
needs to create an editable install.

``get_requires_for_build_editable``
-----------------------------------

.. code::

   def get_requires_for_build_editable(config_settings=None):
       ...

This hook MUST return an additional sequence of strings containing :pep:`508`
dependency specifications, above and beyond those specified in the
``pyproject.toml`` file. The frontend must ensure that these dependencies are
available in the build environment in which the ``build_editable`` hook is
called.

If not defined, the default implementat


id: pep-0676.rst
content:
PEP: 676
Title: PEP Infrastructure Process
Author: Adam Turner <python@quite.org.uk>
Sponsor: Mariatta <mariatta@python.org>
PEP-Delegate: Barry Warsaw <barry@python.org>
Discussions-To: https://discuss.python.org/t/10774
Status: Active
Type: Process
Created: 01-Nov-2021
Post-History: 23-Sep-2021, 30-Nov-2021
Resolution: https://discuss.python.org/t/10774/99


Abstract
========

This PEP addresses the infrastructure around rendering PEP files from
`reStructuredText`_ files to HTML webpages. We aim to specify a self-contained
and maintainable solution for PEP readers, authors, and editors.


Motivation
==========

As of November 2021, Python Enhancement Proposals (PEPs) are rendered in a
multi-system, multi-stage process. A continuous integration (CI) task runs a
`docutils`_ script to render all PEP files individually. The CI task then
uploads a tar archive to a server, where it is retrieved and rendered into the
`python.org`_ website periodically.

This places a constraint on the `python.org`_ website to handle raw HTML
uploads and handle PEP rendering, and makes the appropriate place to raise
issues unclear in some cases [1]_.

This PEP provides a specification for self-contained rendering of PEPs. This
would:

* reduce the amount of distributed configuration for supporting PEPs
* enable quality-of-life improvements for those who read, write, and review
  PEPs
* solve a number of outstanding issues, and lay the path for improvements
* save volunteer maintainers' time

We propose that PEPs are accessed through `peps.python.org`_ at the top-level
(for example `peps.python.org/pep-0008`_), and that all custom tooling to
support rendering PEPs is hosted in the `python/peps`_ repository.


Rationale
=========

Simplifying and Centralising Infrastructure
-------------------------------------------

As of November 2021, to locally render a PEP file, a PEP author or editor needs
to create a full local instance of the `python.org`_ website and run a number
of disparate scripts, following `documentation`_ that lives outside of the
`python/peps`_ repository.

By contrast, the proposed implementation provides a single `Makefile`_ and a
Python script to render all PEP files, with options to target a web-server or
the local filesystem.

Using a single repository to host all tooling will clarify where to raise
issues, reducing volunteer time spent in triage.

Simplified and centralised tooling may also reduce the barrier to entry to
further improvements, as the scope of the PEP rendering infrastructure is well
defined.


Quality-of-Life Improvements and Resolving Issues
-------------------------------------------------

There are several requests for additional features in reading PEPs, such as:

* syntax highlighting [2]_
* use of ``.. code-block::`` directives [2]_
* support for SVG images [3]_
* typographic quotation marks [4]_
* additional footer information [5]_
* intersphinx functionality [6]_
* dark mode theme [7]_

These are "easy wins" from this proposal, and would serve to improve the
quality-of-life for consumers of PEPs (including reviewers and writers).

For example, the current (as of November 2021) system runs periodically on a
schedule. This means that updates to PEPs cannot be circulated immediately,
reducing productivity. The reference implementation renders and publishes all
PEPs on every commit to the repository, solving the issue by design.

The reference implementation fixes several issues [8]_. For example:

* list styles are currently not respected by `python.org`_'s stylesheets
* support for updating images in PEPs is challenging in `python.org`_

Third-party providers such as `Read the Docs`_ or `Netlify`_ can enhance this
experience with features such as automatic rendering of pull requests.


Specification
=============

The proposed specification for rendering the PEP files to HTML is as per the
`reference implementation`_.

The rendered PEPs MUST be available at `peps.python.org`_. These SHOULD be
hosted as static files, and MAY be behind a content delivery network (CDN).

A service to render previews of pull requests SHOULD be provided. This service
MAY be integrated with the hosting and deployment solution.

The following redirect rules MUST be created for the `python.org`_ domain:

* ``/peps/``            -> https://peps.python.org/
* ``/dev/peps/``        -> https://peps.python.org/
* ``/peps/(.*)\.html``  -> https://peps.python.org/$1
* ``/dev/peps/(.*)``    -> https://peps.python.org/$1

The following nginx configuration would achieve this:

.. code-block:: nginx

    location ~ ^/dev/peps/?(.*)$ {
        return 308 https://peps.python.org/$1/;
    }

    location ~ ^/peps/(.*)\.html$ {
        return 308 https://peps.python.org/$1/;
    }

    location ^/(dev/)?peps(/.*)?$ {
        return 308 https://peps.python.org/;
    }

Redirects MUST be implemented to preserve `URL fragments`_ for backward
compatibility purposes.

Backwards Compatibility
=======================

Due to server-side redirects to the new canonical URLs, links in previously
published materials referring to the old URL schemes will be guaranteed to work.
All PEPs will continue to render correctly, and a custom stylesheet in the
reference implementation improves presentation for some elements (most notably
code blocks and block quotes). Therefore, this PEP presents no backwards
compatibility issues.


Security Implications
=====================

The main `python.org`_ website will no longer process raw HTML uploads,
closing a potential threat vector. PEP rendering and deployment processes will
use modern, well-maintained code and secure automated platforms, further
reducing the potential attack surface. Therefore, we see no negative security
impact.


How to Teach This
=================

The new canonical URLs will be publicised in the documentation. However, this
is mainly a backend infrastructure change, and there should be minimal
end-user impact. PEP 1 and PEP 12 will be updated as needed.


Reference Implementation
========================

The proposed implementation has been merged into the `python/peps`_ repository
in a series of pull requests [9]_. It uses the `Sphinx`_ documentation system
with a custom theme (supporting light and dark colour schemes) and extensions.

This already automatically renders all PEPs on every commit, and publishes them
to `python.github.io/peps`_. The high level documentation for the system covers
`how to render PEPs locally <https://python.github.io/peps/docs/build>`__ and
`the implementation of the system <https://python.github.io/peps/docs/rendering_system>`__.


Rejected Ideas
==============

It would likely be possible to amend the current (as of November 2021)
rendering process to include a subset of the quality-of-life improvements and
issue mitigations mentioned above. However, we do not believe that this would
solve the distributed tooling issue.

It would be possible to use the output from the proposed rendering system and
import it into `python.org`_. We would argue that this would be the worst of
both worlds, as a great deal of complexity is added whilst none is removed.


Acknowledgements
================

- Hugo van Kemenade
- Pablo Galindo Salgado
- Ãric Araujo
- Mariatta
- C.A.M. Gerlach


Footnotes
=========

.. _documentation: https://pythondotorg.readthedocs.io/pep_generation.html
.. _docutils: https://docutils.sourceforge.io
.. _Makefile: https://www.gnu.org/software/make/manual/make.html#Introduction
.. _Netlify: https://www.netlify.com/
.. _peps.python.org: https://peps.python.org/
.. _peps.python.org/pep-0008: https://peps.python.org/pep-0008/
.. _python.github.io/peps: https://python.github.io/peps
.. _python.org: https://www.python.org
.. _python/peps: https://github.com/python/peps
.. _Read the Docs: https://readthedocs.org
.. _reStructuredText: https://docutils.sourceforge.io/rst.html
.. _Sphinx: https://www.sphinx-doc.org/en/master/
.. _URL fragments: https://url.spec.whatwg.org/#concept-url


id: pep-0689.rst
content:
PEP: 689
Title: Unstable C API tier
Author: Petr Viktorin <encukou@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-689-unstable-c-api-tier/20452
Status: Final
Type: Standards Track
Requires: 523
Created: 22-Apr-2022
Python-Version: 3.12
Post-History: `27-Apr-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/PQXSP7E2B6KNXTJ2AERWMKKX42YP5D6O/>`__,
              `25-Aug-2022 <https://discuss.python.org/t/c-api-what-should-the-leading-underscore-py-mean/18486>`__,
              `27-Oct-2022 <https://discuss.python.org/t/pep-689-unstable-c-api-tier/20452>`__,
Resolution: https://discuss.python.org/t/pep-689-unstable-c-api-tier/20452/13

.. canonical-doc:: :ref:`devguide:c-api`

   User-facing documentation is at :ref:`py3.12:unstable-c-api`.


Abstract
========

Some functions and types of the C-API are designated *unstable*,
meaning that they will not change in patch (bugfix/security) releases,
but may change between minor releases (e.g. between 3.11 and 3.12) without
deprecation warnings.

Any C API with a leading underscore is designated *internal*, meaning that it
may change or disappear without any notice.


Motivation & Rationale
======================

Unstable C API tier
-------------------

The Python C-API is currently divided into `three stability tiers <https://devguide.python.org/developer-workflow/c-api/index.html>`__:

- Limited API, with high compatibility expectations
- Public API, which follows the :pep:`backwards compatibility policy
  <387>`, and requires deprecation warnings before changes
- Internal (private) API, which can change at any time.

Tools requiring access to CPython internals (e.g. advanced
debuggers and JIT compilers) are often built for minor series releases
of CPython, and assume that the C-API internals used do not change
in patch releases. To support these tools, we need a tier between the
Public and Private C-API, with guarantees on stability throughout
the minor-series release: the proposed *Unstable tier*.

Some functions, like ``PyCode_New()``, are documented as unstable
(âCalling [it] directly can bind you to a precise Python versionâ),
and also often change in practice.
The unstable tier should make their status obvious even to people who don't
read the docs carefully enough, making them hard to use accidentally.


Reserving leading underscores for Private API
---------------------------------------------

Currently, CPython developers don't agree on the exact meaning of a leading
underscore in API names.
It is used to mean two different things:

- API that may change between minor releases, as in the Unstable tier proposed
  here (e.g. functions introduced in :pep:`523`).
- API that is *private* and should not be used outside of CPython at all
  (e.g. because it may change without notice, or it relies on undocumented
  assumptions that non-CPython code cannot guarantee).

The unclear meaning makes the underscore less useful than it could be.
If it only marked *private* API, CPython developers could change underscored
functions, or remove unused ones, without researching how they're
documented or used outside CPython.

With the introduction of a dedicated unstable tier, we can clarify the meaning
of the leading underscore. It should mark private API only.


Not breaking code unnecessarily
-------------------------------

This PEP specifies that API in the unstable tier should have a special name
prefix. This means functions (macros, etc.) will need to be renamed.
After a rename, the old name should continue to be available until
an incompatible change is made (i.e. until call sites need to be updated
anyway).
In other words, just changing the tier of a function shouldn't break users'
code.


Specification
=============

The C API is divided by stability expectations into `three âsectionsâ <https://devguide.python.org/developer-workflow/c-api/index.html>`__
(internal, public, and limited).
We'll now call these *stability tiers*, or *tiers* for short.

An *Unstable tier* will be added.

APIs (functions, types, etc.) in this tier will named with the ``PyUnstable_``
prefix, with no leading underscore.

They will be declared in headers used for public API (``Include/*.h``,
rather than in a subdirectory like ``Include/unstable/``).

Several rules for dealing with the unstable tier will be introduced:

-  Unstable API should have no backwards-incompatible
   changes across patch releases, but may change or be removed in minor
   releases (3.x.0, including Alpha and Beta releases of 3.x.0).
   Such changes must be documented and mentioned in the What's New document.

-  Backwards-incompatible changes to these APIs should be made so that
   code that uses them will need to be updated to compile with
   the new version (e.g. arguments should be added/removed, or a function should
   be renamed, but the semantic meaning of an argument should not change).

-  Unstable API should be documented and tested.

-  To move an API from the public tier to the unstable tier, it should be
   exposed under the new ``PyUnstable_*`` name.

   The old name should be deprecated (e.g. with ``Py_DEPRECATED``), but
   continue to be available until an incompatible change is made to the API.
   Per Python's backwards compatibility policy (:pep:`387`), this deprecation
   needs to last *at least* two releases (without an SC exceptions).
   But it can also last indefinitely -- for example, if :pep:`590`'s
   :pep:`âprovisionalâ <590#finalizing-the-api>`
   ``_PyObject_Vectorcall`` was added today, it would be initially named
   ``PyUnstable_Object_Vectorcall`` and there would be no plan to remove
   this name.

   In the following cases, an incompatible change (and thus removing the
   deprecated name) is allowed without an SC exception, as if the function was
   already part of the Unstable tier:

   -  Any API introduced before Python 3.12 that is *documented* to be less
      stable than default.
   -  Any API introduced before Python 3.12 that was named with a leading
      underscore.

   For examples, see the `initial unstable API`_
   specified in this PEP.

-  To move an *internal* API to the unstable tier, it should be
   exposed under the new ``PyUnstable_*`` name.

   If the old name is documented, or widely used externally,
   it should continue to be available until an
   incompatible change is made (and call sites need to be updated).
   It should start raising deprecation warnings (e.g. using ``Py_DEPRECATED``).

-  To move an API from the unstable tier to the public tier, it should be
   exposed without the ``PyUnstable_*`` prefix.

   The old name should remain available until the API is deprecated or removed.

-  Adding new unstable API *for existing features* is allowed even after
   Beta feature freeze, up until the first Release Candidate.
   Consensus on Core Development Discourse or is needed in the Beta period.

These rules will be documented in the `devguide <https://devguide.python.org/developer-workflow/c-api/index.html>`__,
and `user documentation <https://docs.python.org/3/c-api/stable.html>`__
will be updated accordingly.

Reference docs for C API named ``PyUnstable_*`` will automatically show
notes with links to the unstable tier documentation.


Leading underscore
------------------

C API named with a leading underscore, as well as API only available with
``Py_BUILD_CORE``, will be considered *internal*.
This means:

-  It may change or be removed *without notice* in minor
   releases (3.x.0, including Alpha and Beta releases of 3.x.0).
   API changes in patch releases or Release Candidates should only be done if
   absolutely necessary.

-  It should be documented in source comments or Devguide only, not in the
   public documentation.

-  API introduced before Python 3.12 that is documented or widely used
   externally should be moved to the Unstable tier as explained above.

   This might happen long after this PEP is accepted.
   Consequently, for a few years 


id: pep-0460.rst
content:
PEP: 460
Title: Add binary interpolation and formatting
Author: Antoine Pitrou <solipsis@pitrou.net>
Status: Withdrawn
Type: Standards Track
Created: 06-Jan-2014
Python-Version: 3.5


Abstract
========

This PEP proposes to add minimal formatting operations to bytes and
bytearray objects.  The proposed additions are:

* ``bytes % ...`` and ``bytearray % ...`` for percent-formatting,
  similar in syntax to percent-formatting on ``str`` objects
  (accepting a single object, a tuple or a dict).

* ``bytes.format(...)`` and ``bytearray.format(...)`` for a formatting
  similar in syntax to ``str.format()`` (accepting positional as well as
  keyword arguments).

* ``bytes.format_map(...)`` and ``bytearray.format_map(...)`` for an
  API similar to ``str.format_map(...)``, with the same formatting
  syntax and semantics as ``bytes.format()`` and ``bytearray.format()``.


Rationale
=========

In Python 2, ``str % args`` and ``str.format(args)`` allow the formatting
and interpolation of bytestrings.  This feature has commonly been used
for the assembling of protocol messages when protocols are known to use
a fixed encoding.

Python 3 generally mandates that text be stored and manipulated as unicode
(i.e. ``str`` objects, not ``bytes``).  In some cases, though, it makes
sense to manipulate ``bytes`` objects directly.  Typical usage is binary
network protocols, where you can want to interpolate and assemble several
bytes object (some of them literals, some of them compute) to produce
complete protocol messages.  For example, protocols such as HTTP or SIP
have headers with ASCII names and opaque "textual" values using a varying
and/or sometimes ill-defined encoding.  Moreover, those headers can be
followed by a binary body... which can be chunked and decorated with ASCII
headers and trailers!

While there are reasonably efficient ways to accumulate binary data
(such as using a ``bytearray`` object, the ``bytes.join`` method or
even ``io.BytesIO``), none of them leads to the kind of readable and
intuitive code that is produced by a %-formatted or {}-formatted template
and a formatting operation.


Binary formatting features
==========================

Supported features
------------------

In this proposal, percent-formatting for ``bytes`` and ``bytearray``
supports the following features:

* Looking up formatting arguments by position as well as by name (i.e.,
  ``%s`` as well as ``%(name)s``).
* ``%s`` will try to get a ``Py_buffer`` on the given value, and fallback
  on calling ``__bytes__``.  The resulting binary data is inserted at
  the given point in the string.  This is expected to work with bytes,
  bytearray and memoryview objects (as well as a couple others such
  as pathlib's path objects).
* ``%c`` will accept an integer between 0 and 255, and insert a byte of the
  given value.

Braces-formatting for ``bytes`` and ``bytearray`` supports the following
features:

* All the kinds of argument lookup supported by ``str.format()`` (explicit
  positional lookup, auto-incremented positional lookup, keyword lookup,
  attribute lookup, etc.)
* Insertion of binary data when no modifier or layout is specified
  (e.g. ``{}``, ``{0}``, ``{name}``).  This has the same semantics as
  ``%s`` for percent-formatting (see above).
* The ``c`` modifier will accept an integer between 0 and 255, and insert a
  byte of the given value (same as ``%c`` above).

Unsupported features
--------------------

All other features present in formatting of ``str`` objects (either
through the percent operator or the ``str.format()`` method) are
unsupported.  Those features imply treating the recipient of the
operator or method as text, which goes counter to the text / bytes
separation (for example, accepting ``%d`` as a format code would imply
that the bytes object really is an ASCII-compatible text string).

Amongst those unsupported features are not only most type-specific
format codes, but also the various layout specifiers such as padding
or alignment.  Besides, ``str`` objects are not acceptable as arguments
to the formatting operations, even when using e.g. the ``%s`` format code.

``__format__`` isn't called.


Criticisms
==========

* The development cost and maintenance cost.
* In 3.3 encoding to ASCII or latin-1 is as fast as memcpy (but it still
  creates a separate object).
* Developers will have to work around the lack of binary formatting anyway,
  if they want to support Python 3.4 and earlier.
* bytes.join() is consistently faster than format to join bytes strings
  (XXX *is it?*).
* Formatting functions could be implemented in a third party module,
  rather than added to builtin types.


Other proposals
===============

A new type datatype
-------------------

It was proposed to create a new datatype specialized for "network
programming".  The authors of this PEP believe this is counter-productive.
Python 3 already has several major types dedicated to manipulation of
binary data: ``bytes``, ``bytearray``, ``memoryview``, ``io.BytesIO``.

Adding yet another type would make things more confusing for users, and
interoperability between libraries more painful (also potentially
sub-optimal, due to the necessary conversions).

Moreover, not one type would be needed, but two: one immutable type (to
allow for hashing), and one mutable type (as efficient accumulation is
often necessary when working with network messages).


Resolution
==========

This PEP is made obsolete by the `acceptance
<https://mail.python.org/pipermail/python-dev/2014-March/133621.html>`_
of :pep:`461`, which introduces a more extended formatting language for
bytes objects in conjunction with the modulo operator.


References
==========

* `Issue #3982: support .format for bytes
  <http://bugs.python.org/issue3982>`_
* `Mercurial project
  <http://mercurial.selenic.com/>`_
* `Twisted project
  <http://twistedmatrix.com/trac/>`_
* `Documentation of Python 2 formatting (str % args)
  <http://docs.python.org/2/library/stdtypes.html#string-formatting>`_
* `Documentation of Python 2 formatting (str.format)
  <http://docs.python.org/2/library/string.html#formatstrings>`_

Copyright
=========

This document has been placed in the public domain.


id: pep-0306.rst
content:
PEP: 306
Title: How to Change Python's Grammar
Author: Michael Hudson <mwh@python.net>, Jack Diederich <jackdied@gmail.com>, Alyssa Coghlan <ncoghlan@gmail.com>, Benjamin Peterson <benjamin@python.org>
Status: Withdrawn
Type: Informational
Created: 29-Jan-2003
Post-History: 30-Jan-2003


Note
====

This PEP has been moved to the Python dev guide [1]_.


Abstract
========

There's more to changing Python's grammar than editing
``Grammar/Grammar`` and ``Python/compile.c``.  This PEP aims to be a
checklist of places that must also be fixed.

It is probably incomplete.  If you see omissions, just add them if
you can -- you are not going to offend the author's sense of
ownership.  Otherwise submit a bug or patch and assign it to mwh.

This PEP is not intended to be an instruction manual on Python
grammar hacking, for several reasons.


Rationale
=========

People are getting this wrong all the time; it took well over a
year before someone noticed [2]_ that adding the floor division
operator (``//``) broke the ``parser`` module.


Checklist
=========

- ``Grammar/Grammar``: OK, you'd probably worked this one out :)

- ``Parser/Python.asdl`` may need changes to match the ``Grammar``.  Run
  ``make`` to regenerate ``Include/Python-ast.h`` and
  ``Python/Python-ast.c``.

- ``Python/ast.c`` will need changes to create the AST objects
  involved with the ``Grammar`` change.  ``Lib/compiler/ast.py`` will
  need matching changes to the pure-python AST objects.

- ``Parser/pgen`` needs to be rerun to regenerate ``Include/graminit.h``
  and ``Python/graminit.c``. (make should handle this for you.)

- ``Python/symbtable.c``: This handles the symbol collection pass
  that happens immediately before the compilation pass.

- ``Python/compile.c``: You will need to create or modify the
  ``compiler_*`` functions to generate opcodes for your productions.

- You may need to regenerate ``Lib/symbol.py`` and/or ``Lib/token.py``
  and/or ``Lib/keyword.py``.

- The ``parser`` module.  Add some of your new syntax to ``test_parser``,
  bang on ``Modules/parsermodule.c`` until it passes.

- Add some usage of your new syntax to ``test_grammar.py``.

- The ``compiler`` package.  A good test is to compile the standard
  library and test suite with the ``compiler`` package and then check
  it runs.  Note that this only needs to be done in Python 2.x.

- If you've gone so far as to change the token structure of
  Python, then the ``Lib/tokenizer.py`` library module will need to
  be changed.

- Certain changes may require tweaks to the library module
  ``pyclbr``.

- Documentation must be written!

- After everything's been checked in, you're likely to see a new
  change to ``Python/Python-ast.c``.  This is because this
  (generated) file contains the SVN version of the source from
  which it was generated.  There's no way to avoid this; you just
  have to submit this file separately.


References
==========

.. [1] CPython Developer's Guide: Changing CPython's Grammar
       https://devguide.python.org/grammar/

.. [2] SF Bug #676521, parser module validation failure
       https://bugs.python.org/issue676521


Copyright
=========

This document has been placed in the public domain.


id: pep-0312.rst
content:
PEP: 312
Title: Simple Implicit Lambda
Author: Roman Suzi <rnd@onego.ru>, Alex Martelli <aleaxit@gmail.com>
Status: Deferred
Type: Standards Track
Created: 11-Feb-2003
Python-Version: 2.4
Post-History:


Abstract
========

This PEP proposes to make argumentless lambda keyword optional in
some cases where it is not grammatically ambiguous.


Deferral
========

The BDFL hates the unary colon syntax.  This PEP needs to go back
to the drawing board and find a more Pythonic syntax (perhaps an
alternative unary operator).  See python-dev discussion on
17 June 2005 [1]_.

Also, it is probably a good idea to eliminate the alternative
propositions which have no chance at all.  The examples section
is good and highlights the readability improvements.  It would
carry more weight with additional examples and with real-world
referents (instead of the abstracted dummy calls to ``:A`` and ``:B``).


Motivation
==========

Lambdas are useful for defining anonymous functions, e.g. for use
as callbacks or (pseudo)-lazy evaluation schemes.  Often, lambdas
are not used when they would be appropriate, just because the
keyword "lambda" makes code look complex.  Omitting lambda in some
special cases is possible, with small and backwards compatible
changes to the grammar, and provides a cheap cure against such
"lambdaphobia".


Rationale
=========

Sometimes people do not use lambdas because they fear to introduce
a term with a theory behind it.  This proposal makes introducing
argumentless lambdas easier, by omitting the "lambda" keyword.
itself.  Implementation can be done simply changing grammar so it
lets the "lambda" keyword be implied in a few well-known cases.
In particular, adding surrounding brackets lets you specify
nullary lambda anywhere.


Syntax
======

An argumentless "lambda" keyword can be omitted in the following
cases:

* immediately after "=" in named parameter assignment or default
  value assignment;

* immediately after "(" in any expression;

* immediately after a "," in a function argument list;

* immediately after a ":" in a dictionary literal; (not
  implemented)

* in an assignment statement; (not implemented)


Examples of Use
===============

1) Inline ``if``::

    def ifelse(cond, true_part, false_part):
        if cond:
            return true_part()
        else:
            return false_part()

    # old syntax:
    print ifelse(a < b, lambda:A, lambda:B)

    # new syntax:
    print ifelse(a < b, :A, :B)

    # parts A and B may require extensive processing, as in:
    print ifelse(a < b, :ext_proc1(A), :ext_proc2(B))

2) Locking::

    def with(alock, acallable):
        alock.acquire()
        try:
            acallable()
        finally:
            alock.release()

    with(mylock, :x(y(), 23, z(), 'foo'))


Implementation
==============

Implementation requires some tweaking of the ``Grammar/Grammar`` file
in the Python sources, and some adjustment of
``Modules/parsermodule.c`` to make syntactic and pragmatic changes.

(Some grammar/parser guru is needed to make a full
implementation.)

Here are the changes needed to ``Grammar`` to allow implicit lambda::

    varargslist: (fpdef ['=' imptest] ',')* ('*' NAME [',' '**'
    NAME] | '**' NAME) | fpdef ['=' imptest] (',' fpdef ['='
    imptest])* [',']

    imptest: test | implambdef

    atom: '(' [imptestlist] ')' | '[' [listmaker] ']' |
    '{' [dictmaker] '}' | '`' testlist1 '`' | NAME | NUMBER | STRING+

    implambdef: ':' test

    imptestlist: imptest (',' imptest)* [',']

    argument: [test '='] imptest

Three new non-terminals are needed: ``imptest`` for the place where
implicit lambda may occur, ``implambdef`` for the implicit lambda
definition itself, ``imptestlist`` for a place where ``imptest``'s may
occur.

This implementation is not complete. First, because some files in
Parser module need to be updated. Second, some additional places
aren't implemented, see Syntax section above.


Discussion
==========

This feature is not a high-visibility one (the only novel part is
the absence of lambda). The feature is intended to make null-ary
lambdas more appealing syntactically, to provide lazy evaluation
of expressions in some simple cases. This proposal is not targeted
at more advanced cases (demanding arguments for the lambda).

There is an alternative proposition for implicit lambda: implicit
lambda with unused arguments. In this case the function defined by
such lambda can accept any parameters, i.e. be equivalent to:
``lambda *args: expr``. This form would be more powerful.  Grep in the
standard library revealed that such lambdas are indeed in use.

One more extension can provide a way to have a list of parameters
passed to a function defined by implicit lambda. However, such
parameters need some special name to be accessed and are unlikely
to be included in the language. Possible local names for such
parameters are: ``_``, ``__args__``, ``__``. For example::

    reduce(:_[0] + _[1], [1,2,3], 0)
    reduce(:__[0] + __[1], [1,2,3], 0)
    reduce(:__args__[0] + __args__[1], [1,2,3], 0)

These forms do not look very nice, and in the PEP author's opinion
do not justify the removal of the lambda keyword in such cases.


Credits
=======

The idea of dropping lambda was first coined by Paul Rubin at 08
Feb 2003 16:39:30 -0800 in comp.lang.python while discussing the
thread "For review: PEP 308 - If-then-else expression" [2]_.


References
==========

.. [1] Guido van Rossum, Recommend accepting PEP 312 -- Simple Implicit Lambda
       https://mail.python.org/pipermail/python-dev/2005-June/054304.html

.. [2] Guido van Rossum, For review: PEP 308 - If-then-else expression
       https://mail.python.org/pipermail/python-dev/2003-February/033178.html



Copyright
=========

This document has been placed in the public domain.


id: pep-0474.rst
content:
PEP: 474
Title: Creating forge.python.org
Author: Alyssa Coghlan <ncoghlan@gmail.com>
Status: Withdrawn
Type: Process
Created: 19-Jul-2014
Post-History: 19-Jul-2014, 08-Jan-2015, 01-Feb-2015


Abstract
========

This PEP proposes setting up a new PSF provided resource, forge.python.org,
as a location for maintaining various supporting repositories
(such as the repository for Python Enhancement Proposals) in a way that is
more accessible to new contributors, and easier to manage for core
developers.

This PEP does *not* propose any changes to the core development workflow
for CPython itself (see :pep:`462` in relation to that).


PEP Withdrawal
==============

This PEP has been `withdrawn by the author
<https://mail.python.org/pipermail/core-workflow/2015-October/000227.html>`_
in favour of the GitLab based proposal in :pep:`507`.

If anyone else would like to take over championing this PEP, contact the
`core-workflow mailing list <https://mail.python.org/mailman/listinfo/core-workflow>`_


Proposal
========

This PEP proposes that an instance of the self-hosted Kallithea code
repository management system be deployed as "forge.python.org".

Individual repositories (such as the developer guide or the PEPs repository)
may then be migrated from the existing hg.python.org infrastructure to the
new forge.python.org infrastructure on a case-by-case basis. Each migration
will need to decide whether to retain a read-only mirror on hg.python.org,
or whether to just migrate wholesale to the new location.

In addition to supporting read-only mirrors on hg.python.org,
forge.python.org will also aim to support hosting mirrors on popular
proprietary hosting sites like GitHub and BitBucket. The aim will be to
allow users familiar with these sites to submit and discuss pull requests
using their preferred workflow, with forge.python.org automatically bringing
those contributions over to the master repository.

Given the availability and popularity of commercially backed "free for open
source projects" repository hosting services, this would not be a general
purpose hosting site for arbitrary Python projects. The initial focus will be
specifically on CPython and other repositories currently hosted on
hg.python.org. In the future, this could potentially be expanded to
consolidating other PSF managed repositories that are currently externally
hosted to gain access to a pull request based workflow, such as the
repository for the python.org Django application. As with the initial
migrations, any such future migrations would be considered on a case-by-case
basis, taking into account the preferences of the primary users of each
repository.


Rationale
=========

Currently, hg.python.org hosts more than just the core CPython repository,
it also hosts other repositories such as those for the CPython developer
guide and for Python Enhancement Proposals, along with various "sandbox"
repositories for core developer experimentation.

While the simple "pull request" style workflow made popular by code hosting
sites like GitHub and BitBucket isn't adequate for the complex branching
model needed for parallel maintenance and development of the various
CPython releases, it's a good fit for several of the ancillary projects
that surround CPython that we don't wish to move to a proprietary hosting
site.

The key requirements proposed for a PSF provided software forge are:

* MUST support simple "pull request" style workflows
* MUST support online editing for simple changes
* MUST be backed by an active development organisation (community or
  commercial)
* MUST support self-hosting of the master repository on PSF infrastructure
  without ongoing fees

Additional recommended requirements that are satisfied by this proposal,
but may be negotiable if a sufficiently compelling alternative is presented:

* SHOULD be a fully open source application written in Python
* SHOULD support Mercurial (for consistency with existing tooling)
* SHOULD support Git (to provide that option to users that prefer it)
* SHOULD allow users of git and Mercurial clients to transparently
  collaborate on the same repository
* SHOULD allow users of GitHub and BitBucket to submit proposed changes using
  the standard pull request workflows offered by those tools
* SHOULD be open to customisation to meet the needs of CPython core
  development, including providing a potential path forward for the
  proposed migration to a core reviewer model in :pep:`462`

The preference for self-hosting without ongoing fees rules out the
free-as-in-beer providers like GitHub and BitBucket, in addition to the
various proprietary source code management offerings.

The preference for Mercurial support not only rules out GitHub, but also
other Git-only solutions like GitLab and Gitorious.

The hard requirement for online editing support rules out the Apache
Allura/HgForge combination.

The preference for a fully open source solution rules out RhodeCode.

Of the various options considered by the author of this proposal, that
leaves `Kallithea SCM <https://kallithea-scm.org/>`__ as the proposed
foundation for a forge.python.org service.

Kallithea is a full GPLv3 application (derived from the clearly
and unambiguously GPLv3 licensed components of RhodeCode) that is being
developed under the auspices of the `Software Freedom Conservancy
<http://sfconservancy.org/news/2014/jul/04/kallithea-joins/>`__. The
Conservancy has `affirmed
<http://sfconservancy.org/blog/2014/jul/15/why-kallithea/>`__ that the
Kallithea codebase is completely and validly licensed under GPLv3. In
addition to their role in building the initial Kallithea community, the
Conservancy is also the legal home of both the Mercurial and Git projects.
Other SFC member projects that may be familiar to Python users include
Twisted, Gevent, BuildBot and PyPy.


Intended Benefits
==================

The primary benefit of deploying Kallithea as forge.python.org is that
supporting repositories such as the developer guide and the PEP repo could
potentially be managed using pull requests and online editing. This would be
much simpler than the current workflow which requires PEP editors and
other core developers to act as intermediaries to apply updates suggested
by other users.

The richer administrative functionality would also make it substantially
easier to grant users access to particular repositories for collaboration
purposes, without having to grant them general access to the entire
installation. This helps lower barriers to entry, as trust can more
readily be granted and earned incrementally, rather than being an
all-or-nothing decision around granting core developer access.

Sustaining Engineering Considerations
=====================================

Even with its current workflow, CPython itself remains one of the largest
open source projects in the world (in the
`top 2%
<https://www.openhub.net/p/python/factoids#FactoidTeamSizeVeryLarge>`__
of projects tracked on OpenHub). Unfortunately, we have been significantly
less effective at encouraging contributions to the projects that make up
CPython's workflow infrastructure, including ensuring that our installations
track upstream, and that wherever feasible, our own customisations are
contributed back to the original project.

As such, a core component of this proposal is to actively engage with the
upstream Kallithea community to lower the barriers to working with and on
the Kallithea SCM, as well as with the PSF Infrastructure team to ensure
the forge.python.org service integrates cleanly with the PSF's infrastructure
automation.

This approach aims to provide a number of key benefits:

* allowing those of us contributing to maintenance of this service to be
  as productive as possible in the time we have available
* offering a compelling professional development opportunity to those
  volunteers that choose to participate in maintenance of this service
* making the Kallithea project itself more attractive to other potential


id: pep-8000.rst
content:
PEP: 8000
Title: Python Language Governance Proposal Overview
Author: Barry Warsaw <barry@python.org>
Status: Final
Type: Informational
Topic: Governance
Created: 24-Aug-2018


Abstract
========

This PEP provides an overview of the selection process for a new model of
Python language governance in the wake of `Guido's retirement
<https://mail.python.org/pipermail/python-committers/2018-July/005664.html>`_.
Once the governance model is selected, it will be codified in :pep:`13`.

Here is a list of PEPs related to the governance model selection process.
PEPs in the lower 8000s describe the general process for selecting a
governance model.

* :pep:`8001` - Python Governance Voting Process

  This PEP describes how the vote for the new governance model will be
  conducted.  It outlines the voting method, timeline, criteria for
  participation, and explicit list of eligible voters.

* :pep:`8002` - Open Source Governance Survey

  Surveys will be conducted of governance models for similar open source and
  free software projects, and summaries of these models will be outlined in
  this PEP.  These surveys will serve as useful barometers for how such
  projects can be successfully governed, and may serve as inspiration for
  Python's own governance model.  Python is unique, so it's expected that it
  will have its own spin on governance, rather than directly adopting any of
  those surveyed.

PEPs in the 801Xs describe the actual proposals for Python governance.  It is
expected that these PEPs will cover the broad scope of governance, and that
differences in details (such as the size of a governing council) will be
covered in the same PEP, rather than in potentially vote-splitting individual
PEPs.

* :pep:`8010` - The Technical Leader Governance Model

  This PEP proposes a continuation of the singular technical project
  leader model.  Also within scope is whether an advisory council aids
  or supports the BDFL.  This PEP does *not* name either the next
  BDFL, nor members of such an advisory council.  For that, see PEP
  13.

* :pep:`8011` - Python Governance Model Lead by Trio of Pythonistas

  This PEP describes a new model of Python governance lead by a Trio of Pythonistas
  (TOP).  It describes the role and responsibilities of the Trio.
  This PEP does *not* name members of the Trio.  For that, see :pep:`13`.

* :pep:`8012` - The Community Governance Model

  This is a placeholder PEP for a new model of Python governance based on
  consensus and voting, without the role of a centralized singular leader or a
  governing council.  It describes how, when, and why votes are conducted for
  decisions affecting the Python language.  It also describes the criteria for
  voting eligibility.

* :pep:`8013` - The External Governance Model

  This PEP describes a new model of Python governance based on an external
  council who are responsible for ensuring good process.  Elected by the core
  development team, this council may reject proposals that are not
  sufficiently detailed, do not consider all affected users, or are not
  appropriate for the upcoming release.  This PEP does *not* name members of
  such a council.  For that, see :pep:`13`.

* :pep:`8014` - The Commons Governance Model

  This PEP describes a new model of Python governance based on a council of
  elders who are responsible for ensuring a PEP is supported by a sufficient
  majority of the Python community before being accepted. Unlike some of the
  other governance PEPs it explicitly does *not* specify who has voting
  rights and what a majority vote consists of. In stead this is determined
  by the council of elders on a case-by-case basis.

* :pep:`8015` - Organization of the Python community

  This PEP formalizes the current organization of the Python community
  and proposes 3 main changes: formalize the existing concept of
  "Python teams"; give more autonomy to Python teams; replace the BDFL
  (Guido van Rossum) with a new "Python board" of 3 members which has
  limited roles, mostly decide how a PEP is approved (or rejected).

* :pep:`8016` - The Steering Council Model

  This PEP proposes a model of Python governance based around a
  steering council. The council has broad authority, which they seek
  to exercise as rarely as possible; instead, they use this power to
  establish standard processes, like those proposed in the other
  801x-series PEPs.  This follows the general philosophy that it's
  better to split up large changes into a series of small changes that
  can be reviewed independently: instead of trying to do everything in
  one PEP, we focus on providing a minimal-but-solid foundation for
  further governance decisions.

Additional governance models may be added before the final selection.


Copyright
=========

This document has been placed in the public domain.


id: pep-0448.rst
content:
PEP: 448
Title: Additional Unpacking Generalizations
Author: Joshua Landau <joshua@landau.ws>
Discussions-To: python-ideas@python.org
Status: Final
Type: Standards Track
Created: 29-Jun-2013
Python-Version: 3.5
Post-History:


Abstract
========

This PEP proposes extended usages of the ``*`` iterable unpacking
operator and ``**`` dictionary unpacking operators
to allow unpacking in more positions, an arbitrary number of
times, and in function calls and displays.

Function calls are proposed to support an arbitrary number of
unpackings rather than just one::

    >>> print(*[1], *[2], 3)
    1 2 3
    >>> dict(**{'x': 1}, y=2, **{'z': 3})
    {'x': 1, 'y': 2, 'z': 3}

Unpacking is proposed to be allowed inside tuple, list, set,
and dictionary displays::

    >>> *range(4), 4
    (0, 1, 2, 3, 4)
    >>> [*range(4), 4]
    [0, 1, 2, 3, 4]
    >>> {*range(4), 4}
    {0, 1, 2, 3, 4}
    >>> {'x': 1, **{'y': 2}}
    {'x': 1, 'y': 2}

In dictionaries, later values will always override earlier ones::

    >>> {'x': 1, **{'x': 2}}
    {'x': 2}

    >>> {**{'x': 2}, 'x': 1}
    {'x': 1}

This PEP does not include unpacking operators inside list, set and
dictionary comprehensions although this has not been ruled out for
future proposals.


Rationale
=========

Current usage of the ``*`` iterable unpacking operator features
unnecessary restrictions that can harm readability.

Unpacking multiple times has an obvious rationale.  When you want to
unpack several iterables into a function definition or follow an unpack
with more positional arguments, the most natural way would be to write::

    function(**kw_arguments, **more_arguments)

    function(*arguments, argument)

Simple examples where this is useful are ``print`` and ``str.format``.
Instead, you could be forced to write::

    kwargs = dict(kw_arguments)
    kwargs.update(more_arguments)
    function(**kwargs)

    args = list(arguments)
    args.append(arg)
    function(*args)

or, if you know to do so::

    from collections import ChainMap
    function(**ChainMap(more_arguments, arguments))

    from itertools import chain
    function(*chain(args, [arg]))

which add unnecessary line-noise and, with the first methods, causes
duplication of work.


There are two primary rationales for unpacking inside of containers.
Firstly there is a symmetry of assignment, where ``fst, *other, lst =
elems`` and ``elems = fst, *other, lst`` are approximate inverses,
ignoring the specifics of types.  This, in effect, simplifies the
language by removing special cases.

Secondly, it vastly simplifies types of "addition" such as combining
dictionaries, and does so in an unambiguous and well-defined way::

    combination = {**first_dictionary, "x": 1, "y": 2}

instead of::

    combination = first_dictionary.copy()
    combination.update({"x": 1, "y": 2})

which is especially important in contexts where expressions are
preferred.  This is also useful as a more readable way of summing
iterables into a list, such as ``my_list + list(my_tuple) +
list(my_range)`` which is now equivalent to just ``[*my_list,
*my_tuple, *my_range]``.


Specification
=============

Function calls may accept an unbounded number of ``*`` and ``**``
unpackings.  There will be no restriction of the order of positional
arguments with relation to ``*`` unpackings nor any restriction of the
order of keyword arguments with relation to ``**`` unpackings.

Function calls continue to have the restriction that keyword arguments
must follow positional arguments and ``**`` unpackings must additionally
follow ``*`` unpackings.

Currently, if an argument is given multiple times â such as a
positional argument given both positionally and by keyword â a
``TypeError`` is raised.  This remains true for duplicate arguments
provided through multiple ``**`` unpackings,
e.g. ``f(**{'x': 2}, **{'x': 3})``, except that the error will be
detected at runtime.

A function looks like this::

    function(
        argument or *args, argument or *args, ...,
        kwargument or *args, kwargument or *args, ...,
        kwargument or **kwargs, kwargument or **kwargs, ...
    )

Tuples, lists, sets and dictionaries will allow unpacking.  This will
act as if the elements from unpacked items were inserted in order at
the site of unpacking, much as happens in unpacking in a function-call.
Dictionaries require ``**`` unpacking; all the others require ``*`` unpacking.

The keys in a dictionary remain in a right-to-left priority order, so
``{**{'a': 1}, 'a': 2, **{'a': 3}}`` evaluates to ``{'a': 3}``.  There
is no restriction on the number or position of unpackings.


Disadvantages
=============

The allowable orders for arguments in a function call are more
complicated than before.  The simplest explanation for the rules
may be "positional arguments precede keyword arguments and ``**``
unpacking; ``*`` unpacking precedes ``**`` unpacking".

Whilst ``*elements, = iterable`` causes ``elements`` to be a list,
``elements = *iterable,`` causes ``elements`` to be a tuple.  The
reason for this may confuse people unfamiliar with the construct.

Concerns have been raised about the unexpected difference between
duplicate keys in dictionaries being allowed but duplicate keys
in function call syntax raising an error.  Although this is already
the case with current syntax, this proposal might exacerbate the
issue.  It remains to be seen how much of an issue this is in practice.


Variations
==========

The PEP originally considered whether the ordering of argument types
in a function call (positional, keyword, ``*`` or ``**``) could become
less strict. This met little support so the idea was shelved.

Earlier iterations of this PEP allowed unpacking operators inside
list, set, and dictionary comprehensions as a flattening operator
over iterables of containers::

    >>> ranges = [range(i) for i in range(5)]
    >>> [*item for item in ranges]
    [0, 0, 1, 0, 1, 2, 0, 1, 2, 3]

    >>> {*item for item in ranges}
    {0, 1, 2, 3}

This was met with a mix of strong concerns about readability and mild
support. In order not to disadvantage the less controversial aspects
of the PEP, this was not accepted with the rest of the proposal.

Unbracketed comprehensions in function calls, such as ``f(x for x in it)``,
are already valid.  These could be extended to::

    f(*x for x in it) == f((*x for x in it))
    f(**x for x in it) == f({**x for x in it})

However, it wasn't clear if this was the best behaviour or if it should
unpack into the arguments of the call to ``f``. Since this is likely to be
confusing and is of only very marginal utility, it is not included in this
PEP.  Instead, these will throw a ``SyntaxError`` and comprehensions with
explicit brackets should be used instead.


Approval
========

This PEP was accepted by Guido on February 25, 2015 [1]_.


Implementation
==============

An implementation for Python 3.5 is found at Issue 2292 on bug tracker [2]_.
This currently includes support for unpacking inside comprehensions, which
should be removed.


References
==========

.. [1] PEP accepted, "PEP 448 review", Guido van Rossum
   (https://mail.python.org/pipermail/python-dev/2015-February/138564.html)

.. [2] Issue 2292, "Missing ``*``-unpacking generalizations", Thomas Wouters
   (https://github.com/python/cpython/issues/46545)

[3] Discussion on Python-ideas list,
\   "list / array comprehensions extension", Alexander Heger
\   (https://mail.python.org/pipermail/python-ideas/2011-December/013097.html)

Copyright
=========

This document has been placed in the public domain.


id: pep-8014.rst
content:
PEP: 8014
Title: The Commons Governance Model
Author: Jack Jansen
Status: Rejected
Type: Informational
Topic: Governance
Created: 16-Sep-2018

Abstract
========

This PEP proposes a governance model with as few procedures, defined terms and
percentages as possible. It may also be called *The Anarchist Governance Model*
but uses *Commons* for now because of possible negative connotations of the
term *Anarchist* to some audiences.

The basic idea is that all decisions are in principle voted on by the whole
community, but in practice voted on by only a subset of the
community. A subset, because although the whole community is
entitled to vote in practice it will always be only a small subset that vote
on a specific decision. The vote is overseen by an impartial council that
judges whether the decision has passed or not. The intention is that this
council bases its decision not only on the ratio of yes and no votes but
also on the total number of votes,  on the gravity of the proposal being
voted on and possibly the individual voters and how they voted. Thereby this
council becomes responsible for ensuring that each individual decision is
carried by a sufficient majority.

PEP Rejection
=============

:pep:`8014` was rejected `by a core developer vote
<https://discuss.python.org/t/python-governance-vote-december-2018-results/546/>`__
described in :pep:`8001` on Monday, December 17, 2018.

:pep:`8016` and the governance model it describes were chosen instead.

Introduction
============

The Commons Governance Model tries to ensure that all decisions are endorsed
by, or at least is acceptable to, a sufficient majority of the Python
community.

Unfortunately the previous paragraph has two terms that are very hard to
quantify in the general case: *sufficient majority* and *Python community*.
This is because both terms in reality depend on the *specific* case that is
being decided. To give an example of this difficulty: for a PEP that
proposes a backward-compatible change to some API a simple majority of the
core developers that were interested in voting on the PEP in the first place
is probably sufficient. But for a change that has more farreaching
consequences such as a Python3 to Python4 transition a real majority may be
wanted, and a demonstration that at least there seems to be sufficient
support in the user base. And for a change that transcends the
Python-the-language, such as decisions on abolishing non-inclusive language,
it becomes very vague.

The Commons Governance Model attempts to sidestep this issue by *not*
defining what the terms *sufficient majority* and *Python community* mean in
the general case, by proposing a body that will decide so in *specific*
cases.

The model proposes creating a *Council of Elders* that oversees the decision
process, determining whether a specific proposal has enough support on a
case-by-case basis. There will be a vote on every individual PEP,
and the Council of Elders will declare whether the
outcome of the vote is sufficient to carry the decision *in this specific case*.

The model addresses only the roles traditionally held by the BDFL in the
decision process, not other roles.

The term Commons_ in the model name is loosely based on its historic use as
a shared resource to be used by all and cared for by all. The picture you
should have in mind with this model is a sizeable group of peasants
discussing some plan for the future on the village green on a warm summer
evening, after which the vote is taken and the village elders pronounce
the outcome. Then the banquet begins.

.. _Commons: https://en.wikipedia.org/wiki/Commons

The Commons Governance Model is different from most of the other governance
proposals (with the possible exception of 8012), because it explicitly places
supreme power with the whole community.

Rationale
=========

The rationale for the model is that a model that casts everything in concrete will
have unintended negative side effects. For example, a governance model that
assigns voting rights to Python committers may cause an individual not
to be accepted as a committer because there are already a lot of committers
from the company the new candidate works for.

As another example, setting a fixed percentage for PEP acceptance may lead
to party-formation amongst the voters and individual PEPs no longer be being
judged on individual merit but along party lines (if you support my PEP I
will support yours).

There is also the issue that one-person-one-vote is not the best model for
something like Python. Again an example: in case of a split vote (or a vote
sufficiently close to being split) the opinion of core developer Guido
van Rossum should probably outweigh the opinion of core developer Jack
Jansen. Trying to formalize this in a voting model is going to lead to a
very complex model, that is going to be wrong on boundary cases anyway. The
model presented here leaves deciding on such issues to the (hopefully
sensible) council of elders.

Decision Process
================

All important decisions go through a PEP process. Each PEP has someone
responsible for it, called the *author* here, but that does not have to be a
single person, and it does not have to be the person that actually wrote the
text. So for author you could also read *champion* or *shepherd* or
something like that.

The PEP author is responsible for organizing a vote on the PEP. This vote is
public, i.e. the voters are identified and the results are known to all.
Voting may be simple +1/0/-1, but might also be extended with +2/-2 with a
very terse explanation why the voter feels very strong about the issue. Such
an annotation would serve as an explanation to the Council of Elders. Voters
are annotated with their community status (core developer, etc).

The vote is clearly separated from the discussion, by using a well-defined Discourse
category or tag, a special mailing list or a similar technical method
(such as a website vote.python.org where people have to log in so their
community status can be automatically added, and their identity can be somewhat
confirmed).

The PEP author presents the PEP and the vote results to the Council of Elders.
The council ponders two things:

- the PEP gravity and its implications,
- the measurable vote results (how many people voted, which individuals voted, what they voted).

They pronounce a tentative decision on whether the vote passed and this decision is published.

If the decision is that the vote results do not demonstrate enough support
from the community for the decision the burden is on the author to try and
gather more support and resubmit the vote at a later date. Alternatively the
author can retract the proposal. The period for gathering more support is
time-limited, a month seems a reasonable time, if no vote has been resubmitted
after that period the proposal is rejected.

If the tentative decision is that the results *do* demonstrate enough support
a fairly short waiting period starts (in the order of weeks). During this
period anyone can appeal to the Council of Elders, but *only* on the grounds
that the vote does not reflect a sufficient majority of the community.
After the waiting period the council pronounces a final decision. The PEP
is either accepted or, if the council is swayed by an appeal, goes back to
the state where more support has to be demonstrated.


Council of Elders
=================

The intention of the Council of Elders is that they, together, are capable
of judging whether the will of the Python community is upheld in a specific
vote.

The Council of Elders is *not* a replacement of the BDFL by a group of
people with the same power as the BDFL: it will not provide guidance on the
direction of Python, it only attempts to ensure the outcome of a vote
represents the will of the community.

The Council of Elders is *not* like the US Supreme Court, which has actual
decision power, the council only oversees the voting process to ensure that
the community is re


