# embeddings

id: pep-8015.rst
content:
PEP: 8015
Title: Organization of the Python community
Author: Victor Stinner
Status: Rejected
Type: Informational
Topic: Governance
Created: 04-Oct-2018

Abstract
========

This PEP formalizes the current organization of the Python community and
proposes 3 main changes:

* Formalize the existing concept of "Python teams";
* Give more autonomy to Python teams;
* Replace the BDFL (Guido van Rossum) with a new "Python Steering
  Committee" of 5 members which has limited roles: basically decide how
  decisions are taken, but don't take decisions.

PEPs are approved by a PEP delegate or by a vote (reserved to core
developers, need ``>= 2/3`` majority).


PEP Rejection
=============

:pep:`8015` was rejected `by a core developer vote
<https://discuss.python.org/t/python-governance-vote-december-2018-results/546/>`__
described in :pep:`8001` on Monday, December 17, 2018.

:pep:`8016` and the governance model it describes were chosen instead.


Rationale
=========

This PEP describes the organization of the whole Python development
community, from Python users to the Python Steering Committee.
Describing all groups and all roles in the same document helps to make
the organization more consistent.

The number of governance changes is minimized to get a smooth transition
from the old BDFL organization to the new Steering Committee
organization.

One key design of the organization is to avoid decision bottlenecks.
Discussions and decisions are distributed into Python teams where
experts in each topic can be found. The expectation is smoother
discussions on PEPs: fewer people with better knowledge of the topic.

Previously, most decisions have been taken by the Benevolent
Dictator For Life (BDFL), Guido van Rossum. The growing popularity of
Python increased the pressure on a single person. The proposed
organization distributes decisions and responsibilities to reduce the
pressure and avoid wearing any individual down.

To keep most of the decision power within the hands of the community,
the Python Steering Committee has very limited roles. The idea is to reduce the risk
that a group of people or companies "takes over" the Python project
through just a couple individuals. The project must remain autonomous
and open to everybody.

The most sensitives PEPs are decided by democracy: a vote reserved to
core developers, see the `PEP process`_ section below for the voting
method.


Common Guidelines
=================

* The Python community is open to everyone.
* Members must respect the `Python Community Code of Conduct
  <https://www.python.org/psf/codeofconduct/>`_ which ensures that
  discussions remain constructive and that everybody feels welcomed.
* Python is and will remain an autonomous project.
* People with decisions power should reflect the diversity of its users
  and contributors.


Community Organization
======================

Right now, there are different group of people involved in the Python
project. The more involved you are, the more decisions power you get. It
is important that the people acceding to the deepest group are the most
trusted ones.

This PEP formalizes the following groups:

* Python Users
* Python Contributors
* Python Teams Members
* Python Core Developers
* Python Steering Committee Members
* PSF Code of Conduct Workgroup


Python Users
============

This is the largest group: anyone who uses Python.


Python Contributors
===================

Once a Python user sends an email to a Python mailing list, comments on
the Python bug tracker, proposes or reviews a Python change, they become
a Python contributor.


Python Teams
============

Python became too big to work as a unique team anymore, people
naturally have grouped themselves as teams to work more closely on
specific topics, sometimes called "Special Interest Group" (SIG).

When enough developers are interested by a specific topic, they can
create a new team. Usually, the main action is to ask the Python
postmaster to create a new "SIG" mailing list, but the team can choose
to use a different communication channel.

Team members are Python contributors and Python core developers. The
team is self-organized and is responsible to select who can join the
team and how.

Team members can get the bug triage permission on the team bug tracker
component. The more involved in a team you are, the more decisions power
and responsibilities you get.

A team might become allowed to decide on their own PEPs, but only the
Python Steering Committee can allow that (and it has the power to revoke
it as well). Such a case is exceptional, currently a single team has
such permission: the Packaging Team.

See `Annex: Examples of Python Teams`_.


Python Core Developers
======================

One restricted definition of a core developer is the ability to merge a
change (anywhere in the code) and have the bug triage permission
(on all bug tracker components).

Core developers are developers who are proven to have the required skills to
decide if a change can be approved or must be rejected, but also (and
this is more important) what changes should not be made. Python has a
long history, big constraints on backward compatibility, high quality
standards (ex: changes require new tests). For these reasons, becoming
a core can take several months or longer.

Becoming a core developer means more responsibilities. For example, if a
developer merges a change, they become responsible for regressions and
for the maintenance of that modified code.

Core developers are expected to be exemplary when it comes to the Code
of Conduct. They are encouraged to mentor contributors.

Promote a contributor as core developer
---------------------------------------

Once an existing core developer considers that a contributor is ready to
join the core group, to become a core developer, that core developer
asks the contributor if they would like to become a core developer. If
the contributor is interested in such new responsibilities, a vote is
organized.

The vote is reserved to core developers, is public, and is open for 1
week.  Usually the core developer who proposes the promotion has to
describe the work and skills of the candidate in the description of the
vote. A contributor is only promoted if two thirds (``>= 2/3``) of
votes approve ("+1") the promotion. Only "+1" and "-1" votes are
accounted; other votes (ex: null, "-0", "+0.5") are ignored.

If the candidate is promoted, usually they get a mentor for 1 month to
help them to handle new responsibilities.

If the candidate is not promoted, a new vote can be organized later,
when the candidate gets the missing skills, for example 6 months later.


Python Steering Committee
=========================

The Python Steering Committee is made of the most trusted core
developers since it has the most decision power. The roles of this group
are strictly limited to ensure that Python keeps its autonomy and
remains open.

The Python Steering Committee is composed of 5 members. They are elected
for 3 years and 1/3 is replaced every year (first year: 1, second year:
2, third year: 2). This way, a member will stay for one full Python
release and the committee composition will be updated frequently. A
committee member can be a candidate for the seat they are leaving.
There are no term limits.

Committee members must be Python core developers. It is important that
the members of the committee reflect the diversity of Python' users and
contributors. A small step to ensure that is to enforce that only 2
members (strictly less than 50% of the 5 members) can work for the same
employer (same company or subsidiaries of the same company).

The size of 5 members has been chosen for the members diversity and to
ensure that the committee can continue to work even if a member becomes
unavailable for an unknown duration.

Python Steering Committee Roles
-------------------------------

Python Steering Committee roles:

* Decide how a PEP is approved (or rejected or deferred).
* Grant or rev


id: pep-0449.rst
content:
PEP: 449
Title: Removal of the PyPI Mirror Auto Discovery and Naming Scheme
Author: Donald Stufft <donald@stufft.io>
BDFL-Delegate: Richard Jones <richard@python.org>
Discussions-To: distutils-sig@python.org
Status: Final
Type: Process
Topic: Packaging
Created: 04-Aug-2013
Post-History: 04-Aug-2013
Replaces: 381
Resolution: https://mail.python.org/pipermail/distutils-sig/2013-August/022518.html


Abstract
========

This PEP provides a path to deprecate and ultimately remove the auto discovery
of PyPI mirrors as well as the hard coded naming scheme which requires
delegating a domain name under pypi.python.org to a third party.


Rationale
=========

The PyPI mirroring infrastructure (defined in :pep:`381`) provides a means to
mirror the content of PyPI used by the automatic installers. It also provides
a method for auto discovery of mirrors and a consistent naming scheme.

There are a number of problems with the auto discovery protocol and the
naming scheme:

* They give control over a \*.python.org domain name to a third party,
  allowing that third party to set or read cookies on the pypi.python.org and
  python.org domain name.
* The use of a sub domain of pypi.python.org means that the mirror operators
  will never be able to get a SSL certificate of their own, and giving them
  one for a python.org domain name is unlikely to happen.
* The auto discovery uses an unauthenticated protocol (DNS).
* The lack of a TLS certificate on these domains means that clients can not
  be sure that they have not been a victim of DNS poisoning or a MITM attack.
* The auto discovery protocol was designed to enable a client to automatically
  select a mirror for use. This is no longer a requirement because the CDN
  that PyPI is now using a globally distributed network of servers which will
  automatically select one close to the client without any effort on the
  clients part.
* The auto discovery protocol and use of the consistent naming scheme has only
  ever been implemented by one installer (pip), and its implementation, besides
  being insecure, has serious issues with performance and is slated for removal
  with its next release (1.5).
* While there are provisions in :pep:`381` that would solve *some* of these
  issues for a dedicated client it would not solve the issues that affect a
  users browser. Additionally these provisions have not been implemented by
  any installer to date.

Due to the number of issues, some of them very serious, and the CDN which
provides most of the benefit of the auto discovery and consistent naming scheme
this PEP proposes to first deprecate and then remove the [a..z].pypi.python.org
names for mirrors and the last.pypi.python.org name for the auto discovery
protocol. The ability to mirror and the method of mirror will not be affected
and will continue to exist as written in :pep:`381`. Operators of existing
mirrors are encouraged to acquire their own domains and certificates to use for
their mirrors if they wish to continue hosting them.


Plan for Deprecation & Removal
==============================

Immediately upon acceptance of this PEP documentation on PyPI will be updated
to reflect the deprecated nature of the official public mirrors and will
direct users to external resources like http://www.pypi-mirrors.org/ to
discover unofficial public mirrors if they wish to use one.

Mirror operators, if they wish to continue operating their mirror, should
acquire a domain name to represent their mirror and, if they are able, a TLS
certificate. Once they have acquired a domain they should redirect their
assigned N.pypi.python.org domain name to their new domain. On Feb 15th, 2014
the DNS entries for [a..z].pypi.python.org and last.pypi.python.org will be
removed. At any time prior to Feb 15th, 2014 a mirror operator may request
that their domain name be reclaimed by PyPI and pointed back at the master.


Why Feb 15th, 2014
------------------

The most critical decision of this PEP is the final cut off date. If the date
is too soon then it needlessly punishes people by forcing them to drop
everything to update their deployment scripts. If the date is too far away then
the extended period of time does not help with the migration effort and merely
puts off the migration until a later date.

The date of Feb 15th, 2014 has been chosen because it is roughly 6 months from
the date of the PEP. This should ensure a lengthy period of time to enable
people to update their deployment procedures to point to the new domains names
without merely padding the cut off date.


Why the DNS entries must be removed
-----------------------------------

While it would be possible to simply reclaim the domain names used in mirror
and direct them back at PyPI in order to prevent users from needing to update
configurations to point away from those domains this has a number of issues.

* Anyone who currently has these names hard coded in their configuration has
  them hard coded as HTTP. This means that by allowing these names to continue
  resolving we make it simple for a MITM operator to attack users by rewriting
  the redirect to HTTPS prior to giving it to the client.
* The overhead of maintaining several domains pointing at PyPI has proved
  troublesome for the small number of N.pypi.python.org domains that have
  already been reclaimed. They oftentimes get mis-configured when things
  change on the service which often leaves them broken for months at a time
  until somebody notices. By leaving them in we leave users of these domains
  open to random breakages which are less likely to get caught or noticed.
* People using these domains have explicitly chosen to use them for one reason
  or another. One such reason may be because they do not wish to deploy from
  a host located in a particular country. If these domains continue to resolve
  but do not point at their existing locations we have silently removed this
  choice from the existing users of those domains.

That being said, removing the entries *will* require users who have modified
their configuration to either point back at the master (PyPI) or select a new
mirror name to point at. This is regarded as a regrettable requirement to
protect PyPI itself and the users of the mirrors from the attacks outlined
above or, at the very least, require them to make an informed decision about
the insecurity.


Public or Private Mirrors
=========================

The mirroring protocol will continue to exist as defined in :pep:`381` and
people are encouraged to host public and private mirrors if they so desire.
The recommended mirroring client is `Bandersnatch`_.


.. _PyPI: https://pypi.python.org/
.. _Bandersnatch: https://pypi.python.org/pypi/bandersnatch


Copyright
=========

This document has been placed in the public domain.


id: pep-8001.rst
content:
PEP: 8001
Title: Python Governance Voting Process
Author: Brett Cannon <brett@python.org>,
        Christian Heimes <christian@python.org>,
        Donald Stufft <donald@stufft.io>,
        Eric Snow <ericsnowcurrently@gmail.com>,
        Gregory P. Smith <greg@krypto.org>,
        Åukasz Langa <lukasz@python.org>,
        Mariatta <mariatta@python.org>,
        Nathaniel J. Smith <njs@pobox.com>,
        Pablo Galindo Salgado <pablogsal@python.org>,
        Raymond Hettinger <python@rcn.com>,
        Tal Einat <tal@python.org>,
        Tim Peters <tim.peters@gmail.com>,
        Zachary Ware <zach@python.org>
Status: Final
Type: Process
Topic: Governance
Created: 24-Aug-2018


Abstract
========

This PEP outlines the process for how the new model of Python governance is
selected, in the wake of `Guido's retirement
<https://mail.python.org/pipermail/python-committers/2018-July/005664.html>`_.
Once the model is chosen by the procedures outlined here, it will be codified
in :pep:`13`.


Motivation and Rationale
========================

Guido's stepping down from the BDFL role left us with a meta-problem of
having to choose *how we will choose* how the Python project should be
governed from now on.

This document presents a concrete proposal how this choice can be made.
It summarizes discussion and conclusions of the proceedings of a working
group at the core sprint in Redmond in September 2018 (names of all
attendees are listed as authors). This PEP also summarizes a
`subsequent thread <https://discuss.python.org/t/python-governance-electoral-system/290>`_
that took place on discuss.python.org .

The governance situation should be resolved in a timely fashion.
Ideally that should happen by the end of the 2018 which unblocks
substantial improvements to be merged in time for Python 3.8.  At the
latest, the governance situation needs to be resolved by PyCon US 2019 to
avoid a PR crisis.


Implementation
==============

What are we voting for?
-----------------------

We are voting to choose which governance PEP should be implemented by
the Python project.  The list of candidate PEPs is listed in :pep:`8000`
and consists of all PEPs numbered in the 801X range.

To ensure the vote is legitimate, the aforementioned PEPs must not be
modified during the voting period.

Who gets to vote?
-----------------

Every CPython core developer is invited to vote.  In the interest of
transparency and fairness, we are asking core developers to self-select
based on whether the governance situation will affect them directly.
In other words, we are recommending for inactive core developers *who
intend to remain inactive* to abstain from voting.

When is the vote?
-----------------

November 16th, 2018 to November 30th, 2018 is the official governance
PEP review period.  We discourage the PEP authors from making major
substantive changes during this period, although it is expected that
minor tweaks may occur, as the result of this discussion period.

The vote will happen in a 2-week-long window from December 1st, 2018
to December 16th, 2018
(`Anywhere on Earth <https://en.wikipedia.org/wiki/Anywhere_on_Earth>`_).

Where is the vote?
------------------

The vote will happen using a "private" poll on the
`Condorcet Internet Voting Service <https://civs.cs.cornell.edu/>`_. Every committer
will receive an email with a link allowing them to rank the PEPs in their order of
preference.

The election will be supervised by Ee Durbin, The PSF Director of Infrastructure.

The results of the election, including anonymized ballots, will be made public on
December 17th, after the election has closed.

The following settings will be used for the vote in the CIVS system:

Name of the poll: ``Python governance vote (December 2018)``

Description of the poll::

    This is the vote to choose how the CPython project will govern
    itself, now that Guido has announced his retirement as BDFL. For
    full details, see <a
    href="https://peps.python.org/pep-8001/">PEP
    8001</a>. Many discussions have occurred under <a
    href="https://discuss.python.org/tags/governance">the "governance"
    tag</a> on discuss.python.org.
    <p>
    All votes must be received <b>by the end of December 16th, 2018, <a
    href="https://en.wikipedia.org/wiki/Anywhere_on_Earth">Anywhere on
    Earth</a></b>. All CPython core developers are <a
    href="https://github.com/python/voters">eligible to vote</a>.
    It is asked that inactive core developers <i>who intend to remain
    inactive</i> abstain from voting.
    <p>
    <b>Note: You can only vote once, and all votes are final.</b> Once
    you click "Submit ranking", it's too late to change your mind.
    <p>
    All ballots will be published at the end of voting, but <b>without
    any names attached</b>. No-one associated with the Python project or
    the PSF will know how you voted, or even whether you voted.
    <p>
    If you have any questions, you can post in <a
    href="https://discuss.python.org/c/committers">the Committers
    topic</a>, on <a href="mailto:python-committers@python.org">the
    python-committers list</a>, or <a
    href="mailto:ee@python.org">contact the vote administrator
    directly</a>.
    <p>
    <h1>Options</h1>
    <p>
    We're selecting between seven PEPs, each proposing a different
    governance model.
    <p>
    The options below include links to the text of each PEP, as well
    as their complete change history. The text of these PEPs was
    frozen on December 1, when the vote started. But if you looked at
    the PEPs before that, they might have changed. Please take the
    time to check the current text of the PEPs if you read an older
    draft.
    <p>
    A "Further discussion" option is also included. It represents the
    option of not making a choice at all at this time, and continuing
    the discussion instead. Including this option lets us demonstrate
    the core team's readiness to move forward.
    <p>
    If you think a proposal is a particularly bad idea, you can
    express that by ranking it below "Further discussion". If you
    think all of the proposals are better than further discussion,
    then you should rank "Further discussion" last.

Candidates (note: linebreaks are significant here)::

    <a href="https://peps.python.org/pep-8010/">PEP 8010: The Technical Leader Governance Model</a> (Warsaw) (<a href="https://github.com/python/peps/commits/main/pep-8010.rst">changelog</a>)
    <a href="https://peps.python.org/pep-8011/">PEP 8011: Python Governance Model Lead by Trio of Pythonistas</a> (Mariatta, Warsaw) (<a href="https://github.com/python/peps/commits/main/pep-8011.rst">changelog</a>)
    <a href="https://peps.python.org/pep-8012/">PEP 8012: The Community Governance Model</a> (Langa) (<a href="https://github.com/python/peps/commits/main/pep-8012.rst">changelog</a>)
    <a href="https://peps.python.org/pep-8013/">PEP 8013: The External Council Governance Model</a> (Dower) (<a href="https://github.com/python/peps/commits/main/pep-8013.rst">changelog</a>)
    <a href="https://peps.python.org/pep-8014/">PEP 8014: The Commons Governance Model</a> (Jansen) (<a href="https://github.com/python/peps/commits/main/pep-8014.rst">changelog</a>)
    <a href="https://peps.python.org/pep-8015/">PEP 8015: Organization of the Python community</a> (Stinner) (<a href="https://github.com/python/peps/commits/main/pep-8015.rst">changelog</a>)
    <a href="https://peps.python.org/pep-8016/">PEP 8016: The Steering Council Model</a> (Smith, Stufft) (<a href="https://github.com/python/peps/commits/main/pep-8016.rst">changelog</a>)
    Further discussion

Options::

    [x] Private
    [ ] Make this a test poll: read all votes from a file.
    [ ] Do not release results to all voters.
    [x] Enable detailed ballot reporting.
        [ ] In detailed ballot report, also reveal the identity of the voter with each ballot.
    [ ] Allow voters to write in new choices.
    [ ] Present choices on voting page in e


id: pep-0313.rst
content:
PEP: 313
Title: Adding Roman Numeral Literals to Python
Author: Mike Meyer <mwm@mired.org>
Status: Rejected
Type: Standards Track
Created: 01-Apr-2003
Python-Version: 2.4
Post-History:


Abstract
========

This PEP (also known as PEP CCCXIII) proposes adding Roman
numerals as a literal type.  It also proposes the new built-in
function "roman", which converts an object to an integer, then
converts the integer to a string that is the Roman numeral literal
equivalent to the integer.


BDFL Pronouncement
==================

This PEP is rejected.  While the majority of Python users deemed this
to be a nice-to-have feature, the community was unable to reach a
consensus on whether nine should be represented as IX, the modern
form, or VIIII, the classic form.  Likewise, no agreement was
reached on whether MXM or MCMXC would be considered a well-formed
representation of 1990.  A vocal minority of users has also requested
support for lower-cased numerals for use in (i) powerpoint slides,
(ii) academic work, and (iii) Perl documentation.


Rationale
=========

Roman numerals are used in a number of areas, and adding them to
Python as literals would make computations in those areas easier.
For instance, Super Bowls are counted with Roman numerals, and many
older movies have copyright dates in Roman numerals.  Further,
LISP provides a Roman numerals literal package, so adding Roman
numerals to Python will help ease the LISP-envy sometimes seen in
comp.lang.python.  Besides, the author thinks this is the easiest
way to get his name on a PEP.


Syntax for Roman literals
=========================

Roman numeral literals will consist of the characters M, D, C, L,
X, V and I, and only those characters.  They must be in upper
case, and represent an integer with the following rules:

1.  Except as noted below, they must appear in the order M, D, C,
    L, X, V then I.  Each occurrence of each character adds 1000, 500,
    100, 50, 10, 5 and 1 to the value of the literal, respectively.

2.  Only one D, V or L may appear in any given literal.

3.  At most three each of Is, Xs and Cs may appear consecutively
    in any given literal.

4.  A single I may appear immediately to the left of the single V,
    followed by no Is, and adds 4 to the value of the literal.

5.  A single I may likewise appear before the last X, followed by
    no Is or Vs, and adds 9 to the value.

6.  X is to L and C as I is to V and X, except the values are 40
    and 90, respectively.

7.  C is to D and M as I is to V and X, except the values are 400
    and 900, respectively.

Any literal composed entirely of M, D, C, L, X, V and I characters
that does not follow this format will raise a syntax error,
because explicit is better than implicit.


Built-In "roman" Function
=========================

The new built-in function "roman" will aide the translation from
integers to Roman numeral literals.  It will accept a single
object as an argument, and return a string containing the literal
of the same value.  If the argument is not an integer or a
rational (see :pep:`239`) it will passed through the existing
built-in "int" to obtain the value.  This may cause a loss of
information if the object was a float.  If the object is a
rational, then the result will be formatted as a rational literal
(see :pep:`240`) with the integers in the string being Roman
numeral literals.


Compatibility Issues
====================

No new keywords are introduced by this proposal.  Programs that
use variable names that are all upper case and contain only the
characters M, D, C, L, X, V and I will be affected by the new
literals.  These programs will now have syntax errors when those
variables are assigned, and either syntax errors or subtle bugs
when those variables are referenced in expressions.  Since such
variable names violate :pep:`8`, the code is already broken, it
just wasn't generating exceptions. This proposal corrects that
oversight in the language.


Copyright
=========

This document has been placed in the public domain.


id: pep-0475.rst
content:
PEP: 475
Title: Retry system calls failing with EINTR
Author: Charles-FranÃ§ois Natali <cf.natali@gmail.com>, Victor Stinner <vstinner@python.org>
BDFL-Delegate: Antoine Pitrou <solipsis@pitrou.net>
Status: Final
Type: Standards Track
Created: 29-Jul-2014
Python-Version: 3.5
Resolution: https://mail.python.org/pipermail/python-dev/2015-February/138018.html


Abstract
========

System call wrappers provided in the standard library should be retried
automatically when they fail with ``EINTR``, to relieve application code
from the burden of doing so.

By system calls, we mean the functions exposed by the standard C library
pertaining to I/O or handling of other system resources.


Rationale
=========

Interrupted system calls
------------------------

On POSIX systems, signals are common.  Code calling system calls must be
prepared to handle them.  Examples of signals:

* The most common signal is ``SIGINT``, the signal sent when CTRL+c is
  pressed. By default, Python raises a ``KeyboardInterrupt`` exception
  when this signal is received.
* When running subprocesses, the ``SIGCHLD`` signal is sent when a
  child process exits.
* Resizing the terminal sends the ``SIGWINCH`` signal to the
  applications running in the terminal.
* Putting the application in background (ex: press CTRL-z and then
  type the ``bg`` command) sends the ``SIGCONT`` signal.

Writing a C signal handler is difficult: only "async-signal-safe"
functions can be called (for example, ``printf()`` and ``malloc()``
are not async-signal safe), and there are issues with reentrancy.
Therefore, when a signal is received by a process during the execution
of a system call, the system call can fail with the ``EINTR`` error to
give the program an opportunity to handle the signal without the
restriction on signal-safe functions.

This behaviour is system-dependent: on certain systems, using the
``SA_RESTART`` flag, some system calls are retried automatically instead
of failing with ``EINTR``.  Regardless, Python's ``signal.signal()``
function clears the ``SA_RESTART`` flag when setting the signal handler:
all system calls will probably fail with ``EINTR`` in Python.

Since receiving a signal is a non-exceptional occurrence, robust POSIX code
must be prepared to handle ``EINTR`` (which, in most cases, means
retry in a loop in the hope that the call eventually succeeds).
Without special support from Python, this can make application code
much more verbose than it needs to be.


Status in Python 3.4
--------------------

In Python 3.4, handling the ``InterruptedError`` exception (``EINTR``'s
dedicated exception class) is duplicated at every call site on a case-by-case
basis.  Only a few Python modules actually handle this exception,
and fixes usually took several years to cover a whole module.  Example of
code retrying ``file.read()`` on ``InterruptedError``::

    while True:
        try:
            data = file.read(size)
            break
        except InterruptedError:
            continue

List of Python modules in the standard library which handle
``InterruptedError``:

* ``asyncio``
* ``asyncore``
* ``io``, ``_pyio``
* ``multiprocessing``
* ``selectors``
* ``socket``
* ``socketserver``
* ``subprocess``

Other programming languages like Perl, Java and Go retry system calls
failing with ``EINTR`` at a lower level, so that libraries and applications
needn't bother.


Use Case 1: Don't Bother With Signals
-------------------------------------

In most cases, you don't want to be interrupted by signals and you
don't expect to get ``InterruptedError`` exceptions.  For example, do
you really want to write such complex code for a "Hello World"
example?

::

    while True:
        try:
            print("Hello World")
            break
        except InterruptedError:
            continue

``InterruptedError`` can happen in unexpected places. For example,
``os.close()`` and ``FileIO.close()`` may raise ``InterruptedError``:
see the article `close() and EINTR
<http://alobbs.com/post/54503240599/close-and-eintr>`_.

The `Python issues related to EINTR`_ section below gives examples of
bugs caused by ``EINTR``.

The expectation in this use case is that Python hides the
``InterruptedError`` and retries system calls automatically.


Use Case 2: Be notified of signals as soon as possible
------------------------------------------------------

Sometimes yet, you expect some signals and you want to handle them as
soon as possible.  For example, you may want to immediately quit a
program using the ``CTRL+c`` keyboard shortcut.

Besides, some signals are not interesting and should not disrupt the
application.  There are two options to interrupt an application on
only *some* signals:

* Set up a custom signal handler which raises an exception, such as
  ``KeyboardInterrupt`` for ``SIGINT``.
* Use a I/O multiplexing function like ``select()`` together with Python's
  signal wakeup file descriptor: see the function ``signal.set_wakeup_fd()``.

The expectation in this use case is for the Python signal handler to be
executed timely, and the system call to fail if the handler raised an
exception -- otherwise restart.


Proposal
========

This PEP proposes to handle EINTR and retries at the lowest level, i.e.
in the wrappers provided by the stdlib (as opposed to higher-level
libraries and applications).

Specifically, when a system call fails with ``EINTR``, its Python wrapper
must call the given signal handler (using ``PyErr_CheckSignals()``).
If the signal handler raises an exception, the Python wrapper bails out
and fails with the exception.

If the signal handler returns successfully, the Python wrapper retries the
system call automatically.  If the system call involves a timeout parameter,
the timeout is recomputed.

Modified functions
------------------

Example of standard library functions that need to be modified to comply
with this PEP:

* ``open()``, ``os.open()``, ``io.open()``
* functions of the ``faulthandler`` module
* ``os`` functions:

  - ``os.fchdir()``
  - ``os.fchmod()``
  - ``os.fchown()``
  - ``os.fdatasync()``
  - ``os.fstat()``
  - ``os.fstatvfs()``
  - ``os.fsync()``
  - ``os.ftruncate()``
  - ``os.mkfifo()``
  - ``os.mknod()``
  - ``os.posix_fadvise()``
  - ``os.posix_fallocate()``
  - ``os.pread()``
  - ``os.pwrite()``
  - ``os.read()``
  - ``os.readv()``
  - ``os.sendfile()``
  - ``os.wait3()``
  - ``os.wait4()``
  - ``os.wait()``
  - ``os.waitid()``
  - ``os.waitpid()``
  - ``os.write()``
  - ``os.writev()``
  - special cases: ``os.close()`` and ``os.dup2()`` now ignore ``EINTR`` error,
    the syscall is not retried

* ``select.select()``, ``select.poll.poll()``, ``select.epoll.poll()``,
  ``select.kqueue.control()``, ``select.devpoll.poll()``
* ``socket.socket()`` methods:

  - ``accept()``
  - ``connect()`` (except for non-blocking sockets)
  - ``recv()``
  - ``recvfrom()``
  - ``recvmsg()``
  - ``send()``
  - ``sendall()``
  - ``sendmsg()``
  - ``sendto()``

* ``signal.sigtimedwait()``, ``signal.sigwaitinfo()``
* ``time.sleep()``

(Note: the ``selector`` module already retries on ``InterruptedError``, but it
doesn't recompute the timeout yet)

``os.close``, ``close()`` methods and ``os.dup2()`` are a special case: they
will ignore ``EINTR`` instead of retrying.  The reason is complex but involves
behaviour under Linux and the fact that the file descriptor may really be
closed even if EINTR is returned. See articles:

* `Returning EINTR from close() <http://lwn.net/Articles/576478/>`_
* `(LKML) Re: [patch 7/7] uml: retry host close() on EINTR
  <http://linux.derkeiler.com/Mailing-Lists/Kernel/2005-09/3000.html>`_
* `close() and EINTR <http://alobbs.com/post/54503240599/close-and-eintr>`_

The ``socket.socket.connect()`` method does not retry ``connect()`` for
non-blocking sockets if it is interrupted by a signal (fails with ``EINTR``).
The connection runs asynchronously in background. The caller is responsible
to wait until the socket becomes writable (ex: using ``select.select()`


id: pep-0461.rst
content:
PEP: 461
Title: Adding % formatting to bytes and bytearray
Author: Ethan Furman <ethan@stoneleaf.us>
Status: Final
Type: Standards Track
Created: 13-Jan-2014
Python-Version: 3.5
Post-History: 14-Jan-2014, 15-Jan-2014, 17-Jan-2014, 22-Feb-2014, 25-Mar-2014,
              27-Mar-2014
Resolution: https://mail.python.org/pipermail/python-dev/2014-March/133621.html


Abstract
========

This PEP proposes adding % formatting operations similar to Python 2's ``str``
type to ``bytes`` and ``bytearray`` [1]_ [2]_.


Rationale
=========

While interpolation is usually thought of as a string operation, there are
cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and the
work needed to make up for this missing functionality detracts from the overall
readability of the code.


Motivation
==========

With Python 3 and the split between ``str`` and ``bytes``, one small but
important area of programming became slightly more difficult, and much more
painful -- wire format protocols [3]_.

This area of programming is characterized by a mixture of binary data and
ASCII compatible segments of text (aka ASCII-encoded text).  Bringing back a
restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in
writing new wire format code, and in porting Python 2 wire format code.

Common use-cases include ``dbf`` and ``pdf`` file formats, ``email``
formats, and ``FTP`` and ``HTTP`` communications, among many others.


Proposed semantics for ``bytes`` and ``bytearray`` formatting
=============================================================

%-interpolation
---------------

All the numeric formatting codes (``d``, ``i``, ``o``, ``u``, ``x``, ``X``,
``e``, ``E``, ``f``, ``F``, ``g``, ``G``, and any that are subsequently added
to Python 3) will be supported, and will work as they do for str, including
the padding, justification and other related modifiers (currently ``#``, ``0``,
``-``, space, and ``+`` (plus any added to Python 3)).  The only
non-numeric codes allowed are ``c``, ``b``, ``a``, and ``s`` (which is a
synonym for b).

For the numeric codes, the only difference between ``str`` and ``bytes`` (or
``bytearray``) interpolation is that the results from these codes will be
ASCII-encoded text, not unicode.  In other words, for any numeric formatting
code ``%x``::

   b"%x" % val

is equivalent to::

   ("%x" % val).encode("ascii")

Examples::

   >>> b'%4x' % 10
   b'   a'

   >>> b'%#4x' % 10
   ' 0xa'

   >>> b'%04X' % 10
   '000A'

``%c`` will insert a single byte, either from an ``int`` in range(256), or from
a ``bytes`` argument of length 1, not from a ``str``.

Examples::

    >>> b'%c' % 48
    b'0'

    >>> b'%c' % b'a'
    b'a'

``%b`` will insert a series of bytes.  These bytes are collected in one of two
ways:

- input type supports ``Py_buffer`` [4]_?
  use it to collect the necessary bytes

- input type is something else?
  use its ``__bytes__`` method [5]_ ; if there isn't one, raise a ``TypeError``

In particular, ``%b`` will not accept numbers nor ``str``.  ``str`` is rejected
as the string to bytes conversion requires an encoding, and we are refusing to
guess; numbers are rejected because:

- what makes a number is fuzzy (float? Decimal? Fraction? some user type?)

- allowing numbers would lead to ambiguity between numbers and textual
  representations of numbers (3.14 vs '3.14')

- given the nature of wire formats, explicit is definitely better than implicit

``%s`` is included as a synonym for ``%b`` for the sole purpose of making 2/3 code
bases easier to maintain.  Python 3 only code should use ``%b``.

Examples::

    >>> b'%b' % b'abc'
    b'abc'

    >>> b'%b' % 'some string'.encode('utf8')
    b'some string'

    >>> b'%b' % 3.14
    Traceback (most recent call last):
    ...
    TypeError: b'%b' does not accept 'float'

    >>> b'%b' % 'hello world!'
    Traceback (most recent call last):
    ...
    TypeError: b'%b' does not accept 'str'


``%a`` will give the equivalent of
``repr(some_obj).encode('ascii', 'backslashreplace')`` on the interpolated
value.  Use cases include developing a new protocol and writing landmarks
into the stream; debugging data going into an existing protocol to see if
the problem is the protocol itself or bad data; a fall-back for a serialization
format; or any situation where defining ``__bytes__`` would not be appropriate
but a readable/informative representation is needed [6]_.

``%r`` is included as a synonym for ``%a`` for the sole purpose of making 2/3
code bases easier to maintain.  Python 3 only code use ``%a`` [7]_.

Examples::

    >>> b'%a' % 3.14
    b'3.14'

    >>> b'%a' % b'abc'
    b"b'abc'"

    >>> b'%a' % 'def'
    b"'def'"



Compatibility with Python 2
===========================

As noted above, ``%s`` and ``%r`` are being included solely to help ease
migration from, and/or have a single code base with, Python 2.  This is
important as there are modules both in the wild and behind closed doors that
currently use the Python 2 ``str`` type as a ``bytes`` container, and hence
are using ``%s`` as a bytes interpolator.

However, ``%b`` and ``%a`` should be used in new, Python 3 only code, so ``%s``
and ``%r`` will immediately be deprecated, but not removed from the 3.x series
[7]_.

Proposed variations
===================

It has been proposed to automatically use ``.encode('ascii','strict')`` for
``str`` arguments to ``%b``.

- Rejected as this would lead to intermittent failures.  Better to have the
  operation always fail so the trouble-spot can be correctly fixed.

It has been proposed to have ``%b`` return the ascii-encoded repr when the
value is a ``str`` (b'%b' % 'abc'  --> b"'abc'").

- Rejected as this would lead to hard to debug failures far from the problem
  site.  Better to have the operation always fail so the trouble-spot can be
  easily fixed.

Originally this PEP also proposed adding format-style formatting, but it was
decided that format and its related machinery were all strictly text (aka
``str``) based, and it was dropped.

Various new special methods were proposed, such as ``__ascii__``,
``__format_bytes__``, etc.; such methods are not needed at this time, but can
be visited again later if real-world use shows deficiencies with this solution.

A competing PEP, :pep:`PEP 460 Add binary interpolation and formatting <460>`,
also exists.


Objections
==========

The objections raised against this PEP were mainly variations on two themes:

- the ``bytes`` and ``bytearray`` types are for pure binary data, with no
  assumptions about encodings

- offering %-interpolation that assumes an ASCII encoding will be an
  attractive nuisance and lead us back to the problems of the Python 2
  ``str``/``unicode`` text model

As was seen during the discussion, ``bytes`` and ``bytearray`` are also used
for mixed binary data and ASCII-compatible segments: file formats such as
``dbf`` and ``pdf``, network protocols such as ``ftp`` and ``email``, etc.

``bytes`` and ``bytearray`` already have several methods which assume an ASCII
compatible encoding.  ``upper()``, ``isalpha()``, and ``expandtabs()`` to name
just a few.  %-interpolation, with its very restricted mini-language, will not
be any more of a nuisance than the already existing methods.

Some have objected to allowing the full range of numeric formatting codes with
the claim that decimal alone would be sufficient.  However, at least two
formats (dbf and pdf) make use of non-decimal numbers.


Footnotes
=========

.. [1] http://docs.python.org/2/library/stdtypes.html#string-formatting
.. [2] neither string.Template, format, nor str.format are under consideration
.. [3] https://mail.python.org/pipermail/python-dev/2014-January/131518.html
.. [4] http://docs.python.org/3/c-api/buffer.html
       examples:  ``memoryview``, ``array.array``, ``bytearray``, ``bytes``
.. [5] http://docs.python.org/3/reference/datamodel.html#object.__bytes__
.. [6] https://mail.python.org/pipermail/python-dev/2014-February/132750.html
.. [7] htt


id: pep-0307.rst
content:
PEP: 307
Title: Extensions to the pickle protocol
Author: Guido van Rossum, Tim Peters
Status: Final
Type: Standards Track
Created: 31-Jan-2003
Python-Version: 2.3
Post-History: 07-Feb-2003

Introduction
============

Pickling new-style objects in Python 2.2 is done somewhat clumsily
and causes pickle size to bloat compared to classic class
instances.  This PEP documents a new pickle protocol in Python 2.3
that takes care of this and many other pickle issues.

There are two sides to specifying a new pickle protocol: the byte
stream constituting pickled data must be specified, and the
interface between objects and the pickling and unpickling engines
must be specified.  This PEP focuses on API issues, although it
may occasionally touch on byte stream format details to motivate a
choice.  The pickle byte stream format is documented formally by
the standard library module ``pickletools.py`` (already checked into
CVS for Python 2.3).

This PEP attempts to fully document the interface between pickled
objects and the pickling process, highlighting additions by
specifying "new in this PEP".  (The interface to invoke pickling
or unpickling is not covered fully, except for the changes to the
API for specifying the pickling protocol to picklers.)


Motivation
==========

Pickling new-style objects causes serious pickle bloat.  For
example::

   class C(object): # Omit "(object)" for classic class
       pass
   x = C()
   x.foo = 42
   print len(pickle.dumps(x, 1))

The binary pickle for the classic object consumed 33 bytes, and for
the new-style object 86 bytes.

The reasons for the bloat are complex, but are mostly caused by
the fact that new-style objects use ``__reduce__`` in order to be
picklable at all.  After ample consideration we've concluded that
the only way to reduce pickle sizes for new-style objects is to
add new opcodes to the pickle protocol.  The net result is that
with the new protocol, the pickle size in the above example is 35
(two extra bytes are used at the start to indicate the protocol
version, although this isn't strictly necessary).


Protocol versions
=================

Previously, pickling (but not unpickling) distinguished between
text mode and binary mode.  By design, binary mode is a
superset of text mode, and unpicklers don't need to know in
advance whether an incoming pickle uses text mode or binary mode.
The virtual machine used for unpickling is the same regardless of
the mode; certain opcodes simply aren't used in text mode.

Retroactively, text mode is now called protocol 0, and binary mode
protocol 1.  The new protocol is called protocol 2.  In the
tradition of pickling protocols, protocol 2 is a superset of
protocol 1.  But just so that future pickling protocols aren't
required to be supersets of the oldest protocols, a new opcode is
inserted at the start of a protocol 2 pickle indicating that it is
using protocol 2.  To date, each release of Python has been able to
read pickles written by all previous releases.  Of course pickles
written under protocol *N* can't be read by versions of Python
earlier than the one that introduced protocol *N*.

Several functions, methods and constructors used for pickling used
to take a positional argument named 'bin' which was a flag,
defaulting to 0, indicating binary mode.  This argument is renamed
to 'protocol' and now gives the protocol number, still defaulting
to 0.

It so happens that passing 2 for the 'bin' argument in previous
Python versions had the same effect as passing 1.  Nevertheless, a
special case is added here:  passing a negative number selects the
highest protocol version supported by a particular implementation.
This works in previous Python versions, too, and so can be used to
select the highest protocol available in a way that's both backward
and forward compatible.  In addition, a new module constant
``HIGHEST_PROTOCOL`` is supplied by both ``pickle`` and ``cPickle``, equal to
the highest protocol number the module can read.  This is cleaner
than passing -1, but cannot be used before Python 2.3.

The ``pickle.py`` module has supported passing the 'bin' value as a
keyword argument rather than a positional argument.  (This is not
recommended, since ``cPickle`` only accepts positional arguments, but
it works...)  Passing 'bin' as a keyword argument is deprecated,
and a ``PendingDeprecationWarning`` is issued in this case.  You have
to invoke the Python interpreter with ``-Wa`` or a variation on that
to see ``PendingDeprecationWarning`` messages.  In Python 2.4, the
warning class may be upgraded to ``DeprecationWarning``.


Security issues
===============

In previous versions of Python, unpickling would do a "safety
check" on certain operations, refusing to call functions or
constructors that weren't marked as "safe for unpickling" by
either having an attribute ``__safe_for_unpickling__`` set to 1, or by
being registered in a global registry, ``copy_reg.safe_constructors``.

This feature gives a false sense of security: nobody has ever done
the necessary, extensive, code audit to prove that unpickling
untrusted pickles cannot invoke unwanted code, and in fact bugs in
the Python 2.2 ``pickle.py`` module make it easy to circumvent these
security measures.

We firmly believe that, on the Internet, it is better to know that
you are using an insecure protocol than to trust a protocol to be
secure whose implementation hasn't been thoroughly checked.  Even
high quality implementations of widely used protocols are
routinely found flawed; Python's pickle implementation simply
cannot make such guarantees without a much larger time investment.
Therefore, as of Python 2.3, all safety checks on unpickling are
officially removed, and replaced with this warning:

.. warning::

   Do not unpickle data received from an untrusted or
   unauthenticated source.

The same warning applies to previous Python versions, despite the
presence of safety checks there.


Extended ``__reduce__`` API
===========================

There are several APIs that a class can use to control pickling.
Perhaps the most popular of these are ``__getstate__`` and
``__setstate__``; but the most powerful one is ``__reduce__``.  (There's
also ``__getinitargs__``, and we're adding ``__getnewargs__`` below.)

There are several ways to provide ``__reduce__`` functionality: a
class can implement a ``__reduce__`` method or a ``__reduce_ex__`` method
(see next section), or a reduce function can be declared in
``copy_reg`` (``copy_reg.dispatch_table`` maps classes to functions).  The
return values are interpreted exactly the same, though, and we'll
refer to these collectively as ``__reduce__``.

**Important:** pickling of classic class instances does not look for a
``__reduce__`` or ``__reduce_ex__`` method or a reduce function in the
``copy_reg`` dispatch table, so that a classic class cannot provide
``__reduce__`` functionality in the sense intended here.  A classic
class must use ``__getinitargs__`` and/or ``__getstate__`` to customize
pickling.  These are described below.

``__reduce__`` must return either a string or a tuple.  If it returns
a string, this is an object whose state is not to be pickled, but
instead a reference to an equivalent object referenced by name.
Surprisingly, the string returned by ``__reduce__`` should be the
object's local name (relative to its module); the ``pickle`` module
searches the module namespace to determine the object's module.

The rest of this section is concerned with the tuple returned by
``__reduce__``.  It is a variable size tuple, of length 2 through 5.
The first two items (function and arguments) are required.  The
remaining items are optional and may be left off from the end;
giving ``None`` for the value of an optional item acts the same as
leaving it off.  The last two items are new in this PEP.  The items
are, in order:

+-----------+---------------------------------------------------------------+
| function  | Required.                                                   


id: pep-0688.rst
content:
PEP: 688
Title: Making the buffer protocol accessible in Python
Author: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://discuss.python.org/t/19756
Status: Final
Type: Standards Track
Topic: Typing
Created: 23-Apr-2022
Python-Version: 3.12
Post-History: `23-Apr-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/CX7GPSIYQEL23RXMYL66GAKGP4RLUD7P/>`__,
              `25-Apr-2022 <https://discuss.python.org/t/15265>`__,
              `06-Oct-2022 <https://discuss.python.org/t/19756>`__,
              `26-Oct-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/XH5ZK2MSZIQLL62PYZ6I5532SQKKVCBL/>`__
Resolution: `07-Mar-2023 <https://discuss.python.org/t/pep-688-making-the-buffer-protocol-accessible-in-python/15265/35>`__

.. canonical-doc:: :ref:`python:python-buffer-protocol`

Abstract
========

This PEP proposes a Python-level API for the buffer protocol,
which is currently accessible only to C code. This allows type
checkers to evaluate whether objects implement the protocol.


Motivation
==========

The CPython C API provides a versatile mechanism for accessing the
underlying memory of an objectâthe `buffer protocol <https://docs.python.org/3/c-api/buffer.html>`__
introduced in :pep:`3118`.
Functions that accept binary data are usually written to handle any
object implementing the buffer protocol. For example, at the time of writing,
there are around 130 functions in CPython using the Argument Clinic
``Py_buffer`` type, which accepts the buffer protocol.

Currently, there is no way for Python code to inspect whether an object
supports the buffer protocol. Moreover, the static type system
does not provide a type annotation to represent the protocol.
This is a `common problem <https://github.com/python/typing/issues/593>`__
when writing type annotations for code that accepts generic buffers.

Similarly, it is impossible for a class written in Python to support
the buffer protocol. A buffer class in
Python would give users the ability to easily wrap a C buffer object, or to test
the behavior of an API that consumes the buffer protocol. Granted, this is not
a particularly common need. However, there has been a
`CPython feature request <https://github.com/python/cpython/issues/58006>`__
for supporting buffer classes written in Python that has been open since 2012.


Rationale
=========

Current options
---------------

There are two known workarounds for annotating buffer types in
the type system, but neither is adequate.

First, the `current workaround <https://github.com/python/typeshed/blob/2a0fc1b582ef84f7a82c0beb39fa617de2539d3d/stdlib/_typeshed/__init__.pyi#L194>`__
for buffer types in typeshed is a type alias
that lists well-known buffer types in the standard library, such as
``bytes``, ``bytearray``, ``memoryview``, and ``array.array``. This
approach works for the standard library, but it does not extend to
third-party buffer types.

Second, the `documentation <https://docs.python.org/3.10/library/typing.html#typing.ByteString>`__
for ``typing.ByteString`` currently states:

   This type represents the types ``bytes``, ``bytearray``, and
   ``memoryview`` of byte sequences.

   As a shorthand for this type, ``bytes`` can be used to annotate
   arguments of any of the types mentioned above.

Although this sentence has been in the documentation
`since 2015 <https://github.com/python/cpython/commit/2a19d956ab92fc9084a105cc11292cb0438b322f>`__,
the use of ``bytes`` to include these other types is not specified
in any of the typing PEPs. Furthermore, this mechanism has a number of
problems. It does not include all possible buffer types, and it
makes the ``bytes`` type ambiguous in type annotations. After all,
there are many operations that are valid on ``bytes`` objects, but
not on ``memoryview`` objects, and it is perfectly possible for
a function to accept ``bytes`` but not ``memoryview`` objects.
A mypy user
`reports <https://github.com/python/mypy/issues/12643#issuecomment-1105914159>`__
that this shortcut has caused significant problems for the ``psycopg`` project.

Kinds of buffers
----------------

The C buffer protocol supports
`many options <https://docs.python.org/3.10/c-api/buffer.html#buffer-request-types>`__,
affecting strides, contiguity, and support for writing to the buffer. Some of these
options would be useful in the type system. For example, typeshed
currently provides separate type aliases for writable and read-only
buffers.

However, in the C buffer protocol, most of these options cannot be
queried directly on the type object. The only way to figure out
whether an object supports a particular flag is to actually
ask for the buffer. For some types, such as ``memoryview``,
the supported flags depend on the instance. As a result, it would
be difficult to represent support for these flags in the type system.


Specification
=============

Python-level buffer protocol
----------------------------

We propose to add two Python-level special methods, ``__buffer__``
and ``__release_buffer__``. Python
classes that implement these methods are usable as buffers from C
code. Conversely, classes implemented in C that support the
buffer protocol acquire synthesized methods accessible from Python
code.

The ``__buffer__`` method is called to create a buffer from a Python
object, for example by the ``memoryview()`` constructor.
It corresponds to the ``bf_getbuffer`` C slot.
The Python signature for this method is
``def __buffer__(self, flags: int, /) -> memoryview: ...``. The method
must return a ``memoryview`` object. If the ``bf_getbuffer`` slot
is invoked on a Python class with a ``__buffer__`` method,
the interpreter extracts the underlying ``Py_buffer`` from the
``memoryview`` returned by the method
and returns it to the C caller. Similarly, if Python code calls the
``__buffer__`` method on an instance of a C class that
implements ``bf_getbuffer``, the returned buffer is wrapped in a
``memoryview`` for consumption by Python code.

The ``__release_buffer__`` method should be called when a caller no
longer needs the buffer returned by ``__buffer__``. It corresponds to the
``bf_releasebuffer`` C slot. This is an
optional part of the buffer protocol.
The Python signature for this method is
``def __release_buffer__(self, buffer: memoryview, /) -> None: ...``.
The buffer to be released is wrapped in a ``memoryview``. When this
method is invoked through CPython's buffer API (for example, through
calling ``memoryview.release`` on a ``memoryview`` returned by
``__buffer__``), the passed ``memoryview`` is the same object
as was returned by ``__buffer__``. It is
also possible to call ``__release_buffer__`` on a C class that
implements ``bf_releasebuffer``.

If ``__release_buffer__`` exists on an object,
Python code that calls ``__buffer__`` directly on the object must
call ``__release_buffer__`` on the same object when it is done
with the buffer. Otherwise, resources used by the object may
not be reclaimed. Similarly, it is a programming error
to call ``__release_buffer__`` without a previous call to
``__buffer__``, or to call it multiple times for a single call
to ``__buffer__``. For objects that implement the C buffer protocol,
calls to ``__release_buffer__`` where the argument is not a
``memoryview`` wrapping the same object will raise an exception.
After a valid call to ``__release_buffer__``, the ``memoryview``
is invalidated (as if its ``release()`` method had been called),
and any subsequent calls to ``__release_buffer__`` with the same
``memoryview`` will raise an exception.
The interpreter will ensure that misuse
of the Python API will not break invariants at the C level -- for
example, it will not cause memory safety violations.

``inspect.BufferFlags``
-----------------------

To help implementations of ``__buffer__``, we add ``inspect.BufferFlags``,
a subclass of ``enum.IntFlag``. This enum contains all flags defined in the
C buffer protocol. For example, ``inspect.BufferFlags.SIMP


id: pep-0677.rst
content:
PEP: 677
Title: Callable Type Syntax
Author: Steven Troxler <steven.troxler@gmail.com>,
        Pradeep Kumar Srinivasan <gohanpra@gmail.com>
Sponsor: Guido van Rossum <guido at python.org>
Discussions-To: python-dev@python.org
Status: Rejected
Type: Standards Track
Topic: Typing
Created: 13-Dec-2021
Python-Version: 3.11
Post-History: 16-Dec-2021
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/NHCLHCU2XCWTBGF732WESMN42YYVKOXB/

Abstract
========

This PEP introduces a concise and friendly syntax for callable types,
supporting the same functionality as ``typing.Callable`` but with an
arrow syntax inspired by the syntax for typed function
signatures. This allows types like ``Callable[[int, str], bool]`` to
be written as ``(int, str) -> bool``.

The proposed syntax supports all the functionality provided by
``typing.Callable`` and ``typing.Concatenate``, and is intended to
work as a drop-in replacement.


Motivation
==========

One way to make code safer and easier to analyze is by making sure
that functions and classes are well-typed.  In Python we have type
annotations, the framework for which is defined in :pep:`484`, to provide
type hints that can find bugs as well as helping with editor tooling
like tab completion, static analysis tooling, and code review.

Consider the following untyped code::

    def flat_map(func, l):
        out = []
        for element in l:
            out.extend(func(element))
        return out


    def wrap(x: int) -> list[int]:
        return [x]

    def add(x: int, y: int) -> int:
        return x + y

    flat_map(wrap, [1, 2, 3])  # no runtime error, output is [1, 2, 3]
    flat_map(add, [1, 2, 3])   # runtime error: `add` expects 2 arguments, got 1


We can add types to this example to detect the runtime error::

    from typing import Callable

    def flat_map(
        func: Callable[[int], list[int]],
        l: list[int]
    ) -> list[int]:
        ....

    ...


    flat_map(wrap, [1, 2, 3])  # type checks okay, output is [1, 2, 3]
    flat_map(add, [1, 2, 3])   # type check error

There are a few usability challenges with ``Callable`` we can see here:

- It is verbose, particularly for more complex function signatures.
- It relies on two levels of nested brackets, unlike any other generic
  type. This can be especially hard to read when some of the type
  parameters are themselves generic types.
- The bracket structure is not visually similar to how function signatures
  are written.
- It requires an explicit import, unlike many of the other most common
  types like ``list`` and ``dict``.

Possibly as a result, `programmers often fail to write complete
Callable types
<https://github.com/pradeep90/annotation_collector#typed-projects---callable-type>`_.
Such untyped or partially-typed callable types do not check the
parameter types or return types of the given callable and thus negate
the benefits of static typing. For example, they might write this::


    from typing import Callable

    def flat_map(
        func: Callable[..., Any],
        l: list[int]
    ) -> list[int]:
        ....

    ...


    flat_map(add, [1, 2, 3])  # oops, no type check error!

There's some partial type information here - we at least know that ``func``
needs to be callable. But we've dropped too much type information for
type checkers to find the bug.

With our proposal, the example looks like this::

    def flat_map(
        func: (int) -> list[int],
        l: list[int]
    ) -> list[int]:
        out = []
        for element in l:
            out.extend(f(element))
        return out

    ...

The type ``(int) -> list[int]`` is more concise, uses an arrow similar
to the one indicating a return type in a function header, avoids
nested brackets, and does not require an import.


Rationale
=========

The ``Callable`` type is widely used. For example, `as of October 2021
it was
<https://github.com/pradeep90/annotation_collector#overall-stats-in-typeshed>`_
the fifth most common complex type in typeshed, after ``Optional``,
``Tuple``, ``Union``, and ``List``.

The others have had their syntax improved and the need for imports
eliminated by either :pep:`604` or :pep:`585`:

- ``typing.Optional[int]`` is written ``int | None``
- ``typing.Union[int, str]`` is written ``int | str``
- ``typing.List[int]`` is written ``list[int]``
- ``typing.Tuple[int, str]`` is written ``tuple[int, str]``

The ``typing.Callable`` type is used almost as often as these other
types, is more complicated to read and write, and still requires an
import and bracket-based syntax.

In this proposal, we chose to support all the existing semantics of
``typing.Callable``, without adding support for new features. We made
this decision after examining how frequently each feature might be
used in existing typed and untyped open-source code. We determined
that the vast majority of use cases are covered.

We considered adding support for named, optional, and variadic
arguments. However, we decided against including these features, as
our analysis showed they are infrequently used. When they are really
needed, it is possible to type these using `callback protocols
<https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols>`_.

An Arrow Syntax for Callable Types
----------------------------------

We are proposing a succinct, easy-to-use syntax for
``typing.Callable`` that looks similar to function headers in Python.
Our proposal closely follows syntax used by several popular languages
such as `Typescript
<https://basarat.gitbook.io/typescript/type-system/callable#arrow-syntax>`_,
`Kotlin <https://kotlinlang.org/docs/lambdas.html>`_, and `Scala
<https://docs.scala-lang.org/tour/higher-order-functions.html>`_.

Our goals are that:

- Callable types using this syntax will be easier to learn and use,
  particularly for developers with experience in other languages.
- Library authors will be more likely to use expressive types for
  callables that enable type checkers to better understand code and
  find bugs, as in the ``decorator`` example above.

Consider this simplified real-world example from a web server, written
using the existing ``typing.Callable``::

    from typing import Awaitable, Callable
    from app_logic import Response, UserSetting


    def customize_response(
        response: Response,
        customizer: Callable[[Response, list[UserSetting]], Awaitable[Response]]
    ) -> Response:
       ...

With our proposal, this code can be abbreviated to::

    from app_logic import Response, UserSetting

    def customize_response(
        response: Response,
        customizer: async (Response, list[UserSetting]) -> Response,
    ) -> Response:
        ...

This is shorter and requires fewer imports. It also has far less
nesting of square brackets - only one level, as opposed to three in
the original code.

Compact Syntax for ``ParamSpec``
--------------------------------

A particularly common case where library authors leave off type information
for callables is when defining decorators. Consider the following::


    from typing import Any, Callable

    def with_retries(
        f: Callable[..., Any]
    ) -> Callable[..., Any]:
        def wrapper(retry_once, *args, **kwargs):
            if retry_once:
                try: return f(*args, **kwargs)
                except Exception: pass
            return f(*args, **kwargs)
        return wrapper

    @with_retries
    def f(x: int) -> int:
        return x


    f(y=10)  # oops - no type error!

In the code above, it is clear that the decorator should produce a
function whose signature is like that of the argument ``f`` other
than an additional ``retry_once`` argument. But the use of ``...``
prevents a type checker from seeing this and alerting a user that
``f(y=10)`` is invalid.


With :pep:`612` it is possible to type decorators like this correctly
as follows::

    from typing import Any, Callable, Concatenate, ParamSpec, TypeVar

    R = TypeVar("R")
 


id: pep-0663.rst
content:
PEP: 663
Title: Standardizing Enum str(), repr(), and format() behaviors
Author: Ethan Furman <ethan@stoneleaf.us>
Discussions-To: python-dev@python.org
Status: Rejected
Type: Informational
Created: 30-Jun-2021
Python-Version: 3.11
Post-History: 20-Jul-2021, 02-Nov-2021
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/RN3WCRZSTQR55DOHJTZ2KIO6CZPJPCU7/


Abstract
========

Update the ``repr()``, ``str()``, and ``format()`` of the various Enum types
to better match their intended purpose.  For example, ``IntEnum`` will have
its ``str()`` change to match its ``format()``, while a user-mixed int-enum
will have its ``format()`` match its ``str()``.  In all cases, an enum's
``str()`` and ``format()`` will be the same (unless the user overrides
``format()``).

Add a global enum decorator which changes the ``str()`` and ``repr()``  (and
``format()``) of the decorated enum to be a valid global reference: i.e.
``re.IGNORECASE`` instead of ``<RegexFlag.IGNORECASE: 2>``.


Motivation
==========

Having the ``str()`` of ``IntEnum`` and ``IntFlag`` not be the value causes
bugs and extra work when replacing existing constants.

Having the ``str()`` and ``format()`` of an enum member be different can be
confusing.

The addition of ``StrEnum`` with its requirement to have its ``str()`` be its
``value`` is inconsistent with other provided Enum's ``str``.

The iteration of ``Flag`` members, which directly affects their ``repr()``, is
inelegant at best, and buggy at worst.


Rationale
=========

Enums are becoming more common in the standard library; being able to recognize
enum members by their ``repr()``, and having that ``repr()`` be easy to parse, is
useful and can save time and effort in understanding and debugging code.

However, the enums with mixed-in data types (``IntEnum``, ``IntFlag``, and the new
``StrEnum``) need to be more backwards compatible with the constants they are
replacing -- specifically, ``str(replacement_enum_member) == str(original_constant)``
should be true (and the same for ``format()``).

IntEnum, IntFlag, and StrEnum should be as close to a drop-in replacement of
existing integer and string constants as is possible.  Towards that goal, the
``str()`` output of each should be its inherent value; e.g. if ``Color`` is an
``IntEnum``::

    >>> Color.RED
    <Color.RED: 1>
    >>> str(Color.RED)
    '1'
    >>> format(Color.RED)
    '1'

Note that ``format()`` already produces the correct output, only ``str()`` needs
updating.

As much as possible, the ``str()``, ``repr()``, and ``format()`` of enum members
should be standardized across the standard library.  However, up to Python 3.10
several enums in the standard library have a custom ``str()`` and/or ``repr()``.

The ``repr()`` of Flag currently includes aliases, which it should not; fixing that
will, of course, already change its ``repr()`` in certain cases.


Specification
=============

There are three broad categories of enum usage:

- simple: ``Enum`` or ``Flag``
  a new enum class is created with no data type mixins

- drop-in replacement: ``IntEnum``, ``IntFlag``, ``StrEnum``
  a new enum class is created which also subclasses ``int`` or ``str`` and uses
  ``int.__str__`` or ``str.__str__``

- user-mixed enums and flags
  the user creates their own integer-, float-, str-, whatever-enums instead of
  using enum.IntEnum, etc.

There are also two styles:

- normal: the enumeration members remain in their classes and are accessed as
  ``classname.membername``, and the class name shows in their ``repr()`` and
  ``str()`` (where appropriate)

- global: the enumeration members are copied into their module's global
  namespace, and their module name shows in their ``repr()`` and ``str()``
  (where appropriate)

Some sample enums::

    # module: tools.py

    class Hue(Enum):  # or IntEnum
        LIGHT = -1
        NORMAL = 0
        DARK = +1

    class Color(Flag):  # or IntFlag
        RED = 1
        GREEN = 2
        BLUE = 4

    class Grey(int, Enum):  # or (int, Flag)
       BLACK = 0
       WHITE = 1

Using the above enumerations, the following two tables show the old and new
output (blank cells indicate no change):

+--------+------------------------+-----------------+------------+-----------------------+
| style  | category               | enum repr()     | enum str() | enum format()         |
+--------+-------------+----------+-----------------+------------+-----------------------+
| normal | simple      | 3.10     |                 |            |                       |
|        |             +----------+-----------------+------------+-----------------------+
|        |             | new      |                 |            |                       |
|        +-------------+----------+-----------------+------------+-----------------------+
|        | user mixed  | 3.10     |                 |            | 1                     |
|        |             +----------+-----------------+------------+-----------------------+
|        |             | new      |                 |            | Grey.WHITE            |
|        +-------------+----------+-----------------+------------+-----------------------+
|        | int drop-in | 3.10     |                 | Hue.LIGHT  |                       |
|        |             +----------+-----------------+------------+-----------------------+
|        |             | new      |                 | -1         |                       |
+--------+-------------+----------+-----------------+------------+-----------------------+
| global | simple      | 3.10     | <Hue.LIGHT: -1> | Hue.LIGHT  | Hue.LIGHT             |
|        |             +----------+-----------------+------------+-----------------------+
|        |             | new      | tools.LIGHT     | LIGHT      | LIGHT                 |
|        +-------------+----------+-----------------+------------+-----------------------+
|        | user mixed  | 3.10     | <Grey.WHITE: 1  | Grey.WHITE | Grey.WHITE            |
|        |             +----------+-----------------+------------+-----------------------+
|        |             | new      | tools.WHITE     | WHITE      | WHITE                 |
|        +-------------+----------+-----------------+------------+-----------------------+
|        | int drop-in | 3.10     | <Hue.LIGHT: -1> | Hue.LIGHT  |                       |
|        |             +----------+-----------------+------------+-----------------------+
|        |             | new      | tools.LIGHT     | -1         |                       |
+--------+-------------+----------+-----------------+------------+-----------------------+

+--------+------------------------+-----------------------+------------------------+-----------------------+
| style  | category               | flag repr()           | flag str()             | flag format()         |
+--------+-------------+----------+-----------------------+------------------------+-----------------------+
| normal | simple      | 3.10     | <Color.RED|GREEN: 3>  | Color.RED|GREEN        | Color.RED|GREEN       |
|        |             +----------+-----------------------+------------------------+-----------------------+
|        |             | new      | <Color(3): RED|GREEN> | Color.RED|Color.GREEN  | Color.RED|Color.GREEN |
|        +-------------+----------+-----------------------+------------------------+-----------------------+
|        | user mixed  | 3.10     | <Grey.WHITE: 1>       |                        | 1                     |
|        |             +----------+-----------------------+------------------------+-----------------------+
|        |             | new      | <Grey(1): WHITE>      |                        | Grey.WHITE            |
|        +-------------+----------+-----------------------+------------------------+-----------------------+
|        | int drop-in | 3.10     | <Color.RED|GREEN: 3>  | Color.RED|GREEN        |                       |
|        |             +----------+-------


id: pep-0703.rst
content:
PEP: 703
Title: Making the Global Interpreter Lock Optional in CPython
Author: Sam Gross <colesbury at gmail.com>
Sponsor: Åukasz Langa <lukasz at python.org>
Discussions-To: https://discuss.python.org/t/22606
Status: Accepted
Type: Standards Track
Created: 09-Jan-2023
Python-Version: 3.13
Post-History: `09-Jan-2023 <https://discuss.python.org/t/22606>`__,
              `04-May-2023 <https://discuss.python.org/t/26503>`__
Resolution: `24-Oct-2023 <https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-in-cpython-acceptance/37075>`__

.. note::
   The Steering Council accepts PEP 703, but with clear proviso: that
   the rollout be gradual and break as little as possible, and that we can roll
   back any changes that turn out to be too disruptive â which includes
   potentially rolling back all of PEP 703 entirely if necessary
   (however unlikely or undesirable we expect that to be).


Abstract
========

CPython's global interpreter lock ("GIL") prevents multiple threads
from executing Python code at the same time.  The GIL is an obstacle
to using multi-core CPUs from Python efficiently.  This PEP proposes
adding a build configuration (``--disable-gil``) to CPython to let it
run Python code without the global interpreter lock and with the
necessary changes needed to make the interpreter thread-safe.

Motivation
==========

The GIL is a major obstacle to concurrency.  For scientific computing
tasks, this lack of concurrency is often a bigger issue than speed of
executing Python code, since most of the processor cycles are spent
in optimized CPU or GPU kernels.  The GIL introduces a global
bottleneck that can prevent other threads from making progress if
they call any Python code.  There are existing ways to enable
parallelism in CPython today, but those techniques come with
significant limitations (see `Alternatives`_).

This section focuses on the GIL's impact on scientific computing,
particular AI/ML workloads because that is the area with which this
author has the most experience, but the GIL also affects other users
of Python.


The GIL Makes Many Types of Parallelism Difficult to Express
------------------------------------------------------------

Neural network-based AI models expose multiple opportunities for
parallelism.  For example, individual operations may be parallelized
internally ("intra-operator"), multiple operations may be executed
simultaneously ("inter-operator"), and requests (spanning multiple
operations) may also be parallelized.  Efficient execution requires
exploiting multiple types of parallelism [#yuemmwang2019]_.

The GIL makes it difficult to express inter-operator parallelism, as
well as some forms of request parallelism, efficiently in Python. In
other programming languages, a system might use threads to run
different parts of a neural network on separate CPU cores, but this is
inefficient in Python due to the GIL. Similarly, latency-sensitive
inference workloads frequently use threads to parallelize across
requests, but face the same scaling bottlenecks in Python.

The challenges the GIL poses to exploiting parallelism in Python
frequently come up in reinforcement learning.  Heinrich Kuttler,
author of the NetHack Learning Environment and Member of Technical
Staff at Inflection AI, writes:

  Recent breakthroughs in reinforcement learning, such as on `Dota
  2`_, `StarCraft`_, and `NetHack`_ rely on running multiple
  environments (simulated games) in parallel using asynchronous
  actor-critic methods. Straightforward multithreaded implementations
  in Python don't scale beyond more than a few parallel environments
  due to GIL contention. Multiprocessing, with communication via
  shared memory or UNIX sockets, adds much complexity and in effect
  rules out interacting with CUDA from different workers, severely
  restricting the design space.

.. _Dota 2: https://openai.com/five/
.. _StarCraft: https://www.deepmind.com/blog/alphastar-grandmaster-level-in-starcraft-ii-using-multi-agent-reinforcement-learning
.. _NetHack: https://ai.facebook.com/blog/nethack-learning-environment-to-advance-deep-reinforcement-learning/

Manuel Kroiss, software engineer at DeepMind on the reinforcement
learning team, describes how the bottlenecks posed by the GIL lead to
rewriting Python codebases in C++, making the code less accessible:

  We frequently battle issues with the Python GIL at DeepMind. In many
  of our applications, we would like to run on the order of 50-100
  threads per process. However, we often see that even with fewer
  than 10 threads the GIL becomes the bottleneck. To work around this
  problem, we sometimes use subprocesses, but in many cases the
  inter-process communication becomes too big of an overhead.  To
  deal with the GIL, we usually end up translating large parts of our
  Python codebase into C++. This is undesirable because it makes the
  code less accessible to researchers.


Projects that involve interfacing with multiple hardware devices face
similar challenges: efficient communication requires use of multiple
CPU cores.  The `Dose-3D`_ project aims to improve cancer
radiotherapy with precise dose planning.  It uses medical phantoms
(stand-ins for human tissue) together with custom hardware and a
server application written in Python.  PaweÅ Jurgielewicz, lead
software architect for the data acquisition system on the Dose-3D
project, describes the scaling challenges posed by the GIL and how
using a fork of Python without the GIL simplified the project:

  In the Dose-3D project, the key challenge was to maintain a stable,
  non-trivial concurrent communication link with hardware units while
  utilizing a 1 Gbit/s UDP/IP connection to the maximum. Naturally,
  we started with the multiprocessing package, but at some point, it
  became clear that most CPU time was consumed by the data transfers
  between the data processing stages, not by data processing itself.
  The CPython multithreading implementation based on GIL was a dead
  end too. When we found out about the "nogil" fork of Python it took
  a single person less than half a working day to adjust the codebase
  to use this fork and the results were astonishing. Now we can focus
  on data acquisition system development rather than fine-tuning data
  exchange algorithms.

.. _Dose-3D: https://dose3d.fis.agh.edu.pl/en/projekt-dose-3d-z-programu-team-net-fnp-eng/


Allen Goodman, author of `CellProfiler`_ and staff engineer at
Prescient Design and Genentech, describes how the GIL makes
biological methods research more difficult in Python:

  Issues with Python's global interpreter lock are a frequent source
  of frustration throughout biological methods research.

  I wanted to better understand the current multithreading situation
  so I reimplemented parts of `HMMER`_, a standard method for
  multiple-sequence alignment. I chose this method because it
  stresses both single-thread performance (scoring) and
  multi-threaded performance (searching a database of sequences). The
  GIL became the bottleneck when using only eight threads. This is a
  method where the current popular implementations rely on 64 or
  even 128 threads per process. I tried moving to subprocesses but
  was blocked by the prohibitive IPC costs.  HMMER is a relatively
  elementary bioinformatics method and newer methods have far bigger
  multi-threading demands.

  Method researchers are begging to use Python (myself included),
  because of its ease of use, the Python ecosystem, and because "it's
  what people know."  Many biologists only know a little bit of
  programming (and that's almost always Python). Until Python's
  multithreading situation is addressed, C and C++ will remain the
  lingua franca of the biological methods research community.

.. _CellProfiler: https://cellprofiler.org/
.. _HMMER: http://hmmer.org/


The GIL Affects Python Library Usability
----------------------------------------

The GIL is a CPython implementation detail tha


id: pep-0298.rst
content:
PEP: 298
Title: The Locked Buffer Interface
Author: Thomas Heller <theller@python.net>
Status: Withdrawn
Type: Standards Track
Created: 26-Jul-2002
Python-Version: 2.3
Post-History: 30-Jul-2002, 01-Aug-2002


Abstract
========

This PEP proposes an extension to the buffer interface called the
'locked buffer interface'.

The locked buffer interface avoids the flaws of the 'old' buffer
interface [1]_ as defined in Python versions up to and including
2.2, and has the following semantics:

- The lifetime of the retrieved pointer is clearly defined and
  controlled by the client.

- The buffer size is returned as a 'size_t' data type, which
  allows access to large buffers on platforms where ``sizeof(int)
  != sizeof(void *)``.

(Guido comments: This second sounds like a change we could also
make to the "old" buffer interface, if we introduce another flag
bit that's *not* part of the default flags.)


Specification
=============

The locked buffer interface exposes new functions which return the
size and the pointer to the internal memory block of any python
object which chooses to implement this interface.

Retrieving a buffer from an object puts this object in a locked
state during which the buffer may not be freed, resized, or
reallocated.

The object must be unlocked again by releasing the buffer if it's
no longer used by calling another function in the locked buffer
interface.  If the object never resizes or reallocates the buffer
during its lifetime, this function may be NULL.  Failure to call
this function (if it is != NULL) is a programming error and may
have unexpected results.

The locked buffer interface omits the memory segment model which
is present in the old buffer interface - only a single memory
block can be exposed.

The memory blocks can be accessed without holding the global
interpreter lock.


Implementation
==============

Define a new flag in Include/object.h::

    /* PyBufferProcs contains bf_acquirelockedreadbuffer,
       bf_acquirelockedwritebuffer, and bf_releaselockedbuffer */
    #define Py_TPFLAGS_HAVE_LOCKEDBUFFER (1L<<15)


This flag would be included in ``Py_TPFLAGS_DEFAULT``::

    #define Py_TPFLAGS_DEFAULT  ( \
                        ....
                        Py_TPFLAGS_HAVE_LOCKEDBUFFER | \
                        ....
                        0)


Extend the ``PyBufferProcs`` structure by new fields in
Include/object.h::

    typedef size_t (*acquirelockedreadbufferproc)(PyObject *,
                                                  const void **);
    typedef size_t (*acquirelockedwritebufferproc)(PyObject *,
                                                   void **);
    typedef void (*releaselockedbufferproc)(PyObject *);

    typedef struct {
        getreadbufferproc bf_getreadbuffer;
        getwritebufferproc bf_getwritebuffer;
        getsegcountproc bf_getsegcount;
        getcharbufferproc bf_getcharbuffer;
        /* locked buffer interface functions */
        acquirelockedreadbufferproc bf_acquirelockedreadbuffer;
        acquirelockedwritebufferproc bf_acquirelockedwritebuffer;
        releaselockedbufferproc bf_releaselockedbuffer;
    } PyBufferProcs;


The new fields are present if the ``Py_TPFLAGS_HAVE_LOCKEDBUFFER``
flag is set in the object's type.

The ``Py_TPFLAGS_HAVE_LOCKEDBUFFER`` flag implies the
``Py_TPFLAGS_HAVE_GETCHARBUFFER`` flag.

The ``acquirelockedreadbufferproc`` and ``acquirelockedwritebufferproc``
functions return the size in bytes of the memory block on success,
and fill in the passed void \* pointer on success.  If these
functions fail - either because an error occurs or no memory block
is exposed - they must set the void \* pointer to NULL and raise an
exception.  The return value is undefined in these cases and
should not be used.

If calls to these functions succeed, eventually the buffer must be
released by a call to the ``releaselockedbufferproc``, supplying the
original object as argument.  The ``releaselockedbufferproc`` cannot
fail.  For objects that actually maintain an internal lock count
it would be a fatal error if the ``releaselockedbufferproc`` function
would be called too often, leading to a negative lock count.

Similar to the 'old' buffer interface, any of these functions may
be set to NULL, but it is strongly recommended to implement the
``releaselockedbufferproc`` function (even if it does nothing) if any
of the ``acquireread``/``writelockedbufferproc`` functions are
implemented, to discourage extension writers from checking for a
NULL value and not calling it.

These functions aren't supposed to be called directly, they are
called through convenience functions declared in
Include/abstract.h::

    int PyObject_AcquireLockedReadBuffer(PyObject *obj,
                                        const void **buffer,
                                        size_t *buffer_len);

    int PyObject_AcquireLockedWriteBuffer(PyObject *obj,
                                          void **buffer,
                                          size_t *buffer_len);

    void PyObject_ReleaseLockedBuffer(PyObject *obj);

The former two functions return 0 on success, set buffer to the
memory location and buffer_len to the length of the memory block
in bytes. On failure, or if the locked buffer interface is not
implemented by obj, they return -1 and set an exception.

The latter function doesn't return anything, and cannot fail.


Backward Compatibility
======================

The size of the ``PyBufferProcs`` structure changes if this proposal
is implemented, but the type's ``tp_flags`` slot can be used to
determine if the additional fields are present.


Reference Implementation
========================

An implementation has been uploaded to the SourceForge patch
manager as https://bugs.python.org/issue652857.


Additional Notes/Comments
=========================

Python strings, unicode strings, mmap objects, and array objects
would expose the locked buffer interface.

mmap and array objects would actually enter a locked state while
the buffer is active, this is not needed for strings and unicode
objects.  Resizing locked array objects is not allowed and will
raise an exception. Whether closing a locked mmap object is an
error or will only be deferred until the lock count reaches zero
is an implementation detail.

Guido recommends

    But I'm still very concerned that if most built-in types
    (e.g. strings, bytes) don't implement the release
    functionality, it's too easy for an extension to seem to work
    while forgetting to release the buffer.

    I recommend that at least some built-in types implement the
    acquire/release functionality with a counter, and assert that
    the counter is zero when the object is deleted -- if the
    assert fails, someone DECREF'ed their reference to the object
    without releasing it.  (The rule should be that you must own a
    reference to the object while you've acquired the object.)

    For strings that might be impractical because the string
    object would have to grow 4 bytes to hold the counter; but the
    new bytes object (:pep:`296`) could easily implement the counter,
    and the array object too -- that way there will be plenty of
    opportunity to test proper use of the protocol.


Community Feedback
==================

Greg Ewing doubts the locked buffer interface is needed at all, he
thinks the normal buffer interface could be used if the pointer is
(re)fetched each time it's used.  This seems to be dangerous,
because even innocent looking calls to the Python API like
``Py_DECREF()`` may trigger execution of arbitrary Python code.

The first version of this proposal didn't have the release
function, but it turned out that this would have been too
restrictive: mmap and array objects wouldn't have been able to
implement it, because mmap objects can be closed anytime if not
locked, and array objects could resize or reallocate the buffer.

This PEP will probably be rejected because nobody except the
author needs i


id: pep-0529.rst
content:
PEP: 529
Title: Change Windows filesystem encoding to UTF-8
Author: Steve Dower <steve.dower@python.org>
Status: Final
Type: Standards Track
Created: 27-Aug-2016
Python-Version: 3.6
Post-History: 01-Sep-2016, 04-Sep-2016
Resolution: https://mail.python.org/pipermail/python-dev/2016-September/146277.html

Abstract
========

Historically, Python uses the ANSI APIs for interacting with the Windows
operating system, often via C Runtime functions. However, these have been long
discouraged in favor of the UTF-16 APIs. Within the operating system, all text
is represented as UTF-16, and the ANSI APIs perform encoding and decoding using
the active code page. See `Naming Files, Paths, and Namespaces`_ for
more details.

This PEP proposes changing the default filesystem encoding on Windows to utf-8,
and changing all filesystem functions to use the Unicode APIs for filesystem
paths. This will not affect code that uses strings to represent paths, however
those that use bytes for paths will now be able to correctly round-trip all
valid paths in Windows filesystems. Currently, the conversions between Unicode
(in the OS) and bytes (in Python) were lossy and would fail to round-trip
characters outside of the user's active code page.

Notably, this does not impact the encoding of the contents of files. These will
continue to default to ``locale.getpreferredencoding()`` (for text files) or
plain bytes (for binary files). This only affects the encoding used when users
pass a bytes object to Python where it is then passed to the operating system as
a path name.

Background
==========

File system paths are almost universally represented as text with an encoding
determined by the file system. In Python, we expose these paths via a number of
interfaces, such as the ``os`` and ``io`` modules. Paths may be passed either
direction across these interfaces, that is, from the filesystem to the
application (for example, ``os.listdir()``), or from the application to the
filesystem (for example, ``os.unlink()``).

When paths are passed between the filesystem and the application, they are
either passed through as a bytes blob or converted to/from str using
``os.fsencode()`` and ``os.fsdecode()`` or explicit encoding using
``sys.getfilesystemencoding()``. The result of encoding a string with
``sys.getfilesystemencoding()`` is a blob of bytes in the native format for the
default file system.

On Windows, the native format for the filesystem is utf-16-le. The recommended
platform APIs for accessing the filesystem all accept and return text encoded in
this format. However, prior to Windows NT (and possibly further back), the
native format was a configurable machine option and a separate set of APIs
existed to accept this format. The option (the "active code page") and these
APIs (the "\*A functions") still exist in recent versions of Windows for
backwards compatibility, though new functionality often only has a utf-16-le API
(the "\*W functions").

In Python, str is recommended because it can correctly round-trip all characters
used in paths (on POSIX with surrogateescape handling; on Windows because str
maps to the native representation). On Windows bytes cannot round-trip all
characters used in paths, as Python internally uses the \*A functions and hence
the encoding is "whatever the active code page is". Since the active code page
cannot represent all Unicode characters, the conversion of a path into bytes can
lose information without warning or any available indication.

As a demonstration of this::

    >>> open('test\uAB00.txt', 'wb').close()
    >>> import glob
    >>> glob.glob('test*')
    ['test\uab00.txt']
    >>> glob.glob(b'test*')
    [b'test?.txt']

The Unicode character in the second call to glob has been replaced by a '?',
which means passing the path back into the filesystem will result in a
``FileNotFoundError``. The same results may be observed with ``os.listdir()`` or
any function that matches the return type to the parameter type.

While one user-accessible fix is to use str everywhere, POSIX systems generally
do not suffer from data loss when using bytes exclusively as the bytes are the
canonical representation. Even if the encoding is "incorrect" by some standard,
the file system will still map the bytes back to the file. Making use of this
avoids the cost of decoding and reencoding, such that (theoretically, and only
on POSIX), code such as this may be faster because of the use of ``b'.'``
compared to using ``'.'``::

    >>> for f in os.listdir(b'.'):
    ...     os.stat(f)
    ...

As a result, POSIX-focused library authors prefer to use bytes to represent
paths. For some authors it is also a convenience, as their code may receive
bytes already known to be encoded correctly, while others are attempting to
simplify porting their code from Python 2. However, the correctness assumptions
do not carry over to Windows where Unicode is the canonical representation, and
errors may result. This potential data loss is why the use of bytes paths on
Windows was deprecated in Python 3.3 - all of the above code snippets produce
deprecation warnings on Windows.

Proposal
========

Currently the default filesystem encoding is 'mbcs', which is a meta-encoder
that uses the active code page. However, when bytes are passed to the filesystem
they go through the \*A APIs and the operating system handles encoding. In this
case, paths are always encoded using the equivalent of 'mbcs:replace' with no
opportunity for Python to override or change this.

This proposal would remove all use of the \*A APIs and only ever call the \*W
APIs. When Windows returns paths to Python as ``str``, they will be decoded from
utf-16-le and returned as text (in whatever the minimal representation is). When
Python code requests paths as ``bytes``, the paths will be transcoded from
utf-16-le into utf-8 using surrogatepass (Windows does not validate surrogate
pairs, so it is possible to have invalid surrogates in filenames). Equally, when
paths are provided as ``bytes``, they are transcoded from utf-8 into utf-16-le
and passed to the \*W APIs.

The use of utf-8 will not be configurable, except for the provision of a
"legacy mode" flag to revert to the previous behaviour.

The ``surrogateescape`` error mode does not apply here, as the concern is not
about retaining nonsensical bytes. Any path returned from the operating system
will be valid Unicode, while invalid paths created by the user should raise a
decoding error (currently these would raise ``OSError`` or a subclass).

The choice of utf-8 bytes (as opposed to utf-16-le bytes) is to ensure the
ability to round-trip path names and allow basic manipulation (for example,
using the ``os.path`` module) when assuming an ASCII-compatible encoding. Using
utf-16-le as the encoding is more pure, but will cause more issues than are
resolved.

This change would also undeprecate the use of bytes paths on Windows. No change
to the semantics of using bytes as a path is required - as before, they must be
encoded with the encoding specified by ``sys.getfilesystemencoding()``.

Specific Changes
================

Update sys.getfilesystemencoding
--------------------------------

Remove the default value for ``Py_FileSystemDefaultEncoding`` and set it in
``initfsencoding()`` to utf-8, or if the legacy-mode switch is enabled to mbcs.

Update the implementations of ``PyUnicode_DecodeFSDefaultAndSize()`` and
``PyUnicode_EncodeFSDefault()`` to use the utf-8 codec, or if the legacy-mode
switch is enabled the existing mbcs codec.

Add sys.getfilesystemencodeerrors
---------------------------------

As the error mode may now change between ``surrogatepass`` and ``replace``,
Python code that manually performs encoding also needs access to the current
error mode. This includes the implementation of ``os.fsencode()`` and
``os.fsdecode()``, which currently assume an error mode based on the codec.

Add a public ``Py_FileSystemDefaultEncodeErrors``, similar to the existing
``Py_FileS


id: pep-0267.rst
content:
PEP: 267
Title: Optimized Access to Module Namespaces
Author: Jeremy Hylton <jeremy@alum.mit.edu>
Status: Deferred
Type: Standards Track
Created: 23-May-2001
Python-Version: 2.2
Post-History:


Deferral
========

While this PEP is a nice idea, no-one has yet emerged to do the work of
hashing out the differences between this PEP, :pep:`266` and :pep:`280`.
Hence, it is being deferred.


Abstract
========

This PEP proposes a new implementation of global module namespaces
and the builtin namespace that speeds name resolution.  The
implementation would use an array of object pointers for most
operations in these namespaces.  The compiler would assign indices
for global variables and module attributes at compile time.

The current implementation represents these namespaces as
dictionaries.  A global name incurs a dictionary lookup each time
it is used; a builtin name incurs two dictionary lookups, a failed
lookup in the global namespace and a second lookup in the builtin
namespace.

This implementation should speed Python code that uses
module-level functions and variables.  It should also eliminate
awkward coding styles that have evolved to speed access to these
names.

The implementation is complicated because the global and builtin
namespaces can be modified dynamically in ways that are impossible
for the compiler to detect.  (Example: A module's namespace is
modified by a script after the module is imported.)  As a result,
the implementation must maintain several auxiliary data structures
to preserve these dynamic features.


Introduction
============

This PEP proposes a new implementation of attribute access for
module objects that optimizes access to module variables known at
compile time.  The module will store these variables in an array
and provide an interface to lookup attributes using array offsets.
For globals, builtins, and attributes of imported modules, the
compiler will generate code that uses the array offsets for fast
access.

[describe the key parts of the design: dlict, compiler support,
stupid name trick workarounds, optimization of other module's
globals]

The implementation will preserve existing semantics for module
namespaces, including the ability to modify module namespaces at
runtime in ways that affect the visibility of builtin names.


DLict design
============

The namespaces are implemented using a data structure that has
sometimes gone under the name ``dlict``.  It is a dictionary that has
numbered slots for some dictionary entries.  The type must be
implemented in C to achieve acceptable performance.  The new
type-class unification work should make this fairly easy.  The
``DLict`` will presumably be a subclass of dictionary with an
alternate storage module for some keys.

A Python implementation is included here to illustrate the basic
design::

    """A dictionary-list hybrid"""

    import types

    class DLict:
        def __init__(self, names):
            assert isinstance(names, types.DictType)
            self.names = {}
            self.list = [None] * size
            self.empty = [1] * size
            self.dict = {}
            self.size = 0

        def __getitem__(self, name):
            i = self.names.get(name)
            if i is None:
                return self.dict[name]
            if self.empty[i] is not None:
                raise KeyError, name
            return self.list[i]

        def __setitem__(self, name, val):
            i = self.names.get(name)
            if i is None:
                self.dict[name] = val
            else:
                self.empty[i] = None
                self.list[i] = val
                self.size += 1

        def __delitem__(self, name):
            i = self.names.get(name)
            if i is None:
                del self.dict[name]
            else:
                if self.empty[i] is not None:
                    raise KeyError, name
                self.empty[i] = 1
                self.list[i] = None
                self.size -= 1

        def keys(self):
            if self.dict:
                return self.names.keys() + self.dict.keys()
            else:
                return self.names.keys()

        def values(self):
            if self.dict:
                return self.names.values() + self.dict.values()
            else:
                return self.names.values()

        def items(self):
            if self.dict:
                return self.names.items()
            else:
                return self.names.items() + self.dict.items()

        def __len__(self):
            return self.size + len(self.dict)

        def __cmp__(self, dlict):
            c = cmp(self.names, dlict.names)
            if c != 0:
                return c
            c = cmp(self.size, dlict.size)
            if c != 0:
                return c
            for i in range(len(self.names)):
                c = cmp(self.empty[i], dlict.empty[i])
            if c != 0:
                return c
            if self.empty[i] is None:
                c = cmp(self.list[i], dlict.empty[i])
                if c != 0:
                    return c
            return cmp(self.dict, dlict.dict)

        def clear(self):
            self.dict.clear()
            for i in range(len(self.names)):
                if self.empty[i] is None:
                    self.empty[i] = 1
                    self.list[i] = None

        def update(self):
            pass

        def load(self, index):
            """dlict-special method to support indexed access"""
            if self.empty[index] is None:
                return self.list[index]
            else:
                raise KeyError, index # XXX might want reverse mapping

        def store(self, index, val):
            """dlict-special method to support indexed access"""
            self.empty[index] = None
            self.list[index] = val

        def delete(self, index):
            """dlict-special method to support indexed access"""
            self.empty[index] = 1
            self.list[index] = None


Compiler issues
===============

The compiler currently collects the names of all global variables
in a module.  These are names bound at the module level or bound
in a class or function body that declares them to be global.

The compiler would assign indices for each global name and add the
names and indices of the globals to the module's code object.
Each code object would then be bound irrevocably to the module it
was defined in.  (Not sure if there are some subtle problems with
this.)

For attributes of imported modules, the module will store an
indirection record.  Internally, the module will store a pointer
to the defining module and the offset of the attribute in the
defining module's global variable array.  The offset would be
initialized the first time the name is looked up.


Runtime model
=============

The PythonVM will be extended with new opcodes to access globals
and module attributes via a module-level array.

A function object would need to point to the module that defined
it in order to provide access to the module-level global array.

For module attributes stored in the ``dlict`` (call them static
attributes), the get/delattr implementation would need to track
access to these attributes using the old by-name interface.  If a
static attribute is updated dynamically, e.g.::

   mod.__dict__["foo"] = 2

The implementation would need to update the array slot instead of
the backup dict.


Backwards compatibility
=======================

The ``dlict`` will need to maintain meta-information about whether a
slot is currently used or not.  It will also need to maintain a
pointer to the builtin namespace.  When a name is not currently
used in the global namespace, the lookup will have to fail over to
the builtin namespace.

In the reverse case, each module may need a special accessor
function for the builtin namespace that checks to see if a global
shadowing the builtin has been added dynamically.  This check
would only 


id: pep-0501.rst
content:
PEP: 501
Title: General purpose template literal strings
Author: Alyssa Coghlan <ncoghlan@gmail.com>, Nick Humrich <nick@humrich.us>
Discussions-To: https://discuss.python.org/t/pep-501-reopen-general-purpose-string-template-literals/24625
Status: Withdrawn
Type: Standards Track
Requires: 701
Created: 08-Aug-2015
Python-Version: 3.12
Post-History: `08-Aug-2015 <https://mail.python.org/archives/list/python-dev@python.org/thread/EAZ3P2M3CDDIQFR764NF6FXQHWXYMKJF/>`__,
              `05-Sep-2015 <https://mail.python.org/archives/list/python-dev@python.org/thread/ILVRPS6DTFZ7IHL5HONDBB6INVXTFOZ2/>`__,
              `09-Mar-2023 <https://discuss.python.org/t/pep-501-reopen-general-purpose-string-template-literals/24625>`__,
Superseded-By: 750

.. superseded:: 750

Abstract
========

Though easy and elegant to use, Python :term:`f-string`\s
can be vulnerable to injection attacks when used to construct
shell commands, SQL queries, HTML snippets and similar
(for example, ``os.system(f"echo {message_from_user}")``).
This PEP introduces template literal strings (or "t-strings"),
which have syntax and semantics that are similar to f-strings,
but with rendering deferred until :func:`format` or another
template rendering function is called on them.
This will allow standard library calls, helper functions
and third party tools to safety and intelligently perform
appropriate escaping and other string processing on inputs
while retaining the usability and convenience of f-strings.


PEP Withdrawal
==============

When :pep:`750` was first published as a "tagged strings" proposal
(allowing for arbitrary string prefixes), this PEP was kept open to
continue championing the simpler "template literal" approach that
used a single dedicated string prefix to produce instances of a new
"interpolation template" type.

The `October 2024 updates <https://github.com/python/peps/pull/4062>`__
to :pep:`750` agreed that template strings were a better fit for Python
than the broader tagged strings concept.

All of the other concerns the authors of this PEP had with :pep:`750`
were also either addressed in those updates, or else left in a state
where they could reasonably be addressed in a future change proposal.

Due to the clear improvements in the updated :pep:`750` proposal,
this PEP has been withdrawn in favour of :pep:`750`.

.. important::

  The remainder of this PEP still reflects the state of the tagged strings
  proposal in August 2024. It has *not* been updated to reflect the
  October 2024 changes to :pep:`750`, since the PEP withdrawal makes doing
  so redundant.


Relationship with other PEPs
============================

This PEP is inpired by and builds on top of the f-string syntax first implemented
in :pep:`498` and formalised in :pep:`701`.

This PEP complements the literal string typing support added to Python's formal type
system in :pep:`675` by introducing a *safe* way to do dynamic interpolation of runtime
values into security sensitive strings.

This PEP competes with some aspects of the tagged string proposal in :pep:`750`
(most notably in whether template rendering is expressed as ``render(t"template literal")``
or as ``render"template literal"``), but also shares *many* common features (after
:pep:`750` was published, this PEP was updated with
`several new changes <https://github.com/python/peps/issues/3904>`__
inspired by the tagged strings proposal).

This PEP does NOT propose an alternative to :pep:`292` for user interface
internationalization use cases (but does note the potential for future syntactic
enhancements aimed at that use case that would benefit from the compiler-supported
value interpolation machinery that this PEP and :pep:`750` introduce).


Motivation
==========

:pep:`498` added new syntactic support for string interpolation that is
transparent to the compiler, allowing name references from the interpolation
operation full access to containing namespaces (as with any other expression),
rather than being limited to explicit name references. These are referred
to in the PEP (and elsewhere) as "f-strings" (a mnemonic for "formatted strings").

Since acceptance of :pep:`498`, f-strings have become well-established and very popular.
f-strings became even more useful and flexible with the formalised grammar in :pep:`701`.
While f-strings are great, eager rendering has its limitations. For example, the
eagerness of f-strings has made code like the following unfortunately plausible:

.. code-block:: python

    os.system(f"echo {message_from_user}")

This kind of code is superficially elegant, but poses a significant problem
if the interpolated value ``message_from_user`` is in fact provided by an
untrusted user: it's an opening for a form of code injection attack, where
the supplied user data has not been properly escaped before being passed to
the ``os.system`` call.

While the ``LiteralString`` type annotation introduced in :pep:`675` means that typecheckers
are able to report a type error for this kind of unsafe function usage, those errors don't
help make it easier to write code that uses safer alternatives (such as
:func:`subprocess.run`).

To address that problem (and a number of other concerns), this PEP proposes
the complementary introduction of "t-strings" (a mnemonic for "template literal strings"),
where ``format(t"Message with {data}")`` would produce the same result as
``f"Message with {data}"``, but the template literal instance can instead be passed
to other template rendering functions which process the contents of the template
differently.


Proposal
========

Dedicated template literal syntax
---------------------------------

This PEP proposes a new string prefix that declares the
string to be a template literal rather than an ordinary string:

.. code-block:: python

    template = t"Substitute {names:>{field_width}} and {expressions()!r} at runtime"

This would be effectively interpreted as:

.. code-block:: python

    template = TemplateLiteral(
        r"Substitute {names:>{field_width}} and {expressions()} at runtime",
        TemplateLiteralText(r"Substitute "),
        TemplateLiteralField("names", names, f">{field_width}", ""),
        TemplateLiteralText(r" and "),
        TemplateLiteralField("expressions()", expressions(), f"", "r"),
    )

(Note: this is an illustrative example implementation. The exact compile time construction
syntax of ``types.TemplateLiteral`` is considered an implementation detail not specified by
the PEP. In particular, the compiler may bypass the default constructor's runtime logic that
detects consecutive text segments and merges them into a single text segment, as well as
checking the runtime types of all supplied arguments).

The ``__format__`` method on ``types.TemplateLiteral`` would then
implement the following :meth:`str.format` inspired semantics:

.. code-block:: python-console

  >>> import datetime
  >>> name = 'Jane'
  >>> age = 50
  >>> anniversary = datetime.date(1991, 10, 12)
  >>> format(t'My name is {name}, my age next year is {age+1}, my anniversary is {anniversary:%A, %B %d, %Y}.')
  'My name is Jane, my age next year is 51, my anniversary is Saturday, October 12, 1991.'
  >>> format(t'She said her name is {name!r}.')
  "She said her name is 'Jane'."

The syntax of template literals would be based on :pep:`701`, and largely use the same
syntax for the string portion of the template. Aside from using a different prefix, the one
other syntactic change is in the definition and handling of conversion specifiers, both to
allow ``!()`` as a standard conversion specifier to request evaluation of a field at
rendering time, and to allow custom renderers to also define custom conversion specifiers.

This PEP does not propose to remove or deprecate any of the existing
string formatting mechanisms, as those will remain valuable when formatting
strings that are not present directly in the source code of the application.


Lazy field evaluation conversion specifier
-----------------


id: pep-0515.rst
content:
PEP: 515
Title: Underscores in Numeric Literals
Author: Georg Brandl, Serhiy Storchaka
Status: Final
Type: Standards Track
Created: 10-Feb-2016
Python-Version: 3.6
Post-History: 10-Feb-2016, 11-Feb-2016

Abstract and Rationale
======================

This PEP proposes to extend Python's syntax and number-from-string
constructors so that underscores can be used as visual separators for
digit grouping purposes in integral, floating-point and complex number
literals.

This is a common feature of other modern languages, and can aid
readability of long literals, or literals whose value should clearly
separate into parts, such as bytes or words in hexadecimal notation.

Examples::

    # grouping decimal numbers by thousands
    amount = 10_000_000.0

    # grouping hexadecimal addresses by words
    addr = 0xCAFE_F00D

    # grouping bits into nibbles in a binary literal
    flags = 0b_0011_1111_0100_1110

    # same, for string conversions
    flags = int('0b_1111_0000', 2)


Specification
=============

The current proposal is to allow one underscore between digits, and
after base specifiers in numeric literals.  The underscores have no
semantic meaning, and literals are parsed as if the underscores were
absent.

Literal Grammar
---------------

The production list for integer literals would therefore look like
this::

   integer: decinteger | bininteger | octinteger | hexinteger
   decinteger: nonzerodigit (["_"] digit)* | "0" (["_"] "0")*
   bininteger: "0" ("b" | "B") (["_"] bindigit)+
   octinteger: "0" ("o" | "O") (["_"] octdigit)+
   hexinteger: "0" ("x" | "X") (["_"] hexdigit)+
   nonzerodigit: "1"..."9"
   digit: "0"..."9"
   bindigit: "0" | "1"
   octdigit: "0"..."7"
   hexdigit: digit | "a"..."f" | "A"..."F"

For floating-point and complex literals::

   floatnumber: pointfloat | exponentfloat
   pointfloat: [digitpart] fraction | digitpart "."
   exponentfloat: (digitpart | pointfloat) exponent
   digitpart: digit (["_"] digit)*
   fraction: "." digitpart
   exponent: ("e" | "E") ["+" | "-"] digitpart
   imagnumber: (floatnumber | digitpart) ("j" | "J")

Constructors
------------

Following the same rules for placement, underscores will be allowed in
the following constructors:

- ``int()`` (with any base)
- ``float()``
- ``complex()``
- ``Decimal()``

Further changes
---------------

The new-style number-to-string formatting language will be extended to
allow ``_`` as a thousands separator, where currently only ``,`` is
supported.  This can be used to easily generate code with more
readable literals.  [11]_

The syntax would be the same as for the comma, e.g. ``{:10_}`` for a
width of 10 with ``_`` separator.

For the ``b``, ``x`` and ``o`` format specifiers, ``_`` will be
allowed and group by 4 digits.


Prior Art
=========

Those languages that do allow underscore grouping implement a large
variety of rules for allowed placement of underscores.  In cases where
the language spec contradicts the actual behavior, the actual behavior
is listed.  ("single" or "multiple" refer to allowing runs of
consecutive underscores.)

* Ada: single, only between digits [8]_
* C# (open proposal for 7.0): multiple, only between digits [6]_
* C++14: single, between digits (different separator chosen) [1]_
* D: multiple, anywhere, including trailing [2]_
* Java: multiple, only between digits [7]_
* Julia: single, only between digits (but not in float exponent parts)
  [9]_
* Perl 5: multiple, basically anywhere, although docs say it's
  restricted to one underscore between digits [3]_
* Ruby: single, only between digits (although docs say "anywhere")
  [10]_
* Rust: multiple, anywhere, except for between exponent "e" and digits
  [4]_
* Swift: multiple, between digits and trailing (although textual
  description says only "between digits") [5]_


Alternative Syntax
==================

Underscore Placement Rules
--------------------------

Instead of the relatively strict rule specified above, the use of
underscores could be less limited.  As seen in other languages, common
rules include:

* Only one consecutive underscore allowed, and only between digits.
* Multiple consecutive underscores allowed, but only between digits.
* Multiple consecutive underscores allowed, in most positions except
  for the start of the literal, or special positions like after a
  decimal point.

The syntax in this PEP has ultimately been selected because it covers
the common use cases, and does not allow for syntax that would have to
be discouraged in style guides anyway.

A less common rule would be to allow underscores only every N digits
(where N could be 3 for decimal literals, or 4 for hexadecimal ones).
This is unnecessarily restrictive, especially considering the
separator placement is different in different cultures.

Different Separators
--------------------

A proposed alternate syntax was to use whitespace for grouping.
Although strings are a precedent for combining adjoining literals, the
behavior can lead to unexpected effects which are not possible with
underscores.  Also, no other language is known to use this rule,
except for languages that generally disregard any whitespace.

C++14 introduces apostrophes for grouping (because underscores
introduce ambiguity with user-defined literals), which is not
considered because of the use in Python's string literals. [1]_


Implementation
==============

A preliminary patch that implements the specification given above has
been posted to the issue tracker. [12]_


References
==========

.. [1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3499.html

.. [2] https://dlang.org/spec/lex.html#integerliteral

.. [3] https://perldoc.perl.org/perldata#Scalar-value-constructors

.. [4] https://web.archive.org/web/20160304121349/http://doc.rust-lang.org/reference.html#integer-literals

.. [5] https://docs.swift.org/swift-book/ReferenceManual/LexicalStructure.html

.. [6] https://github.com/dotnet/roslyn/issues/216

.. [7] https://docs.oracle.com/javase/7/docs/technotes/guides/language/underscores-literals.html

.. [8] http://archive.adaic.com/standards/83lrm/html/lrm-02-04.html#2.4

.. [9] https://web.archive.org/web/20160223175334/http://docs.julialang.org/en/release-0.4/manual/integers-and-floating-point-numbers/

.. [10] https://ruby-doc.org/core-2.3.0/doc/syntax/literals_rdoc.html#label-Numbers

.. [11] https://mail.python.org/pipermail/python-dev/2016-February/143283.html

.. [12] http://bugs.python.org/issue26331


Copyright
=========

This document has been placed in the public domain.


id: pep-0273.rst
content:
PEP: 273
Title: Import Modules from Zip Archives
Author: James C. Ahlstrom <jim@interet.com>
Status: Final
Type: Standards Track
Created: 11-Oct-2001
Python-Version: 2.3
Post-History: 26-Oct-2001


Abstract
========

This PEP adds the ability to import Python modules
``*.py``, ``*.py[co]`` and packages from zip archives.  The
same code is used to speed up normal directory imports
provided ``os.listdir`` is available.


Note
====

Zip imports were added to Python 2.3, but the final implementation
uses an approach different from the one described in this PEP.
The 2.3 implementation is SourceForge patch #652586 [1]_, which adds
new import hooks described in :pep:`302`.

The rest of this PEP is therefore only of historical interest.


Specification
=============

Currently, ``sys.path`` is a list of directory names as strings.  If
this PEP is implemented, an item of ``sys.path`` can be a string
naming a zip file archive.  The zip archive can contain a
subdirectory structure to support package imports.  The zip
archive satisfies imports exactly as a subdirectory would.

The implementation is in C code in the Python core and works on
all supported Python platforms.

Any files may be present in the zip archive, but only files
``*.py`` and ``*.py[co]`` are available for import.  Zip import of
dynamic modules (``*.pyd``, ``*.so``) is disallowed.

Just as ``sys.path`` currently has default directory names, a default
zip archive name is added too.  Otherwise there is no way to
import all Python library files from an archive.


Subdirectory Equivalence
========================

The zip archive must be treated exactly as a subdirectory tree so
we can support package imports based on current and future rules.
All zip data is taken from the Central Directory, the data must be
correct, and brain dead zip files are not accommodated.

Suppose ``sys.path`` contains "/A/B/SubDir" and "/C/D/E/Archive.zip",
and we are trying to import ``modfoo`` from the ``Q`` package.  Then
``import.c`` will generate a list of paths and extensions and will
look for the file.  The list of generated paths does not change
for zip imports.  Suppose ``import.c`` generates the path
"/A/B/SubDir/Q/R/modfoo.pyc".  Then it will also generate the path
"/C/D/E/Archive.zip/Q/R/modfoo.pyc".  Finding the SubDir path is
exactly equivalent to finding "Q/R/modfoo.pyc" in the archive.

Suppose you zip up /A/B/SubDir/* and all its subdirectories.  Then
your zip file will satisfy imports just as your subdirectory did.

Well, not quite.  You can't satisfy dynamic modules from a zip
file.  Dynamic modules have extensions like ``.dll``, ``.pyd``, and ``.so``.
They are operating system dependent, and probably can't be loaded
except from a file.  It might be possible to extract the dynamic
module from the zip file, write it to a plain file and load it.
But that would mean creating temporary files, and dealing with all
the ``dynload_*.c``, and that's probably not a good idea.

When trying to import ``*.pyc``, if it is not available then
``*.pyo`` will be used instead.  And vice versa when looking for ``*.pyo``.
If neither ``*.pyc`` nor ``*.pyo`` is available, or if the magic numbers
are invalid, then ``*.py`` will be compiled and used to satisfy the
import, but the compiled file will not be saved.  Python would
normally write it to the same directory as ``*.py``, but surely we
don't want to write to the zip file.  We could write to the
directory of the zip archive, but that would clutter it up, not
good if it is ``/usr/bin`` for example.

Failing to write the compiled files will make zip imports very slow,
and the user will probably not figure out what is wrong.  So it
is best to put ``*.pyc`` and ``*.pyo`` in the archive with the ``*.py``.


Efficiency
==========

The only way to find files in a zip archive is linear search.  So
for each zip file in ``sys.path``, we search for its names once, and
put the names plus other relevant data into a static Python
dictionary.  The key is the archive name from ``sys.path`` joined with
the file name (including any subdirectories) within the archive.
This is exactly the name generated by ``import.c``, and makes lookup
easy.

This same mechanism is used to speed up directory (non-zip) imports.
See below.


zlib
====

Compressed zip archives require ``zlib`` for decompression.  Prior to
any other imports, we attempt an import of ``zlib``.  Import of
compressed files will fail with a message "missing ``zlib``" unless
``zlib`` is available.


Booting
=======

Python imports ``site.py`` itself, and this imports ``os``, ``nt``, ``ntpath``,
``stat``, and ``UserDict``.  It also imports ``sitecustomize.py`` which may
import more modules.  Zip imports must be available before ``site.py``
is imported.

Just as there are default directories in ``sys.path``, there must be
one or more default zip archives too.

The problem is what the name should be.  The name should be linked
with the Python version, so the Python executable can correctly
find its corresponding libraries even when there are multiple
Python versions on the same machine.

We add one name to ``sys.path``.  On Unix, the directory is
``sys.prefix + "/lib"``, and the file name is
``"python%s%s.zip" % (sys.version[0], sys.version[2])``.
So for Python 2.2 and prefix ``/usr/local``, the path
``/usr/local/lib/python2.2/`` is already on ``sys.path``, and
``/usr/local/lib/python22.zip`` would be added.
On Windows, the file is the full path to ``python22.dll``, with
"dll" replaced by "zip".  The zip archive name is always inserted
as the second item in ``sys.path``.  The first is the directory of the
``main.py`` (thanks Tim).


Directory Imports
=================

The static Python dictionary used to speed up zip imports can be
used to speed up normal directory imports too.  For each item in
``sys.path`` that is not a zip archive, we call ``os.listdir``, and add
the directory contents to the dictionary.  Then instead of calling
``fopen()`` in a double loop, we just check the dictionary.  This
greatly speeds up imports.  If ``os.listdir`` doesn't exist, the
dictionary is not used.


Benchmarks
==========

====  =================  =================  ==========  ==========
Case  Original 2.2a3     Using os.listdir   Zip Uncomp  Zip Compr
====  =================  =================  ==========  ==========
1     3.2 2.5 3.2->1.02  2.3 2.5 2.3->0.87  1.66->0.93  1.5->1.07
2     2.8 3.9 3.0->1.32  Same as Case 1.
3     5.7 5.7 5.7->5.7   2.1 2.1 2.1->1.8   1.25->0.99  1.19->1.13
4     9.4 9.4 9.3->9.35  Same as Case 3.
====  =================  =================  ==========  ==========

Case 1: Local drive C:, ``sys.path`` has its default value.
Case 2: Local drive C:, directory with files is at the end of ``sys.path``.
Case 3: Network  drive, ``sys.path`` has its default value.
Case 4: Network  drive, directory with files is at the end of ``sys.path``.

Benchmarks were performed on a Pentium 4 clone, 1.4 GHz, 256 Meg.
The machine was running Windows 2000 with a Linux/Samba network server.
Times are in seconds, and are the time to import about 100 Lib modules.
Case 2 and 4 have the "correct" directory moved to the end of ``sys.path``.
"Uncomp" means uncompressed zip archive, "Compr" means compressed.

Initial times are after a re-boot of the system; the time after
"->" is the time after repeated runs.  Times to import from C:
after a re-boot are rather highly variable for the "Original" case,
but are more realistic.


Custom Imports
==============

The logic demonstrates the ability to import using default searching
until a needed Python module (in this case, ``os``) becomes available.
This can be used to bootstrap custom importers.  For example, if
"``importer()``" in ``__init__.py`` exists, then it could be used for imports.
The "``importer()``" can freely import os and other modules, and these
will be satisfied from the default mechanism.  This PEP does not
define any custom importers, and this note is for information only.


Implementa


id: pep-0503.rst
content:
PEP: 503
Title: Simple Repository API
Author: Donald Stufft <donald@stufft.io>
BDFL-Delegate: Donald Stufft <donald@stufft.io>
Discussions-To: distutils-sig@python.org
Status: Final
Type: Standards Track
Topic: Packaging
Created: 04-Sep-2015
Post-History: 04-Sep-2015
Resolution: https://mail.python.org/pipermail/distutils-sig/2015-September/026899.html


Abstract
========

There are many implementations of a Python package repository and many tools
that consume them. Of these, the canonical implementation that defines what
the "simple" repository API looks like is the implementation that powers
PyPI. This document will specify that API, documenting what the correct
behavior for any implementation of the simple repository API.


Specification
=============

A repository that implements the simple API is defined by its base URL, this is
the top level URL that all additional URLs are below. The API is named the
"simple" repository due to the fact that PyPI's base URL is
``https://pypi.org/simple/``.

.. note:: All subsequent URLs in this document will be relative to this base
          URL (so given PyPI's URL, a URL of ``/foo/`` would be
          ``https://pypi.org/simple/foo/``.


Within a repository, the root URL (``/`` for this PEP which represents the base
URL) **MUST** be a valid HTML5 page with a single anchor element per project in
the repository. The text of the anchor tag **MUST** be the name of
the project and the href attribute **MUST** link to the URL for that particular
project. As an example::

   <!DOCTYPE html>
   <html>
     <body>
       <a href="/frob/">frob</a>
       <a href="/spamspamspam/">spamspamspam</a>
     </body>
   </html>

Below the root URL is another URL for each individual project contained within
a repository. The format of this URL is ``/<project>/`` where the ``<project>``
is replaced by the normalized name for that project, so a project named
"HolyGrail" would have a URL like ``/holygrail/``. This URL must respond with
a valid HTML5 page with a single anchor element per file for the project. The
href attribute **MUST** be a URL that links to the location of the file for
download, and the text of the anchor tag **MUST** match the final path
component (the filename) of the URL. The URL **SHOULD** include a hash in the
form of a URL fragment with the following syntax: ``#<hashname>=<hashvalue>``,
where ``<hashname>`` is the lowercase name of the hash function (such as
``sha256``) and ``<hashvalue>`` is the hex encoded digest.

In addition to the above, the following constraints are placed on the API:

* All URLs which respond with an HTML5 page **MUST** end with a ``/`` and the
  repository **SHOULD** redirect the URLs without a ``/`` to add a ``/`` to the
  end.

* URLs may be either absolute or relative as long as they point to the correct
  location.

* There are no constraints on where the files must be hosted relative to the
  repository.

* There may be any other HTML elements on the API pages as long as the required
  anchor elements exist.

* Repositories **MAY** redirect unnormalized URLs to the canonical normalized
  URL (e.g. ``/Foobar/`` may redirect to ``/foobar/``), however clients
  **MUST NOT** rely on this redirection and **MUST** request the normalized
  URL.

* Repositories **SHOULD** choose a hash function from one of the ones
  guaranteed to be available via the ``hashlib`` module in the Python standard
  library (currently ``md5``, ``sha1``, ``sha224``, ``sha256``, ``sha384``,
  ``sha512``). The current recommendation is to use ``sha256``.

* If there is a GPG signature for a particular distribution file it **MUST**
  live alongside that file with the same name with a ``.asc`` appended to it.
  So if the file ``/packages/HolyGrail-1.0.tar.gz`` existed and had an
  associated signature, the signature would be located at
  ``/packages/HolyGrail-1.0.tar.gz.asc``.

* A repository **MAY** include a ``data-gpg-sig`` attribute on a file link with
  a value of either ``true`` or ``false`` to indicate whether or not there is a
  GPG signature. Repositories that do this **SHOULD** include it on every link.

* A repository **MAY** include a ``data-requires-python`` attribute on a file
  link. This exposes the *Requires-Python* metadata field, specified in :pep:`345`,
  for the corresponding release. Where this is present, installer tools
  **SHOULD** ignore the download when installing to a Python version that
  doesn't satisfy the requirement. For example::

      <a href="..." data-requires-python="&gt;=3">...</a>

  In the attribute value, < and > have to be HTML encoded as ``&lt;`` and
  ``&gt;``, respectively.

Normalized Names
----------------

This PEP references the concept of a "normalized" project name. As per :pep:`426`
the only valid characters in a name are the ASCII alphabet, ASCII numbers,
``.``, ``-``, and ``_``. The name should be lowercased with all runs of the
characters ``.``, ``-``, or ``_`` replaced with a single ``-`` character. This
can be implemented in Python with the ``re`` module::

   import re

   def normalize(name):
       return re.sub(r"[-_.]+", "-", name).lower()

Changes
-------

* The optional ``data-requires-python`` attribute was added in July 2016.


Copyright
=========

This document has been placed in the public domain.


id: pep-0265.rst
content:
PEP: 265
Title: Sorting Dictionaries by Value
Author: Grant Griffin <g2@iowegian.com>
Status: Rejected
Type: Standards Track
Created: 08-Aug-2001
Python-Version: 2.2
Post-History:


Abstract
========

This PEP suggests a "sort by value" operation for dictionaries.
The primary benefit would be in terms of "batteries included"
support for a common Python idiom which, in its current form, is
both difficult for beginners to understand and cumbersome for all
to implement.


BDFL Pronouncement
==================

This PEP is rejected because the need for it has been largely
fulfilled by Py2.4's ``sorted()`` builtin function::

    >>> sorted(d.iteritems(), key=itemgetter(1), reverse=True)
    [('b', 23), ('d', 17), ('c', 5), ('a', 2), ('e', 1)]

or for just the keys::

    sorted(d, key=d.__getitem__, reverse=True)
    ['b', 'd', 'c', 'a', 'e']

Also, Python 2.5's ``heapq.nlargest()`` function addresses the common use
case of finding only a few of the highest valued items::

    >>> nlargest(2, d.iteritems(), itemgetter(1))
    [('b', 23), ('d', 17)]


Motivation
==========

A common use of dictionaries is to count occurrences by setting
the value of ``d[key]`` to 1 on its first occurrence, then increment
the value on each subsequent occurrence.  This can be done several
different ways, but the ``get()`` method is the most succinct::

        d[key] = d.get(key, 0) + 1

Once all occurrences have been counted, a common use of the
resulting dictionary is to print the occurrences in
occurrence-sorted order, often with the largest value first.

This leads to a need to sort a dictionary's items by value.  The
canonical method of doing so in Python is to first use ``d.items()``
to get a list of the dictionary's items, then invert the ordering
of each item's tuple from (key, value) into (value, key), then
sort the list; since Python sorts the list based on the first item
of the tuple, the list of (inverted) items is therefore sorted by
value.  If desired, the list can then be reversed, and the tuples
can be re-inverted back to (key, value).  (However, in my
experience, the inverted tuple ordering is fine for most purposes,
e.g. printing out the list.)

For example, given an occurrence count of::

    >>> d = {'a':2, 'b':23, 'c':5, 'd':17, 'e':1}

we might do::

    >>> items = [(v, k) for k, v in d.items()]
    >>> items.sort()
    >>> items.reverse()             # so largest is first
    >>> items = [(k, v) for v, k in items]

resulting in::

    >>> items
    [('b', 23), ('d', 17), ('c', 5), ('a', 2), ('e', 1)]

which shows the list in by-value order, largest first.  (In this
case, ``'b'`` was found to have the most occurrences.)

This works fine, but is "hard to use" in two aspects.  First,
although this idiom is known to veteran Pythoneers, it is not at
all obvious to newbies -- either in terms of its algorithm
(inverting the ordering of item tuples) or its implementation
(using list comprehensions -- which are an advanced Python
feature.)  Second, it requires having to repeatedly type a lot of
"grunge", resulting in both tedium and mistakes.

We therefore would rather Python provide a method of sorting
dictionaries by value which would be both easy for newbies to
understand (or, better yet, not to *have to* understand) and
easier for all to use.


Rationale
=========

As Tim Peters has pointed out, this sort of thing brings on the
problem of trying to be all things to all people.  Therefore, we
will limit its scope to try to hit "the sweet spot".  Unusual
cases (e.g. sorting via a custom comparison function) can, of
course, be handled "manually" using present methods.

Here are some simple possibilities:

The ``items()`` method of dictionaries can be augmented with new
parameters having default values that provide for full
backwards-compatibility::

    (1) items(sort_by_values=0, reversed=0)

or maybe just::

    (2) items(sort_by_values=0)

since reversing a list is easy enough.

Alternatively, ``items()`` could simply let us control the (key, value)
order::

    (3) items(values_first=0)

Again, this is fully backwards-compatible.  It does less work than
the others, but it at least eases the most complicated/tricky part
of the sort-by-value problem: inverting the order of item tuples.
Using this is very simple::

    items = d.items(1)
    items.sort()
    items.reverse()         # (if desired)

The primary drawback of the preceding three approaches is the
additional overhead for the parameter-less ``items()`` case, due to
having to process default parameters.  (However, if one assumes
that ``items()`` gets used primarily for creating sort-by-value lists,
this is not really a drawback in practice.)

Alternatively, we might add a new dictionary method which somehow
embodies "sorting".  This approach offers two advantages.  First,
it avoids adding overhead to the ``items()`` method.  Second, it is
perhaps more accessible to newbies: when they go looking for a
method for sorting dictionaries, they hopefully run into this one,
and they will not have to understand the finer points of tuple
inversion and list sorting to achieve sort-by-value.

To allow the four basic possibilities of sorting by key/value and in
forward/reverse order, we could add this method::

    (4) sorted_items(by_value=0, reversed=0)

I believe the most common case would actually be ``by_value=1,
reversed=1``, but the defaults values given here might lead to
fewer surprises by users: ``sorted_items()`` would be the same as
``items()`` followed by ``sort()``.

Finally (as a last resort), we could use::

    (5) items_sorted_by_value(reversed=0)


Implementation
==============

The proposed dictionary methods would necessarily be implemented
in C.  Presumably, the implementation would be fairly simple since
it involves just adding a few calls to Python's existing
machinery.


Concerns
========

Aside from the run-time overhead already addressed in
possibilities 1 through 3, concerns with this proposal probably
will fall into the categories of "feature bloat" and/or "code
bloat".  However, I believe that several of the suggestions made
here will result in quite minimal bloat, resulting in a good
tradeoff between bloat and "value added".

Tim Peters has noted that implementing this in C might not be
significantly faster than implementing it in Python today.
However, the major benefits intended here are "accessibility" and
"ease of use", not "speed".  Therefore, as long as it is not
noticeably slower (in the case of plain ``items()``, speed need not be
a consideration.


References
==========

A related thread called "counting occurrences" appeared on
comp.lang.python in August, 2001.  This included examples of
approaches to systematizing the sort-by-value problem by
implementing it as reusable Python functions and classes.


Copyright
=========

This document has been placed in the public domain.


id: pep-0271.rst
content:
PEP: 271
Title: Prefixing sys.path by command line option
Author: FrÃ©dÃ©ric B. Giacometti <fred@arakne.com>
Status: Rejected
Type: Standards Track
Created: 15-Aug-2001
Python-Version: 2.2
Post-History:


Abstract
========

At present, setting the ``PYTHONPATH`` environment variable is the
only method for defining additional Python module search
directories.

This PEP introduces the '-P' valued option to the python command
as an alternative to ``PYTHONPATH``.


Rationale
=========

On Unix::

    python -P $SOMEVALUE

will be equivalent to::

   env PYTHONPATH=$SOMEVALUE python

On Windows 2K::

    python -P %SOMEVALUE%

will (almost) be equivalent to::

   set __PYTHONPATH=%PYTHONPATH% && set PYTHONPATH=%SOMEVALUE%\
      && python && set PYTHONPATH=%__PYTHONPATH%


Other Information
=================

This option is equivalent to the 'java -classpath' option.


When to use this option
=======================

This option is intended to ease and make more robust the use of
Python in test or build scripts, for instance.


Reference Implementation
========================

A patch implementing this is available from SourceForge::

    http://sourceforge.net/tracker/download.php?group_id=5470&atid=305470&file_id=6916&aid=429614

with the patch discussion at::

    http://sourceforge.net/tracker/?func=detail&atid=305470&aid=429614&group_id=5470


Copyright
=========

This document has been placed in the public domain.


id: pep-0517.rst
content:
PEP: 517
Title: A build-system independent format for source trees
Author: Nathaniel J. Smith <njs@pobox.com>,
        Thomas Kluyver <thomas@kluyver.me.uk>
BDFL-Delegate: Alyssa Coghlan <ncoghlan@gmail.com>
Discussions-To: distutils-sig@python.org
Status: Final
Type: Standards Track
Topic: Packaging
Created: 30-Sep-2015
Post-History: 01-Oct-2015, 25-Oct-2015, 19-May-2017, 11-Sep-2017
Resolution: https://mail.python.org/pipermail/distutils-sig/2017-September/031548.html

==========
 Abstract
==========

While ``distutils`` / ``setuptools`` have taken us a long way, they
suffer from three serious problems: (a) they're missing important
features like usable build-time dependency declaration,
autoconfiguration, and even basic ergonomic niceties like `DRY
<https://en.wikipedia.org/wiki/Don%27t_repeat_yourself>`_-compliant
version number management, and (b) extending them is difficult, so
while there do exist various solutions to the above problems, they're
often quirky, fragile, and expensive to maintain, and yet (c) it's
very difficult to use anything else, because distutils/setuptools
provide the standard interface for installing packages expected by
both users and installation tools like ``pip``.

Previous efforts (e.g. distutils2 or setuptools itself) have attempted
to solve problems (a) and/or (b). This proposal aims to solve (c).

The goal of this PEP is get distutils-sig out of the business of being
a gatekeeper for Python build systems. If you want to use distutils,
great; if you want to use something else, then that should be easy to
do using standardized methods. The difficulty of interfacing with
distutils means that there aren't many such systems right now, but to
give a sense of what we're thinking about see `flit
<https://github.com/takluyver/flit>`_ or `bento
<https://cournape.github.io/Bento/>`_. Fortunately, wheels have now
solved many of the hard problems here -- e.g. it's no longer necessary
that a build system also know about every possible installation
configuration -- so pretty much all we really need from a build system
is that it have some way to spit out standard-compliant wheels and
sdists.

We therefore propose a new, relatively minimal interface for
installation tools like ``pip`` to interact with package source trees
and source distributions.


=======================
 Terminology and goals
=======================

A *source tree* is something like a VCS checkout. We need a standard
interface for installing from this format, to support usages like
``pip install some-directory/``.

A *source distribution* is a static snapshot representing a particular
release of some source code, like ``lxml-3.4.4.tar.gz``. Source
distributions serve many purposes: they form an archival record of
releases, they provide a stupid-simple de facto standard for tools
that want to ingest and process large corpora of code, possibly
written in many languages (e.g. code search), they act as the input to
downstream packaging systems like Debian/Fedora/Conda/..., and so
forth. In the Python ecosystem they additionally have a particularly
important role to play, because packaging tools like ``pip`` are able
to use source distributions to fulfill binary dependencies, e.g. if
there is a distribution ``foo.whl`` which declares a dependency on
``bar``, then we need to support the case where ``pip install bar`` or
``pip install foo`` automatically locates the sdist for ``bar``,
downloads it, builds it, and installs the resulting package.

Source distributions are also known as *sdists* for short.

A *build frontend* is a tool that users might run that takes arbitrary
source trees or source distributions and builds wheels from them. The
actual building is done by each source tree's *build backend*. In a
command like ``pip wheel some-directory/``, pip is acting as a build
frontend.

An *integration frontend* is a tool that users might run that takes a
set of package requirements (e.g. a requirements.txt file) and
attempts to update a working environment to satisfy those
requirements. This may require locating, building, and installing a
combination of wheels and sdists. In a command like ``pip install
lxml==2.4.0``, pip is acting as an integration frontend.


==============
 Source trees
==============

There is an existing, legacy source tree format involving
``setup.py``. We don't try to specify it further; its de facto
specification is encoded in the source code and documentation of
``distutils``, ``setuptools``, ``pip``, and other tools. We'll refer
to it as the ``setup.py``\-style.

Here we define a new style of source tree based around the
``pyproject.toml`` file defined in :pep:`518`, extending the
``[build-system]`` table in that file with one additional key,
``build-backend``. Here's an example of how it would look::

    [build-system]
    # Defined by PEP 518:
    requires = ["flit"]
    # Defined by this PEP:
    build-backend = "flit.api:main"

``build-backend`` is a string naming a Python object that will be
used to perform the build (see below for details). This is formatted
following the same ``module:object`` syntax as a ``setuptools`` entry
point. For instance, if the string is ``"flit.api:main"`` as in the
example above, this object would be looked up by executing the
equivalent of::

    import flit.api
    backend = flit.api.main

It's also legal to leave out the ``:object`` part, e.g. ::

    build-backend = "flit.api"

which acts like::

    import flit.api
    backend = flit.api

Formally, the string should satisfy this grammar::

    identifier = (letter | '_') (letter | '_' | digit)*
    module_path = identifier ('.' identifier)*
    object_path = identifier ('.' identifier)*
    entry_point = module_path (':' object_path)?

And we import ``module_path`` and then lookup
``module_path.object_path`` (or just ``module_path`` if
``object_path`` is missing).

When importing the module path, we do *not* look in the directory containing the
source tree, unless that would be on ``sys.path`` anyway (e.g. because it is
specified in PYTHONPATH). Although Python automatically adds the working
directory to ``sys.path`` in some situations, code to resolve the backend should
not be affected by this.

If the ``pyproject.toml`` file is absent, or the ``build-backend``
key is missing, the source tree is not using this specification, and
tools should revert to the legacy behaviour of running ``setup.py`` (either
directly, or by implicitly invoking the ``setuptools.build_meta:__legacy__``
backend).

Where the ``build-backend`` key exists, this takes precedence and the source tree follows the format and
conventions of the specified backend (as such no ``setup.py`` is needed unless the backend requires it).
Projects may still wish to include a ``setup.py`` for compatibility with tools that do not use this spec.

This PEP also defines a ``backend-path`` key for use in ``pyproject.toml``, see
the "In-Tree Build Backends" section below. This key would be used as follows::

    [build-system]
    # Defined by PEP 518:
    requires = ["flit"]
    # Defined by this PEP:
    build-backend = "local_backend"
    backend-path = ["backend"]


Build requirements
==================

This PEP places a number of additional requirements on the "build requirements"
section of ``pyproject.toml``. These are intended to ensure that projects do
not create impossible to satisfy conditions with their build requirements.

- Project build requirements will define a directed graph of requirements
  (project A needs B to build, B needs C and D, etc.) This graph MUST NOT
  contain cycles.  If (due to lack of co-ordination between projects, for
  example) a cycle is present, front ends MAY refuse to build the project.
- Where build requirements are available as wheels, front ends SHOULD use these
  where practical, to avoid deeply nested builds.  However front ends MAY have
  modes where they do not consider wheels when locating build requirements, and
  so projects MUST NOT assume th


id: pep-0259.rst
content:
PEP: 259
Title: Omit printing newline after newline
Author: Guido van Rossum <guido@python.org>
Status: Rejected
Type: Standards Track
Created: 11-Jun-2001
Python-Version: 2.2
Post-History: 11-Jun-2001


Abstract
========

Currently, the ``print`` statement always appends a newline, unless a
trailing comma is used.  This means that if we want to print data
that already ends in a newline, we get two newlines, unless
special precautions are taken.

I propose to skip printing the newline when it follows a newline
that came from data.

In order to avoid having to add yet another magic variable to file
objects, I propose to give the existing 'softspace' variable an
extra meaning: a negative value will mean "the last data written
ended in a newline so no space *or* newline is required."


Problem
=======

When printing data that resembles the lines read from a file using
a simple loop, double-spacing occurs unless special care is taken::

    >>> for line in open("/etc/passwd").readlines():
    ... print line
    ...
    root:x:0:0:root:/root:/bin/bash

    bin:x:1:1:bin:/bin:

    daemon:x:2:2:daemon:/sbin:

    (etc.)

    >>>

While there are easy work-arounds, this is often noticed only
during testing and requires an extra edit-test roundtrip; the
fixed code is uglier and harder to maintain.


Proposed Solution
=================

In the ``PRINT_ITEM`` opcode in ``ceval.c``, when a string object is
printed, a check is already made that looks at the last character
of that string.  Currently, if that last character is a whitespace
character other than space, the softspace flag is reset to zero;
this suppresses the space between two items if the first item is a
string ending in newline, tab, etc. (but not when it ends in a
space).  Otherwise the softspace flag is set to one.

The proposal changes this test slightly so that softspace is set
to:

- ``-1`` -- if the last object written is a string ending in a
  newline

- ``0`` -- if the last object written is a string ending in a
  whitespace character that's neither space nor newline

- ``1`` -- in all other cases (including the case when the last
  object written is an empty string or not a string)

Then, the ``PRINT_NEWLINE`` opcode, printing of the newline is
suppressed if the value of softspace is negative; in any case the
softspace flag is reset to zero.


Scope
=====

This only affects printing of 8-bit strings.  It doesn't affect
Unicode, although that could be considered a bug in the Unicode
implementation.  It doesn't affect other objects whose string
representation happens to end in a newline character.


Risks
=====

This change breaks some existing code.  For example::

    print "Subject: PEP 259\n"
    print message_body

In current Python, this produces a blank line separating the
subject from the message body; with the proposed change, the body
begins immediately below the subject.  This is not very robust
code anyway; it is better written as::

    print "Subject: PEP 259"
    print
    print message_body

In the test suite, only ``test_StringIO`` (which explicitly tests for
this feature) breaks.


Implementation
==============

A patch relative to current CVS is here::

    http://sourceforge.net/tracker/index.php?func=detail&aid=432183&group_id=5470&atid=305470


Rejected
========

The user community unanimously rejected this, so I won't pursue
this idea any further.  Frequently heard arguments against
included:

- It is likely to break thousands of CGI scripts.

- Enough magic already (also: no more tinkering with 'print'
  please).


Copyright
=========

This document has been placed in the public domain.


id: pep-0701.rst
content:
PEP: 701
Title: Syntactic formalization of f-strings
Author: Pablo Galindo <pablogsal@python.org>,
        Batuhan Taskaya <batuhan@python.org>,
        Lysandros Nikolaou <lisandrosnik@gmail.com>,
        Marta GÃ³mez MacÃ­as <cyberwitch@google.com>
Discussions-To: https://discuss.python.org/t/pep-701-syntactic-formalization-of-f-strings/22046
Status: Accepted
Type: Standards Track
Created: 15-Nov-2022
Python-Version: 3.12
Post-History: `19-Dec-2022 <https://discuss.python.org/t/pep-701-syntactic-formalization-of-f-strings/22046>`__,
Resolution: `14-Mar-2023 <https://discuss.python.org/t/pep-701-syntactic-formalization-of-f-strings/22046/119>`__


Abstract
========

This document proposes to lift some of the restrictions originally formulated in
:pep:`498` and to provide a formalized grammar for f-strings that can be
integrated into the parser directly. The proposed syntactic formalization of
f-strings will have some small side-effects on how f-strings are parsed and
interpreted, allowing for a considerable number of advantages for end users and
library developers, while also dramatically reducing the maintenance cost of
the code dedicated to parsing f-strings.


Motivation
==========

When f-strings were originally introduced in :pep:`498`, the specification was
provided without providing a formal grammar for f-strings. Additionally, the
specification contains several restrictions that are imposed so the parsing of
f-strings could be implemented into CPython without modifying the existing
lexer. These limitations have been recognized previously and previous attempts
have been made to lift them in :pep:`536`, but `none of this work was ever implemented`_.
Some of these limitations (collected originally by :pep:`536`) are:

#. It is impossible to use the quote character delimiting the f-string
   within the expression portion::

    >>> f'Magic wand: { bag['wand'] }'
                                 ^
    SyntaxError: invalid syntax

#. A previously considered way around it would lead to escape sequences
   in executed code and is prohibited in f-strings::

    >>> f'Magic wand { bag[\'wand\'] } string'
    SyntaxError: f-string expression portion cannot include a backslash

#. Comments are forbidden even in multi-line f-strings::

    >>> f'''A complex trick: {
    ... bag['bag']  # recursive bags!
    ... }'''
    SyntaxError: f-string expression part cannot include '#'

#. Arbitrary nesting of expressions without expansion of escape sequences is
   available in many other languages that employ a string interpolation
   method that uses expressions instead of just variable names. Some examples:

   .. code-block:: text

      # Ruby
      "#{ "#{1+2}" }"

      # JavaScript
      `${`${1+2}`}`

      # Swift
      "\("\(1+2)")"

      # C#
      $"{$"{1+2}"}"

These limitations serve no purpose from a language user perspective and
can be lifted by giving f-string literals a regular grammar without exceptions
and implementing it using dedicated parse code.

The other issue that f-strings have is that the current implementation in
CPython relies on tokenising f-strings as ``STRING`` tokens and a post processing of
these tokens. This has the following problems:

#. It adds a considerable maintenance cost to the CPython parser. This is because
   the parsing code needs to be written by hand, which has historically led to a
   considerable number of inconsistencies and bugs. Writing and maintaining parsing
   code by hand in C has always been considered error prone and dangerous as it needs
   to deal with a lot of manual memory management over the original lexer buffers.

#. The f-string parsing code is not able to use the new improved error message mechanisms
   that the new PEG parser, originally introduced in :pep:`617`, has allowed. The
   improvements that these error messages brought has been greatly celebrated but
   unfortunately f-strings cannot benefit from them because they are parsed in a
   separate piece of the parsing machinery. This is especially unfortunate, since
   there are several syntactical features of f-strings that can be confusing due
   to the different implicit tokenization that happens inside the expression
   part (for instance ``f"{y:=3}"`` is not an assignment expression).

#. Other Python implementations have no way to know if they have implemented
   f-strings correctly because contrary to other language features, they are not
   part of the :ref:`official Python grammar <python:f-strings>`.
   This is important because several prominent
   alternative implementations are using CPython's PEG parser, `such as PyPy`_,
   and/or are basing their grammars on the official PEG grammar. The
   fact that f-strings use a separate parser prevents these alternative implementations
   from leveraging the official grammar and benefiting from improvements in error messages derived
   from the grammar.


A version of this proposal was originally `discussed on Python-Dev`_  and
`presented at the Python Language Summit 2022`_ where it was enthusiastically
received.

Rationale
=========

By building on top of the new Python PEG Parser (:pep:`617`), this PEP proposes
to redefine âf-stringsâ, especially emphasizing the clear separation of the
string component and the expression (or replacement, ``{...}``) component. :pep:`498`
summarizes the syntactical part of âf-stringsâ as the following:

    In Python source code, an f-string is a literal string, prefixed with âfâ, which
    contains expressions inside braces. The expressions are replaced with their values.

However, :pep:`498` also contained a formal list of exclusions on what
can or cannot be contained inside the expression component (primarily due to the
limitations of the existing parser). By clearly establishing the formal grammar, we
now also have the ability to define the expression component of an f-string as truly "any
applicable Python expression" (in that particular context) without being bound
by the limitations imposed by the details of our implementation.

The formalization effort and the premise above also has a significant benefit for
Python programmers due to its ability to simplify and eliminate the obscure
limitations. This reduces the mental burden and the cognitive complexity of
f-string literals (as well as the Python language in general).

#. The expression component can include any string literal that a normal Python expression
   can include. This opens up the possibility of nesting string literals (formatted or
   not) inside the expression component of an f-string with the same quote type (and length)::

    >>> f"These are the things: {", ".join(things)}"

    >>> f"{source.removesuffix(".py")}.c: $(srcdir)/{source}"

    >>> f"{f"{f"infinite"}"}" + " " + f"{f"nesting!!!"}"

   This "feature" is not universally agreed to be desirable, and some users find this unreadable.
   For a discussion on the different views on this, see the `considerations regarding quote reuse`_ section.

#. Another issue that has felt unintuitive to most is the lack of support for backslashes
   within the expression component of an f-string. One example that keeps coming up is including
   a newline character in the expression part for joining containers. For example::

    >>> a = ["hello", "world"]
    >>> f"{'\n'.join(a)}"
    File "<stdin>", line 1
        f"{'\n'.join(a)}"
                        ^
    SyntaxError: f-string expression part cannot include a backslash

   A common work-around for this was to either assign the newline to an intermediate variable or
   pre-create the whole string prior to creating the f-string::

    >>> a = ["hello", "world"]
    >>> joined = '\n'.join(a)
    >>> f"{joined}"
    'hello\nworld'

   It only feels natural to allow backslashes in the expression part now that the new PEG parser
   can easily support it.

    >>> a = ["hello", "world"]
    >>> f"{'\n'.join(a)}"
    'hello\nworld'

#. Before the changes proposed in this do


id: pep-0715.rst
content:
PEP: 715
Title: Disabling bdist_egg distribution uploads on PyPI
Author: William Woodruff <william@yossarian.net>
Sponsor: Donald Stufft <donald@stufft.io>
PEP-Delegate: Donald Stufft <donald@stufft.io>
Discussions-To: https://discuss.python.org/t/27610
Status: Final
Type: Standards Track
Topic: Packaging
Created: 06-Jun-2023
Post-History: `09-Jun-2023 <https://discuss.python.org/t/27610>`__
Resolution: `24-Jun-2023 <https://discuss.python.org/t/pep-715-disabling-bdist-egg-distribution-uploads-on-pypi/27610/13>`__

Abstract
========

This PEP recommends deprecating and then disabling new uploads of the
``bdist_egg`` distribution type on PyPI. In a parallel move, this PEP recommends
deprecating and then disabling new uploads of distribution filenames that have
the ``.egg`` suffix.

After this PEP, PyPI will only accept new uploads of the ``sdist``
and ``bdist_wheel`` types, corresponding to files with ``.tar.gz``/``.zip`` and
``.whl`` suffixes respectively.

This PEP does not recommend removing or otherwise affecting any previously
uploaded ``bdist_egg`` distributions or files with the ``.egg`` suffix.

Rationale
=========

Previous Work
-------------

The groundwork for this proposal was established with :pep:`527`, which
proposed deprecating and eventually removing upload support for a handful
of un(der)used file extensions and distribution types.

In particular, :pep:`527` proposed the removal of the ``bdist_dumb``,
``bdist_rpm``, ``bdist_dmg``, ``bdist_msi``, and ``bdist_wininst`` distribution
types, as well as the ``.tar``, ``.tar.bz2``, ``.tar.xz``, ``.tar.Z``,
``.tgz``, and ``.tbz`` file extensions on distribution filenames.

:pep:`527` was fully enacted with
`PR #7529 <https://github.com/pypi/warehouse/pull/7529>`_ to Warehouse,
which was merged on 13 April 2020.

The ``bdist_egg`` format
------------------------

The ``bdist_egg`` filetype identifies distributions in the
:term:`egg format <packaging:Egg>`. The
egg format was introduced by setuptools in 2004 and is roughly equivalent
in functionality to the
:term:`wheel format <packaging:Wheel>`
first introduced by :pep:`427` in 2012
as :ref:`the standardized format <packaging:binary-distribution-format>`
for :term:`built distributions <packaging:Built Distribution>`.

Despite its longevity, the egg format has had
`limited adoption on PyPI <https://github.com/pypi/warehouse/issues/10653>`_.
Some observations from that issue:

* In the month of May 2023, ``bdist_egg`` uploads accounted for 0.2% of all
  distribution uploads to PyPI;
* ``pip`` deprecated its ``--egg`` option in 2016;
* ``setuptools`` has considered egg support deprecated since 2019;
* ``build`` only supports the ``sdist`` and ``bdist_wheel`` filetypes.

Given the above, this PEP proposes the removal of the ``bdist_egg`` format
under the same justifications presented in :pep:`527`, namely:

* Egg distributions are of limited use to the broader ecosystem and
  therefore represent a non-reciprocal maintenance burden;
* Having an additional built distribution format
  is confusing to end users, who may
  mistakenly pick it over the wheel format;

This PEP *additionally* offers an argument for removal rooted in
standardization and duplication: the egg format is not standardized by any
PEP or other community standard, and overlaps heavily with its standardized
and well-supported alternative (wheel).

The ``.egg`` file extension
---------------------------

The ``.egg`` file extension is used exclusively for distributions of the
``bdist_egg`` format. As such, it serves no purpose in a scenario where
PyPI disables new distribution uploads for ``bdist_egg`` distributions.

Removal Process
===============

This PEP does **NOT** propose removing any existing files from PyPI, only
disallowing new ones from being uploaded.

PyPI will provide a deprecation period of one month. At the beginning
of the deprecation period, maintainers of projects that have uploaded one or
more egg distributions since 1 Jan 2023 will receive a one-time email informing
them of the upcoming end of support for egg distribution uploads.

During the deprecation period, users will continue to be allowed to upload egg
distributions to new and existing projects. Uploading an egg distribution
during this period will also send all maintainers of the project
a similar email as above, reminding them of the upcoming end of support.

After the deprecation period, support for uploading egg distributions will
cease to exist on PyPI.

Prior Art
---------

The removal process above was based on that of :pep:`527`, with the following
changes:

* All projects will be continue to be allowed to upload eggs
  during the deprecation period, not just those that have done so previously.
* Uploading an egg during the deprecation period will also trigger an email
  to maintainers, in addition to the one-time email at the beginning.

Backwards Compatibility
=======================

Limited Impact
--------------

As noted in the rationale section, this PEP is expected to have no impact
on the overwhelming majority of PyPI users and projects, and there has been
`substantial community coordination <https://github.com/pypi/warehouse/issues/10653>`_
over the past 1 1/2 years to minimize the impact on the few last use cases.

Hosted Files
------------

This PEP does **NOT** propose the removal of any egg distributions that have
already been uploaded to PyPI. All previously uploaded egg distributions will
remain downloadable, ensuring that existing users will continue to be able
to download them.

Deprecation Period
------------------

This PEP uses the removal process documented above, which specifies
a deprecation period of 1 month for projects that have previously uploaded
egg distributions to PyPI.

After the end of the deprecation period, support for uploading new egg
distributions will cease to exist on PyPI.

Security Implications
=====================

This PEP does not identify any positive or negative security implications
associated with removing upload support for egg distributions.

How To Teach This
=================

As part of the removal process, PyPI will send emails to all maintainers of
projects that have previously uploaded egg distributions in 2023.

Additionally, PyPI will write a post on the
`PyPI blog <https://blog.pypi.org/>`_ that publicly announces the deprecation
period's start and end.

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0729.rst
content:
PEP: 729
Title: Typing governance process
Author: Jelle Zijlstra <jelle.zijlstra@gmail.com>, Shantanu Jain <hauntsaninja at gmail.com>
Discussions-To: https://discuss.python.org/t/pep-729-typing-governance-process/35362
Status: Active
Type: Process
Topic: Governance, Typing
Created: 19-Sep-2023
Post-History: `04-Oct-2023 <https://discuss.python.org/t/pep-729-typing-governance-process/35362>`__,
              `20-Sep-2023 <https://discuss.python.org/t/proposed-new-typing-governance-process/34244>`__
Resolution: `20-Nov-2023 <https://discuss.python.org/t/pep-729-typing-governance-process/35362/12>`__

Abstract
========

This PEP proposes a new way to govern the Python type system: a council that is responsible
for maintaining and developing the Python type system. The council will maintain a
specification and conformance test suite and will initially be appointed by the Python Steering Council.

Motivation
==========

The Python type system was created by :pep:`484`, almost ten years ago. The type
system is now widely used, and typing has become an important tool for writing
good, maintainable Python code. Many changes have been made to the type system to cover
more use cases and improve usability. Several type checkers have been created, each
with their own strengths. The type annotation syntax has driven several major innovations
in the Python ecosystem, such as the popular :mod:`dataclasses` module, runtime type
checking and validation by packages such as `Pydantic <https://docs.pydantic.dev/latest/>`__,
and static compilation by tools such as `mypyc <https://mypyc.readthedocs.io/en/latest/>`__.

However, as the type system has grown, several interrelated problems with the current
way to manage the type system have become apparent.

PEPs are the only specification
--------------------------------

The Python type system was initially created by a PEP (:pep:`484`), and
changes to the type system are still made by PEPs. The specification for
the Python type system, to the extent there is one, consists of this series
of PEPs. But Standards Track PEPs aren't meant to be living documents
or specifications; they are change proposals.

An example may illustrate the problem here. Around the same time
as the introduction of the :mod:`typing` module by :pep:`484`, :pep:`3156`
introduced the :mod:`asyncio` module, another major new feature that has
been instrumental to the success of Python 3. Both modules
have evolved greatly since their initial creation and inspired changes to the
core language.

However, :mod:`asyncio` and :mod:`typing` are different in an essential aspect:
a user who uses :mod:`asyncio` interacts only with the standard library itself,
while a user of :mod:`typing` has to also think about an external tool, the type
checker. The Python language reference covers the symbols in the typing module, but does
not (and should not) go into detail on how the full type system should be
interpreted by type checkers. That material currently exists only in the PEPs.

This problem is shared by the packaging ecosystem, which attempts to solve it
by maintaining a separate set of `PyPA specifications <https://packaging.python.org/en/latest/specifications/>`_.

It's hard to evolve the specification
-------------------------------------

Because the PEPs are the only specification we have, anything that could be seen
as a change to the specification would theoretically require a new PEP. But that
is often too heavy a process for a small change. Sometimes changes are made
directly to old PEPs instead, but that goes against the idea that accepted and
implemented PEPs become historical documents that should no longer be changed.

Some concrete examples include:

* :pep:`484` explicitly says that :data:`typing.NoReturn` cannot be used in
  argument annotations. Nevertheless, type checkers have long accepted such
  usage.
* A `2023 discussion <https://discuss.python.org/t/pep-561-clarification-regarding-n/32875>`_
  noted that :pep:`561`'s description of partial stubs is unclear, and
  major type checkers did not implement it exactly as specified.
* The widely used third-party ``typing_extensions`` package provides backports of new
  type system features. Type checkers are expected to treat symbols in this
  module the same as symbols in :mod:`typing`, but this is not explicitly
  specified in any of the PEPs.

The type system is underspecified
---------------------------------

While the PEPs provide a specification, they are often not sufficiently precise
(sometimes intentionally so). This is especially true as the combinatorial
complexity of the type system has grown.

It ends up falling to individual type checkers to decide how to navigate
underspecified areas. In cases where type checkers informally coordinate, this
results in de facto standards that aren't clearly recorded anywhere, making
the type system less accessible to newcomers. For example:

* How ``@overload`` matching works
* How ``ParamSpec`` `should work <https://github.com/python/typing/discussions/946>`_ `with methods <https://github.com/microsoft/pyright/issues/3954#issuecomment-1250098464>`_
* The concept of `recursive aliases <https://github.com/python/typing/issues/182>`_
* Semantics of `variable initialization <https://mail.python.org/archives/list/typing-sig@python.org/thread/GYVM5KEE6URE6PAH7UTK6324M7GWSFQS/#SY3KPJCAW4UTOOCH3XRJYROSGDEGOTWI>`_
* `Reachability semantics <https://github.com/python/mypy/issues/15158#issuecomment-1677915108>`__
  of annotations on ``__exit__``
* `Symbol visibility <https://mail.python.org/archives/list/typing-sig@python.org/thread/YLJPWECBNPD2K4TRIBRIPISNUZJCRREY/#OX4GLBQOOCMRE5YPZEY3R3XNV6DD7XLW>`_
* Use of `NoReturn for exhaustiveness checking <https://github.com/python/mypy/issues/5818>`_

The Steering Council is not well-placed to solve the above problems
-------------------------------------------------------------------

The SC has the entire language in its remit, and is not well-placed to make
decisions that are purely about the type system -- if only because they don't have
the time to deal with type system arcana alongside their other responsibilities.
This is similar in spirit to the reasons why the Steering Council sometimes uses
PEP delegation.

Endorsements
------------

This PEP was endorsed by maintainers of all major
type checkers, including `Rebecca Chen (pytype) <https://discuss.python.org/t/pep-729-typing-governance-process/35362/3>`__,
`Eric Traut (Pyright) <https://discuss.python.org/t/pep-729-typing-governance-process/35362/4>`__,
and privately by maintainers of mypy and Pyre.

Specification
=============

We propose the creation of a new group, the Typing Council. This group will
be responsible for developing and maintaining the Python type system, and
for solving the above problems.

The "operations and process" section describes how this group would operate and
be governed.

The more exciting "projects" section describes solutions to the above problems
that the Typing Council could shepherd.

Mandate
-------

The Typing Council's mandate is to ensure that the Python type system is:

* **Useful**: The type system should serve common use cases. As identified
  by :pep:`484`, the primary use case is static analysis, but there are others,
  such as runtime type checking, static compilation, IDE support, and documentation.
  The Typing Council should consider all of these use cases when making decisions,
  and be open to supporting additional use cases as they come up.
* **Usable**: The type system should be easy to use for Python developers. It
  should be ergonomic to write well-typed Python code that is accepted by type
  checkers. There should be good documentation for the type system.
* **Stable**: As the type system matures, users should be able to rely on their
  typed code continuing to work and be able to trust their mental model for the
  type system. Changes should be made with care and in a way
  that minimizes disruption. Ne


id: pep-0675.rst
content:
PEP: 675
Title: Arbitrary Literal String Type
Author: Pradeep Kumar Srinivasan <gohanpra@gmail.com>, Graham Bleaney <gbleaney@gmail.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://mail.python.org/archives/list/typing-sig@python.org/thread/VB74EHNM4RODDFM64NEEEBJQVAUAWIAW/
Status: Final
Type: Standards Track
Topic: Typing
Created: 30-Nov-2021
Python-Version: 3.11
Post-History: 07-Feb-2022
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/XEOOSSPNYPGZ5NXOJFPLXG2BTN7EVRT5/

.. canonical-typing-spec:: :ref:`typing:literalstring` and
                           :py:data:`typing.LiteralString`

Abstract
========

There is currently no way to specify, using type annotations, that a
function parameter can be of any literal string type. We have to
specify a precise literal string type, such as
``Literal["foo"]``. This PEP introduces a supertype of literal string
types: ``LiteralString``. This allows a function to accept arbitrary
literal string types, such as ``Literal["foo"]`` or
``Literal["bar"]``.


Motivation
==========

Powerful APIs that execute SQL or shell commands often recommend that
they be invoked with literal strings, rather than arbitrary user
controlled strings. There is no way to express this recommendation in
the type system, however, meaning security vulnerabilities sometimes
occur when developers fail to follow it. For example, a naive way to
look up a user record from a database is to accept a user id and
insert it into a predefined SQL query:

::

    def query_user(conn: Connection, user_id: str) -> User:
        query = f"SELECT * FROM data WHERE user_id = {user_id}"
        conn.execute(query)
        ...  # Transform data to a User object and return it

    query_user(conn, "user123")  # OK.

However, the user-controlled data ``user_id`` is being mixed with the
SQL command string, which means a malicious user could run arbitrary
SQL commands:

.. code-block::
   :class: bad

    # Delete the table.
    query_user(conn, "user123; DROP TABLE data;")

    # Fetch all users (since 1 = 1 is always true).
    query_user(conn, "user123 OR 1 = 1")


To prevent such SQL injection attacks, SQL APIs offer parameterized
queries, which separate the executed query from user-controlled data
and make it impossible to run arbitrary queries. For example, with
`sqlite3 <https://docs.python.org/3/library/sqlite3.html>`_, our
original function would be written safely as a query with parameters:

::

    def query_user(conn: Connection, user_id: str) -> User:
        query = "SELECT * FROM data WHERE user_id = ?"
        conn.execute(query, (user_id,))
        ...

The problem is that there is no way to enforce this
discipline. sqlite3's own `documentation
<https://docs.python.org/3/library/sqlite3.html>`_ can only admonish
the reader to not dynamically build the ``sql`` argument from external
input; the API's authors cannot express that through the type
system. Users can (and often do) still use a convenient f-string as
before and leave their code vulnerable to SQL injection.

Existing tools, such as the popular security linter `Bandit
<https://github.com/PyCQA/bandit/blob/aac3f16f45648a7756727286ba8f8f0cf5e7d408/bandit/plugins/django_sql_injection.py#L102>`_,
attempt to detect unsafe external data used in SQL APIs, by inspecting
the AST or by other semantic pattern-matching. These tools, however,
preclude common idioms like storing a large multi-line query in a
variable before executing it, adding literal string modifiers to the
query based on some conditions, or transforming the query string using
a function. (We survey existing tools in the `Rejected Alternatives`_
section.) For example, many tools will detect a false positive issue
in this benign snippet:


::

    def query_data(conn: Connection, user_id: str, limit: bool) -> None:
        query = """
            SELECT
                user.name,
                user.age
            FROM data
            WHERE user_id = ?
        """
        if limit:
            query += " LIMIT 1"

        conn.execute(query, (user_id,))

We want to forbid harmful execution of user-controlled data while
still allowing benign idioms like the above and not requiring extra
user work.

To meet this goal, we introduce the ``LiteralString`` type, which only
accepts string values that are known to be made of literals. This is a
generalization of the ``Literal["foo"]`` type from :pep:`586`.
A string of type
``LiteralString`` cannot contain user-controlled data. Thus, any API
that only accepts ``LiteralString`` will be immune to injection
vulnerabilities (with `pragmatic limitations <Appendix B:
Limitations_>`_).

Since we want the ``sqlite3`` ``execute`` method to disallow strings
built with user input, we would make its `typeshed stub
<https://github.com/python/typeshed/blob/1c88ceeee924ec6cfe05dd4865776b49fec299e6/stdlib/sqlite3/dbapi2.pyi#L153>`_
accept a ``sql`` query that is of type ``LiteralString``:

::

    from typing import LiteralString

    def execute(self, sql: LiteralString, parameters: Iterable[str] = ...) -> Cursor: ...


This successfully forbids our unsafe SQL example. The variable
``query`` below is inferred to have type ``str``, since it is created
from a format string using ``user_id``, and cannot be passed to
``execute``:

::

    def query_user(conn: Connection, user_id: str) -> User:
        query = f"SELECT * FROM data WHERE user_id = {user_id}"
        conn.execute(query)  # Error: Expected LiteralString, got str.
        ...

The method remains flexible enough to allow our more complicated
example:

::

    def query_data(conn: Connection, user_id: str, limit: bool) -> None:
        # This is a literal string.
        query = """
            SELECT
                user.name,
                user.age
            FROM data
            WHERE user_id = ?
        """

        if limit:
            # Still has type LiteralString because we added a literal string.
            query += " LIMIT 1"

        conn.execute(query, (user_id,))  # OK

Notice that the user did not have to change their SQL code at all. The
type checker was able to infer the literal string type and complain
only in case of violations.

``LiteralString`` is also useful in other cases where we want strict
command-data separation, such as when building shell commands or when
rendering a string into an HTML response without escaping (see
`Appendix A: Other Uses`_). Overall, this combination of strictness
and flexibility makes it easy to enforce safer API usage in sensitive
code without burdening users.

Usage statistics
----------------

In a sample of open-source projects using ``sqlite3``, we found that
``conn.execute`` was called `~67% of the time
<https://grep.app/search?q=conn%5C.execute%5C%28%5Cs%2A%5B%27%22%5D&regexp=true&filter[lang][0]=Python>`_
with a safe string literal and `~33% of the time
<https://grep.app/search?current=3&q=conn%5C.execute%5C%28%5Ba-zA-Z_%5D%2B%5C%29&regexp=true&filter[lang][0]=Python>`_
with a potentially unsafe, local string variable. Using this PEP's
literal string type along with a type checker would prevent the unsafe
portion of that 33% of cases (ie. the ones where user controlled data
is incorporated into the query), while seamlessly allowing the safe
ones to remain.

Rationale
=========

Firstly, why use *types* to prevent security vulnerabilities?

Warning users in documentation is insufficient - most users either
never see these warnings or ignore them. Using an existing dynamic or
static analysis approach is too restrictive - these prevent natural
idioms, as we saw in the `Motivation`_ section (and will discuss more
extensively in the `Rejected Alternatives`_ section). The typing-based
approach in this PEP strikes a user-friendly balance between
strictness and flexibility.

Runtime approaches do not work because, at runtime, the query string
is a plain ``str``. While we could prevent some exploits using
heuristics, such as regex-f


id: pep-0661.rst
content:
PEP: 661
Title: Sentinel Values
Author: Tal Einat <tal@python.org>
Discussions-To: https://discuss.python.org/t/pep-661-sentinel-values/9126
Status: Deferred
Type: Standards Track
Created: 06-Jun-2021
Post-History: `20-May-2021 <https://discuss.python.org/t/sentinel-values-in-the-stdlib/8810>`__, `06-Jun-2021 <https://discuss.python.org/t/pep-661-sentinel-values/9126>`__


TL;DR: See the `Specification`_ and `Reference Implementation`_.


Abstract
========

Unique placeholder values, commonly known as "sentinel values", are common in
programming.  They have many uses, such as for:

* Default values for function arguments, for when a value was not given::

    def foo(value=None):
        ...

* Return values from functions when something is not found or unavailable::

    >>> "abc".find("d")
    -1

* Missing data, such as NULL in relational databases or "N/A" ("not
  available") in spreadsheets

Python has the special value ``None``, which is intended to be used as such
a sentinel value in most cases.  However, sometimes an alternative sentinel
value is needed, usually when it needs to be distinct from ``None`` since
``None`` is a valid value in that context.  Such cases are common enough that
several idioms for implementing such sentinels have arisen over the years, but
uncommon enough that there hasn't been a clear need for standardization.
However, the common implementations, including some in the stdlib, suffer from
several significant drawbacks.

This PEP proposes adding a utility for defining sentinel values, to be used
in the stdlib and made publicly available as part of the stdlib.

Note: Changing all existing sentinels in the stdlib to be implemented this
way is not deemed necessary, and whether to do so is left to the discretion
of the maintainers.


Motivation
==========

In May 2021, a question was brought up on the python-dev mailing list
[1]_ about how to better implement a sentinel value for
``traceback.print_exception``.  The existing implementation used the
following common idiom::

    _sentinel = object()

However, this object has an uninformative and overly verbose repr, causing the
function's signature to be overly long and hard to read::

    >>> help(traceback.print_exception)
    Help on function print_exception in module traceback:

    print_exception(exc, /, value=<object object at
    0x000002825DF09650>, tb=<object object at 0x000002825DF09650>,
    limit=None, file=None, chain=True)

Additionally, two other drawbacks of many existing sentinels were brought up
in the discussion:

1. Some do not have a distinct type, hence it is impossible to define clear
   type signatures for functions with such sentinels as default values.
2. They behave unexpectedly after being copied or unpickled, due to a separate
   instance being created and thus comparisons using ``is`` failing.

In the ensuing discussion, Victor Stinner supplied a list of currently used
sentinel values in the Python standard library [2]_.  This showed that the
need for sentinels is fairly common, that there are various implementation
methods used even within the stdlib, and that many of these suffer from at
least one of the three above drawbacks.

The discussion did not lead to any clear consensus on whether a standard
implementation method is needed or desirable, whether the drawbacks mentioned
are significant, nor which kind of implementation would be good.  The author
of this PEP created an issue on bugs.python.org (now a GitHub issue [3]_)
suggesting options for improvement, but that focused on only a single
problematic aspect of a few cases, and failed to gather any support.

A poll [4]_ was created on discuss.python.org to get a clearer sense of
the community's opinions. After nearly two weeks, significant further,
discussion, and 39 votes, the poll's results were not conclusive. 40% had
voted for "The status-quo is fine / thereâs no need for consistency in
this", but most voters had voted for one or more standardized solutions.
Specifically, 37% of the voters chose "Consistent use of a new, dedicated
sentinel factory / class / meta-class, also made publicly available in the
stdlib".

With such mixed opinions, this PEP was created to facilitate making a decision
on the subject.

While working on this PEP, iterating on various options and implementations
and continuing discussions, the author has come to the opinion that a simple,
good implementation available in the standard library would be worth having,
both for use in the standard library itself and elsewhere.


Rationale
=========

The criteria guiding the chosen implementation were:

1. The sentinel objects should behave as expected by a sentinel object: When
   compared using the ``is`` operator, it should always be considered
   identical to itself but never to any other object.
2. Creating a sentinel object should be a simple, straightforward one-liner.
3. It should be simple to define as many distinct sentinel values as needed.
4. The sentinel objects should have a clear and short repr.
5. It should be possible to use clear type signatures for sentinels.
6. The sentinel objects should behave correctly after copying and/or
   unpickling.
7. Such sentinels should work when using CPython 3.x and PyPy3, and ideally
   also with other implementations of Python.
8. As simple and straightforward as possible, in implementation and especially
   in use.  Avoid this becoming one more special thing to learn when learning
   Python.  It should be easy to find and use when needed, and obvious enough
   when reading code that one would normally not feel a need to look up its
   documentation.

With so many uses in the Python standard library [2]_, it would be useful to
have an implementation in the standard library, since the stdlib cannot use
implementations of sentinel objects available elsewhere (such as the
``sentinels`` [5]_ or ``sentinel`` [6]_ PyPI packages).

After researching existing idioms and implementations, and going through many
different possible implementations, an implementation was written which meets
all of these criteria (see `Reference Implementation`_).


Specification
=============

A new ``Sentinel`` class will be added to a new ``sentinellib`` module.

    >>> from sentinellib import Sentinel
    >>> MISSING = Sentinel('MISSING')
    >>> MISSING
    MISSING

Checking if a value is such a sentinel *should* be done using the ``is``
operator, as is recommended for ``None``.  Equality checks using ``==`` will
also work as expected, returning ``True`` only when the object is compared
with itself.  Identity checks such as ``if value is MISSING:`` should usually
be used rather than boolean checks such as ``if value:`` or ``if not value:``.

Sentinel instances are "truthy", i.e. boolean evaluation will result in
``True``.  This parallels the default for arbitrary classes, as well as the
boolean value of ``Ellipsis``. This is unlike ``None``, which is "falsy".

The names of sentinels are unique within each module.  When calling
``Sentinel()`` in a module where a sentinel with that name was already
defined, the existing sentinel with that name will be returned.  Sentinels
with the same name defined in different modules will be distinct from each
other.

Creating a copy of a sentinel object, such as by using ``copy.copy()`` or by
pickling and unpickling, will return the same object.

``Sentinel()`` will also accept a single optional argument, ``module_name``.
This should normally not need to be supplied, as ``Sentinel()`` will usually
be able to recognize the module in which it was called.  ``module_name``
should be supplied only in unusual cases when this automatic recognition does
not work as intended, such as perhaps when using Jython or IronPython.  This
parallels the designs of ``Enum`` and ``namedtuple``.  For more details, see
:pep:`435`.

The ``Sentinel`` class may not be sub-classed, to avoid the greater complexity
of supporting subclassing.

Ordering comparisons are undefined for se


id: pep-0649.rst
content:
PEP: 649
Title: Deferred Evaluation Of Annotations Using Descriptors
Author: Larry Hastings <larry@hastings.org>
Discussions-To: https://discuss.python.org/t/pep-649-deferred-evaluation-of-annotations-tentatively-accepted/21331/
Status: Accepted
Type: Standards Track
Topic: Typing
Created: 11-Jan-2021
Python-Version: 3.14
Post-History: `11-Jan-2021 <https://mail.python.org/archives/list/python-dev@python.org/thread/5QMMCRF4HTRRNJV56CGHVI5GRHVBDGQO/>`__,
              `12-Apr-2021 <https://mail.python.org/archives/list/python-dev@python.org/thread/QSASX6PZ3LIIFIANHQQFS752BJYFUFPY/>`__,
              `18-Apr-2021 <https://mail.python.org/archives/list/python-dev@python.org/thread/WUZGTGE43T7XV3EUGT6AN2N52OD3U7AE/>`__,
              `09-Aug-2021 <https://mail.python.org/archives/list/python-dev@python.org/thread/2MEOWHCVDLPABOBLYUGRXVOOOBYOLLU6/>`__,
              `20-Oct-2021 <https://mail.python.org/archives/list/python-dev@python.org/thread/SZLWVYV2HPLU6AH7DOUD7DWFUGBJGQAY/>`__,
              `20-Oct-2021 <https://discuss.python.org/t/type-annotations-pep-649-and-pep-563/11363>`__,
              `17-Nov-2021 <https://mail.python.org/archives/list/python-dev@python.org/thread/VIZEBX5EYMSYIJNDBF6DMUMZOCWHARSO/>`__,
              `15-Mar-2022 <https://discuss.python.org/t/finding-edge-cases-for-peps-484-563-and-649-type-annotations/14314>`__,
              `23-Nov-2022 <https://discuss.python.org/t/pep-649-deferred-evaluation-of-annotations-tentatively-accepted/21331>`__,
              `07-Feb-2023 <https://discuss.python.org/t/two-polls-on-how-to-revise-pep-649/23628>`__,
              `11-Apr-2023 <https://discuss.python.org/t/a-massive-pep-649-update-with-some-major-course-corrections/25672>`__,
Replaces: 563
Resolution: `08-May-2023 <https://discuss.python.org/t/pep-649-deferred-evaluation-of-annotations-tentatively-accepted/21331/43>`__

********
Abstract
********

Annotations are a Python technology that allows expressing
type information and other metadata about Python functions,
classes, and modules.  But Python's original semantics
for annotations required them to be eagerly evaluated,
at the time the annotated object was bound.  This caused
chronic problems for static type analysis users using
"type hints", due to forward-reference and circular-reference
problems.

Python solved this by accepting :pep:`563`, incorporating
a new approach called "stringized annotations" in which
annotations were automatically converted into strings by
Python.  This solved the forward-reference and circular-reference
problems, and also fostered intriguing new uses for annotation
metadata.  But stringized annotations in turn caused chronic
problems for runtime users of annotations.

This PEP proposes a new and comprehensive third approach
for representing and computing annotations.  It adds a new
internal mechanism for lazily computing annotations on demand,
via a new object method called ``__annotate__``.
This approach, when combined with a novel technique for
coercing annotation values into alternative formats, solves
all the above problems, supports all existing use cases,
and should foster future innovations in annotations.


********
Overview
********

This PEP adds a new dunder attribute to the objects that
support annotations--functions, classes, and modules.
The new attribute is called ``__annotate__``, and is
a reference to a function which computes and returns
that object's annotations dict.

At compile time, if the definition of an object includes
annotations, the Python compiler will write the expressions
computing the annotations into its own function.  When run,
the function will return the annotations dict.  The Python
compiler then stores a reference to this function in
``__annotate__`` on the object.

Furthermore, ``__annotations__`` is redefined to be a
"data descriptor" which calls this annotation function once
and caches the result.

This mechanism delays the evaluation of annotations expressions
until the annotations are examined, which solves many circular
reference problems.

This PEP also defines new functionality for two functions
in the Python standard library:
``inspect.get_annotations`` and ``typing.get_type_hints``.
The functionality is accessed via a new keyword-only parameter,
``format``.  ``format`` allows the user to request
the annotations from these functions
in a specific format.
Format identifiers are always predefined integer values.
The formats defined by this PEP are:


* ``inspect.VALUE = 1``

  The default value.
  The function will return the conventional Python
  values for the annotations.  This format is identical
  to the return value for these functions under Python 3.11.

* ``inspect.FORWARDREF = 2``

  The function will attempt to return the conventional
  Python values for the annotations.  However, if it
  encounters an undefined name, or a free variable that
  has not yet been associated with a value, it dynamically
  creates a proxy object (a ``ForwardRef``) that substitutes
  for that value in the expression, then continues evaluation.
  The resulting dict may contain a mixture of proxies and
  real values.  If all real values are defined at the time
  the function is called, ``inspect.FORWARDREF`` and
  ``inspect.VALUE`` produce identical results.

* ``inspect.SOURCE = 3``

  The function will produce an annotation dictionary
  where the values have been replaced by strings containing
  the original source code for the annotation expressions.
  These strings may only be approximate, as they may be
  reverse-engineered from another format, rather than
  preserving the original source code, but the differences
  will be minor.

If accepted, this PEP would *supersede* :pep:`563`,
and :pep:`563`'s behavior would be deprecated and
eventually removed.


Comparison Of Annotation Semantics
==================================

.. note:: The code presented in this section is simplified
   for clarity, and is intentionally inaccurate in some
   critical aspects.  This example is intended merely to
   communicate the high-level concepts involved without
   getting lost in the details.  But readers should note
   that the actual implementation is quite different in
   several important ways.  See the Implementation_
   section later in this PEP for a far more accurate
   description of what this PEP proposes from a technical
   level.

Consider this example code:

.. code-block::

    def foo(x: int = 3, y: MyType = None) -> float:
        ...
    class MyType:
        ...
    foo_y_annotation = foo.__annotations__['y']

As we see here, annotations are available at runtime through an
``__annotations__`` attribute on functions, classes, and modules.
When annotations are specified on one of these objects,
``__annotations__`` is a dictionary mapping the names of the
fields to the value specified as that field's annotation.

The default behavior in Python is to evaluate the expressions
for the annotations, and build the annotations dict, at the time
the function, class, or module is bound.  At runtime the above
code actually works something like this:

.. code-block::

    annotations = {'x': int, 'y': MyType, 'return': float}
    def foo(x = 3, y = "abc"):
        ...
    foo.__annotations__ = annotations
    class MyType:
        ...
    foo_y_annotation = foo.__annotations__['y']

The crucial detail here is that the values ``int``, ``MyType``,
and ``float`` are looked up at the time the function object is
bound, and these values are stored in the annotations dict.
But this code doesn't runâit throws a ``NameError`` on the first
line, because ``MyType`` hasn't been defined yet.

:pep:`563`'s solution is to decompile the expressions back
into strings during compilation and store those strings as the
values in the annotations dict.  The equivalent runtime code
would look something like this:

.. code-block::

    annotations = {'x': 'int', 'y': 'MyType', 'return': 'float'}
    def foo(x = 3, y = "abc"):
        ...
    foo


id: pep-0477.rst
content:
PEP: 477
Title: Backport ensurepip (PEP 453) to Python 2.7
Author: Donald Stufft <donald@stufft.io>,
        Alyssa Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: Benjamin Peterson <benjamin@python.org>
Status: Final
Type: Standards Track
Created: 26-Aug-2014
Post-History: 01-Sep-2014
Resolution: https://mail.python.org/pipermail/python-dev/2014-September/136238.html


Abstract
========

This PEP proposes that the ``ensurepip`` module, added to Python 3.4 by PEP
453, be backported to Python 2.7. It also proposes that automatic invocation
of ``ensurepip`` be added to the Python 2.7 Windows and OSX installers. However
it does **not** propose that automatic invocation be added to the ``Makefile``.

It also proposes that the documentation changes for the package distribution
and installation guides be updated to match that in 3.4, which references using
the ``ensurepip`` module to bootstrap the installer.


Rationale
=========

Python 2.7 is effectively a LTS release of Python which represents the end of
the 2.x series and there is still a very large contingent of users whom are
still using Python 2.7 as their primary version. These users, in order to
participate in the wider Python ecosystem, must manually attempt to go out and
find the correct way to bootstrap the packaging tools.

It is the opinion of this PEP that making it as easy as possible for end users
to participate in the wider Python ecosystem is important for 3 primary
reasons:

1. The Python 2.x to 3.x migration has a number of painpoints that are eased by
   a number of third party modules such as six [#six]_, modernize [#modernize]_,
   or future [#future]_. However relying on these tools requires that everyone
   who uses the project have a tool to install these packages.
2. In addition to tooling to aid in migration from Python 2.x to 3.x, there are
   also a number of modules that are *new* in Python 3 for which there are
   backports available on PyPI. This can also aid in the ability for people
   to write 2.x and 3.x compatible software as well as enable them to use some
   of the newer features of Python 3 on Python 2.
3. Users also will need a number of tools in order to create python packages
   that conform to the newer standards that are being proposed. Things like
   setuptools [#setuptools]_, Wheel [#wheel]_, and twine [#twine]_ are enabling
   a safer, faster, and more reliable packaging tool chain. These tools can be
   difficult for people to use if first they must be told how to go out and
   install the package manager.
4. One of Pythons biggest strengths is in the huge ecosystem of libraries and
   projects that have been built on top of it, most of which are distributed
   through PyPI. However, in order to benefit from this wide ecosystem
   meaningfully requires end users, some of which are going to be new, to make
   a decision on which package manager they should get, how to get it, and then
   finally actually installing it first.

Furthermore, alternative implementations of Python are recognizing the benefits
of :pep:`453` and both PyPy and Jython have plans to backport ensurepip to their
2.7 runtimes.


Automatic Invocation
====================

:pep:`453` has ``ensurepip`` automatically invoked by default in the ``Makefile``
and the Windows and OSX Installers. This allowed it to ensure that, by default,
all users would get Python with pip already installed. This PEP however
believes that while this is fine for the Python 2.7 Windows and Mac OS X
installers it is *not* ok for the Python 2.7 ``Makefile`` in general.

The primary consumers of the ``Makefile`` are downstream package managers which
distribute Python themselves. These downstream distributors typically do not
want pip to be installed via ``ensurepip`` and would prefer that end users
install it with their own package manager. Not invoking ``ensurepip``
automatically from the ``Makefile`` would allow these distributors to simply
ignore the fact that ``ensurepip`` has been backported and still not end up
with pip installed via it.

The primary consumers of the OSX and Windows installers are end users who are
attempting to install Python on their own machine. There is not a package
manager available where these users can install pip into their Python through
a more supported mechanism. For this reason it is the belief of this PEP that
installing by default on OSX and Windows is the best course of action.


Documentation
=============

As part of this PEP, the updated packaging distribution and installation
guides for Python 3.4 would be backported to Python 2.7.


Disabling ensurepip by Downstream Distributors
==============================================

Due to its use in the ``venv`` module, downstream distributors cannot disable
the ``ensurepip`` module in Python 3.4. However, since Python 2.7 has no such
module it is explicitly allowed for downstream distributors to patch the
``ensurepip`` module to prevent it from installing anything.

If a downstream distributor wishes to disable ``ensurepip`` completely in
Python 2.7, they should still at least provide the module and allow
``python -m ensurepip`` style invocation. However it should raise errors or
otherwise exit with a non-zero exit code and print out an error on stderr
directing users to what they can/should use instead of ``ensurepip``.


References
==========

.. [#six] `six.py <https://pypi.python.org/pypi/six>`__
.. [#modernize] `modernize <https://pypi.python.org/pypi/modernize>`__
.. [#future] `python-future <https://pypi.python.org/pypi/future>`__
.. [#setuptools] `setuptools <https://pypi.python.org/pypi/setuptools>`__
.. [#wheel] `Wheel <https://pypi.python.org/pypi/wheel>`__
.. [#twine] `twine <https://pypi.python.org/pypi/twine>`__


Copyright
=========

This document has been placed in the public domain.


id: pep-0311.rst
content:
PEP: 311
Title: Simplified Global Interpreter Lock Acquisition for Extensions
Author: Mark Hammond <mhammond@skippinet.com.au>
Status: Final
Type: Standards Track
Created: 05-Feb-2003
Python-Version: 2.3
Post-History: 05-Feb-2003, 14-Feb-2003, 19-Apr-2003


Abstract
========

This PEP proposes a simplified API for access to the Global
Interpreter Lock (GIL) for Python extension modules.
Specifically, it provides a solution for authors of complex
multi-threaded extensions, where the current state of Python
(i.e., the state of the GIL is unknown.

This PEP proposes a new API, for platforms built with threading
support, to manage the Python thread state.  An implementation
strategy is proposed, along with an initial, platform independent
implementation.


Rationale
=========

The current Python interpreter state API is suitable for simple,
single-threaded extensions, but quickly becomes incredibly complex
for non-trivial, multi-threaded extensions.

Currently Python provides two mechanisms for dealing with the GIL:

- ``Py_BEGIN_ALLOW_THREADS`` and ``Py_END_ALLOW_THREADS`` macros.
  These macros are provided primarily to allow a simple Python
  extension that already owns the GIL to temporarily release it
  while making an "external" (ie, non-Python), generally
  expensive, call.  Any existing Python threads that are blocked
  waiting for the GIL are then free to run.  While this is fine
  for extensions making calls from Python into the outside world,
  it is no help for extensions that need to make calls into Python
  when the thread state is unknown.

- ``PyThreadState`` and ``PyInterpreterState`` APIs.
  These API functions allow an extension/embedded application to
  acquire the GIL, but suffer from a serious boot-strapping
  problem - they require you to know the state of the Python
  interpreter and of the GIL before they can be used.  One
  particular problem is for extension authors that need to deal
  with threads never before seen by Python, but need to call
  Python from this thread.  It is very difficult, delicate and
  error prone to author an extension where these "new" threads
  always know the exact state of the GIL, and therefore can
  reliably interact with this API.

For these reasons, the question of how such extensions should
interact with Python is quickly becoming a FAQ.  The main impetus
for this PEP, a thread on python-dev [1]_, immediately identified
the following projects with this exact issue:

- The win32all extensions
- Boost
- ctypes
- Python-GTK bindings
- Uno
- PyObjC
- Mac toolbox
- PyXPCOM

Currently, there is no reasonable, portable solution to this
problem, forcing each extension author to implement their own
hand-rolled version.  Further, the problem is complex, meaning
many implementations are likely to be incorrect, leading to a
variety of problems that will often manifest simply as "Python has
hung".

While the biggest problem in the existing thread-state API is the
lack of the ability to query the current state of the lock, it is
felt that a more complete, simplified solution should be offered
to extension authors.  Such a solution should encourage authors to
provide error-free, complex extension modules that take full
advantage of Python's threading mechanisms.


Limitations and Exclusions
==========================

This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single "PyInterpreterState".  There is no attempt to cater for
extensions that require multiple interpreter states.  At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.

This API will not perform automatic initialization of Python, or
initialize Python for multi-threaded operation.  Extension authors
must continue to call ``Py_Initialize()``, and for multi-threaded
applications, ``PyEval_InitThreads()``.  The reason for this is that
the first thread to call ``PyEval_InitThreads()`` is nominated as the
"main thread" by Python, and so forcing the extension author to
specify the main thread (by requiring them to make this first call)
removes ambiguity.  As ``Py_Initialize()`` must be called before
``PyEval_InitThreads()``, and as both of these functions currently
support being called multiple times, the burden this places on
extension authors is considered reasonable.

It is intended that this API be all that is necessary to acquire
the Python GIL.  Apart from the existing, standard
``Py_BEGIN_ALLOW_THREADS`` and ``Py_END_ALLOW_THREADS`` macros, it is
assumed that no additional thread state API functions will be used
by the extension.  Extensions with such complicated requirements
are free to continue to use the existing thread state API.


Proposal
========

This proposal recommends a new API be added to Python to simplify
the management of the GIL.  This API will be available on all
platforms built with ``WITH_THREAD`` defined.

The intent is that assuming Python has correctly been initialized,
an extension author be able to use a small, well-defined "prologue
dance", at any time and on any thread, which will ensure Python
is ready to be used on that thread.  After the extension has
finished with Python, it must also perform an "epilogue dance" to
release any resources previously acquired.  Ideally, these dances
can be expressed in a single line.

Specifically, the following new APIs are proposed::

   /* Ensure that the current thread is ready to call the Python
      C API, regardless of the current state of Python, or of its
      thread lock.  This may be called as many times as desired
      by a thread so long as each call is matched with a call to
      PyGILState_Release().  In general, other thread-state APIs may
      be used between _Ensure() and _Release() calls, so long as the
      thread-state is restored to its previous state before the Release().
      For example, normal use of the Py_BEGIN_ALLOW_THREADS/
      Py_END_ALLOW_THREADS macros are acceptable.

      The return value is an opaque "handle" to the thread state when
      PyGILState_Acquire() was called, and must be passed to
      PyGILState_Release() to ensure Python is left in the same state. Even
      though recursive calls are allowed, these handles can *not* be
      shared - each unique call to PyGILState_Ensure must save the handle
      for its call to PyGILState_Release.

      When the function returns, the current thread will hold the GIL.

      Failure is a fatal error.
   */
   PyAPI_FUNC(PyGILState_STATE) PyGILState_Ensure(void);

   /* Release any resources previously acquired.  After this call, Python's
      state will be the same as it was prior to the corresponding
      PyGILState_Acquire call (but generally this state will be unknown to
      the caller, hence the use of the GILState API.)

      Every call to PyGILState_Ensure must be matched by a call to
      PyGILState_Release on the same thread.
   */
   PyAPI_FUNC(void) PyGILState_Release(PyGILState_STATE);

Common usage will be::

   void SomeCFunction(void)
   {
       /* ensure we hold the lock */
       PyGILState_STATE state = PyGILState_Ensure();
       /* Use the Python API */
       ...
       /* Restore the state of Python */
       PyGILState_Release(state);
   }


Design and Implementation
=========================

The general operation of ``PyGILState_Ensure()`` will be:

- assert Python is initialized.

- Get a ``PyThreadState`` for the current thread, creating and saving
  if necessary.

- remember the current state of the lock (owned/not owned)

- If the current state does not own the GIL, acquire it.

- Increment a counter for how many calls to ``PyGILState_Ensure`` have been
  made on the current thread.

- return

The general operation of ``PyGILState_Release()`` will be:

- assert our thread currently holds the lock.

- If old state indicates lock was previously unlocked, rel


id: pep-0305.rst
content:
PEP: 305
Title: CSV File API
Author: Kevin Altis <altis@semi-retired.com>,
        Dave Cole <djc@object-craft.com.au>,
        Andrew McNamara <andrewm@object-craft.com.au>,
        Skip Montanaro <skip@pobox.com>,
        Cliff Wells <LogiplexSoftware@earthlink.net>
Discussions-To: csv@python.org
Status: Final
Type: Standards Track
Created: 26-Jan-2003
Python-Version: 2.3
Post-History: 31-Jan-2003, 13-Feb-2003


Abstract
========

The Comma Separated Values (CSV) file format is the most common import
and export format for spreadsheets and databases.  Although many CSV
files are simple to parse, the format is not formally defined by a
stable specification and is subtle enough that parsing lines of a CSV
file with something like ``line.split(",")`` is eventually bound to
fail.  This PEP defines an API for reading and writing CSV files.  It
is accompanied by a corresponding module which implements the API.


To Do (Notes for the Interested and Ambitious)
==============================================

- Better motivation for the choice of passing a file object to the
  constructors.  See
  https://mail.python.org/pipermail/csv/2003-January/000179.html

- Unicode.  ugh.


Application Domain
==================

This PEP is about doing one thing well: parsing tabular data which may
use a variety of field separators, quoting characters, quote escape
mechanisms and line endings.  The authors intend the proposed module
to solve this one parsing problem efficiently.  The authors do not
intend to address any of these related topics:

- data interpretation (is a field containing the string "10" supposed
  to be a string, a float or an int? is it a number in base 10, base
  16 or base 2? is a number in quotes a number or a string?)

- locale-specific data representation (should the number 1.23 be
  written as "1.23" or "1,23" or "1 23"?) -- this may eventually be
  addressed.

- fixed width tabular data - can already be parsed reliably.


Rationale
=========

Often, CSV files are formatted simply enough that you can get by
reading them line-by-line and splitting on the commas which delimit
the fields.  This is especially true if all the data being read is
numeric.  This approach may work for a while, then come back to bite
you in the butt when somebody puts something unexpected in the data
like a comma.  As you dig into the problem you may eventually come to
the conclusion that you can solve the problem using regular
expressions.  This will work for a while, then break mysteriously one
day.  The problem grows, so you dig deeper and eventually realize that
you need a purpose-built parser for the format.

CSV formats are not well-defined and different implementations have a
number of subtle corner cases.  It has been suggested that the "V" in
the acronym stands for "Vague" instead of "Values".  Different
delimiters and quoting characters are just the start.  Some programs
generate whitespace after each delimiter which is not part of the
following field.  Others quote embedded quoting characters by doubling
them, others by prefixing them with an escape character.  The list of
weird ways to do things can seem endless.

All this variability means it is difficult for programmers to reliably
parse CSV files from many sources or generate CSV files designed to be
fed to specific external programs without a thorough understanding of
those sources and programs.  This PEP and the software which accompany
it attempt to make the process less fragile.


Existing Modules
================

This problem has been tackled before.  At least three modules
currently available in the Python community enable programmers to read
and write CSV files:

- Object Craft's CSV module [2]_

- Cliff Wells' Python-DSV module [3]_

- Laurence Tratt's ASV module [4]_

Each has a different API, making it somewhat difficult for programmers
to switch between them.  More of a problem may be that they interpret
some of the CSV corner cases differently, so even after surmounting
the differences between the different module APIs, the programmer has
to also deal with semantic differences between the packages.


Module Interface
================

This PEP supports three basic APIs, one to read and parse CSV files,
one to write them, and one to identify different CSV dialects to the
readers and writers.


Reading CSV Files
-----------------

CSV readers are created with the reader factory function::

    obj = reader(iterable [, dialect='excel']
                 [optional keyword args])

A reader object is an iterator which takes an iterable object
returning lines as the sole required parameter.  If it supports a
binary mode (file objects do), the iterable argument to the reader
function must have been opened in binary mode.  This gives the reader
object full control over the interpretation of the file's contents.
The optional dialect parameter is discussed below.  The reader
function also accepts several optional keyword arguments which define
specific format settings for the parser (see the section "Formatting
Parameters").  Readers are typically used as follows::

    csvreader = csv.reader(file("some.csv"))
    for row in csvreader:
        process(row)

Each row returned by a reader object is a list of strings or Unicode
objects.

When both a dialect parameter and individual formatting parameters are
passed to the constructor, first the dialect is queried for formatting
parameters, then individual formatting parameters are examined.


Writing CSV Files
-----------------

Creating writers is similar::

    obj = writer(fileobj [, dialect='excel'],
                 [optional keyword args])

A writer object is a wrapper around a file-like object opened for
writing in binary mode (if such a distinction is made).  It accepts
the same optional keyword parameters as the reader constructor.

Writers are typically used as follows::

    csvwriter = csv.writer(file("some.csv", "w"))
    for row in someiterable:
        csvwriter.writerow(row)

To generate a set of field names as the first row of the CSV file, the
programmer must explicitly write it, e.g.::

    csvwriter = csv.writer(file("some.csv", "w"), fieldnames=names)
    csvwriter.write(names)
    for row in someiterable:
        csvwriter.write(row)

or arrange for it to be the first row in the iterable being written.


Managing Different Dialects
---------------------------

Because CSV is a somewhat ill-defined format, there are plenty of ways
one CSV file can differ from another, yet contain exactly the same
data.  Many tools which can import or export tabular data allow the
user to indicate the field delimiter, quote character, line
terminator, and other characteristics of the file.  These can be
fairly easily determined, but are still mildly annoying to figure out,
and make for fairly long function calls when specified individually.

To try and minimize the difficulty of figuring out and specifying a
bunch of formatting parameters, reader and writer objects support a
dialect argument which is just a convenient handle on a group of these
lower level parameters.  When a dialect is given as a string it
identifies one of the dialects known to the module via its
registration functions, otherwise it must be an instance of the
Dialect class as described below.

Dialects will generally be named after applications or organizations
which define specific sets of format constraints.  Two dialects are
defined in the module as of this writing, "excel", which describes the
default format constraints for CSV file export by Excel 97 and Excel
2000, and "excel-tab", which is the same as "excel" but specifies an
ASCII TAB character as the field delimiter.

Dialects are implemented as attribute only classes to enable users to
construct variant dialects by subclassing.  The "excel" dialect is a
subclass of Dialect and is defined as follows::

    class Dialect:
        # placeholders
        delimiter = None
        quotechar = None
        escapechar = None
        d


id: pep-0463.rst
content:
PEP: 463
Title: Exception-catching expressions
Author: Chris Angelico <rosuav@gmail.com>
Status: Rejected
Type: Standards Track
Created: 15-Feb-2014
Python-Version: 3.5
Post-History: 20-Feb-2014, 16-Feb-2014
Resolution: https://mail.python.org/pipermail/python-dev/2014-March/133118.html


Rejection Notice
================

From https://mail.python.org/pipermail/python-dev/2014-March/133118.html:

"""
I want to reject this PEP. I think the proposed syntax is acceptable given
the desired semantics, although it's still a bit jarring. It's probably no
worse than the colon used with lambda (which echoes the colon used in a def
just like the colon here echoes the one in a try/except) and definitely
better than the alternatives listed.

But the thing I can't get behind are the motivation and rationale. I don't
think that e.g. dict.get() would be unnecessary once we have except
expressions, and I disagree with the position that EAFP is better than
LBYL, or "generally recommended" by Python. (Where do you get that? From
the same sources that are so obsessed with DRY they'd rather introduce a
higher-order-function than repeat one line of code? :-)

This is probably the most you can get out of me as far as a pronouncement.
Given that the language summit is coming up I'd be happy to dive deeper in
my reasons for rejecting it there (if there's demand).

I do think that (apart from never explaining those dreadful acronyms :-)
this was a well-written and well-researched PEP, and I think you've done a
great job moderating the discussion, collecting objections, reviewing
alternatives, and everything else that is required to turn a heated debate
into a PEP. Well done Chris (and everyone who helped), and good luck with
your next PEP!
"""

Abstract
========

Just as :pep:`308` introduced a means of value-based conditions in an
expression, this system allows exception-based conditions to be used
as part of an expression.


Motivation
==========

A number of functions and methods have parameters which will cause
them to return a specified value instead of raising an exception.  The
current system is ad-hoc and inconsistent, and requires that each
function be individually written to have this functionality; not all
support this.

* dict.get(key, default) - second positional argument in place of
  KeyError

* next(iter, default) - second positional argument in place of
  StopIteration

* list.pop() - no way to return a default

* seq[index] - no way to handle a bounds error

* min(sequence, default=default) - keyword argument in place of
  ValueError

* statistics.mean(data) - no way to handle an empty iterator

Had this facility existed early in Python's history, there would have been
no need to create dict.get() and related methods; the one obvious way to
handle an absent key would be to respond to the exception.  One method is
written which signals the absence in one way, and one consistent technique
is used to respond to the absence.  Instead, we have dict.get(), and as of
Python 3.4, we also have min(... default=default), and myriad others.  We
have a LBYL syntax for testing inside an expression, but there is currently
no EAFP notation; compare the following::

    # LBYL:
    if key in dic:
        process(dic[key])
    else:
        process(None)
    # As an expression:
    process(dic[key] if key in dic else None)

    # EAFP:
    try:
        process(dic[key])
    except KeyError:
        process(None)
    # As an expression:
    process(dic[key] except KeyError: None)

Python generally recommends the EAFP policy, but must then proliferate
utility functions like dic.get(key,None) to enable this.


Rationale
=========

The current system requires that a function author predict the need
for a default, and implement support for it.  If this is not done, a
full try/except block is needed.

Since try/except is a statement, it is impossible to catch exceptions
in the middle of an expression.  Just as if/else does for conditionals
and lambda does for function definitions, so does this allow exception
catching in an expression context.

This provides a clean and consistent way for a function to provide a
default: it simply raises an appropriate exception, and the caller
catches it.

With some situations, an LBYL technique can be used (checking if some
sequence has enough length before indexing into it, for instance). This is
not safe in all cases, but as it is often convenient, programmers will be
tempted to sacrifice the safety of EAFP in favour of the notational brevity
of LBYL. Additionally, some LBYL techniques (eg involving getattr with
three arguments) warp the code into looking like literal strings rather
than attribute lookup, which can impact readability. A convenient EAFP
notation solves all of this.

There's no convenient way to write a helper function to do this; the
nearest is something ugly using either lambda::

    def except_(expression, exception_list, default):
        try:
            return expression()
        except exception_list:
            return default()
    value = except_(lambda: 1/x, ZeroDivisionError, lambda: float("nan"))

which is clunky, and unable to handle multiple exception clauses; or
eval::

    def except_(expression, exception_list, default):
        try:
            return eval(expression, globals_of_caller(), locals_of_caller())
        except exception_list as exc:
            l = locals_of_caller().copy()
            l['exc'] = exc
            return eval(default, globals_of_caller(), l)

    def globals_of_caller():
        return sys._getframe(2).f_globals

    def locals_of_caller():
        return sys._getframe(2).f_locals

    value = except_("""1/x""",ZeroDivisionError,""" "Can't divide by zero" """)

which is even clunkier, and relies on implementation-dependent hacks.
(Writing globals_of_caller() and locals_of_caller() for interpreters
other than CPython is left as an exercise for the reader.)

Raymond Hettinger `expresses`__ a desire for such a consistent
API. Something similar has been `requested`__ `multiple`__ `times`__
in the past.

__ https://mail.python.org/pipermail/python-ideas/2014-February/025443.html
__ https://mail.python.org/pipermail/python-ideas/2013-March/019760.html
__ https://mail.python.org/pipermail/python-ideas/2009-August/005441.html
__ https://mail.python.org/pipermail/python-ideas/2008-August/001801.html


Proposal
========

Just as the 'or' operator and the three part 'if-else' expression give
short circuiting methods of catching a falsy value and replacing it,
this syntax gives a short-circuiting method of catching an exception
and replacing it.

This currently works::

    lst = [1, 2, None, 3]
    value = lst[2] or "No value"

The proposal adds this::

    lst = [1, 2]
    value = (lst[2] except IndexError: "No value")

Specifically, the syntax proposed is::

    (expr except exception_list: default)

where expr, exception_list, and default are all expressions.  First,
expr is evaluated.  If no exception is raised, its value is the value
of the overall expression.  If any exception is raised, exception_list
is evaluated, and should result in either a type or a tuple, just as
with the statement form of try/except.  Any matching exception will
result in the corresponding default expression being evaluated and
becoming the value of the expression.  As with the statement form of
try/except, non-matching exceptions will propagate upward.

Parentheses are required around the entire expression, unless they
would be completely redundant, according to the same rules as generator
expressions follow. This guarantees correct interpretation of nested
except-expressions, and allows for future expansion of the syntax -
see below on multiple except clauses.

Note that the current proposal does not allow the exception object to
be captured. Where this is needed, the statement form must be used.
(See below for discussion and elaboration on this.)

This ternary operator would be between lambda and if/else i


id: pep-0339.rst
content:
PEP: 339
Title: Design of the CPython Compiler
Author: Brett Cannon <brett@python.org>
Status: Withdrawn
Type: Informational
Created: 02-Feb-2005
Post-History:


.. note::
    This PEP has been withdrawn and moved to the Python
    developer's guide.


Abstract
--------

Historically (through 2.4), compilation from source code to bytecode
involved two steps:

1. Parse the source code into a parse tree (Parser/pgen.c)
2. Emit bytecode based on the parse tree (Python/compile.c)

Historically, this is not how a standard compiler works.  The usual
steps for compilation are:

1. Parse source code into a parse tree (Parser/pgen.c)
2. Transform parse tree into an Abstract Syntax Tree (Python/ast.c)
3. Transform AST into a Control Flow Graph (Python/compile.c)
4. Emit bytecode based on the Control Flow Graph (Python/compile.c)

Starting with Python 2.5, the above steps are now used.  This change
was done to simplify compilation by breaking it into three steps.
The purpose of this document is to outline how the latter three steps
of the process works.

This document does not touch on how parsing works beyond what is needed
to explain what is needed for compilation.  It is also not exhaustive
in terms of the how the entire system works.  You will most likely need
to read some source to have an exact understanding of all details.


Parse Trees
-----------

Python's parser is an LL(1) parser mostly based on the
implementation laid out in the Dragon Book [Aho86]_.

The grammar file for Python can be found in Grammar/Grammar with the
numeric value of grammar rules are stored in Include/graminit.h.  The
numeric values for types of tokens (literal tokens, such as ``:``,
numbers, etc.) are kept in Include/token.h).  The parse tree made up of
``node *`` structs (as defined in Include/node.h).

Querying data from the node structs can be done with the following
macros (which are all defined in Include/token.h):

- ``CHILD(node *, int)``
        Returns the nth child of the node using zero-offset indexing
- ``RCHILD(node *, int)``
        Returns the nth child of the node from the right side; use
        negative numbers!
- ``NCH(node *)``
        Number of children the node has
- ``STR(node *)``
        String representation of the node; e.g., will return ``:`` for a
        COLON token
- ``TYPE(node *)``
        The type of node as specified in ``Include/graminit.h``
- ``REQ(node *, TYPE)``
        Assert that the node is the type that is expected
- ``LINENO(node *)``
        retrieve the line number of the source code that led to the
        creation of the parse rule; defined in Python/ast.c

To tie all of this example, consider the rule for 'while'::

  while_stmt: 'while' test ':' suite ['else' ':' suite]

The node representing this will have ``TYPE(node) == while_stmt`` and
the number of children can be 4 or 7 depending on if there is an 'else'
statement.  To access what should be the first ':' and require it be an
actual ':' token, ``(REQ(CHILD(node, 2), COLON)``.


Abstract Syntax Trees (AST)
---------------------------

The abstract syntax tree (AST) is a high-level representation of the
program structure without the necessity of containing the source code;
it can be thought of as an abstract representation of the source code.  The
specification of the AST nodes is specified using the Zephyr Abstract
Syntax Definition Language (ASDL) [Wang97]_.

The definition of the AST nodes for Python is found in the file
Parser/Python.asdl .

Each AST node (representing statements, expressions, and several
specialized types, like list comprehensions and exception handlers) is
defined by the ASDL.  Most definitions in the AST correspond to a
particular source construct, such as an 'if' statement or an attribute
lookup.  The definition is independent of its realization in any
particular programming language.

The following fragment of the Python ASDL construct demonstrates the
approach and syntax::

  module Python
  {
        stmt = FunctionDef(identifier name, arguments args, stmt* body,
                            expr* decorators)
              | Return(expr? value) | Yield(expr value)
              attributes (int lineno)
  }

The preceding example describes three different kinds of statements;
function definitions, return statements, and yield statements.  All
three kinds are considered of type stmt as shown by '|' separating the
various kinds.  They all take arguments of various kinds and amounts.

Modifiers on the argument type specify the number of values needed; '?'
means it is optional, '*' means 0 or more, no modifier means only one
value for the argument and it is required.  FunctionDef, for instance,
takes an identifier for the name, 'arguments' for args, zero or more
stmt arguments for 'body', and zero or more expr arguments for
'decorators'.

Do notice that something like 'arguments', which is a node type, is
represented as a single AST node and not as a sequence of nodes as with
stmt as one might expect.

All three kinds also have an 'attributes' argument; this is shown by the
fact that 'attributes' lacks a '|' before it.

The statement definitions above generate the following C structure type::

  typedef struct _stmt *stmt_ty;

  struct _stmt {
        enum { FunctionDef_kind=1, Return_kind=2, Yield_kind=3 } kind;
        union {
                struct {
                        identifier name;
                        arguments_ty args;
                        asdl_seq *body;
                } FunctionDef;

                struct {
                        expr_ty value;
                } Return;

                struct {
                        expr_ty value;
                } Yield;
        } v;
        int lineno;
   }

Also generated are a series of constructor functions that allocate (in
this case) a stmt_ty struct with the appropriate initialization.  The
'kind' field specifies which component of the union is initialized.  The
FunctionDef() constructor function sets 'kind' to FunctionDef_kind and
initializes the 'name', 'args', 'body', and 'attributes' fields.


Memory Management
-----------------

Before discussing the actual implementation of the compiler, a discussion of
how memory is handled is in order.  To make memory management simple, an arena
is used.  This means that a memory is pooled in a single location for easy
allocation and removal.  What this gives us is the removal of explicit memory
deallocation.  Because memory allocation for all needed memory in the compiler
registers that memory with the arena, a single call to free the arena is all
that is needed to completely free all memory used by the compiler.

In general, unless you are working on the critical core of the compiler, memory
management can be completely ignored.  But if you are working at either the
very beginning of the compiler or the end, you need to care about how the arena
works.  All code relating to the arena is in either Include/pyarena.h or
Python/pyarena.c .

PyArena_New() will create a new arena.  The returned PyArena structure will
store pointers to all memory given to it.  This does the bookkeeping of what
memory needs to be freed when the compiler is finished with the memory it used.
That freeing is done with PyArena_Free().  This needs to only be called in
strategic areas where the compiler exits.

As stated above, in general you should not have to worry about memory
management when working on the compiler.  The technical details have been
designed to be hidden from you for most cases.

The only exception comes about when managing a PyObject.  Since the rest
of Python uses reference counting, there is extra support added
to the arena to cleanup each PyObject that was allocated.  These cases
are very rare.  However, if you've allocated a PyObject, you must tell
the arena about it by calling PyArena_AddPyObject().


Parse Tree to AST
-----------------

The AST is generated from the parse tree (see Python/ast.c) using the
function ``PyAST_FromNode()``.

The functi


id: pep-0488.rst
content:
PEP: 488
Title: Elimination of PYO files
Author: Brett Cannon <brett@python.org>
Status: Final
Type: Standards Track
Created: 20-Feb-2015
Python-Version: 3.5
Post-History:
    06-Mar-2015,
    13-Mar-2015,
    20-Mar-2015

Abstract
========

This PEP proposes eliminating the concept of PYO files from Python.
To continue the support of the separation of bytecode files based on
their optimization level, this PEP proposes extending the PYC file
name to include the optimization level in the bytecode repository
directory when there are optimizations applied.


Rationale
=========

As of today, bytecode files come in two flavours: PYC and PYO. A PYC
file is the bytecode file generated and read from when no
optimization level is specified at interpreter startup (i.e., ``-O``
is not specified). A PYO file represents the bytecode file that is
read/written when **any** optimization level is specified (i.e., when
``-O`` **or** ``-OO`` is specified). This means that while PYC
files clearly delineate the optimization level used when they were
generated -- namely no optimizations beyond the peepholer -- the same
is not true for PYO files. To put this in terms of optimization
levels and the file extension:

- 0: ``.pyc``
- 1 (``-O``): ``.pyo``
- 2 (``-OO``): ``.pyo``

The reuse of the ``.pyo`` file extension for both level 1 and 2
optimizations means that there is no clear way to tell what
optimization level was used to generate the bytecode file. In terms
of reading PYO files, this can lead to an interpreter using a mixture
of optimization levels with its code if the user was not careful to
make sure all PYO files were generated using the same optimization
level (typically done by blindly deleting all PYO files and then
using the ``compileall`` module to compile all-new PYO files [1]_).
This issue is only compounded when people optimize Python code beyond
what the interpreter natively supports, e.g., using the astoptimizer
project [2]_.

In terms of writing PYO files, the need to delete all PYO files
every time one either changes the optimization level they want to use
or are unsure of what optimization was used the last time PYO files
were generated leads to unnecessary file churn. The change proposed
by this PEP also allows for **all** optimization levels to be
pre-compiled for bytecode files ahead of time, something that is
currently impossible thanks to the reuse of the ``.pyo`` file
extension for multiple optimization levels.

As for distributing bytecode-only modules, having to distribute both
``.pyc`` and ``.pyo`` files is unnecessary for the common use-case
of code obfuscation and smaller file deployments. This means that
bytecode-only modules will only load from their non-optimized
``.pyc`` file name.


Proposal
========

To eliminate the ambiguity that PYO files present, this PEP proposes
eliminating the concept of PYO files and their accompanying ``.pyo``
file extension. To allow for the optimization level to be unambiguous
as well as to avoid having to regenerate optimized bytecode files
needlessly in the ``__pycache__`` directory, the optimization level
used to generate the bytecode file will be incorporated into the
bytecode file name. When no optimization level is specified, the
pre-PEP ``.pyc`` file name will be used (i.e., no optimization level
will be specified in the file name). For example, a source file named
``foo.py`` in CPython 3.5 could have the following bytecode files
based on the interpreter's optimization level (none, ``-O``, and
``-OO``):

- 0: ``foo.cpython-35.pyc`` (i.e., no change)
- 1: ``foo.cpython-35.opt-1.pyc``
- 2: ``foo.cpython-35.opt-2.pyc``

Currently bytecode file names are created by
``importlib.util.cache_from_source()``, approximately using the
following expression defined by :pep:`3147` [3]_, [4]_::

    '{name}.{cache_tag}.pyc'.format(name=module_name,
                                    cache_tag=sys.implementation.cache_tag)

This PEP proposes to change the expression when an optimization
level is specified to::

    '{name}.{cache_tag}.opt-{optimization}.pyc'.format(
            name=module_name,
            cache_tag=sys.implementation.cache_tag,
            optimization=str(sys.flags.optimize))

The "opt-" prefix was chosen so as to provide a visual separator
from the cache tag. The placement of the optimization level after
the cache tag was chosen to preserve lexicographic sort order of
bytecode file names based on module name and cache tag which will
not vary for a single interpreter. The "opt-" prefix was chosen over
"o" so as to be somewhat self-documenting. The "opt-" prefix was
chosen over "O" so as to not have any confusion in case "0" was the
leading prefix of the optimization level.

A period was chosen over a hyphen as a separator so as to distinguish
clearly that the optimization level is not part of the interpreter
version as specified by the cache tag. It also lends to the use of
the period in the file name to delineate semantically different
concepts.

For example, if ``-OO`` had been passed to the interpreter then
instead of ``importlib.cpython-35.pyo`` the file name would be
``importlib.cpython-35.opt-2.pyc``.

Leaving out the new ``opt-`` tag when no optimization level is
applied should increase backwards-compatibility. This is also more
understanding of Python implementations which have no use for
optimization levels (e.g., PyPy [10]_).

It should be noted that this change in no way affects the performance
of import. Since the import system looks for a single bytecode file
based on the optimization level of the interpreter already and
generates a new bytecode file if it doesn't exist, the introduction
of potentially more bytecode files in the ``__pycache__`` directory
has no effect in terms of stat calls. The interpreter will continue
to look for only a single bytecode file based on the optimization
level and thus no increase in stat calls will occur.

The only potentially negative result of this PEP is the probable
increase in the number of ``.pyc`` files and thus increase in storage
use. But for platforms where this is an issue,
``sys.dont_write_bytecode`` exists to turn off bytecode generation so
that it can be controlled offline.


Implementation
==============

An implementation of this PEP is available [11]_.

importlib
---------

As ``importlib.util.cache_from_source()`` is the API that exposes
bytecode file paths as well as being directly used by importlib, it
requires the most critical change. As of Python 3.4, the function's
signature is::

  importlib.util.cache_from_source(path, debug_override=None)

This PEP proposes changing the signature in Python 3.5 to::

  importlib.util.cache_from_source(path, debug_override=None, *, optimization=None)

The introduced ``optimization`` keyword-only parameter will control
what optimization level is specified in the file name. If the
argument is ``None`` then the current optimization level of the
interpreter will be assumed (including no optimization). Any argument
given for ``optimization`` will be passed to ``str()`` and must have
``str.isalnum()`` be true, else ``ValueError`` will be raised (this
prevents invalid characters being used in the file name). If the
empty string is passed in for ``optimization`` then the addition of
the optimization will be suppressed, reverting to the file name
format which predates this PEP.

It is expected that beyond Python's own two optimization levels,
third-party code will use a hash of optimization names to specify the
optimization level, e.g.
``hashlib.sha256(','.join(['no dead code', 'const folding'])).hexdigest()``.
While this might lead to long file names, it is assumed that most
users never look at the contents of the __pycache__ directory and so
this won't be an issue.

The ``debug_override`` parameter will be deprecated. A ``False``
value will be equivalent to ``optimization=1`` while a ``True``
value will represent ``optimization=''`` (a ``None`` argument will
continue to mean the same as for ``opt


id: pep-0489.rst
content:
PEP: 489
Title: Multi-phase extension module initialization
Author: Petr Viktorin <encukou@gmail.com>,
        Stefan Behnel <stefan_ml@behnel.de>,
        Alyssa Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: Eric Snow <ericsnowcurrently@gmail.com>
Discussions-To: import-sig@python.org
Status: Final
Type: Standards Track
Created: 11-Aug-2013
Python-Version: 3.5
Post-History: 23-Aug-2013, 20-Feb-2015, 16-Apr-2015, 07-May-2015, 18-May-2015
Resolution: https://mail.python.org/pipermail/python-dev/2015-May/140108.html


Abstract
========

This PEP proposes a redesign of the way in which built-in and extension modules
interact with the import machinery. This was last revised for Python 3.0 in PEP
3121, but did not solve all problems at the time. The goal is to solve
import-related problems by bringing extension modules closer to the way Python
modules behave; specifically to hook into the ModuleSpec-based loading
mechanism introduced in :pep:`451`.

This proposal draws inspiration from PyType_Spec of :pep:`384` to allow extension
authors to only define features they need, and to allow future additions
to extension module declarations.

Extensions modules are created in a two-step process, fitting better into
the ModuleSpec architecture, with parallels to __new__ and __init__ of classes.

Extension modules can safely store arbitrary C-level per-module state in
the module that is covered by normal garbage collection and supports
reloading and sub-interpreters.
Extension authors are encouraged to take these issues into account
when using the new API.

The proposal also allows extension modules with non-ASCII names.

Not all problems tackled in :pep:`3121` are solved in this proposal.
In particular, problems with run-time module lookup (PyState_FindModule)
are left to a future PEP.


Motivation
==========

Python modules and extension modules are not being set up in the same way.
For Python modules, the module object is created and set up first, then the
module code is being executed (:pep:`302`).
A ModuleSpec object (:pep:`451`) is used to hold information about the module,
and passed to the relevant hooks.

For extensions (i.e. shared libraries) and built-in modules, the module
init function is executed straight away and does both the creation and
initialization. The initialization function is not passed the ModuleSpec,
or any information it contains, such as the __file__ or fully-qualified
name. This hinders relative imports and resource loading.

In Py3, modules are also not being added to sys.modules, which means that a
(potentially transitive) re-import of the module will really try to re-import
it and thus run into an infinite loop when it executes the module init function
again. Without access to the fully-qualified module name, it is not trivial to
correctly add the module to sys.modules either.
This is specifically a problem for Cython generated modules, for which it's
not uncommon that the module init code has the same level of complexity as
that of any 'regular' Python module. Also, the lack of __file__ and __name__
information hinders the compilation of "__init__.py" modules, i.e. packages,
especially when relative imports are being used at module init time.

Furthermore, the majority of currently existing extension modules has
problems with sub-interpreter support and/or interpreter reloading, and, while
it is possible with the current infrastructure to support these
features, it is neither easy nor efficient.
Addressing these issues was the goal of :pep:`3121`, but many extensions,
including some in the standard library, took the least-effort approach
to porting to Python 3, leaving these issues unresolved.
This PEP keeps backwards compatibility, which should reduce pressure and give
extension authors adequate time to consider these issues when porting.


The current process
===================

Currently, extension and built-in modules export an initialization function
named "PyInit_modulename", named after the file name of the shared library.
This function is executed by the import machinery and must return a fully
initialized module object.
The function receives no arguments, so it has no way of knowing about its
import context.

During its execution, the module init function creates a module object
based on a PyModuleDef object. It then continues to initialize it by adding
attributes to the module dict, creating types, etc.

In the back, the shared library loader keeps a note of the fully qualified
module name of the last module that it loaded, and when a module gets
created that has a matching name, this global variable is used to determine
the fully qualified name of the module object. This is not entirely safe as it
relies on the module init function creating its own module object first,
but this assumption usually holds in practice.


The proposal
============

The initialization function (PyInit_modulename) will be allowed to return
a pointer to a PyModuleDef object. The import machinery will be in charge
of constructing the module object, calling hooks provided in the PyModuleDef
in the relevant phases of initialization (as described below).

This multi-phase initialization is an additional possibility. Single-phase
initialization, the current practice of returning a fully initialized module
object, will still be accepted, so existing code will work unchanged,
including binary compatibility.

The PyModuleDef structure will be changed to contain a list of slots,
similarly to :pep:`384`'s PyType_Spec for types.
To keep binary compatibility, and avoid needing to introduce a new structure
(which would introduce additional supporting functions and per-module storage),
the currently unused m_reload pointer of PyModuleDef will be changed to
hold the slots. The structures are defined as::

    typedef struct {
        int slot;
        void *value;
    } PyModuleDef_Slot;

    typedef struct PyModuleDef {
        PyModuleDef_Base m_base;
        const char* m_name;
        const char* m_doc;
        Py_ssize_t m_size;
        PyMethodDef *m_methods;
        PyModuleDef_Slot *m_slots;  /* changed from `inquiry m_reload;` */
        traverseproc m_traverse;
        inquiry m_clear;
        freefunc m_free;
    } PyModuleDef;

The *m_slots* member must be either NULL, or point to an array of
PyModuleDef_Slot structures, terminated by a slot with id set to 0
(i.e. ``{0, NULL}``).

To specify a slot, a unique slot ID must be provided.
New Python versions may introduce new slot IDs, but slot IDs will never be
recycled. Slots may get deprecated, but will continue to be supported
throughout Python 3.x.

A slot's value pointer may not be NULL, unless specified otherwise in the
slot's documentation.

The following slots are currently available, and described later:

* Py_mod_create
* Py_mod_exec

Unknown slot IDs will cause the import to fail with SystemError.

When using multi-phase initialization, the *m_name* field of PyModuleDef will
not be used during importing; the module name will be taken from the ModuleSpec.

Before it is returned from PyInit_*, the PyModuleDef object must be initialized
using the newly added PyModuleDef_Init function. This sets the object type
(which cannot be done statically on certain compilers), refcount, and internal
bookkeeping data (m_index).
For example, an extension module "example" would be exported as::

    static PyModuleDef example_def = {...}

    PyMODINIT_FUNC
    PyInit_example(void)
    {
        return PyModuleDef_Init(&example_def);
    }

The PyModuleDef object must be available for the lifetime of the module created
from it â usually, it will be declared statically.

Pseudo-code Overview
--------------------

Here is an overview of how the modified importers will operate.
Details such as logging or handling of errors and invalid states
are left out, and C code is presented with a concise Python-like syntax.

The framework that calls the importers is explained in
:pep:`451#how-loading-will-work`.

impor


id: pep-0338.rst
content:
PEP: 338
Title: Executing modules as scripts
Author: Alyssa Coghlan <ncoghlan@gmail.com>
Status: Final
Type: Standards Track
Created: 16-Oct-2004
Python-Version: 2.5
Post-History: 08-Nov-2004, 11-Feb-2006, 12-Feb-2006, 18-Feb-2006


Abstract
========

This PEP defines semantics for executing any Python module as a
script, either with the ``-m`` command line switch, or by invoking
it via ``runpy.run_module(modulename)``.

The ``-m`` switch implemented in Python 2.4 is quite limited. This
PEP proposes making use of the :pep:`302` import hooks to allow any
module which provides access to its code object to be executed.

Rationale
=========

Python 2.4 adds the command line switch ``-m`` to allow modules to be
located using the Python module namespace for execution as scripts.
The motivating examples were standard library modules such as ``pdb``
and ``profile``, and the Python 2.4 implementation is fine for this
limited purpose.

A number of users and developers have requested extension of the
feature to also support running modules located inside packages.  One
example provided is pychecker's ``pychecker.checker`` module.  This
capability was left out of the Python 2.4 implementation because the
implementation of this was significantly more complicated, and the most
appropriate strategy was not at all clear.

The opinion on python-dev was that it was better to postpone the
extension to Python 2.5, and go through the PEP process to help make
sure we got it right.

Since that time, it has also been pointed out that the current version
of ``-m`` does not support ``zipimport`` or any other kind of
alternative import behaviour (such as frozen modules).

Providing this functionality as a Python module is significantly easier
than writing it in C, and makes the functionality readily available to
all Python programs, rather than being specific to the CPython
interpreter. CPython's command line switch can then be rewritten to
make use of the new module.

Scripts which execute other scripts (e.g. ``profile``, ``pdb``) also
have the option to use the new module to provide ``-m`` style support
for identifying the script to be executed.


Scope of this proposal
==========================

In Python 2.4, a module located using ``-m`` is executed just as if
its filename had been provided on the command line.  The goal of this
PEP is to get as close as possible to making that statement also hold
true for modules inside packages, or accessed via alternative import
mechanisms (such as ``zipimport``).

Prior discussions suggest it should be noted that this PEP is **not**
about changing the idiom for making Python modules also useful as
scripts (see :pep:`299`). That issue is considered orthogonal to the
specific feature addressed by this PEP.

Current Behaviour
=================

Before describing the new semantics, it's worth covering the existing
semantics for Python 2.4 (as they are currently defined only by the
source code and the command line help).

When ``-m`` is used on the command line, it immediately terminates the
option list (like ``-c``).  The argument is interpreted as the name of
a top-level Python module (i.e. one which can be found on
``sys.path``).

If the module is found, and is of type ``PY_SOURCE`` or
``PY_COMPILED``, then the command line is effectively reinterpreted
from ``python <options> -m <module> <args>`` to ``python <options>
<filename> <args>``.  This includes setting ``sys.argv[0]`` correctly
(some scripts rely on this - Python's own ``regrtest.py`` is one
example).

If the module is not found, or is not of the correct type, an error
is printed.


Proposed Semantics
==================

The semantics proposed are fairly simple: if ``-m`` is used to execute
a module the :pep:`302` import mechanisms are used to locate the module and
retrieve its compiled code, before executing the module in accordance
with the semantics for a top-level module. The interpreter does this by
invoking a new standard library function ``runpy.run_module``.

This is necessary due to the way Python's import machinery locates
modules inside packages.  A package may modify its own __path__
variable during initialisation.  In addition, paths may be affected by
``*.pth`` files, and some packages will install custom loaders on
``sys.metapath``.  Accordingly, the only way for Python to reliably
locate the module is by importing the containing package and
using the :pep:`302` import hooks to gain access to the Python code.

Note that the process of locating the module to be executed may require
importing the containing package.  The effects of such a package import
that will be visible to the executed module are:

- the containing package will be in sys.modules

- any external effects of the package initialisation (e.g. installed
  import hooks, loggers, atexit handlers, etc.)


Reference Implementation
========================

A reference implementation is available on SourceForge ([2]_), along
with documentation for the library reference ([5]_).  There are
two parts to this implementation. The first is a proposed standard
library module ``runpy``. The second is a modification to the code
implementing the ``-m`` switch to always delegate to
``runpy.run_module`` instead of trying to run the module directly.
The delegation has the form::

  runpy.run_module(sys.argv[0], run_name="__main__", alter_sys=True)

``run_module`` is the only function ``runpy`` exposes in its public API.

``run_module(mod_name[, init_globals][, run_name][, alter_sys])``

    Execute the code of the specified module and return the resulting
    module globals dictionary. The module's code is first located using
    the standard import mechanism (refer to :pep:`302` for details) and
    then executed in a fresh module namespace.

    The optional dictionary argument ``init_globals`` may be used to
    pre-populate the globals dictionary before the code is executed.
    The supplied dictionary will not be modified. If any of the special
    global variables below are defined in the supplied dictionary, those
    definitions are overridden by the run_module function.

    The special global variables ``__name__``, ``__file__``,
    ``__loader__`` and ``__builtins__`` are set in the globals dictionary
    before the module code is executed.

    ``__name__`` is set to ``run_name`` if this optional argument is
    supplied, and the original ``mod_name`` argument otherwise.

    ``__loader__`` is set to the :pep:`302` module loader used to retrieve
    the code for the module (This loader may be a wrapper around the
    standard import mechanism).

    ``__file__`` is set to the name provided by the module loader. If
    the loader does not make filename information available, this
    argument is set to ``None``.

    ``__builtins__`` is automatically initialised with a reference to
    the top level namespace of the ``__builtin__`` module.

    If the argument ``alter_sys`` is supplied and evaluates to ``True``,
    then ``sys.argv[0]`` is updated with the value of ``__file__``
    and ``sys.modules[__name__]`` is updated with a temporary module
    object for the module being executed. Both ``sys.argv[0]`` and
    ``sys.modules[__name__]`` are restored to their original values
    before this function returns.

When invoked as a script, the ``runpy`` module finds and executes the
module supplied as the first argument.  It adjusts ``sys.argv`` by
deleting ``sys.argv[0]`` (which refers to the ``runpy`` module itself)
and then invokes ``run_module(sys.argv[0], run_name="__main__",
alter_sys=True)``.

Import Statements and the Main Module
=====================================

The release of 2.5b1 showed a surprising  (although obvious in
retrospect) interaction between this PEP and :pep:`328` - explicit
relative imports don't work from a main module. This is due to
the fact that relative imports rely on ``__name__`` to determine
the current module's position in the package hierarchy. In a main
module, the 


id: pep-8002.rst
content:
PEP: 8002
Title: Open Source Governance Survey
Author: Barry Warsaw <barry@python.org>, Åukasz Langa <lukasz@python.org>,
        Antoine Pitrou <solipsis@pitrou.net>, Doug Hellmann <doug@doughellmann.com>,
        Carol Willing <willingc@gmail.com>
Status: Final
Type: Informational
Topic: Governance
Created: 24-Aug-2018


Abstract
========

This PEP surveys existing and similar open source and free software projects
for their governance models, providing summaries that will serve as useful
references for Python's own selection of a new governance model in the wake of
`Guido's retirement
<https://mail.python.org/pipermail/python-committers/2018-July/005664.html>`_.

Rather than an individual PEP for each of these community surveys, they will
all be collected here in this PEP.


Rationale
=========

CPython is not the first open source project to undergo a governance crisis.
Other projects have experimented various governance options, sometimes several
times during their existence.  There are useful lessons to take away of their
experience, which will help inform our own decision.

Project choice
--------------

There are many open source projects out there, but it will be most fruitful
to survey those which are similar enough to CPython on a couple key metrics:

1. the number of contributors and their activity (there are scaling issues that
   don't make the governance models of very small projects very enlightening
   for our purposes) ;
2. being mostly or partly community-driven (company-driven projects can afford
   different governance options, since the company hierarchy has power over
   the main participants) ;
3. being faced with important design decisions that require a somewhat formal
   decision process.


Rust
====

The governance structure is documented in `Rust RFC #1068
<https://github.com/rust-lang/rfcs/blob/master/text/1068-rust-governance.md>`_.

The effective governance process grows organically over time without being entirely
codified as RFCs, especially in case of day-to-day operation details.  One example is
the `formation of Domain Working Groups
<https://internals.rust-lang.org/t/announcing-the-2018-domain-working-groups/6737>`_ in
February 2018.

Key people and their functions
------------------------------

In the Rust project there are teams responsible for certain areas. For language features
there is a "lang team", for tooling there's "dev tools" and "Cargo", and so on.
Contentious issues have facilitators to drive discussion who often aren't the decision
makers.  Typically the facilitators are authors of the proposed changes (see
"Controversial decision process" below).  They ensure all key decision makers are
involved along with interested community members.  They push towards an agreeable
outcome via iteration.

In practice this means decisions are rarely escalated to the core team.

The most common role of a contributor is team membership.  Issue triage/code review
privileges without team membership is rare.  Contributors have full commit access,
code ownership separation is based on trust.  Writing to the compiler repository is
frowned upon, all changes go through pull requests and get merged by an integration
bot after they were reviewed and approved.

New team members are added by nomination by an existing team member.

Regular decision process
------------------------

Primary work happens via GitHub issues and pull requests.  Approving a pull request
by any team member allows it to be merged without further process.  All merged pull
requests end up in the next stable version of Rust.

Notifying relevant people by mentions is important.  Listening to the firehose of
e-mails for all GitHub activity is not popular.

There are planning and triage meetings open to the public happening on IRC and Discord.
They are not very popular because most of work happens through GitHub.  Discussions also
happen on official Rust forums (https://users.rust-lang.org/ and
https://internals.rust-lang.org/).  There is a dedicated moderation team responsible for
taking notes and enforcing `code of conduct
<https://www.rust-lang.org/en-US/conduct.html>`_.

Controversial decision process
------------------------------

Larger or controversial work goes through a `RFC process
<https://github.com/rust-lang/rfcs>`_.  It allows everyone to express their thoughts and
iterates towards a resolution.  At some point when all blocking concerns of relevant
team members are addressed, they sign off on the RFC and it reaches a "final comment
period". That does not require consensus amongst all participants, rather there should
not be a strong consensus against the proposal.

After 10 days the RFC is *merged* unless any new blocking concerns are raised by team
members.  A "merge" signifies that work towards implementing the feature and integrating
it can now happen without interruption.  An RFC doesn't have to have a reference
implementation for it to be accepted.

The other possible results of the "final comment period" are to:

* *postpone* the RFC (similar to the Deferred status in PEPs),
* get it *back into discussion* if blocking concerns can be addressed, or
* *close it* if blocking concerns are not solvable.  When an RFC is marked as
  *closed*, there is a 7-day grace period to debate whether it should be closed.

In practice registering concerns with an RFC happens very often initially but rarely
causes for the RFC to be entirely killed.

This process scales well for small-contention changes and/or smaller changes.  For the
largest controversial changes the discussion gets unwieldy.  This is a topic currently
(as of August 2018) on the minds of the Rust team (see:
`"Listening and Trust, part 1" <http://aturon.github.io/2018/05/25/listening-part-1/>`_,
`"Listening and Trust, part 2" <http://aturon.github.io/2018/06/02/listening-part-2/>`_,
`"Listening and Trust, part 3" <http://aturon.github.io/2018/06/18/listening-part-3/>`_,
`"Proposal for a staged RFC process"
<http://smallcultfollowing.com/babysteps/blog/2018/06/20/proposal-for-a-staged-rfc-process/>`_).

Planning a new release
----------------------

Every six weeks the Rust compiler is released with whatever it contained at the time.
There are no LTS channels or releases yet but this concept is planned to make
redistributors able to keep up with development better.

Every few years a so-called `"Edition"
<https://rust-lang-nursery.github.io/edition-guide/editions/index.html>`_ is released.
Those are milestone releases with full sets of updated documentation and tooling. They
can be backwards incompatible with previous editions.  External packages opt into
breaking changes in their crate metadata.  The Rust compiler supports all editions that
existed prior to its release.  Linking between crates of any supported edition is
possible.

Changes in the process over time
--------------------------------

The Rust programming language was started by Graydon Hoare who developed it as
a personal project for a few years.  When Mozilla started sponsoring the project,
the team slowly grew with Graydon as a BDFL-style figure.  He `left the project
<https://www.reddit.com/r/rust/comments/7qels2/i_wonder_why_graydon_hoare_the_author_of_rust/dsqeh1d/>`_
in 2013.  Rust functions without a BDFL since.  The RFC process was put in place later.
Initially some design discussions happened during closed-door weekly video meetings
which was `shut down
<https://github.com/rust-lang/meeting-minutes/blob/master/weekly-meetings/2015-05-26.md#future-of-weekly-meeting>`_
in May 2015 (before the 1.0 release of Rust), organically replaced with open discussion
and direct influence of teams.

The number of teams is growing in time.  The number of technical decisions made by the
core team is decreasing, instead those get delegated to respective teams.

The concept of a "final comment period" was introduced to encourage more public
discussion and enable reacting to a change *about to* being made, instead of having to
rever


id: pep-8016.rst
content:
PEP: 8016
Title: The Steering Council Model
Author: Nathaniel J. Smith, Donald Stufft
Status: Accepted
Type: Informational
Topic: Governance
Created: 01-Nov-2018

Note
====

This PEP is retained for historical purposes, but the official
governance document is now :pep:`13`.


Abstract
========

This PEP proposes a model of Python governance based around a steering
council. The council has broad authority, which they seek to exercise
as rarely as possible; instead, they use this power to establish
standard processes, like those proposed in the other 801x-series PEPs.
This follows the general philosophy that it's better to split up large
changes into a series of small changes that can be reviewed
independently: instead of trying to do everything in one PEP, we focus
on providing a minimal-but-solid foundation for further governance
decisions.


PEP Acceptance
==============

:pep:`8016` was accepted `by a core developer vote
<https://discuss.python.org/t/python-governance-vote-december-2018-results/546/>`__
described in :pep:`8001` on Monday, December 17, 2018.


Rationale
=========

The main goals of this proposal are:

* Be **boring**: We're not experts in governance, and we don't think
  Python is a good place to experiment with new and untried governance
  models. So this proposal sticks to mature, well-known, previously
  tested processes as much as possible. The high-level approach of a
  mostly-hands-off council is arguably the most common across large
  successful F/OSS projects, and low-level details are derived
  directly from Django's governance.
* Be **simple**: We've attempted to pare things down to the minimum
  needed to make this workable: the council, the core team (who elect
  the council), and the process for changing the document. The goal is
  Minimum Viable Governance.
* Be **comprehensive**: But for the things we need to define, we've
  tried to make sure to cover all the bases, because we don't want to
  go through this kind of crisis again. Having a clear and unambiguous
  set of rules also helps minimize confusion and resentment.
* Be **flexible and light-weight**: We know that it will take time and
  experimentation to find the best processes for working together. By
  keeping this document as minimal as possible, we keep maximal
  flexibility for adjusting things later, while minimizing the need
  for heavy-weight and anxiety-provoking processes like whole-project
  votes.

A number of details were discussed in `this Discourse thread
<https://discuss.python.org/t/working-discussion-for-pep-8016-the-boringest-possible-steering-council-model/333/>`__,
and then `this thread has further discussion
<https://discuss.python.org/t/pep-8016-the-steering-council-model/394>`__. These
may be useful to anyone trying to understand the rationale for various
minor decisions.


Specification
=============

The steering council
--------------------

Composition
~~~~~~~~~~~

The steering council is a 5-person committee.


Mandate
~~~~~~~

The steering council shall work to:

* Maintain the quality and stability of the Python language and
  CPython interpreter,
* Make contributing as accessible, inclusive, and sustainable as
  possible,
* Formalize and maintain the relationship between the core team and
  the PSF,
* Establish appropriate decision-making processes for PEPs,
* Seek consensus among contributors and the core team before acting in
  a formal capacity,
* Act as a "court of final appeal" for decisions where all other
  methods have failed.


Powers
~~~~~~

The council has broad authority to make decisions about the project.
For example, they can:

* Accept or reject PEPs
* Enforce or update the project's code of conduct
* Work with the PSF to manage any project assets
* Delegate parts of their authority to other subcommittees or
  processes

However, they cannot modify this PEP, or affect the membership of the
core team, except via the mechanisms specified in this PEP.

The council should look for ways to use these powers as little as
possible. Instead of voting, it's better to seek consensus. Instead of
ruling on individual PEPs, it's better to define a standard process
for PEP decision making (for example, by accepting one of the other
801x series of PEPs). It's better to establish a Code of Conduct
committee than to rule on individual cases. And so on.

To use its powers, the council votes. Every council member must either
vote or explicitly abstain. Members with conflicts of interest on a
particular vote must abstain. Passing requires support from a majority
of non-abstaining council members.

Whenever possible, the council's deliberations and votes shall be held
in public.


Electing the council
~~~~~~~~~~~~~~~~~~~~

A council election consists of two phases:

* Phase 1: Candidates advertise their interest in serving. Candidates
  must be nominated by a core team member. Self-nominations are
  allowed.

* Phase 2: Each core team member can vote for zero to five of the
  candidates. Voting is performed anonymously. Candidates are ranked
  by the total number of votes they receive. If a tie occurs, it may
  be resolved by mutual agreement among the candidates, or else the
  winner will be chosen at random.

Each phase lasts one to two weeks, at the outgoing council's discretion.
For the initial election, both phases will last two weeks.

The election process is managed by a returns officer nominated by the
outgoing steering council. For the initial election, the returns
officer will be nominated by the PSF Executive Director.

The council should ideally reflect the diversity of Python
contributors and users, and core team members are encouraged to vote
accordingly.


Term
~~~~

A new council is elected after each feature release. Each council's
term runs from when their election results are finalized until the
next council's term starts. There are no term limits.


Vacancies
~~~~~~~~~

Council members may resign their position at any time.

Whenever there is a vacancy during the regular council term, the
council may vote to appoint a replacement to serve out the rest of the
term.

If a council member drops out of touch and cannot be contacted for a
month or longer, then the rest of the council may vote to replace
them.


Conflicts of interest
~~~~~~~~~~~~~~~~~~~~~

While we trust council members to act in the best interests of Python
rather than themselves or their employers, the mere appearance of any
one company dominating Python development could itself be harmful and
erode trust. In order to avoid any appearance of conflict of interest,
at most 2 members of the council can work for any single employer.

In a council election, if 3 of the top 5 vote-getters work for the
same employer, then whichever of them ranked lowest is disqualified
and the 6th-ranking candidate moves up into 5th place; this is
repeated until a valid council is formed.

During a council term, if changing circumstances cause this rule to be
broken (for instance, due to a council member changing employment),
then one or more council members must resign to remedy the issue, and
the resulting vacancies can then be filled as normal.


Ejecting core team members
~~~~~~~~~~~~~~~~~~~~~~~~~~

In exceptional circumstances, it may be necessary to remove someone
from the core team against their will. (For example: egregious and
ongoing code of conduct violations.) This can be accomplished by a
steering council vote, but unlike other steering council votes, this
requires at least a two-thirds majority. With 5 members voting, this
means that a 3:2 vote is insufficient; 4:1 in favor is the minimum
required for such a vote to succeed. In addition, this is the one
power of the steering council which cannot be delegated, and this
power cannot be used while a vote of no confidence is in process.

If the ejected core team member is also on the steering council, then
they are removed from the steering council as well.


Vote of no confidence
~~~~~~~~~~~~~~~~~~~~~

In exception


id: pep-0304.rst
content:
PEP: 304
Title: Controlling Generation of Bytecode Files
Author: Skip Montanaro
Status: Withdrawn
Type: Standards Track
Created: 22-Jan-2003
Post-History: 27-Jan-2003, 31-Jan-2003, 17-Jun-2005

Historical Note
===============

While this original PEP was withdrawn, a variant of this feature
was eventually implemented for Python 3.8 in https://bugs.python.org/issue33499

Several of the issues and concerns originally raised in this PEP were resolved
by other changes in the intervening years:

- the introduction of isolated mode to handle potential security concerns
- the switch to ``importlib``, a fully import-hook based import system implementation
- :pep:`3147`'s change in the bytecode cache layout to use ``__pycache__``
  subdirectories, including the ``source_to_cache(path)`` and
  ``cache_to_source(path)`` APIs that allow the interpreter to automatically
  handle the redirection to a separate cache directory

Abstract
========

This PEP outlines a mechanism for controlling the generation and
location of compiled Python bytecode files.  This idea originally
arose as a patch request [1]_ and evolved into a discussion thread on
the python-dev mailing list [2]_.  The introduction of an environment
variable will allow people installing Python or Python-based
third-party packages to control whether or not bytecode files should
be generated at installation time, and if so, where they should be
written.  It will also allow users to control whether or not bytecode
files should be generated at application run-time, and if so, where
they should be written.


Proposal
========

Add a new environment variable, PYTHONBYTECODEBASE, to the mix of
environment variables which Python understands.  PYTHONBYTECODEBASE is
interpreted as follows:

- If not defined, Python bytecode is generated in exactly the same way
  as is currently done.  sys.bytecodebase is set to the root directory
  (either / on Unix and Mac OSX or the root directory of the startup
  (installation???) drive -- typically ``C:\`` -- on Windows).

- If defined and it refers to an existing directory to which the user
  has write permission, sys.bytecodebase is set to that directory and
  bytecode files are written into a directory structure rooted at that
  location.

- If defined but empty, sys.bytecodebase is set to None and generation
  of bytecode files is suppressed altogether.

- If defined and one of the following is true:

  * it does not refer to a directory,

  * it refers to a directory, but not one for which the user has write
    permission

  a warning is displayed, sys.bytecodebase is set to None and
  generation of bytecode files is suppressed altogether.

After startup initialization, all runtime references are to
sys.bytecodebase, not the PYTHONBYTECODEBASE environment variable.
sys.path is not modified.

From the above, we see sys.bytecodebase can only take on two valid
types of values: None or a string referring to a valid directory on
the system.

During import, this extension works as follows:

- The normal search for a module is conducted.  The search order is
  roughly: dynamically loaded extension module, Python source file,
  Python bytecode file.  The only time this mechanism comes into play
  is if a Python source file is found.

- Once we've found a source module, an attempt to read a byte-compiled
  file in the same directory is made.  (This is the same as before.)

- If no byte-compiled file is found, an attempt to read a
  byte-compiled file from the augmented directory is made.

- If bytecode generation is required, the generated bytecode is written
  to the augmented directory if possible.

Note that this PEP is explicitly *not* about providing
module-by-module or directory-by-directory control over the
disposition of bytecode files.


Glossary
--------

- "bytecode base" refers to the current setting of
  sys.bytecodebase.

- "augmented directory" refers to the directory formed from the
  bytecode base and the directory name of the source file.

- PYTHONBYTECODEBASE refers to the environment variable when necessary
  to distinguish it from "bytecode base".


Locating bytecode files
-----------------------

When the interpreter is searching for a module, it will use sys.path
as usual.  However, when a possible bytecode file is considered, an
extra probe for a bytecode file may be made.  First, a check is made
for the bytecode file using the directory in sys.path which holds the
source file (the current behavior).  If a valid bytecode file is not
found there (either one does not exist or exists but is out-of-date)
and the bytecode base is not None, a second probe is made using the
directory in sys.path prefixed appropriately by the bytecode base.


Writing bytecode files
----------------------

When the bytecode base is not None, a new bytecode file is written to
the appropriate augmented directory, never directly to a directory in
sys.path.


Defining augmented directories
------------------------------

Conceptually, the augmented directory for a bytecode file is the
directory in which the source file exists prefixed by the bytecode
base.  In a Unix environment this would be::

    pcb = os.path.abspath(sys.bytecodebase)
    if sourcefile[0] == os.sep: sourcefile = sourcefile[1:]
    augdir = os.path.join(pcb, os.path.dirname(sourcefile))

On Windows, which does not have a single-rooted directory tree, the
drive letter of the directory containing the source file is treated as
a directory component after removing the trailing colon.  The
augmented directory is thus derived as ::

    pcb = os.path.abspath(sys.bytecodebase)
    drive, base = os.path.splitdrive(os.path.dirname(sourcefile))
    drive = drive[:-1]
    if base[0] == "\\": base = base[1:]
    augdir = os.path.join(pcb, drive, base)


Fixing the location of the bytecode base
----------------------------------------

During program startup, the value of the PYTHONBYTECODEBASE
environment variable is made absolute, checked for validity and added
to the sys module, effectively::

    pcb = os.path.abspath(os.environ["PYTHONBYTECODEBASE"])
    probe = os.path.join(pcb, "foo")
    try:
        open(probe, "w")
    except IOError:
        sys.bytecodebase = None
    else:
        os.unlink(probe)
        sys.bytecodebase = pcb

This allows the user to specify the bytecode base as a relative path,
but not have it subject to changes to the current working directory
during program execution.  (I can't imagine you'd want it to move
around during program execution.)

There is nothing special about sys.bytecodebase.  The user may change
it at runtime if desired, but normally it will not be modified.


Rationale
=========

In many environments it is not possible for non-root users to write
into directories containing Python source files.  Most of the time,
this is not a problem as Python source is generally byte compiled
during installation.  However, there are situations where bytecode
files are either missing or need to be updated.  If the directory
containing the source file is not writable by the current user a
performance penalty is incurred each time a program importing the
module is run. [3]_ Warning messages may also be generated in certain
circumstances.  If the directory is writable, nearly simultaneous
attempts to write the bytecode file by two separate processes
may occur, resulting in file corruption. [4]_

In environments with RAM disks available, it may be desirable for
performance reasons to write bytecode files to a directory on such a
disk.  Similarly, in environments where Python source code resides on
network file systems, it may be desirable to cache bytecode files on
local disks.


Alternatives
============

The only other alternative proposed so far [1]_ seems to be to add a
-R flag to the interpreter to disable writing bytecode files
altogether.  This proposal subsumes that.  Adding a command-line
option is certainly possible, but is probably not sufficient, as the
interpreter's command line is 


id: pep-0462.rst
content:
PEP: 462
Title: Core development workflow automation for CPython
Author: Alyssa Coghlan <ncoghlan@gmail.com>
Status: Withdrawn
Type: Process
Requires: 474
Created: 23-Jan-2014
Post-History: 25-Jan-2014, 27-Jan-2014, 01-Feb-2015


Abstract
========

This PEP proposes investing in automation of several of the tedious,
time-consuming activities that are currently required for the core development
team to incorporate changes into CPython. This proposal is intended to
allow core developers to make more effective use of the time they have
available to contribute to CPython, which should also result in an improved
experience for other contributors that are reliant on the core team to get
their changes incorporated.


PEP Withdrawal
==============

This PEP has been `withdrawn by the author
<https://mail.python.org/pipermail/core-workflow/2015-October/000227.html>`_
in favour of the GitLab based proposal in :pep:`507`.

If anyone else would like to take over championing this PEP, contact the
`core-workflow mailing list <https://mail.python.org/mailman/listinfo/core-workflow>`_


Rationale for changes to the core development workflow
======================================================

The current core developer workflow to merge a new feature into CPython
on a POSIX system "works" as follows:

#. If applying a change submitted to bugs.python.org by another user, first
   check they have signed the PSF Contributor Licensing Agreement. If not,
   request that they sign one before continuing with merging the change.
#. Apply the change locally to a current checkout of the main CPython
   repository (the change will typically have been discussed and reviewed
   as a patch on bugs.python.org first, but this step is not currently
   considered mandatory for changes originating directly from core
   developers).
#. Run the test suite locally, at least ``make test`` or
   ``./python -m test`` (depending on system specs, this takes a few
   minutes in the default configuration, but substantially longer if all
   optional resources, like external network access, are enabled).
#. Run ``make patchcheck`` to fix any whitespace issues and as a reminder
   of other changes that may be needed (such as updating Misc/ACKS or
   adding an entry to Misc/NEWS)
#. Commit the change and push it to the main repository. If hg indicates
   this would create a new head in the remote repository, run
   ``hg pull --rebase`` (or an equivalent). Theoretically, you should
   rerun the tests at this point, but it's *very* tempting to skip that
   step.
#. After pushing, monitor the `stable buildbots
   <http://buildbot.python.org/all/waterfall?category=3.x.stable>`__
   for any new failures introduced by your change. In particular, developers
   on POSIX systems will often break the Windows buildbots, and vice-versa.
   Less commonly, developers on Linux or Mac OS X may break other POSIX
   systems.

The steps required on Windows are similar, but the exact commands used
will be different.

Rather than being simpler, the workflow for a bug fix is *more* complicated
than that for a new feature! New features have the advantage of only being
applied to the ``default`` branch, while bug fixes also need to be considered
for inclusion in maintenance branches.

* If a bug fix is applicable to Python 2.7, then it is also separately
  applied to the 2.7 branch, which is maintained as an independent head
  in Mercurial
* If a bug fix is applicable to the current 3.x maintenance release, then
  it is first applied to the maintenance branch and then merged forward
  to the default branch. Both branches are pushed to hg.python.org at the
  same time.

Documentation patches are simpler than functional patches, but not
hugely so - the main benefit is only needing to check the docs build
successfully rather than running the test suite.

I would estimate that even when everything goes smoothly, it would still
take me at least 20-30 minutes to commit a bug fix patch that applies
cleanly. Given that it should be possible to automate several of these
tasks, I do not believe our current practices are making effective use
of scarce core developer resources.

There are many, many frustrations involved with this current workflow, and
they lead directly to some undesirable development practices.

* Much of this overhead is incurred on a per-patch applied basis. This
  encourages large commits, rather than small isolated changes. The time
  required to commit a 500 line feature is essentially the same as that
  needed to commit a 1 line bug fix - the additional time needed for the
  larger change appears in any preceding review rather than as part of the
  commit process.
* The additional overhead of working on applying bug fixes creates an
  additional incentive to work on new features instead, and new features
  are already *inherently* more interesting to work on - they don't need
  workflow difficulties giving them a helping hand!
* Getting a preceding review on bugs.python.org is *additional* work,
  creating an incentive to commit changes directly, increasing the reliance
  on post-review on the python-checkins mailing list.
* Patches on the tracker that are complete, correct and ready to merge may
  still languish for extended periods awaiting a core developer with the
  time to devote to getting it merged.
* The risk of push races (especially when pushing a merged bug fix) creates
  a temptation to skip doing full local test runs (especially after a push
  race has already been encountered once), increasing the chance of
  breaking the buildbots.
* The buildbots are sometimes red for extended periods, introducing errors
  into local test runs, and also meaning that they sometimes fail to serve
  as a reliable indicator of whether or not a patch has introduced cross
  platform issues.
* Post-conference development sprints are a nightmare, as they collapse
  into a mire of push races. It's tempting to just leave patches on the
  tracker until after the sprint is over and then try to clean them up
  afterwards.

There are also many, many opportunities for core developers to make
mistakes that inconvenience others, both in managing the Mercurial branches
and in breaking the buildbots without being in a position to fix them
promptly. This both makes the existing core development team cautious in
granting new developers commit access, as well as making those new
developers cautious about actually making use of their increased level of
access.

There are also some incidental annoyances (like keeping the NEWS file up to
date) that will also be necessarily addressed as part of this proposal.

One of the most critical resources of a volunteer-driven open source project
is the emotional energy of its contributors. The current approach to change
incorporation doesn't score well on that front for anyone:

* For core developers, the branch wrangling for bug fixes is delicate and
  easy to get wrong. Conflicts on the NEWS file and push races when
  attempting to upload changes add to the irritation of something most of
  us aren't being paid to spend time on (and for those that are, contributing
  to CPython is likely to be only one of our responsibilities). The time we
  spend actually getting a change merged is time we're not spending coding
  additional changes, writing or updating documentation or reviewing
  contributions from others.
* Red buildbots make life difficult for other developers (since a local
  test failure may *not* be due to anything that developer did), release
  managers (since they may need to enlist assistance cleaning up test
  failures prior to a release) and for the developers themselves (since
  it creates significant pressure to fix any failures we inadvertently
  introduce right *now*, rather than at a more convenient time, as well
  as potentially making ``hg bisect`` more difficult to use if
  ``hg annotate`` isn't sufficient to identify the source of a new failure).
* For other contribu


id: pep-0476.rst
content:
PEP: 476
Title: Enabling certificate verification by default for stdlib http clients
Author: Alex Gaynor <alex.gaynor@gmail.com>
Status: Final
Type: Standards Track
Created: 28-Aug-2014
Python-Version: 2.7.9, 3.4.3, 3.5
Resolution: https://mail.python.org/pipermail/python-dev/2014-October/136676.html

Abstract
========

Currently when a standard library http client (the ``urllib``, ``urllib2``,
``http``, and ``httplib`` modules) encounters an ``https://`` URL it will wrap
the network HTTP traffic in a TLS stream, as is necessary to communicate with
such a server. However, during the TLS handshake it will not actually check
that the server has an X509 certificate is signed by a CA in any trust root,
nor will it verify that the Common Name (or Subject Alternate Name) on the
presented certificate matches the requested host.

The failure to do these checks means that anyone with a privileged network
position is able to trivially execute a man in the middle attack against a
Python application using either of these HTTP clients, and change traffic at
will.

This PEP proposes to enable verification of X509 certificate signatures, as
well as hostname verification for Python's HTTP clients by default, subject to
opt-out on a per-call basis. This change would be applied to Python 2.7, Python
3.4, and Python 3.5.

Rationale
=========

The "S" in "HTTPS" stands for secure. When Python's users type "HTTPS" they are
expecting a secure connection, and Python should adhere to a reasonable
standard of care in delivering this. Currently we are failing at this, and in
doing so, APIs which appear simple are misleading users.

When asked, many Python users state that they were not aware that Python failed
to perform these validations, and are shocked.

The popularity of ``requests`` (which enables these checks by default)
demonstrates that these checks are not overly burdensome in any way, and the
fact that it is widely recommended as a major security improvement over the
standard library clients demonstrates that many expect a higher standard for
"security by default" from their tools.

The failure of various applications to note Python's negligence in this matter
is a source of *regular* CVE assignment [#]_ [#]_ [#]_ [#]_ [#]_ [#]_ [#]_ [#]_
[#]_ [#]_ [#]_.

.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4340
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-3533
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-5822
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-5825
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-1909
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2037
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2073
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2191
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-4111
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-6396
.. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-6444

Technical Details
=================

Python would use the system provided certificate database on all platforms.
Failure to locate such a database would be an error, and users would need to
explicitly specify a location to fix it.

This will be achieved by adding a new ``ssl._create_default_https_context``
function, which is the same as ``ssl.create_default_context``.

``http.client`` can then replace its usage of ``ssl._create_stdlib_context``
with the ``ssl._create_default_https_context``.

Additionally ``ssl._create_stdlib_context`` is renamed
``ssl._create_unverified_context`` (an alias is kept around for backwards
compatibility reasons).

Trust database
--------------

This PEP proposes using the system-provided certificate database. Previous
discussions have suggested bundling Mozilla's certificate database and using
that by default. This was decided against for several reasons:

* Using the platform trust database imposes a lower maintenance burden on the
  Python developers -- shipping our own trust database would require doing a
  release every time a certificate was revoked.
* Linux vendors, and other downstreams, would unbundle the Mozilla
  certificates, resulting in a more fragmented set of behaviors.
* Using the platform stores makes it easier to handle situations such as
  corporate internal CAs.

OpenSSL also has a pair of environment variables, ``SSL_CERT_DIR`` and
``SSL_CERT_FILE`` which can be used to point Python at a different certificate
database.

Backwards compatibility
-----------------------

This change will have the appearance of causing some HTTPS connections to
"break", because they will now raise an Exception during handshake.

This is misleading however, in fact these connections are presently failing
silently, an HTTPS URL indicates an expectation of confidentiality and
authentication. The fact that Python does not actually verify that the user's
request has been made is a bug, further: "Errors should never pass silently."

Nevertheless, users who have a need to access servers with self-signed or
incorrect certificates would be able to do so by providing a context with
custom trust roots or which disables validation (documentation should strongly
recommend the former where possible). Users will also be able to add necessary
certificates to system trust stores in order to trust them globally.

Twisted's 14.0 release made this same change, and it has been met with almost
no opposition.

Opting out
----------

For users who wish to opt out of certificate verification on a single
connection, they can achieve this by providing the ``context`` argument to
``urllib.urlopen``::

    import ssl

    # This restores the same behavior as before.
    context = ssl._create_unverified_context()
    urllib.urlopen("https://no-valid-cert", context=context)

It is also possible, **though highly discouraged**, to globally disable
verification by monkeypatching the ``ssl`` module in versions of Python that
implement this PEP::

    import ssl

    try:
        _create_unverified_https_context = ssl._create_unverified_context
    except AttributeError:
        # Legacy Python that doesn't verify HTTPS certificates by default
        pass
    else:
        # Handle target environment that doesn't support HTTPS verification
        ssl._create_default_https_context = _create_unverified_https_context

This guidance is aimed primarily at system administrators that wish to adopt
newer versions of Python that implement this PEP in legacy environments that
do not yet support certificate verification on HTTPS connections. For
example, an administrator may opt out by adding the monkeypatch above to
``sitecustomize.py`` in their Standard Operating Environment for Python.
Applications and libraries SHOULD NOT be making this change process wide
(except perhaps in response to a system administrator controlled configuration
setting).

Particularly security sensitive applications should always provide an explicit
application defined SSL context rather than relying on the default behaviour
of the underlying Python implementation.

Other protocols
===============

This PEP only proposes requiring this level of validation for HTTP clients, not
for other protocols such as SMTP.

This is because while a high percentage of HTTPS servers have correct
certificates, as a result of the validation performed by browsers, for other
protocols self-signed or otherwise incorrect certificates are far more common.
Note that for SMTP at least, this appears to be changing and should be reviewed
for a potential similar PEP in the future:

* https://www.facebook.com/notes/protect-the-graph/the-current-state-of-smtp-starttls-deployment/1453015901605223
* https://www.facebook.com/notes/protect-the-graph/massive-growth-in-smtp-starttls-deployment/1491049534468526

Python Versions
===============

This PEP describes changes that will occur on both the 3.4.x, 3.5 and 2.7.X
branches. For 2.7.X this will requ


id: pep-0310.rst
content:
PEP: 310
Title: Reliable Acquisition/Release Pairs
Author: Michael Hudson <mwh@python.net>,
        Paul Moore <p.f.moore@gmail.com>
Status: Rejected
Type: Standards Track
Created: 18-Dec-2002
Python-Version: 2.4
Post-History:


Abstract
========

It would be nice to have a less typing-intense way of writing::

    the_lock.acquire()
    try:
        ....
    finally:
        the_lock.release()

This PEP proposes a piece of syntax (a 'with' block) and a
"small-i" interface that generalizes the above.


Pronouncement
=============

This PEP is rejected in favor of :pep:`343`.


Rationale
=========

One of the advantages of Python's exception handling philosophy is
that it makes it harder to do the "wrong" thing (e.g. failing to
check the return value of some system call).  Currently, this does
not apply to resource cleanup.  The current syntax for acquisition
and release of a resource (for example, a lock) is::

   the_lock.acquire()
   try:
       ....
   finally:
       the_lock.release()

This syntax separates the acquisition and release by a (possibly
large) block of code, which makes it difficult to confirm "at a
glance" that the code manages the resource correctly.  Another
common error is to code the "acquire" call within the try block,
which incorrectly releases the lock if the acquire fails.


Basic Syntax and Semantics
==========================

The syntax of a 'with' statement is as follows::

    'with' [ var '=' ] expr ':'
        suite

This statement is defined as being equivalent to the following
sequence of statements::

    var = expr

    if hasattr(var, "__enter__"):
        var.__enter__()

    try:
        suite

    finally:
        var.__exit__()

(The presence of an ``__exit__`` method is *not* checked like that of
``__enter__`` to ensure that using inappropriate objects in with:
statements gives an error).

If the variable is omitted, an unnamed object is allocated on the
stack.  In that case, the suite has no access to the unnamed object.


Possible Extensions
===================

A number of potential extensions to the basic syntax have been
discussed on the Python Developers list.  None of these extensions
are included in the solution proposed by this PEP.  In many cases,
the arguments are nearly equally strong in both directions.  In
such cases, the PEP has always chosen simplicity, simply because
where extra power is needed, the existing try block is available.

Multiple expressions
--------------------

One proposal was for allowing multiple expressions within one
'with' statement.  The ``__enter__`` methods would be called left to
right, and the ``__exit__`` methods right to left.  The advantage of
doing so is that where more than one resource is being managed,
nested 'with' statements will result in code drifting towards the
right margin.  The solution to this problem is the same as for any
other deep nesting - factor out some of the code into a separate
function.  Furthermore, the question of what happens if one of the
``__exit__`` methods raises an exception (should the other ``__exit__``
methods be called?) needs to be addressed.

Exception handling
------------------

An extension to the protocol to include an optional ``__except__``
handler, which is called when an exception is raised, and which
can handle or re-raise the exception, has been suggested.  It is
not at all clear that the semantics of this extension can be made
precise and understandable.  For example, should the equivalent
code be ``try ... except ... else`` if an exception handler is
defined, and ``try ... finally`` if not?  How can this be determined
at compile time, in general?  The alternative is to define the
code as expanding to a ``try ... except`` inside a ``try ... finally``.
But this may not do the right thing in real life.

The only use case identified for exception handling is with
transactional processing (commit on a clean finish, and rollback
on an exception).  This is probably just as easy to handle with a
conventional ``try ... except ... else`` block, and so the PEP does
not include any support for exception handlers.


Implementation Notes
====================

There is a potential race condition in the code specified as
equivalent to the with statement.  For example, if a
``KeyboardInterrupt`` exception is raised between the completion of
the ``__enter__`` method call and the start of the try block, the
``__exit__`` method will not be called.  This can lead to resource
leaks, or to deadlocks.  [XXX Guido has stated that he cares about
this sort of race condition, and intends to write some C magic to
handle them.  The implementation of the 'with' statement should
copy this.]


Open Issues
===========

Should existing classes (for example, file-like objects and locks)
gain appropriate ``__enter__`` and ``__exit__`` methods?  The obvious
reason in favour is convenience (no adapter needed).  The argument
against is that if built-in files have this but (say) ``StringIO``
does not, then code that uses "with" on a file object can't be
reused with a ``StringIO`` object.  So ``__exit__ = close`` becomes a part
of the "file-like object" protocol, which user-defined classes may
need to support.

The ``__enter__`` hook may be unnecessary - for many use cases, an
adapter class is needed and in that case, the work done by the
``__enter__`` hook can just as easily be done in the ``__init__`` hook.

If a way of controlling object lifetimes explicitly was available,
the function of the ``__exit__`` hook could be taken over by the
existing ``__del__`` hook.  An email exchange [1]_ with a proponent of
this approach left one of the authors even more convinced that
it isn't the right idea...

It has been suggested [2]_ that the "__exit__" method be called
"close", or that a "close" method should be considered if no
``__exit__`` method is found, to increase the "out-of-the-box utility"
of the "with ..." construct.

There are some similarities in concept between 'with ...' blocks
and generators, which have led to proposals that for loops could
implement the with block functionality [3]_.  While neat on some
levels, we think that for loops should stick to being loops.


Alternative Ideas
=================

IEXEC: Holger Krekel -- generalised approach with XML-like syntax
(no URL found...).

Holger has much more far-reaching ideas about "execution monitors"
that are informed about details of control flow in the monitored
block.  While interesting, these ideas could change the language
in deep and subtle ways and as such belong to a different PEP.

Any Smalltalk/Ruby anonymous block style extension obviously
subsumes this one.

:pep:`319` is in the same area, but did not win support when aired on
python-dev.


Backwards Compatibility
=======================

This PEP proposes a new keyword, so the ``__future__`` game will need
to be played.


Cost of Adoption
================

Those who claim the language is getting larger and more
complicated have something else to complain about.  It's something
else to teach.

For the proposal to be useful, many file-like and lock-like
classes in the standard library and other code will have to have ::

   __exit__ = close

or similar added.


Cost of Non-Adoption
====================

Writing correct code continues to be more effort than writing
incorrect code.


References
==========

There are various python-list and python-dev discussions that
could be mentioned here.

.. [1] Off-list conversation between Michael Hudson and Bill Soudan
       (made public with permission)
       http://starship.python.net/crew/mwh/pep310/

.. [2] Samuele Pedroni on python-dev
       https://mail.python.org/pipermail/python-dev/2003-August/037795.html

.. [3] Thread on python-dev with subject
       ``[Python-Dev] pre-PEP: Resource-Release Support for Generators``
       starting at
       https://mail.python.org/pipermail/python-dev/2003-August/037803.html

Copyright
=========

This document has been placed in the public domain.


id: pep-0648.rst
content:
PEP: 648
Title: Extensible customizations of the interpreter at startup
Author: Mario Corchero <mariocj89@gmail.com>
Sponsor: Pablo Galindo
Discussions-To: https://discuss.python.org/t/pep-648-extensible-customizations-of-the-interpreter-at-startup/6403
Status: Rejected
Type: Standards Track
Created: 30-Dec-2020
Python-Version: 3.11
Post-History: 16-Dec-2020, 18-Dec-2020

Abstract
========

This PEP proposes supporting extensible customization of the interpreter by
allowing users to install files that will be executed at startup.

PEP Rejection
=============

PEP 648 was rejected `by the steering council
<https://mail.python.org/archives/list/python-dev@python.org/message/UHODNOGISLYUKX2K2JCCBYMZFEWZDSPO/>`__
as it has a limited number of use cases and further complicates the startup sequence.

Motivation
==========

System administrators, tools that repackage the interpreter and some
libraries need to customize aspects of the interpreter at startup time.

This is usually achieved via ``sitecustomize.py`` for system administrators
whilst libraries rely on exploiting ``pth`` files. This PEP proposes a way of
achieving the same functionality in a more user-friendly and structured way.

Limitations of ``pth`` files
----------------------------

If a library needs to perform any customization before an import or that
relates to the general working of the interpreter, they often rely on the
fact that ``pth`` files, which are loaded at startup and implemented via the
site module [#site]_, can include Python code that will be executed when the
``pth`` file is evaluated.

Note that ``pth`` files were originally developed to just add additional
directories to ``sys.path``, but they may also contain lines which start
with "import", which will be passed to ``exec()``. Users have exploited this
feature to allow the customizations that they needed. See setuptools
[#setuptools]_ or betterexceptions [#betterexceptions]_ as examples.

Using ``pth`` files for this purpose is far from ideal for library developers,
as they need to inject code into a single line preceded by an import, making
it rather unreadable. Library developers following that practice will usually
create a module that performs all actions on import, as done by
betterexceptions [#betterexceptions]_, but the approach is still not really
user friendly.

Additionally, it is also non-ideal for users of the interpreter if they want
to inspect what is being executed at Python startup as they need to review
all the ``pth`` files for potential code execution which can be spread across
all site paths. Most of those ``pth`` files will be "legitimate" ``pth``
files that just modify the path, answering the question of "what is changing
my interpreter at startup" a rather complex one.

Lastly, there have been multiple suggestions for removing code execution from
``pth`` files, see [#bpo-24534]_ and [#bpo-33944]_.

Limitations of ``sitecustomize.py``
-----------------------------------

Whilst sitecustomize is an acceptable solution, it assumes a single person is
in charge of the system and the interpreter. If both the system administrator
and the responsibility of provisioning the interpreter want to add
customizations at the interpreter startup they need to agree on the contents
of the file and combine all the changes. This is not a major limitation
though, and it is not the main driver of this change. Should the change
happen, it will also improve the situation for these users, as rather than
having a ``sitecustomize.py`` which performs all those actions, they can have
custom isolated files named after the features they want to enhance. As an
example, Ubuntu could change their current ``sitecustomize.py`` to just be
``ubuntu_apport_python_hook``. This not only better represents its intent but
also gives users of the interpreter a better understanding of the
modifications happening on their interpreter.

Rationale
=========

This PEP proposes supporting extensible customization of the interpreter at
startup by executing all files discovered in directories named
``__sitecustomize__`` in sitepackages [#sitepackages-api]_ or
usersitepackages [#usersitepackages-api]_ at startup time.

Why ``__sitecustomize__``
-------------------------

The name aims to follow the already existing concept of ``sitecustomize.py``.
As the directory will be within ``sys.path``, given that it is located in
site paths, we choose to use double underscore around its name, to prevent
colliding with the already existing ``sitecustomize.py``.

Discovering the new ``__sitecustomize__`` directories
-----------------------------------------------------

The Python interpreter will look at startup for directory named
``__sitecustomize__`` within any of the standard site-packages path.

These are commonly the Python system location and the user location, but are
ultimately defined by the site module logic.

Users can use ``site.sitepackages`` [#sitepackages-api]_ and
``site.usersitepackages`` [#usersitepackages-api]_ to know the paths where
the interpreter can discover ``__sitecustomize__`` directories.

Time of ``__sitecustomize__`` discovery
---------------------------------------

The ``__sitecustomize__`` directories will be discovered exactly after ``pth``
files are discovered in a site-packages path as part of ``site.addsitedir``
[#siteaddsitedir]_.

These is repeated for each of the site-packages path in the exact same order
that is being followed today for ``pth`` files.

Order of execution within ``__sitecustomize__``
-----------------------------------------------

The implementation will execute the files within ``__sitecustomize__`` by
sorting them by name when discovering each of the ``__sitecustomize__``
directories. We discourage users to rely on the order of execution though.

We considered executing them in random order, but that could result in
different results depending on how the interpreter chooses to pick up those
files. So even if it won't be a good practice to rely on other files being
executed, we think that is better than having randomly different results on
interpreter startup. We chose to run the files after the ``pth`` files in
case a user needs to add items to the path before running a files.

Interaction with ``pth`` files
------------------------------

``pth`` files can be used to add paths into ``sys.path``, but this should not
affect the ``__sitecustomize__`` discovery process, as those directories are
looked up exclusively in site-packages paths.

Execution of files within ``__sitecustomize__``
-----------------------------------------------

When a ``__sitecustomize__`` directory is discovered, all of the files that
have a ``.py`` extension within it will be read with ``io.open_code`` and
executed by using ``exec`` [#exec]_.

An empty dictionary will be passed as ``globals`` to the ``exec`` function
to prevent unexpected interactions between different files.

Failure handling
----------------

Any error on the execution of any of the files will not be logged unless the
interpreter is run in verbose mode and it should not stop the evaluation of
other files. The user will receive a message in stderr saying that the file
failed to be executed and that verbose mode can be used to get more
information. This behaviour mimics the one existing for ``sitecustomize.py``.

Interaction with virtual environments
-------------------------------------

The customizations applied to an interpreter via the new
``__sitecustomize__`` solutions will continue to work when a user creates a
virtual environment the same way that ``sitecustomize.py``
interact with virtual environments.

This is a difference when compared to ``pth`` files, which are not propagated
into virtual environments unless ``include-system-site-packages`` is enabled.

If library maintainers have features installed via ``__sitecustomize__`` that
they do not want to propagate into virtual environments, they should detect
if they are running within a virtual envi


id: pep-0660.rst
content:
PEP: 660
Title: Editable installs for pyproject.toml based builds (wheel based)
Author: Daniel Holth <dholth@gmail.com>, StÃ©phane Bidoul <stephane.bidoul@gmail.com>
Sponsor: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/draft-pep-editable-installs-for-pep-517-style-build-backends/8510
Status: Final
Type: Standards Track
Topic: Packaging
Created: 30-Mar-2021
Post-History:
Resolution: https://discuss.python.org/t/pronouncement-on-peps-660-and-662-editable-installs/9450


Abstract
========

This document describes a :pep:`517` style method for the installation of packages
in editable mode.

Motivation
==========

Python programmers want to be able to develop packages without having to
install (i.e. copy) them into ``site-packages``, for example, by working in a
checkout of the source repository.

While this can be done by adding the relevant source directories to
``PYTHONPATH``, ``setuptools`` provides the ``setup.py develop`` mechanism that
makes the process easier, and also installs dependencies and entry points such
as console scripts. ``pip`` exposes this mechanism via its ``pip install
--editable`` option.

The installation of projects in such a way that the python code being
imported remains in the source directory is known as the *editable*
installation mode.

Now that :pep:`517` provides a mechanism to create alternatives to setuptools, and
decouple installation front ends from build backends, we need a new mechanism
to install packages in editable mode.

Rationale
=========

:pep:`517` deferred "Editable installs", meaning non-``setup.py``
distributions lacked that feature. The only way to retain ``editable`` installs
for these distributions was to provide a compatible ``setup.py develop``
implementation. By defining an editable hook other build frontends gain
parity with ``setup.py``.

Terminology and goals
=====================

The editable installation mode implies that the source code of the project
being installed is available in a local directory.

Once the project is installed in editable mode, users expect that changes to
the project *python* code in the local source tree become effective without the
need of a new installation step.

Some kind of changes, such as the addition or modification of entry points, or
the addition of new dependencies, require a new installation step to become
effective. These changes are typically made in build backend configuration
files (such as ``pyproject.toml``), so it is consistent with the general user
expectation that *python* source code is imported from the source tree.

The modification of non-python source code such a C extension modules obviously
require a compilation and/or installation step to become effective. The exact
steps to perform will remain specific to the build backend used.

When a project is installed in editable mode, users expect the installation to
behave identically as a regular installation. In particular the code must be
importable by other code, and metadata must be available to standard mechanisms
such as ``importlib.metadata``.

Depending on the way build backends implement this specification, some minor
differences may be visible such as the presence of additional files that are in
the source tree and would not be part of a regular install. Build backends are
encouraged to document such potential differences.

The Mechanism
=============

This PEP adds three optional hooks to the :pep:`517` backend interface. These hooks
are used to build a wheel that, when installed, allows that distribution to be
imported from its source folder.

build_editable
--------------

::

  def build_editable(wheel_directory, config_settings=None, metadata_directory=None):
      ...

Must build a ``.whl`` file, and place it in the specified ``wheel_directory``.
It must return the basename (not the full path) of the .whl file it creates, as
a unicode string.

May do an in-place build of the distribution as a side effect so that any
extension modules or other built artifacts are ready to be used.

The .whl file must comply with the Wheel binary file format specification (PEP
427). In particular it must contain a compliant .dist-info directory.
Metadata must be identical as the one that would have been produced by
``build_wheel`` or ``prepare_metadata_for_build_wheel``, except for
``Requires-Dist`` which may differ slightly as explained below.

Build-backends must produce wheels that have the same dependencies
(``Requires-Dist`` metadata) as wheels produced by the ``build_wheel`` hook,
with the exception that they can add dependencies necessary for their editable
mechanism to function at runtime (such as `editables`_).

The filename for the "editable" wheel needs to be :pep:`427` compliant too. It
does not need to use the same tags as ``build_wheel`` but it must be tagged as
compatible with the system.

If the build frontend has previously called ``prepare_metadata_for_build_editable``
and depends on the wheel resulting from this call to have metadata
matching this earlier call, then it should provide the path to the created
``.dist-info`` directory as the ``metadata_directory`` argument. If this
argument is provided, then ``build_editable`` MUST produce a wheel with identical
metadata. The directory passed in by the build frontend MUST be
identical to the directory created by ``prepare_metadata_for_build_editable``,
including any unrecognized files it created.

An "editable" wheel uses the wheel format not for distribution but as ephemeral
communication between the build system and the front end. This avoids having
the build backend install anything directly. This wheel must not be exposed
to end users, nor cached, nor distributed.

get_requires_for_build_editable
-------------------------------

::

  def get_requires_for_build_editable(config_settings=None):
      ...

This hook MUST return an additional list of strings containing :pep:`508`
dependency specifications, above and beyond those specified in the
``pyproject.toml`` file, to be installed when calling the
``build_editable`` hooks.

If not defined, the default implementation is equivalent to ``return []``.

prepare_metadata_for_build_editable
-----------------------------------

::

  def prepare_metadata_for_build_editable(metadata_directory, config_settings=None):
      ...

Must create a ``.dist-info`` directory containing wheel metadata
inside the specified ``metadata_directory`` (i.e., creates a directory
like ``{metadata_directory}/{package}-{version}.dist-info/``). This
directory MUST be a valid ``.dist-info`` directory as defined in the
wheel specification, except that it need not contain ``RECORD`` or
signatures. The hook MAY also create other files inside this
directory, and a build frontend MUST preserve, but otherwise ignore, such files;
the intention
here is that in cases where the metadata depends on build-time
decisions, the build backend may need to record these decisions in
some convenient format for re-use by the actual wheel-building step.

This must return the basename (not the full path) of the ``.dist-info``
directory it creates, as a unicode string.

If a build frontend needs this information and the method is
not defined, it should call ``build_editable`` and look at the resulting
metadata directly.

What to put in the wheel
------------------------

Build backends must populate the generated wheel with files that when installed will result in an editable install.
Build backends may use different techniques to achieve the goals of an editable
install. This section provides examples and is not normative.

* Build backends may choose to place a ``.pth`` file at the root of the ``.whl`` file,
  containing the root directory of the source tree. This approach is simple but
  not very precise, although it may be considered good enough (especially when
  using the ``src`` layout) and is similar to what ``setup.py develop``
  currently does.
* The `editables`_ library shows how to build 


id: pep-0674.rst
content:
PEP: 674
Title: Disallow using macros as l-values
Author: Victor Stinner <vstinner@python.org>
Status: Deferred
Type: Standards Track
Created: 30-Nov-2021
Python-Version: 3.12


Abstract
========

Disallow using macros as l-values. For example,
``Py_TYPE(obj) = new_type`` now fails with a compiler error.

In practice, the majority of affected projects only have to make two
changes:

* Replace ``Py_TYPE(obj) = new_type``
  with ``Py_SET_TYPE(obj, new_type)``.
* Replace ``Py_SIZE(obj) = new_size``
  with ``Py_SET_SIZE(obj, new_size)``.


PEP Deferral
============

See `SC reply to PEP 674 -- Disallow using macros as l-values
<https://mail.python.org/archives/list/python-dev@python.org/message/CV6KWDRHV5WP6TIDK3Z46PW7HNSHYOWG/>`_
(February 2022).


Rationale
=========

Using a macro as a an l-value
-----------------------------

In the Python C API, some functions are implemented as macro because
writing a macro is simpler than writing a regular function. If a macro
exposes directly a structure member, it is technically possible to use
this macro to not only get the structure member but also set it.

Example with the Python 3.10 ``Py_TYPE()`` macro::

    #define Py_TYPE(ob) (((PyObject *)(ob))->ob_type)

This macro can be used as a **r-value** to **get** an object type::

    type = Py_TYPE(object);

It can also be used as an **l-value** to **set** an object type::

    Py_TYPE(object) = new_type;

It is also possible to set an object reference count and an object size
using ``Py_REFCNT()`` and ``Py_SIZE()`` macros.

Setting directly an object attribute relies on the current exact CPython
implementation. Implementing this feature in other Python
implementations can make their C API implementation less efficient.

CPython nogil fork
------------------

Sam Gross forked Python 3.9 to remove the GIL: the `nogil branch
<https://github.com/colesbury/nogil/>`_. This fork has no
``PyObject.ob_refcnt`` member, but a more elaborated implementation for
reference counting, and so the ``Py_REFCNT(obj) = new_refcnt;`` code
fails with a compiler error.

Merging the nogil fork into the upstream CPython main branch requires
first to fix this C API compatibility issue. It is a concrete example of
a Python optimization blocked indirectly by the C API.

This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro
has been already modified to disallow using it as an l-value.

These statements are endorsed by Sam Gross (nogil developer).

HPy project
-----------

The `HPy project <https://hpyproject.org/>`_ is a brand new C API for
Python using only handles and function calls: handles are opaque,
structure members cannot be accessed directly, and pointers cannot be
dereferenced.

Searching and replacing ``Py_SET_SIZE()`` is easier and safer than
searching and replacing some strange macro uses of ``Py_SIZE()``.
``Py_SIZE()`` can be semi-mechanically replaced by ``HPy_Length()``,
whereas seeing ``Py_SET_SIZE()`` would immediately make clear that the
code needs bigger changes in order to be ported to HPy (for example by
using ``HPyTupleBuilder`` or ``HPyListBuilder``).

The fewer internal details exposed via macros, the easier it will be for
HPy to provide direct equivalents. Any macro that references
"non-public" interfaces effectively exposes those interfaces publicly.

These statements are endorsed by Antonio Cuni (HPy developer).

GraalVM Python
--------------

In GraalVM, when a Python object is accessed by the Python C API, the C API
emulation layer has to wrap the GraalVM objects into wrappers that expose
the internal structure of the CPython structures (PyObject, PyLongObject,
PyTypeObject, etc). This is because when the C code accesses it directly or via
macros, all GraalVM can intercept is a read at the struct offset, which has
to be mapped back to the representation in GraalVM. The smaller the
"effective" number of exposed struct members (by replacing macros with
functions), the simpler GraalVM wrappers can be.

This PEP alone is not enough to get rid of the wrappers in GraalVM, but it
is a step towards this long term goal. GraalVM already supports HPy which is a better
solution in the long term.

These statements are endorsed by Tim Felgentreff (GraalVM Python developer).

Specification
=============

Disallow using macros as l-values
----------------------------------

The following 65 macros are modified to disallow using them as l-values.

PyObject and PyVarObject macros
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* ``Py_TYPE()``: ``Py_SET_TYPE()`` must be used instead
* ``Py_SIZE()``: ``Py_SET_SIZE()`` must be used instead

GET macros
^^^^^^^^^^

* ``PyByteArray_GET_SIZE()``
* ``PyBytes_GET_SIZE()``
* ``PyCFunction_GET_CLASS()``
* ``PyCFunction_GET_FLAGS()``
* ``PyCFunction_GET_FUNCTION()``
* ``PyCFunction_GET_SELF()``
* ``PyCell_GET()``
* ``PyCode_GetNumFree()``
* ``PyDict_GET_SIZE()``
* ``PyFunction_GET_ANNOTATIONS()``
* ``PyFunction_GET_CLOSURE()``
* ``PyFunction_GET_CODE()``
* ``PyFunction_GET_DEFAULTS()``
* ``PyFunction_GET_GLOBALS()``
* ``PyFunction_GET_KW_DEFAULTS()``
* ``PyFunction_GET_MODULE()``
* ``PyHeapType_GET_MEMBERS()``
* ``PyInstanceMethod_GET_FUNCTION()``
* ``PyList_GET_SIZE()``
* ``PyMemoryView_GET_BASE()``
* ``PyMemoryView_GET_BUFFER()``
* ``PyMethod_GET_FUNCTION()``
* ``PyMethod_GET_SELF()``
* ``PySet_GET_SIZE()``
* ``PyTuple_GET_SIZE()``
* ``PyUnicode_GET_DATA_SIZE()``
* ``PyUnicode_GET_LENGTH()``
* ``PyUnicode_GET_LENGTH()``
* ``PyUnicode_GET_SIZE()``
* ``PyWeakref_GET_OBJECT()``

AS macros
^^^^^^^^^

* ``PyByteArray_AS_STRING()``
* ``PyBytes_AS_STRING()``
* ``PyFloat_AS_DOUBLE()``
* ``PyUnicode_AS_DATA()``
* ``PyUnicode_AS_UNICODE()``

PyUnicode macros
^^^^^^^^^^^^^^^^

* ``PyUnicode_1BYTE_DATA()``
* ``PyUnicode_2BYTE_DATA()``
* ``PyUnicode_4BYTE_DATA()``
* ``PyUnicode_DATA()``
* ``PyUnicode_IS_ASCII()``
* ``PyUnicode_IS_COMPACT()``
* ``PyUnicode_IS_READY()``
* ``PyUnicode_KIND()``
* ``PyUnicode_READ()``
* ``PyUnicode_READ_CHAR()``

PyDateTime GET macros
^^^^^^^^^^^^^^^^^^^^^

* ``PyDateTime_DATE_GET_FOLD()``
* ``PyDateTime_DATE_GET_HOUR()``
* ``PyDateTime_DATE_GET_MICROSECOND()``
* ``PyDateTime_DATE_GET_MINUTE()``
* ``PyDateTime_DATE_GET_SECOND()``
* ``PyDateTime_DATE_GET_TZINFO()``
* ``PyDateTime_DELTA_GET_DAYS()``
* ``PyDateTime_DELTA_GET_MICROSECONDS()``
* ``PyDateTime_DELTA_GET_SECONDS()``
* ``PyDateTime_GET_DAY()``
* ``PyDateTime_GET_MONTH()``
* ``PyDateTime_GET_YEAR()``
* ``PyDateTime_TIME_GET_FOLD()``
* ``PyDateTime_TIME_GET_HOUR()``
* ``PyDateTime_TIME_GET_MICROSECOND()``
* ``PyDateTime_TIME_GET_MINUTE()``
* ``PyDateTime_TIME_GET_SECOND()``
* ``PyDateTime_TIME_GET_TZINFO()``

Port C extensions to Python 3.11
--------------------------------

In practice, the majority of projects affected by these PEP only have to
make two changes:

* Replace ``Py_TYPE(obj) = new_type``
  with ``Py_SET_TYPE(obj, new_type)``.
* Replace ``Py_SIZE(obj) = new_size``
  with ``Py_SET_SIZE(obj, new_size)``.

The `pythoncapi_compat project
<https://github.com/pythoncapi/pythoncapi_compat>`_ can be used to
update automatically C extensions: add Python 3.11 support without
losing support with older Python versions. The project provides a header
file which provides ``Py_SET_REFCNT()``, ``Py_SET_TYPE()`` and
``Py_SET_SIZE()`` functions to Python 3.8 and older.

PyTuple_GET_ITEM() and PyList_GET_ITEM() are left unchanged
-----------------------------------------------------------

The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are left
unchanged.

The code patterns ``&PyTuple_GET_ITEM(tuple, 0)`` and
``&PyList_GET_ITEM(list, 0)`` are still commonly used to get access to
the inner ``PyObject**`` array.

Changing these macros is out of the scope of this PEP.

PyDescr_NAME() and PyDescr_TYPE() are left unchanged
----------------------------------------------------

The ``PyDescr_NAME()`` and ``PyDescr_TYPE()`` macros are left unchanged.

These macros give access to ``PyDescrObject.d_name`` and
``PyDescrObject.d_type`` members. They can be used as l


id: pep-0728.rst
content:
PEP: 728
Title: TypedDict with Typed Extra Items
Author: Zixuan James Li <p359101898@gmail.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-728-typeddict-with-typed-extra-items/45443
Status: Draft
Type: Standards Track
Topic: Typing
Created: 12-Sep-2023
Python-Version: 3.15
Post-History: `09-Feb-2024 <https://discuss.python.org/t/pep-728-typeddict-with-typed-extra-items/45443>`__,


Abstract
========

This PEP adds two class parameters, ``closed`` and ``extra_items``
to type the extra items on a :class:`~typing.TypedDict`. This addresses the
need to define closed TypedDict types or to type a subset of keys that might
appear in a ``dict`` while permitting additional items of a specified type.

Motivation
==========

A :py:class:`typing.TypedDict` type can annotate the value type of each known
item in a dictionary. However, due to :term:`typing:structural`
:term:`assignability <typing:assignable>`, a TypedDict can have extra items
that are not visible through its type. There is currently no way to restrict
the types of items that might be present in the TypedDict type's
:term:`consistent subtypes <typing:consistent subtype>`.

Disallowing Extra Items Explicitly
----------------------------------

The current behavior of TypedDict prevents users from defining a
TypedDict type when it is expected that the type contains no extra items.

Due to the possible presence of extra items, type checkers cannot infer more
precise return types for ``.items()`` and ``.values()`` on a TypedDict.
This can be resolved by
`defining a closed TypedDict type <https://github.com/python/mypy/issues/7981>`__.

Another possible use case for this is a sound way to
`enable type narrowing <https://github.com/python/mypy/issues/9953>`__ with the
``in`` check::

    class Movie(TypedDict):
        name: str
        director: str

    class Book(TypedDict):
        name: str
        author: str

    def fun(entry: Movie | Book) -> None:
        if "author" in entry:
            reveal_type(entry)  # Revealed type is still 'Movie | Book'

Nothing prevents a ``dict`` that is assignable with ``Movie`` to have the
``author`` key, and under the current specification it would be incorrect for
the type checker to :term:`typing:narrow` its type.

Allowing Extra Items of a Certain Type
--------------------------------------

For supporting API interfaces or legacy codebase where only a subset of possible
keys are known, it would be useful to explicitly specify extra items of certain
value types.

However, the typing spec is more restrictive when checking the construction of a
TypedDict, `preventing users <https://github.com/python/mypy/issues/4617>`__
from doing this::

    class MovieBase(TypedDict):
        name: str

    def foo(movie: MovieBase) -> None:
        # movie can have extra items that are not visible through MovieBase
        ...

    movie: MovieBase = {"name": "Blade Runner", "year": 1982}  # Not OK
    foo({"name": "Blade Runner", "year": 1982})  # Not OK

While the restriction is enforced when constructing a TypedDict, due to
:term:`typing:structural` :term:`assignability <typing:assignable>`, the
TypedDict may have extra items that are not visible through its type.
For example::

    class Movie(MovieBase):
        year: int

    movie: Movie = {"name": "Blade Runner", "year": 1982}
    foo(movie)  # OK

It is not possible to acknowledge the existence of the extra items through
``in`` checks and access them without breaking type safety, even though they
might exist from some :term:`consistent subtypes <typing:consistent subtype>` of
``MovieBase``::

    def bar(movie: MovieBase) -> None:
        if "year" in movie:
            reveal_type(movie["year"])  # Error: TypedDict 'MovieBase' has no key 'year'

Some workarounds have already been implemented to allow
extra items, but none of them is ideal. For mypy,
``--disable-error-code=typeddict-unknown-key``
`suppresses type checking error <https://github.com/python/mypy/pull/14225>`__
specifically for unknown keys on TypedDict. This sacrifices type safety over
flexibility, and it does not offer a way to specify that the TypedDict type
expects additional keys whose value types are assignable with a certain type.

Support Additional Keys for ``Unpack``
--------------------------------------

:pep:`692` adds a way to precisely annotate the types of individual keyword
arguments represented by ``**kwargs`` using TypedDict with ``Unpack``. However,
because TypedDict cannot be defined to accept arbitrary extra items, it is not
possible to `allow additional keyword arguments
<https://discuss.python.org/t/pep-692-using-typeddict-for-more-precise-kwargs-typing/17314/87>`__
that are not known at the time the TypedDict is defined.

Given the usage of pre-:pep:`692` type annotation for ``**kwargs`` in existing
codebases, it will be valuable to accept and type extra items on TypedDict so
that the old typing behavior can be supported in combination with ``Unpack``.

Rationale
=========

Suppose we want a type that allows extra items of type ``str`` on a TypedDict.

`Index Signatures
<https://www.typescriptlang.org/docs/handbook/2/objects.html#index-signatures>`__
in TypeScript allow this:

.. code-block:: typescript

    type Foo = {
        a: string
        [key: string]: string
    }

This proposal aims to support a similar feature without syntax changes,
offering a natural extension to the existing assignability rules.

We propose to add a class parameter ``extra_items`` to TypedDict.
It accepts a :term:`typing:type expression` as the argument; when it is present,
extra items are allowed, and their value types must be assignable to the
type expression value.

An application of this is to disallow extra items. We propose to add a
``closed`` class parameter, which only accepts a literal ``True`` or ``False``
as the argument. It should be a runtime error when ``closed`` and
``extra_items`` are used at the same time.

Different from index signatures, the types of the known items do not need to be
assignable to the ``extra_items`` argument.

There are some advantages to this approach:

- We can build on top of the `assignability rules defined in the typing spec
  <https://typing.python.org/en/latest/spec/typeddict.html#assignability>`__,
  where ``extra_items`` can be treated as a pseudo-item.

- There is no need to introduce a grammar change to specify the type of the
  extra items.

- We can precisely type the extra items without requiring the value types of the
  known items to be :term:`typing:assignable` to ``extra_items``.

- We do not lose backwards compatibility as both ``extra_items`` and ``closed``
  are opt-in only features.

Specification
=============

This specification is structured to parallel :pep:`589` to highlight changes to
the original TypedDict specification.

If ``extra_items`` is specified, extra items are treated as :ref:`non-required
<typing:required-notrequired>`
items matching the ``extra_items`` argument, whose keys are allowed when
determining `supported and unsupported operations
<https://typing.python.org/en/latest/spec/typeddict.html#supported-and-unsupported-operations>`__.

The ``extra_items`` Class Parameter
-----------------------------------

By default ``extra_items`` is unset.  For a TypedDict type that specifies
``extra_items``, during construction, the value type of each unknown item
is expected to be non-required and assignable to the ``extra_items`` argument.
For example::

    class Movie(TypedDict, extra_items=bool):
        name: str

    a: Movie = {"name": "Blade Runner", "novel_adaptation": True}  # OK
    b: Movie = {
        "name": "Blade Runner",
        "year": 1982,  # Not OK. 'int' is not assignable to 'bool'
    }

Here, ``extra_items=bool`` specifies that items other than ``'name'``
have a value type of ``bool`` and are non-required.

The alternative inline syntax is also supported::

    Movie = TypedDict("Movie", {"name": str


id: pep-0714.rst
content:
PEP: 714
Title: Rename dist-info-metadata in the Simple API
Author: Donald Stufft <donald@stufft.io>
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/27471
Status: Accepted
Type: Standards Track
Topic: Packaging
Created: 06-Jun-2023
Post-History: `06-Jun-2023 <https://discuss.python.org/t/27471>`__
Resolution: `27-Jun-2023 <https://discuss.python.org/t/27471/19>`__


Abstract
========

This PEP renames the metadata provided by :pep:`658` in both HTML and JSON
formats of the Simple API and provides guidelines for both clients and servers
in how to handle the renaming.


Motivation
==========

:pep:`658` specified a mechanism to host the core metadata files from an
artifact available through the Simple API such that a client could fetch the
metadata and use it without having to download the entire artifact. Later
:pep:`691` was written to add the ability to use JSON rather than HTML on the
Simple API, which included support for the :pep:`658` metadata.

Unfortunately, PyPI did not support :pep:`658` until just
`recently <https://github.com/pypi/warehouse/pull/13649>`__, which released with
a `bug <https://github.com/pypi/warehouse/issues/13705>`__ where the
``dist-info-metadata`` key from :pep:`658` was incorrectly named in the JSON
representation, to be ``data-dist-info-metadata``. However, when
attempting to fix that bug, it was discovered that pip *also* had a
`bug <https://github.com/pypa/pip/issues/12042>`__, where any use of
``dist-info-metadata`` in the JSON representation would cause pip to hard fail
with an exception.

The bug in pip has existed since at least ``v22.3``, which means that it has
been released for approximately 8 months, long enough to have been pulled into
Python releases, downstream Linux releases, baked into containers, virtual
environments, etc.

This puts us in an awkward position of having a bug on PyPI that cannot be fixed
without breaking pip, due to a bug in pip, but that version of pip is old enough
to have been widely deployed. To make matters worse, a version of pip that is
broken in this way cannot install *anything* from PyPI once it fixes its bug,
including installing a new, fixed version of pip.


Rationale
=========

There are 3 main options for a path forward for fixing these bugs:

1. Do not change the spec, fix the bug in pip, wait some amount of time, then
   fix the bug in PyPI, breaking anyone using an unfixed pip such that they
   cannot even install a new pip from PyPI.
2. Do the same as (1), but special case PyPI so it does not emit the :pep:`658`
   metadata for pip, even if it is available. This allows people to upgrade pip
   if they're on a broken version, but nothing else.
3. Change the spec to avoid the key that pip can't handle currently, allowing
   PyPI to emit that key and a new version of pip to be released to take
   advantage of that key.

This PEP chooses (3), but goes a little further and also renames the key in the
HTML representation.

Typically we do not change specs because of bugs that only affect one particular
implementation, unless the spec itself is at fault, which isn't the case here:
the spec is fine and these are just genuine bugs in pip and PyPI.

However, we choose to do this for 4 reasons:

1. Bugs that affect pip and PyPI together represent an outsized amount of impact
   compared to any other client or repository combination.
2. The impact of being broken is that installs do not function, at all, rather
   than degrading gracefully in some way.
3. The feature that is being blocked by these bugs is of large importance to
   the ability to quickly and efficiently resolve dependencies from PyPI with
   pip, and having to delay it for a long period of time while we wait for the
   broken versions of pip to fall out of use would be of detriment to the entire
   ecosystem.
4. The downsides of changing the spec are fairly limited, given that we do not
   believe that support for this is widespread, so it affects only a limited
   number of projects.


Specification
=============

The keywords "**MUST**", "**MUST NOT**", "**REQUIRED**", "**SHALL**",
"**SHALL NOT**", "**SHOULD**", "**SHOULD NOT**", "**RECOMMENDED**", "**MAY**",
and "**OPTIONAL**"" in this document are to be interpreted as described in
:rfc:`RFC 2119 <2119>`.


Servers
-------

The :pep:`658` metadata, when used in the HTML representation of the Simple API,
**MUST** be emitted using the attribute name ``data-core-metadata``, with the
supported values remaining the same.

The :pep:`658` metadata, when used in the :pep:`691` JSON representation of the
Simple API, **MUST** be emitted using the key ``core-metadata``, with the
supported values remaining the same.

To support clients that used the previous key names, the HTML representation
**MAY** also be emitted using the ``data-dist-info-metadata``, and if it does
so it **MUST** match the value of ``data-core-metadata``.



Clients
-------

Clients consuming any of the HTML representations of the Simple API **MUST**
read the :pep:`658` metadata from the key ``data-core-metadata`` if it is
present. They **MAY** optionally use the legacy ``data-dist-info-metadata`` if
it is present but ``data-core-metadata`` is not.

Clients consuming the JSON representation of the Simple API **MUST** read the
:pep:`658` metadata from the key ``core-metadata`` if it is present. They
**MAY** optionally use the legacy ``dist-info-metadata`` key if it is present
but ``core-metadata`` is not.


Backwards Compatibility
=======================

There is a minor compatibility break in this PEP, in that clients that currently
correctly handle the existing metadata keys will not automatically understand
the newer metadata keys, but they should degrade gracefully, and simply act
as if the :pep:`658` metadata does not exist.

Otherwise there should be no compatibility problems with this PEP.


Rejected Ideas
==============

Leave the spec unchanged, and cope with fixing in PyPI and/or pip
-----------------------------------------------------------------

We believe that the improvements brought by :pep:`658` are very important to
improving the performance of resolving dependencies from PyPI, and would like to
be able to deploy it as quickly as we can.

Unfortunately the nature of these bugs is that we cannot deploy them as is
without breaking widely deployed and used versions of pip. The breakages in
this case would be bad enough that affected users would not even be able to
directly upgrade their version of pip to fix it, but would have to manually
fetch pip another way first (e.g. ``get-pip.py``).

This is something that PyPI would be unwilling to do without some way to
mitigate those breakages for those users. Without some reasonable mitigation
strategy, we would have to wait until those versions of pip are no longer in use
on PyPI, which would likely be 5+ years from now.

There are a few possible mitigation strategies that we could use, but we've
rejected them as well.


Mitigation: Special Case pip
++++++++++++++++++++++++++++

The breakages are particularly bad in that it prevents users from even upgrading
pip to get an unbroken version of pip, so a command like
``pip install --upgrade pip`` would fail. We could mitigate this by having PyPI
special case pip itself, so that the JSON endpoint never returns the :pep:`658`
metadata and the above still works.

This PEP rejects this idea because while the simple command that only upgrades
pip would work, if the user included *anything* else in that command to upgrade
then the command would go back to failing, which we consider to be still too
large of a breakage.

Additionally, while this bug happens to be getting exposed right now with PyPI,
it is really a bug that would happen with any :pep:`691` repository that
correctly exposed the :pep:`658` metadata. This would mean that every repository
would have to carry this special case for pip.


Mitigation: Have the server use User-Agent Detection
++++++++


id: pep-0700.rst
content:
PEP: 700
Title: Additional Fields for the Simple API for Package Indexes
Author: Paul Moore <p.f.moore@gmail.com>
PEP-Delegate: Donald Stufft <donald@stufft.io>
Discussions-To: https://discuss.python.org/t/pep-700-additional-fields-for-the-simple-api-for-package-indexes/20177
Status: Final
Type: Standards Track
Topic: Packaging
Created: 21-Oct-2022
Post-History: `21-Oct-2022 <https://discuss.python.org/t/pep-700-additional-fields-for-the-simple-api-for-package-indexes/20177>`__
Resolution: `19-Dec-2022 <https://discuss.python.org/t/pep-700-additional-fields-for-the-simple-api-for-package-indexes/20177/42>`__

.. canonical-pypa-spec:: :ref:`packaging:simple-repository-api`

Abstract
========

:pep:`691` defined a JSON form for the "Simple Repository API". This allowed
clients to more easily query the data that was previously only available in
HTML, as defined in :pep:`503`.

This proposal adds three fields to the JSON form, which allow it to be used in
place of PyPI's `JSON API <https://warehouse.pypa.io/api-reference/json.html>`__
in a number of situations.

- A field to allow retrieval of a list of all the published versions of a project.
- Fields containing the size and upload time for a project file.

The new fields are all part of the data returned from the "project details" URL.


Rationale
=========

With the introduction of the JSON form of the simple API in :pep:`691`, the
simple API offers functionality that is almost as complete as the PyPI JSON API.
This PEP adds a number of fields which were previously only available through
the JSON API, in order to allow more clients which were previously Warehouse
specific to support arbitrary standards-compliant indexes.


Specification
=============

This specification defines version 1.1 of the simple repository API. For the
HTML version of the API, there is no change from version 1.0. For the JSON
version of the API, the following changes are made:

- The ``api-version`` must specify version 1.1 or later.
- A new ``versions`` key is added at the top level.
- Two new "file information" keys, ``size`` and ``upload-time``, are added to
  the ``files`` data.
- Keys (at any level) with a leading underscore are reserved as private for
  index server use. No future standard will assign a meaning to any such key.

The ``versions`` and ``size`` keys are mandatory. The ``upload-time`` key is
optional.

Versions
--------

An additional key, ``versions`` MUST be present at the top level, in addition to
the keys ``name``, ``files`` and ``meta`` defined in :pep:`691`. This key MUST
contain a list of version strings specifying all of the project versions uploaded
for this project. The value is logically a set, and as such may not contain
duplicates, and the order of the values is not significant.

All of the files listed in the ``files`` key MUST be associated with one of the
versions in the ``versions`` key. The ``versions`` key MAY contain versions with
no associated files (to represent versions with no files uploaded, if the server
has such a concept).

Note that because servers may hold "legacy" data from before the adoption of
:pep:`440`, version strings currently cannot be required to be valid :pep:`440`
versions, and therefore cannot be assumed to be orderable using the :pep:`440`
rules. However, servers SHOULD use normalised :pep:`440` versions where
possible.


Additional file information
---------------------------

Two new keys are added to the ``files`` key.

- ``size``: This field is mandatory. It MUST contain an integer which is the
  file size in bytes.
- ``upload-time``: This field is optional. If present, it MUST contain a valid
  ISO 8601 date/time string, in the format ``yyyy-mm-ddThh:mm:ss.ffffffZ``,
  which represents the time the file was uploaded to the index. As indicated by
  the ``Z`` suffix, the upload time MUST use the UTC timezone. The fractional
  seconds part of the timestamp (the ``.ffffff`` part) is optional, and if
  present may contain up to 6 digits of precision. If a server does not record
  upload time information for a file, it MAY omit the ``upload-time`` key.


FAQ
===

Why not add this data to the HTML API as well?
----------------------------------------------

It would be possible to add the data to the HTML API, but the vast majority of
consumers for this data are likely to be currently getting it from the PyPI JSON
API, and so will already be expecting to parse JSON. Traditional consumers of
the HTML API have never needed this data previously.

Does this imply that the HTML API is obsolete?
----------------------------------------------

No. The FAQ of :pep:`691` was clear that the HTML API is not being deprecated,
and this PEP does not change that position. However, clients wishing to access
the new data introduced by this PEP will need to use the JSON API to get it. And
indexes wanting to provide it will need to serve the JSON format.

Is the simple API replacing the Warehouse JSON and XML-RPC APIs?
----------------------------------------------------------------

Where possible, clients should prefer the simple API over the JSON or XML-RPC
APIs, as the former is standardised and can be assumed to be available from any
index, whereas the latter are exclusive to the Warehouse project.

However, while this PEP brings the simple API closer to being able to replace
the JSON API, there is no formal policy that the simple API will replicate all
of the functionality covered by the existing Warehouse APIs. Proposed additions
to the simple API will still be considered on their individual merits, and the
requirement that the API should be simple and fast for the primary use case of
locating files for a project will remain the overriding consideration.

Why not allow other date formats?
---------------------------------

The ISO 8601 standard is complex, and there seems little value in requiring
clients to deal with that. The standard library ``datetime`` module provides
methods to parse ISO 8601 strings, but it is possible that users may want to
access index data *without* using Python (for example, piping the output of
``curl`` into ``jq``). Having a single, well-defined format makes this easier,
and doesn't have any significant disadvantages.

What if file sizes are too big for a JSON number?
-------------------------------------------------

The JSON standard does not specify how numbers are to be interpreted. Python can
read and write arbitrary-length integers in a JSON file, so this should not be
an issue for code written in Python. Non-Python implementations may need to take
care to handle large integers correctly, but this is not expected to be a
significant problem.

Why not require PEP 440 versions?
---------------------------------

At the time this PEP was written, PyPI still contains (and serves) projects and
files with "legacy" versions. Requiring :pep:`440` versions would make it
impossible for PyPI to follow this specification while still serving the
existing content.

Ideally, at some future time, the simple index API will be updated to require
:pep:`440` versions, at which time this specification should be updated to
reflect that. However, that change will need to be co-ordinated with existing
index providers including PyPI, to desupport and remove non-conforming projects
and/or files.

Why not provide a "latest version" value?
-----------------------------------------

For :pep:`440` versions, this is easy enough for the client to do (using the
``packaging`` library, ``latest = max(Version(s) for s in proj["versions"])``).
For non-standard versions, there is no well-defined ordering, and clients will
need to decide on what rule is appropriate for their needs. Requiring the server
to supply a latest version value takes the choice away from the client.

Servers with an explicit concept of which version is the "latest", which cannot
be calculated from data available to the client, can provide a non-standard,
underscore-prefixed key to convey that information 


id: pep-0258.rst
content:
PEP: 258
Title: Docutils Design Specification
Author: David Goodger <goodger@python.org>
Discussions-To: doc-sig@python.org
Status: Rejected
Type: Standards Track
Requires: 256, 257
Created: 31-May-2001
Post-History: 13-Jun-2001


================
Rejection Notice
================

While this may serve as an interesting design document for the
now-independent docutils, it is no longer slated for inclusion in the
standard library.


==========
 Abstract
==========

This PEP documents design issues and implementation details for
Docutils, a Python Docstring Processing System (DPS).  The rationale
and high-level concepts of a DPS are documented in :pep:`256`, "Docstring
Processing System Framework".  Also see :pep:`256` for a
"Road Map to the Docstring PEPs".

Docutils is being designed modularly so that any of its components can
be replaced easily.  In addition, Docutils is not limited to the
processing of Python docstrings; it processes standalone documents as
well, in several contexts.

No changes to the core Python language are required by this PEP.  Its
deliverables consist of a package for the standard library and its
documentation.


===============
 Specification
===============

Docutils Project Model
======================

Project components and data flow::

                     +---------------------------+
                     |        Docutils:          |
                     | docutils.core.Publisher,  |
                     | docutils.core.publish_*() |
                     +---------------------------+
                      /            |            \
                     /             |             \
            1,3,5   /        6     |              \ 7
           +--------+       +-------------+       +--------+
           | READER | ----> | TRANSFORMER | ====> | WRITER |
           +--------+       +-------------+       +--------+
            /     \\                                  |
           /       \\                                 |
     2    /      4  \\                             8  |
    +-------+   +--------+                        +--------+
    | INPUT |   | PARSER |                        | OUTPUT |
    +-------+   +--------+                        +--------+

The numbers above each component indicate the path a document's data
takes.  Double-width lines between Reader & Parser and between
Transformer & Writer indicate that data sent along these paths should
be standard (pure & unextended) Docutils doc trees.  Single-width
lines signify that internal tree extensions or completely unrelated
representations are possible, but they must be supported at both ends.


Publisher
---------

The ``docutils.core`` module contains a "Publisher" facade class and
several convenience functions: "publish_cmdline()" (for command-line
front ends), "publish_file()" (for programmatic use with file-like
I/O), and "publish_string()" (for programmatic use with string I/O).
The Publisher class encapsulates the high-level logic of a Docutils
system.  The Publisher class has overall responsibility for
processing, controlled by the ``Publisher.publish()`` method:

1. Set up internal settings (may include config files & command-line
   options) and I/O objects.

2. Call the Reader object to read data from the source Input object
   and parse the data with the Parser object.  A document object is
   returned.

3. Set up and apply transforms via the Transformer object attached to
   the document.

4. Call the Writer object which translates the document to the final
   output format and writes the formatted data to the destination
   Output object.  Depending on the Output object, the output may be
   returned from the Writer, and then from the ``publish()`` method.

Calling the "publish" function (or instantiating a "Publisher" object)
with component names will result in default behavior.  For custom
behavior (customizing component settings), create custom component
objects first, and pass *them* to the Publisher or ``publish_*``
convenience functions.


Readers
-------

Readers understand the input context (where the data is coming from),
send the whole input or discrete "chunks" to the parser, and provide
the context to bind the chunks together back into a cohesive whole.

Each reader is a module or package exporting a "Reader" class with a
"read" method.  The base "Reader" class can be found in the
``docutils/readers/__init__.py`` module.

Most Readers will have to be told what parser to use.  So far (see the
list of examples below), only the Python Source Reader ("PySource";
still incomplete) will be able to determine the parser on its own.

Responsibilities:

* Get input text from the source I/O.

* Pass the input text to the parser, along with a fresh `document
  tree`_ root.

Examples:

* Standalone (Raw/Plain): Just read a text file and process it.
  The reader needs to be told which parser to use.

  The "Standalone Reader" has been implemented in module
  ``docutils.readers.standalone``.

* Python Source: See `Python Source Reader`_ below.  This Reader is
  currently in development in the Docutils sandbox.

* Email: :rfc:`822` headers, quoted excerpts, signatures, MIME parts.

* PEP: :rfc:`822` headers, "PEP xxxx" and "RFC xxxx" conversion to URIs.
  The "PEP Reader" has been implemented in module
  ``docutils.readers.pep``; see :pep:`287` and :pep:`12`.

* Wiki: Global reference lookups of "wiki links" incorporated into
  transforms.  (CamelCase only or unrestricted?)  Lazy
  indentation?

* Web Page: As standalone, but recognize meta fields as meta tags.
  Support for templates of some sort?  (After ``<body>``, before
  ``</body>``?)

* FAQ: Structured "question & answer(s)" constructs.

* Compound document: Merge chapters into a book.  Master manifest
  file?


Parsers
-------

Parsers analyze their input and produce a Docutils `document tree`_.
They don't know or care anything about the source or destination of
the data.

Each input parser is a module or package exporting a "Parser" class
with a "parse" method.  The base "Parser" class can be found in the
``docutils/parsers/__init__.py`` module.

Responsibilities: Given raw input text and a doctree root node,
populate the doctree by parsing the input text.

Example: The only parser implemented so far is for the
reStructuredText markup.  It is implemented in the
``docutils/parsers/rst/`` package.

The development and integration of other parsers is possible and
encouraged.


.. _transforms:

Transformer
-----------

The Transformer class, in ``docutils/transforms/__init__.py``, stores
transforms and applies them to documents.  A transformer object is
attached to every new document tree.  The Publisher_ calls
``Transformer.apply_transforms()`` to apply all stored transforms to
the document tree.  Transforms change the document tree from one form
to another, add to the tree, or prune it.  Transforms resolve
references and footnote numbers, process interpreted text, and do
other context-sensitive processing.

Some transforms are specific to components (Readers, Parser, Writers,
Input, Output).  Standard component-specific transforms are specified
in the ``default_transforms`` attribute of component classes.  After
the Reader has finished processing, the Publisher_ calls
``Transformer.populate_from_components()`` with a list of components
and all default transforms are stored.

Each transform is a class in a module in the ``docutils/transforms/``
package, a subclass of ``docutils.transforms.Transform``.  Transform
classes each have a ``default_priority`` attribute which is used by
the Transformer to apply transforms in order (low to high).  The
default priority can be overridden when adding transforms to the
Transformer object.

Transformer responsibilities:

* Apply transforms to the document tree, in priority order.

* Store a mapping of component type name ('reader', 'writer', etc.) to
  component objects.  These are used by certain transforms (such as
  "components.Filter") to deter


id: pep-0270.rst
content:
PEP: 270
Title: uniq method for list objects
Author: Jason Petrone <jp@demonseed.net>
Status: Rejected
Type: Standards Track
Created: 21-Aug-2001
Python-Version: 2.2
Post-History:


Notice
======

This PEP is withdrawn by the author.  He writes:

    Removing duplicate elements from a list is a common task, but
    there are only two reasons I can see for making it a built-in.
    The first is if it could be done much faster, which isn't the
    case.  The second is if it makes it significantly easier to
    write code.  The introduction of ``sets.py`` eliminates this
    situation since creating a sequence without duplicates is just
    a matter of choosing a different data structure: a set instead
    of a list.

As described in :pep:`218`, sets are being added to the standard
library for Python 2.3.


Abstract
========

This PEP proposes adding a method for removing duplicate elements to
the list object.


Rationale
=========

Removing duplicates from a list is a common task.  I think it is
useful and general enough to belong as a method in list objects.
It also has potential for faster execution when implemented in C,
especially if optimization using hashing or sorted cannot be used.

On comp.lang.python there are many, many, posts [1]_ asking about
the best way to do this task.  It's a little tricky to implement
optimally and it would be nice to save people the trouble of
figuring it out themselves.


Considerations
==============

Tim Peters suggests trying to use a hash table, then trying to
sort, and finally falling back on brute force [2]_.  Should uniq
maintain list order at the expense of speed?

Is it spelled 'uniq' or 'unique'?


Reference Implementation
========================

I've written the brute force version.  It's about 20 lines of code
in ``listobject.c``.  Adding support for hash table and sorted
duplicate removal would only take another hour or so.


References
==========

.. [1] https://groups.google.com/forum/#!searchin/comp.lang.python/duplicates

.. [2] Tim Peters unique() entry in the Python cookbook:
       http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52560/index_txt


Copyright
=========

This document has been placed in the public domain.


id: pep-0516.rst
content:
PEP: 516
Title: Build system abstraction for pip/conda etc
Author: Robert Collins <rbtcollins@hp.com>,
        Nathaniel J. Smith <njs@pobox.com>
BDFL-Delegate: Alyssa Coghlan <ncoghlan@gmail.com>
Discussions-To: distutils-sig@python.org
Status: Rejected
Type: Standards Track
Topic: Packaging
Created: 26-Oct-2015
Resolution: https://mail.python.org/pipermail/distutils-sig/2017-May/030517.html


Abstract
========

This PEP specifies a programmatic interface for pip [#pip]_ and other
distribution or installation tools to use when working with Python
source trees (both the developer tree - e.g. the git tree - and source
distributions).

The programmatic interface allows decoupling of pip from its current
hard dependency on setuptools [#setuptools]_ able for two
key reasons:

1. It enables new build systems that may be much easier to use without
   requiring them to even appear to be setuptools.

2. It facilitates setuptools itself changing its user interface without
   breaking pip, giving looser coupling.

The interface needed to permit pip to install build systems also enables pip to
install build time requirements for packages which is an important step in
getting pip to full feature parity with the installation components of
easy-install.

As :pep:`426` is draft, we cannot utilise the metadata format it
defined. However :pep:`427` wheels are in wide use and fairly well specified, so
we have adopted the METADATA format from that for specifying distribution
dependencies and general project metadata. :pep:`508` provides a
self-contained language for describing a dependency, which we encapsulate in a
thin JSON schema to describe bootstrap dependencies.

Since Python sdists specified in :pep:`314` are also source trees, this
PEP is updating the definition of sdists.


PEP Rejection
=============

The CLI based approach proposed in this PEP has been rejected in favour of the
Python API based approach proposed in :pep:`517`. The specific CLI used to
communicate with build backends running as isolated subprocesses will be
considered an implementation detail of front-end developer tool implementations.


Motivation
==========

There is significant pent-up frustration in the Python packaging ecosystem
around the current lock-in between build system and pip. Breaking that lock-in
is better for pip, for setuptools, and for other build systems like flit
[#flit]_.

Specification
=============

Overview
--------

Build tools will be located by reading a file ``pypa.json`` from the root
directory of the source tree. That file describes how to get the build tool
and the name of the command to run to invoke the tool.

All tools will be expected to conform to a single command line interface
modelled on pip's existing use of the setuptools setup.py interface.

pypa.json
---------

The file ``pypa.json`` acts as a neutral configuration file for pip and other
tools that want to build source trees to consult for configuration. The
absence of a ``pypa.json`` file in a Python source tree implies a setuptools
or setuptools compatible build system.

The JSON has the following schema. Extra keys are ignored, which permits the
use of ``pypa.json`` as a configuration file for other related tools. If doing
that the chosen keys must be namespaced under ``tools``::

  {"tools": {"flit": ["Flits content here"]}}

schema
    The version of the schema. This PEP defines version "1".  Defaults to "1"
    when absent. All tools reading the file must error on an unrecognised
    schema version.

bootstrap_requires
    Optional list of :pep:`508` dependency specifications that must be
    installed before running the build tool. For instance, if using flit, then
    the requirements might be::

        bootstrap_requires: ["flit"]

build_command
    A mandatory key, this is a list of Python format strings [#strformat]_
    describing the command to run. For instance, if using flit then the build
    command might be::

        build_command: ["flit"]

    If using a command which is a runnable module fred::

        build_command: ["{PYTHON}", "-m", "fred"]

Process interface
-----------------

The command to run is defined by a simple Python format string [#strformat]_.

This permits build systems with dedicated scripts and those that are invoked
using "python -m somemodule".

Processes will be run with the current working directory set to the root of
the source tree.

When run, processes should not read from stdin - while pip currently runs
build systems with stdin connected to its own stdin, stdout and stderr are
redirected and no communication with the user is possible.

As usual with processes, a non-zero exit status indicates an error.

Available format variables
--------------------------

PYTHON
    The Python interpreter in use. This is important to enable calling things
    which are just Python entry points.

        {PYTHON} -m foo

Available environment variables
-------------------------------

These variables are set by the caller of the build system and will always be
available.

PATH
    The standard system path.

PYTHON
    As for format variables.

PYTHONPATH
    Used to control sys.path per the normal Python mechanisms.

Subcommands
-----------

There are a number of separate subcommands that build systems must support.
The examples below use a build_command of ``flit`` for illustrative purposes.

build_requires
    Query build requirements. Build requirements are returned as a UTF-8
    encoded JSON document with one key ``build_requires`` consisting of a list
    of :pep:`508` dependency specifications. Additional keys must be
    ignored. The build_requires command is the only command run without
    setting up a build environment.

    Example command::

        flit build_requires

metadata
    Query project metadata.  The metadata and only the metadata should
    be output on stdout in UTF-8 encoding. pip would run metadata just once to
    determine what other packages need to be downloaded and installed. The
    metadata is output as a wheel METADATA file per :pep:`427`.

    Note that the metadata generated by the metadata command, and the metadata
    present in a generated wheel must be identical.

    Example command::

        flit metadata

wheel -d OUTPUT_DIR
    Command to run to build a wheel of the project. OUTPUT_DIR will point to
    an existing directory where the wheel should be output. Stdout and stderr
    have no semantic meaning. Only one file should be output - if more are
    output then pip would pick an arbitrary one to consume.

    Example command::

        flit wheel -d /tmp/pip-build_1234

develop [--prefix PREFIX]
    Command to do an in-place 'development' installation of the project.
    Stdout and stderr have no semantic meaning.

    Not all build systems will be able to perform develop installs. If a build
    system cannot do develop installs, then it should error when run. Note
    that doing so will cause use operations like ``pip install -e foo`` to
    fail.

    The prefix option is used for defining an alternative prefix for the
    installation. While setuptools has ``--root`` and ``--user`` options,
    they can be done equivalently using ``--prefix``, and pip or other
    tools that accept ``--root`` or ``--user`` options should translate
    appropriately.

    The root option is used to define an alternative root within which the
    command should operate.

    For instance::

        flit develop --root /tmp/ --prefix /usr/local

    Should install scripts within ``/tmp/usr/local/bin``, even if the Python
    environment in use reports that the sys.prefix is ``/usr/`` which would
    lead to using ``/tmp/usr/bin/``.
    Similar logic applies for package files etc.

The build environment
---------------------

Except for the build_requires command, all commands are run within a build
environment. No specific implementation is required, but a build environment
must achieve the following requirements.

1. All depende


id: pep-0502.rst
content:
PEP: 502
Title: String Interpolation - Extended Discussion
Author: Mike G. Miller
Status: Rejected
Type: Informational
Created: 10-Aug-2015
Python-Version: 3.6


Abstract
========

:pep:`498`: *Literal String Interpolation*, which proposed "formatted strings" was
accepted September 9th, 2015.
Additional background and rationale given during its design phase is detailed
below.

To recap that PEP,
a string prefix was introduced that marks the string as a template to be
rendered.
These formatted strings may contain one or more expressions
built on `the existing syntax`_ of ``str.format()``. [10]_ [11]_
The formatted string expands at compile-time into a conventional string format
operation,
with the given expressions from its text extracted and passed instead as
positional arguments.

At runtime,
the resulting expressions are evaluated to render a string to given
specifications::

    >>> location = 'World'
    >>> f'Hello, {location} !'      # new prefix: f''
    'Hello, World !'                # interpolated result

Format-strings may be thought of as merely syntactic sugar to simplify traditional
calls to ``str.format()``.

.. _the existing syntax: https://docs.python.org/3/library/string.html#format-string-syntax


PEP Status
==========

This PEP was rejected based on its using an opinion-based tone rather than a factual one.
This PEP was also deemed not critical as :pep:`498` was already written and should be the place
to house design decision details.


Motivation
==========

Though string formatting and manipulation features are plentiful in Python,
one area where it falls short
is the lack of a convenient string interpolation syntax.
In comparison to other dynamic scripting languages
with similar use cases,
the amount of code necessary to build similar strings is substantially higher,
while at times offering lower readability due to verbosity, dense syntax,
or identifier duplication.

These difficulties are described at moderate length in the original
`post to python-ideas`_
that started the snowball (that became :pep:`498`) rolling. [1]_

Furthermore, replacement of the print statement with the more consistent print
function of Python 3 (:pep:`3105`) has added one additional minor burden,
an additional set of parentheses to type and read.
Combined with the verbosity of current string formatting solutions,
this puts an otherwise simple language at an unfortunate disadvantage to its
peers::

    echo "Hello, user: $user, id: $id, on host: $hostname"              # bash
    say  "Hello, user: $user, id: $id, on host: $hostname";             # perl
    puts "Hello, user: #{user}, id: #{id}, on host: #{hostname}\n"      # ruby
                                                                        # 80 ch -->|
    # Python 3, str.format with named parameters
    print('Hello, user: {user}, id: {id}, on host: {hostname}'.format(**locals()))

    # Python 3, worst case
    print('Hello, user: {user}, id: {id}, on host: {hostname}'.format(user=user,
                                                                      id=id,
                                                                      hostname=
                                                                        hostname))

In Python, the formatting and printing of a string with multiple variables in a
single line of code of standard width is noticeably harder and more verbose,
with indentation exacerbating the issue.

For use cases such as smaller projects, systems programming,
shell script replacements, and even one-liners,
where message formatting complexity has yet to be encapsulated,
this verbosity has likely lead a significant number of developers and
administrators to choose other languages over the years.

.. _post to python-ideas: https://mail.python.org/pipermail/python-ideas/2015-July/034659.html


Rationale
=========


Goals
-------------

The design goals of format strings are as follows:

#. Eliminate need to pass variables manually.
#. Eliminate repetition of identifiers and redundant parentheses.
#. Reduce awkward syntax, punctuation characters, and visual noise.
#. Improve readability and eliminate mismatch errors,
   by preferring named parameters to positional arguments.
#. Avoid need for ``locals()`` and ``globals()`` usage,
   instead parsing the given string for named parameters,
   then passing them automatically. [2]_ [3]_


Limitations
-------------

In contrast to other languages that take design cues from Unix and its
shells,
and in common with Javascript,
Python specified both single (``'``) and double (``"``) ASCII quote
characters to enclose strings.
It is not reasonable to choose one of them now to enable interpolation,
while leaving the other for uninterpolated strings.
Other characters,
such as the "Backtick" (or grave accent `````) are also
`constrained by history`_
as a shortcut for ``repr()``.

This leaves a few remaining options for the design of such a feature:

* An operator, as in printf-style string formatting via ``%``.
* A class, such as ``string.Template()``.
* A method or function, such as ``str.format()``.
* New syntax, or
* A new string prefix marker, such as the well-known ``r''`` or ``u''``.

The first three options above are mature.
Each has specific use cases and drawbacks,
yet also suffer from the verbosity and visual noise mentioned previously.
All options are discussed in the next sections.

.. _constrained by history: https://mail.python.org/pipermail/python-ideas/2007-January/000054.html


Background
-------------

Formatted strings build on several existing techniques and proposals and what
we've collectively learned from them.
In keeping with the design goals of readability and error-prevention,
the following examples therefore use named,
not positional arguments.

Let's assume we have the following dictionary,
and would like to print out its items as an informative string for end users::

    >>> params = {'user': 'nobody', 'id': 9, 'hostname': 'darkstar'}


Printf-style formatting, via operator
'''''''''''''''''''''''''''''''''''''

This `venerable technique`_ continues to have its uses,
such as with byte-based protocols,
simplicity in simple cases,
and familiarity to many programmers::

    >>> 'Hello, user: %(user)s, id: %(id)s, on host: %(hostname)s' % params
    'Hello, user: nobody, id: 9, on host: darkstar'

In this form, considering the prerequisite dictionary creation,
the technique is verbose, a tad noisy,
yet relatively readable.
Additional issues are that an operator can only take one argument besides the
original string,
meaning multiple parameters must be passed in a tuple or dictionary.
Also, it is relatively easy to make an error in the number of arguments passed,
the expected type,
have a missing key,
or forget the trailing type, e.g. (``s`` or ``d``).

.. _venerable technique: https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting


string.Template Class
'''''''''''''''''''''

The ``string.Template`` `class from`_ :pep:`292`
(Simpler String Substitutions)
is a purposely simplified design,
using familiar shell interpolation syntax,
with `safe-substitution feature`_,
that finds its main use cases in shell and internationalization tools::

    Template('Hello, user: $user, id: ${id}, on host: $hostname').substitute(params)

While also verbose, the string itself is readable.
Though functionality is limited,
it meets its requirements well.
It isn't powerful enough for many cases,
and that helps keep inexperienced users out of trouble,
as well as avoiding issues with moderately-trusted input (i18n) from
third-parties.
It unfortunately takes enough code to discourage its use for ad-hoc string
interpolation,
unless encapsulated in a `convenience library`_ such as ``flufl.i18n``.

.. _class from: https://docs.python.org/3/library/string.html#template-strings
.. _safe-substitution feature: https://docs.python.org/3/library/string.html#string.Template.safe_substitute
.. _convenience library: http://p


id: pep-0264.rst
content:
PEP: 264
Title: Future statements in simulated shells
Author: Michael Hudson <mwh@python.net>
Status: Final
Type: Standards Track
Requires: 236
Created: 30-Jul-2001
Python-Version: 2.2
Post-History: 30-Jul-2001


Abstract
========

As noted in :pep:`236`, there is no clear way for "simulated
interactive shells" to simulate the behaviour of ``__future__``
statements in "real" interactive shells, i.e. have ``__future__``
statements' effects last the life of the shell.

The PEP also takes the opportunity to clean up the other
unresolved issue mentioned in :pep:`236`, the inability to stop
``compile()`` inheriting the effect of future statements affecting the
code calling ``compile()``.

This PEP proposes to address the first problem by adding an
optional fourth argument to the builtin function "compile", adding
information to the ``_Feature`` instances defined in ``__future__.py`` and
adding machinery to the standard library modules "codeop" and
"code" to make the construction of such shells easy.

The second problem is dealt with by simply adding *another*
optional argument to ``compile()``, which if non-zero suppresses the
inheriting of future statements' effects.


Specification
=============

I propose adding a fourth, optional, "flags" argument to the
builtin "compile" function.  If this argument is omitted,
there will be no change in behaviour from that of Python 2.1.

If it is present it is expected to be an integer, representing
various possible compile time options as a bitfield.  The
bitfields will have the same values as the ``CO_*`` flags already used
by the C part of Python interpreter to refer to future statements.

``compile()`` shall raise a ``ValueError`` exception if it does not
recognize any of the bits set in the supplied flags.

The flags supplied will be bitwise-"or"ed with the flags that
would be set anyway, unless the new fifth optional argument is a
non-zero integer, in which case the flags supplied will be exactly
the set used.

The above-mentioned flags are not currently exposed to Python.  I
propose adding ``.compiler_flag`` attributes to the ``_Feature`` objects
in ``__future__.py`` that contain the necessary bits, so one might
write code such as::

    import __future__
    def compile_generator(func_def):
        return compile(func_def, "<input>", "suite",
                    __future__.generators.compiler_flag)

A recent change means that these same bits can be used to tell if
a code object was compiled with a given feature; for instance ::

   codeob.co_flags & __future__.generators.compiler_flag``

will be non-zero if and only if the code object "codeob" was
compiled in an environment where generators were allowed.

I will also add a ``.all_feature_flags`` attribute to the ``__future__``
module, giving a low-effort way of enumerating all the ``__future__``
options supported by the running interpreter.

I also propose adding a pair of classes to the standard library
module codeop.

One - ``Compile`` - will sport a ``__call__`` method which will act much
like the builtin "compile" of 2.1 with the difference that after
it has compiled a ``__future__`` statement, it "remembers" it and
compiles all subsequent code with the ``__future__`` option in effect.

It will do this by using the new features of the ``__future__`` module
mentioned above.

Objects of the other class added to codeop - ``CommandCompiler`` -
will do the job of the existing ``codeop.compile_command`` function,
but in a ``__future__``-aware way.

Finally, I propose to modify the class ``InteractiveInterpreter`` in
the standard library module code to use a ``CommandCompiler`` to
emulate still more closely the behaviour of the default Python
shell.


Backward Compatibility
======================

Should be very few or none; the changes to compile will make no
difference to existing code, nor will adding new functions or
classes to codeop.  Existing code using
``code.InteractiveInterpreter`` may change in behaviour, but only for
the better in that the "real" Python shell will be being better
impersonated.


Forward Compatibility
=====================

The fiddling that needs to be done to ``Lib/__future__.py`` when
adding a ``__future__`` feature will be a touch more complicated.
Everything else should just work.


Issues
======

I hope the above interface is not too disruptive to implement for
Jython.


Implementation
==============

A series of preliminary implementations are at [1]_.

After light massaging by Tim Peters, they have now been checked in.


References
==========

.. [1] http://sourceforge.net/tracker/?func=detail&atid=305470&aid=449043&group_id=5470

Copyright
=========

This document has been placed in the public domain.


id: pep-0248.rst
content:
PEP: 248
Title: Python Database API Specification v1.0
Author: Greg Stein <gstein@lyra.org>, Marc-AndrÃ© Lemburg <mal@lemburg.com>
Discussions-To: db-sig@python.org
Status: Final
Type: Informational
Created: 08-May-1996
Post-History:
Superseded-By: 249


Introduction
============

This API has been defined to encourage similarity between the
Python modules that are used to access databases.  By doing this,
we hope to achieve a consistency leading to more easily understood
modules, code that is generally more portable across databases,
and a broader reach of database connectivity from Python.

This interface specification consists of several items:

* Module Interface
* Connection Objects
* Cursor Objects
* DBI Helper Objects

Comments and questions about this specification may be directed to
the SIG on Tabular Databases in Python
(http://www.python.org/sigs/db-sig).

This specification document was last updated on: April 9, 1996.
It will be known as Version 1.0 of this specification.


Module Interface
================

The database interface modules should typically be named with
something terminated by ``db``.  Existing examples are: ``oracledb``,
``informixdb``, and ``pg95db``.  These modules should export several
names:

``modulename(connection_string)``
   Constructor for creating a connection to the database.
   Returns a Connection Object.

``error``
   Exception raised for errors from the database module.


Connection Objects
==================

Connection Objects should respond to the following methods:

``close()``
   Close the connection now (rather than whenever ``__del__`` is
   called).  The connection will be unusable from this point
   forward; an exception will be raised if any operation is
   attempted with the connection.

``commit()``
   Commit any pending transaction to the database.

``rollback()``
   Roll the database back to the start of any pending
   transaction.

``cursor()``
   Return a new Cursor Object.  An exception may be thrown if
   the database does not support a cursor concept.

``callproc([params])``
   (Note: this method is not well-defined yet.)  Call a
   stored database procedure with the given (optional)
   parameters.  Returns the result of the stored procedure.

(all Cursor Object attributes and methods)
   For databases that do not have cursors and for simple
   applications that do not require the complexity of a
   cursor, a Connection Object should respond to each of the
   attributes and methods of the Cursor Object.  Databases
   that have cursor can implement this by using an implicit,
   internal cursor.


Cursor Objects
==============

These objects represent a database cursor, which is used to manage
the context of a fetch operation.

Cursor Objects should respond to the following methods and
attributes:

``arraysize``
   This read/write attribute specifies the number of rows to
   fetch at a time with ``fetchmany()``.  This value is also used
   when inserting multiple rows at a time (passing a
   tuple/list of tuples/lists as the params value to
   ``execute()``).  This attribute will default to a single row.

   Note that the arraysize is optional and is merely provided
   for higher performance database interactions.
   Implementations should observe it with respect to the
   ``fetchmany()`` method, but are free to interact with the
   database a single row at a time.

``description``
   This read-only attribute is a tuple of 7-tuples.  Each
   7-tuple contains information describing each result
   column: (name, type_code, display_size, internal_size,
   precision, scale, null_ok). This attribute will be ``None``
   for operations that do not return rows or if the cursor
   has not had an operation invoked via the ``execute()`` method
   yet.

   The 'type_code' is one of the 'dbi' values specified in
   the section below.

   Note: this is a bit in flux. Generally, the first two
   items of the 7-tuple will always be present; the others
   may be database specific.

``close()``
   Close the cursor now (rather than whenever ``__del__`` is
   called).  The cursor will be unusable from this point
   forward; an exception will be raised if any operation is
   attempted with the cursor.

``execute(operation [,params])``
   Execute (prepare) a database operation (query or command).
   Parameters may be provided (as a sequence
   (e.g. tuple/list)) and will be bound to variables in the
   operation.  Variables are specified in a database-specific
   notation that is based on the index in the parameter tuple
   (position-based rather than name-based).

   The parameters may also be specified as a sequence of
   sequences (e.g. a list of tuples) to insert multiple rows
   in a single operation.

   A reference to the operation will be retained by the
   cursor.  If the same operation object is passed in again,
   then the cursor can optimize its behavior.  This is most
   effective for algorithms where the same operation is used,
   but different parameters are bound to it (many times).

   For maximum efficiency when reusing an operation, it is
   best to use the ``setinputsizes()`` method to specify the
   parameter types and sizes ahead of time.  It is legal for
   a parameter to not match the predefined information; the
   implementation should compensate, possibly with a loss of
   efficiency.

   Using SQL terminology, these are the possible result
   values from the ``execute()`` method:

   - If the statement is DDL (e.g. ``CREATE TABLE``), then 1 is
     returned.

   - If the statement is DML (e.g. ``UPDATE`` or ``INSERT``), then the
     number of rows affected is returned (0 or a positive
     integer).

   - If the statement is DQL (e.g. ``SELECT``), ``None`` is returned,
     indicating that the statement is not really complete until
     you use one of the  'fetch' methods.

``fetchone()``
   Fetch the next row of a query result, returning a single
   tuple.

``fetchmany([size])``
   Fetch the next set of rows of a query result, returning as
   a list of tuples. An empty list is returned when no more
   rows are available. The number of rows to fetch is
   specified by the parameter.  If it is ``None``, then the
   cursor's arraysize determines the number of rows to be
   fetched.

   Note there are performance considerations involved with
   the size parameter.  For optimal performance, it is
   usually best to use the arraysize attribute.  If the size
   parameter is used, then it is best for it to retain the
   same value from one ``fetchmany()`` call to the next.

``fetchall()``
   Fetch all rows of a query result, returning as a list of
   tuples.  Note that the cursor's arraysize attribute can
   affect the performance of this operation.

``setinputsizes(sizes)``
   (Note: this method is not well-defined yet.)  This can be
   used before a call to ``execute()`` to predefine memory
   areas for the operation's parameters.  sizes is specified
   as a tuple -- one item for each input parameter.  The item
   should be a Type object that corresponds to the input that
   will be used, or it should be an integer specifying the
   maximum length of a string parameter.  If the item is
   ``None``, then no predefined memory area will be reserved
   for that column (this is useful to avoid predefined areas
   for large inputs).

   This method would be used before the ``execute()`` method is
   invoked.

   Note that this method is optional and is merely provided
   for higher performance database interaction.
   Implementations are free to do nothing and users are free
   to not use it.

``setoutputsize(size [,col])``
   (Note: this method is not well-defined yet.)

   Set a column buffer size for fetches of large columns
   (e.g. LONG).  The column is specified as an index into the
   result tuple.  Using a column of ``None`` will set the default
   size for all large columns in the cursor.

   This method would be used before the ``execute()`` method is
   invoked.

   Note that this m


id: pep-0260.rst
content:
PEP: 260
Title: Simplify xrange()
Author: Guido van Rossum <guido@python.org>
Status: Final
Type: Standards Track
Created: 26-Jun-2001
Python-Version: 2.2
Post-History: 26-Jun-2001


Abstract
========

This PEP proposes to strip the ``xrange()`` object from some rarely
used behavior like ``x[i:j]`` and ``x*n``.


Problem
=======

The ``xrange()`` function has one idiomatic use::

    for i in xrange(...): ...

However, the ``xrange()`` object has a bunch of rarely used behaviors
that attempt to make it more sequence-like.  These are so rarely
used that historically they have has serious bugs (e.g. off-by-one
errors) that went undetected for several releases.

I claim that it's better to drop these unused features.  This will
simplify the implementation, testing, and documentation, and
reduce maintenance and code size.


Proposed Solution
=================

I propose to strip the ``xrange()`` object to the bare minimum.  The
only retained sequence behaviors are ``x[i]``, ``len(x)``, and ``repr(x)``.
In particular, these behaviors will be dropped:

* ``x[i:j]`` (slicing)
* ``x*n``, ``n*x`` (sequence-repeat)
* ``cmp(x1, x2)`` (comparisons)
* ``i in x`` (containment test)
* ``x.tolist()`` method
* ``x.start``, ``x.stop``, ``x.step`` attributes

I also propose to change the signature of the ``PyRange_New()`` C API
to remove the 4th argument (the repetition count).

By implementing a custom iterator type, we could speed up the
common use, but this is optional (the default sequence iterator
does just fine).


Scope
=====

This PEP affects the ``xrange()`` built-in function and the
``PyRange_New()`` C API.


Risks
=====

Somebody's code could be relying on the extended code, and this
code would break.  However, given that historically bugs in the
extended code have gone undetected for so long, it's unlikely that
much code is affected.


Transition
==========

For backwards compatibility, the existing functionality will still
be present in Python 2.2, but will trigger a warning.  A year
after Python 2.2 final is released (probably in 2.4) the
functionality will be ripped out.


Copyright
=========

This document has been placed in the public domain.


