# embeddings

id: pep-0506.rst
content:
PEP: 506
Title: Adding A Secrets Module To The Standard Library
Author: Steven D'Aprano <steve@pearwood.info>
Status: Final
Type: Standards Track
Created: 19-Sep-2015
Python-Version: 3.6
Post-History:


Abstract
========

This PEP proposes the addition of a module for common security-related
functions such as generating tokens to the Python standard library.


Definitions
===========

Some common abbreviations used in this proposal:

* PRNG:

  Pseudo Random Number Generator.  A deterministic algorithm used
  to produce random-looking numbers with certain desirable
  statistical properties.

* CSPRNG:

  Cryptographically Strong Pseudo Random Number Generator.  An
  algorithm used to produce random-looking numbers which are
  resistant to prediction.

* MT:

  Mersenne Twister.  An extensively studied PRNG which is currently
  used by the ``random`` module as the default.


Rationale
=========

This proposal is motivated by concerns that Python's standard library
makes it too easy for developers to inadvertently make serious security
errors.  Theo de Raadt, the founder of OpenBSD, contacted Guido van Rossum
and expressed some concern [#]_ about the use of MT for generating sensitive
information such as passwords, secure tokens, session keys and similar.

Although the documentation for the ``random`` module explicitly states that
the default is not suitable for security purposes [#]_, it is strongly
believed that this warning may be missed, ignored or misunderstood by
many Python developers.  In particular:

* developers may not have read the documentation and consequently
  not seen the warning;

* they may not realise that their specific use of the module has security
  implications; or

* not realising that there could be a problem, they have copied code
  (or learned techniques) from websites which don't offer best
  practises.

The first [#]_ hit when searching for "python how to generate passwords" on
Google is a tutorial that uses the default functions from the ``random``
module [#]_.  Although it is not intended for use in web applications, it is
likely that similar techniques find themselves used in that situation.
The second hit is to a StackOverflow question about generating
passwords [#]_.  Most of the answers given, including the accepted one, use
the default functions.  When one user warned that the default could be
easily compromised, they were told "I think you worry too much." [#]_

This strongly suggests that the existing ``random`` module is an attractive
nuisance when it comes to generating (for example) passwords or secure
tokens.

Additional motivation (of a more philosophical bent) can be found in the
post which first proposed this idea [#]_.


Proposal
========

Alternative proposals have focused on the default PRNG in the ``random``
module, with the aim of providing "secure by default" cryptographically
strong primitives that developers can build upon without thinking about
security.  (See Alternatives below.)  This proposes a different approach:

* The standard library already provides cryptographically strong
  primitives, but many users don't know they exist or when to use them.

* Instead of requiring crypto-naive users to write secure code, the
  standard library should include a set of ready-to-use "batteries" for
  the most common needs, such as generating secure tokens.  This code
  will both directly satisfy a need ("How do I generate a password reset
  token?"), and act as an example of acceptable practises which
  developers can learn from [#]_.

To do this, this PEP proposes that we add a new module to the standard
library, with the suggested name ``secrets``.  This module will contain a
set of ready-to-use functions for common activities with security
implications, together with some lower-level primitives.

The suggestion is that ``secrets`` becomes the go-to module for dealing
with anything which should remain secret (passwords, tokens, etc.)
while the ``random`` module remains backward-compatible.


API and Implementation
======================

This PEP proposes the following functions for the ``secrets`` module:

* Functions for generating tokens suitable for use in (e.g.) password
  recovery, as session keys, etc., in the following formats:

  - as bytes, ``secrets.token_bytes``;
  - as text, using hexadecimal digits, ``secrets.token_hex``;
  - as text, using URL-safe base-64 encoding, ``secrets.token_urlsafe``.

* A limited interface to the system CSPRNG, using either ``os.urandom``
  directly or ``random.SystemRandom``.  Unlike the ``random`` module, this
  does not need to provide methods for seeding, getting or setting the
  state, or any non-uniform distributions.  It should provide the
  following:

  - A function for choosing items from a sequence, ``secrets.choice``.
  - A function for generating a given number of random bits and/or bytes
    as an integer, ``secrets.randbits``.
  - A function for returning a random integer in the half-open range
    0 to the given upper limit, ``secrets.randbelow`` [#]_.

* A function for comparing text or bytes digests for equality while being
  resistant to timing attacks, ``secrets.compare_digest``.

The consensus appears to be that there is no need to add a new CSPRNG to
the ``random`` module to support these uses, ``SystemRandom`` will be
sufficient.

Some illustrative implementations have been given by Alyssa (Nick) Coghlan [#]_
and a minimalist API by Tim Peters [#]_. This idea has also been discussed
on the issue tracker for the "cryptography" module [#]_.  The following
pseudo-code should be taken as the starting point for the real
implementation::

    from random import SystemRandom
    from hmac import compare_digest

    _sysrand = SystemRandom()

    randbits = _sysrand.getrandbits
    choice = _sysrand.choice

    def randbelow(exclusive_upper_bound):
        return _sysrand._randbelow(exclusive_upper_bound)

    DEFAULT_ENTROPY = 32  # bytes

    def token_bytes(nbytes=None):
        if nbytes is None:
            nbytes = DEFAULT_ENTROPY
        return os.urandom(nbytes)

    def token_hex(nbytes=None):
        return binascii.hexlify(token_bytes(nbytes)).decode('ascii')

    def token_urlsafe(nbytes=None):
        tok = token_bytes(nbytes)
        return base64.urlsafe_b64encode(tok).rstrip(b'=').decode('ascii')


The ``secrets`` module itself will be pure Python, and other Python
implementations can easily make use of it unchanged, or adapt it as
necessary. An implementation can be found on BitBucket [#]_.

Default arguments
~~~~~~~~~~~~~~~~~

One difficult question is "How many bytes should my token be?".  We can
help with this question by providing a default amount of entropy for the
"token_*" functions. If the ``nbytes`` argument is None or not given, the
default entropy will be used. This default value should be large enough
to be expected to be secure for medium-security uses, but is expected to
change in the future, possibly even in a maintenance release [#]_.

Naming conventions
~~~~~~~~~~~~~~~~~~

One question is the naming conventions used in the module [#]_, whether to
use C-like naming conventions such as "randrange" or more Pythonic names
such as "random_range".

Functions which are simply bound methods of the private ``SystemRandom``
instance (e.g. ``randrange``), or a thin wrapper around such, should keep
the familiar names. Those which are something new (such as the various
``token_*`` functions) will use more Pythonic names.

Alternatives
============

One alternative is to change the default PRNG provided by the ``random``
module [#]_.  This received considerable scepticism and outright opposition:

* There is fear that a CSPRNG may be slower than the current PRNG (which
  in the case of MT is already quite slow).

* Some applications (such as scientific simulations, and replaying
  gameplay) require the ability to seed the PRNG into a known state,
  which a CSPRNG lacks by design.

* Another major use of the ``random`` module i


id: pep-0512.rst
content:
PEP: 512
Title: Migrating from hg.python.org to GitHub
Author: Brett Cannon <brett@python.org>
Discussions-To: core-workflow@python.org
Status: Final
Type: Process
Created: 17-Jan-2015
Post-History: 17-Jan-2016, 19-Jan-2016, 23-Jan-2016


.. note::

   CPython's development process moved to https://github.com/python/cpython
   on 2017-02-10.

Abstract
========

This PEP outlines the steps required to migrate Python's development
process from Mercurial [#hg]_ as hosted at
hg.python.org [#h.p.o]_ to Git [#git]_ on GitHub [#GitHub]_. Meeting
the minimum goals of this PEP should allow for the development
process of Python to be as productive as it currently is, and meeting
its extended goals should improve the development process from its
status quo.


Rationale
=========

In 2014, it became obvious that Python's custom development
process was becoming a hindrance. As an example, for an external
contributor to submit a fix for a bug that eventually was committed,
the basic steps were:

1. Open an issue for the bug at bugs.python.org [#b.p.o]_.
2. Checkout out the CPython source code from hg.python.org [#h.p.o]_.
3. Make the fix.
4. Upload a patch.
5. Have a core developer review the patch using our fork of the
   Rietveld code review tool [#rietveld]_.
6. Download the patch to make sure it still applies cleanly.
7. Run the test suite manually.
8. Update the ``NEWS``, ``ACKS``, and "What's New" document as necessary
9. Pull changes to avoid a merge race.
10. Commit the change manually.
11. If the change was for a bugfix release, merge into the
    in-development branch.
12. Run the test suite manually again.
13. Commit the merge.
14. Push the changes.

This is a very heavy, manual process for core developers. Even in the
simple case, you could only possibly skip the code review step, as you
would still need to build the documentation. This led to patches
languishing on the issue tracker due to core developers not being
able to work through the backlog fast enough to keep up with
submissions. In turn, that led to a side-effect issue of discouraging
outside contribution due to frustration from lack of attention, which
is a dangerous problem for an open source project with no corporate
backing as it runs counter to having a viable future for the project.
While allowing patches to be uploaded to bugs.python.org [#b.p.o]_ is
potentially simple for an external contributor, it is as slow and
burdensome as it gets for a core developer to work with.

Hence the decision was made in late 2014 that a move to a new
development process was needed. A request for PEPs
proposing new workflows was made, in the end leading to two:
:pep:`481` and :pep:`507` proposing GitHub [#github]_ and
GitLab [#gitlab]_, respectively.

The year 2015 was spent off-and-on working on those proposals and
trying to tease out details of what made them different from each
other on the core-workflow mailing list [#core-workflow]_.
PyCon US 2015 also showed that the community was a bit frustrated
with our process due to both cognitive overhead for new contributors
and how long it was taking for core developers to
look at a patch (see the end of Guido van Rossum's
keynote at PyCon US 2015 [#guido-keynote]_ as an example of the
frustration).

On January 1, 2016, the decision was made by Brett Cannon to move the
development process to GitHub. The key reasons for choosing GitHub
were [#reasons]_:

* Maintaining custom infrastructure has been a burden on volunteers
  (e.g., an unmaintained, custom fork of Rietveld [#rietveld]_
  is currently being used).
* The custom workflow is very time-consuming for core developers
  (not enough automated tooling built to help support it).
* The custom workflow is a hindrance to external contributors
  (acts as a barrier of entry due to time required to ramp up on
  development process unique to CPython itself).
* There is no feature differentiating GitLab from GitHub beyond
  GitLab being open source.
* Familiarity with GitHub is far higher among core developers and
  external contributors than with GitLab.
* Our BDFL prefers GitHub (who would be the first person to tell
  you that his opinion shouldn't matter, but the person making the
  decision felt it was important that the BDFL feel comfortable with
  the workflow of his own programming language to encourage his
  continued participation).

There's even already an unofficial logo to represent the
migration to GitHub [#pythocat]_.

The overarching goal of this migration is to improve the development
process to the extent that a core developer can go from external
contribution submission through all the steps leading to committing
said contribution from within a browser on a tablet with WiFi
using *some* development process (this does not inherently mean
GitHub's default workflow). The final solution will also allow
an external contributor to contribute even if they chose not to use
GitHub (although there is not guarantee in feature parity).


Repositories to Migrate
=======================

While hg.python.org [#h.p.o]_ hosts many repositories, there are only
five key repositories that need to move:

1. devinabox [#devinabox-repo]_ (done)
2. benchmarks [#benchmarks-repo]_ (skipped)
3. peps [#peps-repo]_ (done)
4. devguide [#devguide-repo]_ (done)
5. cpython [#cpython-repo]_

The devinabox repository is code-only.
The peps and devguide repositories involve the generation of webpages.
And the cpython repository has special requirements for integration
with bugs.python.org [#b.p.o]_.


Migration Plan
==============

The migration plan is separated into sections based on what is
required to migrate the repositories listed in the
`Repositories to Migrate`_ section. Completion of requirements
outlined in each section should unblock the migration of the related
repositories. The sections are expected to be completed in order, but
not necessarily the requirements within a section.


Requirements for Code-Only Repositories
---------------------------------------

Completion of the requirements in this section will allow the
devinabox repository to move to GitHub.


Create a 'Python core' team
'''''''''''''''''''''''''''

To manage permissions, a 'Python core' team will be created as part of
the python organization [#github-python-org]_. Any repository that is
moved will have the 'Python core' team added to it with write
permissions [#github-org-perms]_. Anyone who previously had rights to
manage SSH keys on hg.python.org will become a team maintainer for the
'Python core' team.


Define commands to move a Mercurial repository to Git
'''''''''''''''''''''''''''''''''''''''''''''''''''''

Since moving to GitHub also entails moving to Git [#git]_, we must
decide what tools and commands we will run to translate a Mercurial
repository to Git. The tools developed specifically for this migration
are hosted at https://github.com/orsenthil/cpython-hg-to-git .


CLA enforcement
'''''''''''''''

A key part of any open source project is making sure that its source
code can be properly licensed. This requires making sure all people
making contributions have signed a contributor license agreement
(CLA) [#cla]_. Up until now, enforcement of CLA signing of
contributed code has been enforced by core developers checking
whether someone had an ``*`` by their username on
bugs.python.org [#b.p.o]_. With this migration, the plan is to start
off with automated checking and enforcement of contributors signing
the CLA.


Adding GitHub username support to bugs.python.org
+++++++++++++++++++++++++++++++++++++++++++++++++

To keep tracking of CLA signing under the direct control of the PSF,
tracking who has signed the PSF CLA will be continued by marking that
fact as part of someone's bugs.python.org user profile. What this
means is that an association will be needed between a person's
bugs.python.org [#b.p.o]_ account and their GitHub account, which
will be done through a new field in a user's profile. This does
implicitly require that contribu


id: pep-0274.rst
content:
PEP: 274
Title: Dict Comprehensions
Author: Barry Warsaw <barry@python.org>
Status: Final
Type: Standards Track
Created: 25-Oct-2001
Python-Version: 2.7, 3.0
Post-History: 29-Oct-2001


Abstract
========

:pep:`202` introduces a syntactical extension to Python called the
"list comprehension".  This PEP proposes a similar syntactical
extension called the "dictionary comprehension" or "dict
comprehension" for short.  You can use dict comprehensions in ways
very similar to list comprehensions, except that they produce
Python dictionary objects instead of list objects.


Resolution
==========

This PEP was originally written for inclusion in Python 2.3.  It
was withdrawn after observation that substantially all of its
benefits were subsumed by generator expressions coupled with the
``dict()`` constructor.

However, Python 2.7 and 3.0 introduces this exact feature, as well
as the closely related set comprehensions.  On 2012-04-09, the PEP
was changed to reflect this reality by updating its Status to
Accepted, and updating the Python-Version field.  The Open
Questions section was also removed since these have been long
resolved by the current implementation.


Proposed Solution
=================

Dict comprehensions are just like list comprehensions, except that
you group the expression using curly braces instead of square
braces.  Also, the left part before the ``for`` keyword expresses
both a key and a value, separated by a colon.  The notation is
specifically designed to remind you of list comprehensions as
applied to dictionaries.


Rationale
=========

There are times when you have some data arranged as a sequences of
length-2 sequences, and you want to turn that into a dictionary.
In Python 2.2, the ``dict()`` constructor accepts an argument that is
a sequence of length-2 sequences, used as (key, value) pairs to
initialize a new dictionary object.

However, the act of turning some data into a sequence of length-2
sequences can be inconvenient or inefficient from a memory or
performance standpoint.  Also, for some common operations, such as
turning a list of things into a set of things for quick duplicate
removal or set inclusion tests, a better syntax can help code
clarity.

As with list comprehensions, an explicit for loop can always be
used (and in fact was the only way to do it in earlier versions of
Python).  But as with list comprehensions, dict comprehensions can
provide a more syntactically succinct idiom that the traditional
for loop.


Semantics
=========

The semantics of dict comprehensions can actually be demonstrated
in stock Python 2.2, by passing a list comprehension to the
built-in dictionary constructor::

    >>> dict([(i, chr(65+i)) for i in range(4)])

is semantically equivalent to::

    >>> {i : chr(65+i) for i in range(4)}

The dictionary constructor approach has two distinct disadvantages
from the proposed syntax though.  First, it isn't as legible as a
dict comprehension.  Second, it forces the programmer to create an
in-core list object first, which could be expensive.


Examples
========

::

   >>> print {i : chr(65+i) for i in range(4)}
   {0 : 'A', 1 : 'B', 2 : 'C', 3 : 'D'}

::

   >>> print {k : v for k, v in someDict.iteritems()} == someDict.copy()
   1

::

   >>> print {x.lower() : 1 for x in list_of_email_addrs}
   {'barry@zope.com'   : 1, 'barry@python.org' : 1, 'guido@python.org' : 1}

::

   >>> def invert(d):
   ...     return {v : k for k, v in d.iteritems()}
   ...
   >>> d = {0 : 'A', 1 : 'B', 2 : 'C', 3 : 'D'}
   >>> print invert(d)
   {'A' : 0, 'B' : 1, 'C' : 2, 'D' : 3}

::

   >>> {(k, v): k+v for k in range(4) for v in range(4)}
   ... {(3, 3): 6, (3, 2): 5, (3, 1): 4, (0, 1): 1, (2, 1): 3,
      (0, 2): 2, (3, 0): 3, (0, 3): 3, (1, 1): 2, (1, 0): 1,
      (0, 0): 0, (1, 2): 3, (2, 0): 2, (1, 3): 4, (2, 2): 4, (
      2, 3): 5}


Implementation
==============

All implementation details were resolved in the Python 2.7 and 3.0
time-frame.


Copyright
=========

This document has been placed in the public domain.


id: pep-0738.rst
content:
PEP: 738
Title: Adding Android as a supported platform
Author: Malcolm Smith <smith@chaquo.com>
Sponsor: Petr Viktorin <encukou@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-738-adding-android-as-a-supported-platform/40975
Status: Final
Type: Standards Track
Created: 12-Dec-2023
Python-Version: 3.13
Resolution: https://discuss.python.org/t/pep-738-adding-android-as-a-supported-platform/40975/23

.. canonical-doc:: :ref:`python:using-android`


Abstract
========

This PEP proposes adding Android as a supported platform in CPython. The initial
goal is for Android to achieve Tier 3 support in Python 3.13.

This PEP is based on :pep:`730` â "Adding iOS as a supported platform" by
Russell Keith-Magee, and covers many of the same issues. Notable differences
between the two platforms can be found by searching for the word "iOS".


Motivation
==========

Over the last 15 years, mobile platforms have become increasingly important
parts of the computing landscape. Android is the operating system that runs on
about `70% of these devices
<https://gs.statcounter.com/os-market-share/mobile/worldwide>`__. However, there
is no official support for Android in CPython.

The `Chaquopy <https://chaquo.com/chaquopy/>`__, `BeeWare
<https://beeware.org>`__ and `Kivy <https://kivy.org>`__ projects
have all supported Android for many years, and they have all been used to
generate applications that have been accepted for publication in the Google Play
Store. This demonstrates the technical feasibility of Android support.

It is important for the future of Python as a language that it is able to be
used on any platform that has widespread adoption. Otherwise, potential users
will choose other languages that *do* provide support for these platforms. This
is especially true in education, where the next generation of developers is in
many cases already spending more time using mobile platforms than desktop ones.


Rationale
=========

General
-------

Android is broadly a POSIX platform, based on a Linux kernel and the
ELF binary format. It does not use glibc, instead providing its own C
library implementation called Bionic. As a result, it is generally not
binary-compatible with any other Linux distribution, even if the architecture
matches. It also has its own filesystem layout which doesn't resemble any other
Unix.

However, Android's source-compatibility with Linux is quite good. In its early years,
the C library was very incomplete, but most of the gaps were filled by around
2014. Since then, any C code that compiles for Linux can usually be compiled for
Android, unless it involves direct access to hardware devices or operating
system services.

This is also true of CPython. Although it has never officially supported
Android, recent versions (since 3.6) can already be compiled for Android with
minimal patching.

.. _738-os-versions:

OS versions
-----------

Each Android version can be identified in three ways:

* A conventional dotted version number (though recent versions have all used
  whole numbers)
* A sequential integer "API level" (the most common form in developer
  documentation)
* An alphabetic confectionery-themed code name (no longer used for marketing,
  but still appears in developer documentation)

There is no consistent pattern to link one of these to another; they must be
looked up in `a table <https://en.wikipedia.org/wiki/Android_version_history>`__.

A new major Android version is released each year, but the updates available to
each device are entirely under the control of its manufacturer. Unfortunately
many manufacturers stop sending updates to devices long before their users are
ready to dispose of them. For example, as of October 2023, the oldest Android
version still receiving security updates was API level 30, but according to
`Google's own statistics
<https://dl.google.com/android/studio/metadata/distributions.json>`__, only 60%
of devices were on that version or newer.

For Python 3.13 we therefore propose the minimum Android version to be 5.0
(API level 21), which was released in 2014. According to the statistics above,
this would cover 99% of active devices.


Development tools
-----------------

The Android development tools are equally supported on Linux (x86_64), Windows
(x86_64) and macOS (x86_64 and ARM64). For CPython, the most important tools
are:

* The NDK (native development kit) contains a C and C++ compiler (clang),
  linker (lld), and headers for all the system libraries.

  Binary compatibility between libraries compiled with different versions of the
  NDK is generally very good, but for reproducibility it would be best for each
  Python version to stick with one NDK version throughout its life. For Python
  3.13, this would be the current NDK long-term support version, r26.

  Each NDK version can be set to target any of a wide range of Android versions.
  For example, NDK r26 supports :ref:`API levels <738-os-versions>` 21 to 34.
  However, binaries compiled for an older Android version will usually keep on
  working indefinitely on newer versions; exceptions to this rule are only made
  for security reasons.

* Gradle is the tool used to build complete, deployable apps.

* The emulator, based on QEMU, is a simulated Android device running on a
  development machine. Unlike on iOS, an emulator uses the same ABI as a real
  device of the same architecture, and can run the same binaries.

These tools may all be used either from the command line, or through the Android
Studio IDE, which is based on IntelliJ IDEA.


Architectures
-------------

Android currently supports 4 architectures. Their names as used by the Android
tools are:

* ``armeabi-v7a``
* ``arm64-v8a``
* ``x86``
* ``x86_64``

Virtually all current physical devices use one of the ARM architectures. ``x86``
and ``x86_64`` are supported for use in the emulator.

For Python 3.13 we propose that Tier 3 support will only cover the 64-bit platforms
(``arm64-v8a`` and ``x86_64``):

* ``x86`` has not been supported as a development platform since 2020, and no
  new emulator images have been released since then.

* ``armeabi-v7a``'s proportion of active devices is now
  `less than 10% and steadily falling
  <https://github.com/chaquo/chaquopy/issues/709#issuecomment-1744541892>`__.

  It would also be more difficult to cover with a reliable buildbot, since there
  are no native hosts available for the emulator (ARM64 Macs don't have hardware
  support for ARM32 code). Although cross-architecture emulation is possible, it
  has much worse performance and stability, which is why the ``armeabi-v7a``
  emulator images have not been updated since 2016.

  However, it continues to be used for watches and ultra-low-cost phones. If
  this persists, we may need to consider adding it in a future Python version.

Even if 32-bit architectures are not officially supported, no changes should be
made which would impede any downstream projects which still wish to build them.


App lifecycle
-------------

The primary programming language in Android apps is Java, or its modern descendant
Kotlin. As such, an app does not provide its own executable file. Instead, all
apps start off as a Java virtual machine running an executable provided by the
operating system. The app's Java code can then add native code to the process by
loading dynamic libraries and calling them through JNI.

Unlike iOS, creating subprocesses *is* supported on Android. However apps may
only run executables in `certain locations
<https://issuetracker.google.com/issues/128554619#comment4>`__, none of which
are writable at runtime. Long-running subprocesses are `officially discouraged
<https://issuetracker.google.com/issues/128554619#comment4>`__, and are not
guaranteed to be supported in future Android versions.

Android does provide a command-line shell, but this is intended only for use by
developers, and is not available to the typical end user.

For these reasons, the recommended way of running Python on


id: pep-0704.rst
content:
PEP: 704
Title: Require virtual environments by default for package installers
Author: Pradyun Gedam <pradyunsg@gmail.com>
Sponsor: Brett Cannon <brett@python.org>
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/22846
Status: Withdrawn
Type: Standards Track
Topic: Packaging
Created: 16-Jan-2023
Post-History: `16-Jan-2023 <https://discuss.python.org/t/22846>`__


Abstract
========

This PEP recommends that package installers like ``pip``  require a virtual environment by default on Python 3.13+.

PEP Withdrawal
==============

During discussion of this PEP, it became clear that changes to pip's UX are not controlled by PEPs as proposed. It also became clear that a significant number of users rely on being able to mix managed-by-pip dependencies with managed-by-some-other-tool dependencies (most prominently, when using Conda).

Further, a significant subset of the benefits of the proposed change are achievable via :pep:`668` (accepted and implemented at the time of writing). It provides redistributors of the Python interpreter control over whether users should be required to use a virtual environment, what the messaging presented to the user is and how the rollout of that change should happen for their users.

Since enforcement of virtual environments with ``pip`` was the primary focus of this PEP, it felt more appropriate to withdraw this PEP than to refocus it on a different topic.

A future PEP to resolve the virtual environment naming convention question/issues would still be appropriate, but it's worth starting any such effort as a fresh PEP focused on the benefits of such a convention, rather than on the enforcement of it.

Motivation
==========

Python virtual environments are an essential part of the development workflow for Python. However, they require extra effort since they are an opt-in feature, and requires users to either:

- take explicit steps to activate/deactivate a virtual environment
- use :samp:`{<path-to-venv>}/{<bin-path>}/{<executable>}` to run files

For new users, things will seemingly work correctly when virtual environments are not used -â until they don't. Further, activating a virtual environment uses slightly different syntax and mechanisms on different platforms. This complicates the introduction to virtual environments, since now information and context about how/why they are useful needs to explained to justify adding additional steps to the workflow.

It also creates a scope for mistakes, since users need to remember to activate the virtual environment before running an installer like ``pip`` or configure those installers to error out. On certain Linux distributions, forgetting to do so can result in the installer modifying files that are owned by the operating system (which is partially mitigated by :pep:`668` for distributions that opt-in to marking their environments accordingly).


Rationale
=========

Changing the default behaviour of installers like ``pip`` to require a virtual environment to be active would:

- make it easier for new users to get started with Python (since there's a consistent experience and virtual environments are understood as a thing you're required to use)
- reduce the scope for accidental installation issues for all users by default (by explicitly flagging when you're not using a virtual environment).

Setting up a convention of placing the virtual environment in-tree in a directory named ``.venv`` removes a decision point for common workflows and creates a clear convention within the ecosystem.

Specification
=============

Requiring a virtual environment by default
------------------------------------------

When a user runs an installer without an active virtual environment, the installer SHOULD print an error message and exit with a non-zero exit code.

The error message SHOULD inform the user that a virtual environment is required, SHOULD provide shell-specific instructions on how to create and activate a virtual environment named ``.venv``, and SHOULD provide a link to a documentation page that explains how to create and activate a virtual environment.

See `Implementation Notes`_ for more details.

Opting out of virtual environments
----------------------------------

The installer SHOULD also provide a explicit opt-in to disable this requirement, allowing an end user to use it outside of a virtual environment. If the installer does not provide this functionality, it SHOULD mention this in the error message and documentation page.

Consistent timeline for the change
----------------------------------

Installers MAY choose to implement this default behaviour on any Python versions, but SHOULD implement it on Python 3.13 or newer.


Backwards Compatibility
=======================

This PEP is backwards incompatible with workflows where users are using installers outside of virtual environments. Such users will be prompted with an error message and will need to either:

- explicitly opt-in to running the installer outside of a virtual environment, or
- create and use a virtual environment

Users who are already using virtual environments will not be affected by this change.

Workflow tools (which manage virtual environments for the user, under the hood) should be unaffected, since they should already be using a virtual environment for running the installer.


Security Implications
=====================

This PEP does not introduce any new security implications.


How to Teach This
=================

This PEP requires that new users create and use a virtual environment to get started with using Python packages. This is, however, a best practice, as `demonstrated <https://packaging.python.org/en/latest/tutorials/installing-packages/#creating-virtual-environments>`__ by the section on "basics of how to install Python packages" in the Python Packaging User Guide, which explains how/what virtual environments are before discussing using ``pip``.


Reference Implementation
========================

There is no reference implementation for this PEP. However, the proposed behaviour is largely already implemented in ``pip`` and can be activated by setting the ``PIP_REQUIRE_VENV`` environment variable to ``1``. (Leaving it unset results in the proposed opt-in behaviour of not requiring a virtual environment for installation.)


Implementation Notes
====================

Detecting an active virtual environment
---------------------------------------

:pep:`As discussed in PEP 668 <668#backwards-compatibility>`, the logic for robustly detecting a virtual environment is something like::

    def is_virtual_environment():
        return sys.base_prefix != sys.prefix or hasattr(sys, "real_prefix")

Documentation on using a virtual environment
--------------------------------------------

Package installers are expected to provide a link to a documentation page in the error message.

Ideally, such a documentation page would explain what virtual environments are, why they are required, and how to create and activate a virtual environment using ``venv``. It should include instructions for the most common shells and platforms.

Such a documentation page should be made available in the `Python Packaging User Guide <https://packaging.python.org>`__ to reduce duplicated effort across installers for covering this topic.

Rejected Ideas
==============

Do not specify a name for the virtual environment directory
-----------------------------------------------------------

Using a consistent name for the virtual environment directory is important for a few reasons:

1. It makes it easier for users to find the virtual environment directory, and to activate it.
2. It removes a decision point for new users, since they do not need to decide on a name for the virtual environment directory.
3. It creates a clear convention within the ecosystem, which makes it easier for users to find documentation.
4. It ensures consistency across different tools, so that differences 


id: pep-0710.rst
content:
PEP: 710
Title: Recording the provenance of installed packages
Author: FridolÃ­n PokornÃ½ <fridolin.pokorny at gmail.com>
Sponsor: Donald Stufft <donald@stufft.io>
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-710-recording-the-provenance-of-installed-packages/25428
Status: Draft
Type: Standards Track
Topic: Packaging
Created: 27-Mar-2023
Post-History: `03-Dec-2021 <https://discuss.python.org/t/pip-installation-reports/12316>`__,
              `30-Jan-2023 <https://discuss.python.org/t/pre-pep-recording-provenance-of-installed-packages/23340>`__,
              `14-Mar-2023 <https://discuss.python.org/t/draft-pep-recording-provenance-of-installed-packages/24838>`__,
              `03-Apr-2023 <https://discuss.python.org/t/pep-710-recording-the-provenance-of-installed-packages/25428>`__,

Abstract
========

This PEP describes a way to record the provenance of installed Python distributions.
The record is created by an installer and is available to users in
the form of a JSON file ``provenance_url.json`` in the ``.dist-info`` directory.
The mentioned JSON file captures additional metadata to allow recording a URL to a
:term:`distribution package` together with the installed distribution hash.
This proposal is built on top of :pep:`610` following :ref:`its corresponding
canonical PyPA spec <packaging:direct-url>` and complements ``direct_url.json``
with ``provenance_url.json`` for when packages are identified by a name, and
optionally a version.

Motivation
==========

Installing a Python :term:`Project` involves downloading a :term:`Distribution Package`
from a :term:`Package Index`
and extracting its content to an appropriate place. After the installation
process is done, information about the release artifact used as well as its source
is generally lost. However, there are use cases for keeping records of
distributions used for installing packages and their provenance.

Python wheels can be built with different compiler flags or supporting
different wheel tags. In both cases, users might get into a situation in which
multiple wheels might be considered by installers (possibly from different
package indexes) and immediately finding out which wheel file was actually used
during the installation might be helpful. This way, developers can use
information about wheels to debug issues making sure the desired wheel was
actually installed. Another use case could be tools reporting software
installed, such as tools reporting a SBOM (Software Bill of Materials), that might
give more accurate reports. Yet another use case could be reconstruction of the
Python environment by pinning each installed package to a specific distribution
artifact consumed from a Python package index.

Rationale
=========

The motivation described in this PEP is an extension of :ref:`Recording the
Direct URL Origin of installed distributions <packaging:direct-url>`
specification. In addition to recording provenance information for packages
installed using a direct URL, installers should also do so for packages
installed by name (and optionally version) from Python package indexes.

The idea described in this PEP originated in a tool called `micropipenv`_
that is used to install
:term:`distribution packages <Distribution Package>` in containerized
environments (see the reported issue `thoth-station/micropipenv#206`_).
Currently, the assembled containerized application does not implicitly carry
information about the provenance of installed distribution packages
(unless these are installed from full URLs and recorded via ``direct_url.json``).
This requires container image suppliers to link
container images with the corresponding build process, its configuration and
the application source code for checking requirements files in cases when
software present in containerized environments needs to be audited.

The `subsequent discussion in the Discourse thread
<https://discuss.python.org/t/12316>`__ also brought up
pip's new ``--report`` option that can
`generate a detailed JSON report <pip_installation_report_>`__ about
the installation process. This option could help with the provenance problem
this PEP approaches. Nevertheless, this option needs to be *explicitly* passed
to pip to obtain the provenance information, and includes additional metadata that
might not be necessary for checking the provenance (such as Python version
requirements of each distribution package). Also, this option is
specific to pip as of the writing of this PEP.

Note the current :ref:`spec for recording installed packages
<packaging:recording-installed-packages>` defines a ``RECORD`` file that
records installed files, but not the distribution artifact from which these
files were obtained. Auditing installed artifacts can be performed
based on matching the entries listed in the ``RECORD`` file. However, this
technique requires a pre-computed database of files each artifact provides or a
comparison with the actual artifact content. Both approaches are relatively
expensive and time consuming operations which could be eliminated with the
proposed ``provenance_url.json`` file.

Recording provenance information for installed distribution packages,
both those obtained from direct URLs and by name/version from an index,
can simplify auditing Python environments in general, beyond just
the specific use case for containerized applications mentioned earlier.
A community project `pip-audit
<https://github.com/pypa/pip-audit>`__ raised their possible interest in
`pypa/pip-audit#170`_.

Specification
=============

The keywords âMUSTâ, âMUST NOTâ, âREQUIREDâ, âSHOULDâ,
âSHOULD NOTâ, âRECOMMENDEDâ, âMAYâ, and âOPTIONALâ
in this document are to be interpreted as described in :rfc:`2119`.

The ``provenance_url.json`` file SHOULD be created in the ``.dist-info``
directory by installers when installing a :term:`Distribution Package`
specified by name (and optionally by :term:`Version Specifier`).

This file MUST NOT be created when installing a distribution package from a requirement
specifying a direct URL reference (including a VCS URL).

Only one of the files ``provenance_url.json`` and ``direct_url.json`` (from
:ref:`Recording the Direct URL Origin of installed distributions
<packaging:direct-url>` specification and the corresponding specification of
the :ref:`Direct URL Data Structure <packaging:direct-url-data-structure>`),
may be present in a given ``.dist-info`` directory; installers MUST NOT add
both.

The ``provenance_url.json`` JSON file MUST be a dictionary, compliant with
:rfc:`8259` and UTF-8 encoded.

If present, it MUST contain exactly two keys. The first MUST be ``url``, with
type ``string``. The second key MUST be ``archive_info`` with a value defined
below.

The value of the ``url`` key MUST be the URL from which the distribution
package was downloaded. If a wheel is built from a source distribution, the
``url`` value MUST be the URL from which the source distribution was
downloaded. If a wheel is downloaded and installed directly, the ``url`` field
MUST be the URL from which the wheel was downloaded. As in the :ref:`Direct URL
Data Structure <packaging:direct-url-data-structure>` specification, the ``url``
value MUST be stripped of any sensitive authentication information for security
reasons.

The user:password section of the URL MAY however be composed of environment
variables, matching the following regular expression:

.. code-block:: text

    \$\{[A-Za-z0-9-_]+\}(:\$\{[A-Za-z0-9-_]+\})?

Additionally, the user:password section of the URL MAY be a well-known,
non-security sensitive string. A typical example is ``git`` in the case of an
URL such as ``ssh://git@gitlab.com``.

The value of ``archive_info`` MUST be a dictionary with a single key
``hashes``. The value of ``hashes`` is a dictionary mapping hash function
names to a hex-encoded digest of the file referenced by the ``url`` value. At
least one hash MUST be recorded. Multi


id: pep-0658.rst
content:
PEP: 658
Title: Serve Distribution Metadata in the Simple Repository API
Author: Tzu-ping Chung <uranusjr@gmail.com>
Sponsor: Brett Cannon <brett@python.org>
PEP-Delegate: Donald Stufft <donald@stufft.io>
Discussions-To: https://discuss.python.org/t/8651
Status: Accepted
Type: Standards Track
Topic: Packaging
Created: 10-May-2021
Post-History: 10-May-2021
Resolution: https://discuss.python.org/t/8651/48


Abstract
========

This PEP proposes adding an anchor tag to expose the ``METADATA`` file
from distributions in the :pep:`503` "simple" repository API. A
``data-dist-info-metadata`` attribute is introduced to indicate that
the file from a given distribution can be independently fetched.


Motivation
==========

Package management workflows made popular by recent tooling increase
the need to inspect distribution metadata without intending to install
the distribution, and download multiple distributions of a project to
choose from based on their metadata. This means they end up discarding
much downloaded data, which is inefficient and results in a bad user
experience.


Rationale
=========

Tools have been exploring methods to reduce the download size by
partially downloading wheels with HTTP range requests. This, however,
adds additional run-time requirements to the repository server. It
also still adds additional overhead, since a separate request is
needed to fetch the wheel's file listing to find the correct offset to
fetch the metadata file. It is therefore desired to make the server
extract the metadata file in advance, and serve it as an independent
file to avoid the need to perform additional requests and ZIP
inspection.

The metadata file defined by the Core Metadata Specification
[core-metadata]_ will be served directly by repositories since it
contains the necessary information for common use cases. The metadata
must only be served for standards-compliant distributions such as
wheels [wheel]_ and sdists [sdist]_, and must be identical to the
distribution's canonical metadata file, such as a wheel's ``METADATA``
file in the ``.dist-info`` directory [dist-info]_.

An HTML attribute on the distribution file's anchor link is needed to
indicate whether a client is able to choose the separately served
metadata file. The attribute is also used to provide the metadata
content's hash for client-side verification. The attribute's absence
indicates that a separate metadata entry is not available for the
distribution, either because of the distribution's content, or lack of
repository support.


Specification
=============

In a simple repository's project page, each anchor tag pointing to a
distribution **MAY** have a ``data-dist-info-metadata`` attribute. The
presence of the attribute indicates the distribution represented by
the anchor tag **MUST** contain a Core Metadata file that will not be
modified when the distribution is processed and/or installed.

If a ``data-dist-info-metadata`` attribute is present, the repository
**MUST** serve the distribution's Core Metadata file alongside the
distribution with a ``.metadata`` appended to the distribution's file
name. For example, the Core Metadata of a distribution served at
``/files/distribution-1.0-py3.none.any.whl`` would be located at
``/files/distribution-1.0-py3.none.any.whl.metadata``. This is similar
to how :pep:`503` specifies the GPG signature file's location.

The repository **SHOULD** provide the hash of the Core Metadata file
as the ``data-dist-info-metadata`` attribute's value using the syntax
``<hashname>=<hashvalue>``, where ``<hashname>`` is the lower cased
name of the hash function used, and ``<hashvalue>`` is the hex encoded
digest. The repository **MAY** use ``true`` as the attribute's value
if a hash is unavailable.


Backwards Compatibility
=======================

If an anchor tag lacks the ``data-dist-info-metadata`` attribute,
tools are expected to revert to their current behaviour of downloading
the distribution to inspect the metadata.

Older tools not supporting the new ``data-dist-info-metadata``
attribute are expected to ignore the attribute and maintain their
current behaviour of downloading the distribution to inspect the
metadata. This is similar to how prior ``data-`` attribute additions
expect existing tools to operate.


Rejected Ideas
==============

Put metadata content on the project page
----------------------------------------

Since tools generally only need dependency information from a
distribution in addition to what's already available on the project
page, it was proposed that repositories may directly include the
information on the project page, like the ``data-requires-python``
attribute specified in :pep:`503`.

This approach was abandoned since a distribution may contain
arbitrarily long lists of dependencies (including required and
optional), and it is unclear whether including the information for
every distribution in a project would result in net savings since the
information for most distributions generally ends up unneeded. By
serving the metadata separately, performance can be better estimated
since data usage will be more proportional to the number of
distributions inspected.


Expose more files in the distribution
-------------------------------------

It was proposed to provide the entire ``.dist-info`` directory as a
separate part, instead of only the metadata file. However, serving
multiple files in one entity through HTTP requires re-archiving them
separately after they are extracted from the original distribution
by the repository server, and there are no current use cases for files
other than ``METADATA`` when the distribution itself is not going to
be installed.

It should also be noted that the approach taken here does not
preclude other files from being introduced in the future, whether we
want to serve them together or individually.


Explicitly specify the metadata file's URL on the project page
--------------------------------------------------------------

An early version of this draft proposed putting the metadata file's
URL in the ``data-dist-info-metadata`` attribute. But people feel it
is better for discoverability to require the repository to serve the
metadata file at a determined location instead. The current approach
also has an additional benefit of making the project page smaller.


References
==========

.. [core-metadata] https://packaging.python.org/specifications/core-metadata/

.. [dist-info] https://packaging.python.org/specifications/recording-installed-packages/

.. [wheel] https://packaging.python.org/specifications/binary-distribution-format/

.. [sdist] https://packaging.python.org/specifications/source-distribution-format/


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0670.rst
content:
PEP: 670
Title: Convert macros to functions in the Python C API
Author: Erlend Egeberg Aasland <erlend@python.org>,
        Victor Stinner <vstinner@python.org>
Status: Final
Type: Standards Track
Created: 19-Oct-2021
Python-Version: 3.11
Post-History: `20-Oct-2021 <https://mail.python.org/archives/list/python-dev@python.org/thread/2GN646CGWGTO6ZHHU7JTA5XWDF4ULM77/>`__,
              `08-Feb-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/IJ3IBVY3JDPROKX55YNDT6XZTVTTPGOP/>`__,
              `22-Feb-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/VM6I3UHVMME6QRSUOYLK6N2OZHP454W6/>`__
Resolution: https://mail.python.org/archives/list/python-dev@python.org/thread/QQFCJ7LR36RUZSC3WI6WZZMQVQ3ZI4MS/


Abstract
========

Macros in the C API will be converted to static inline functions or
regular functions. This will help avoid macro pitfalls in C/C++, and
make the functions usable from other programming languages.

To avoid compiler warnings, function arguments of pointer types
will be cast to appropriate types using additional macros.
The cast will not be done in the limited C API version 3.11:
users who opt in to the new limited API may need to add casts to
the exact expected type.

To avoid introducing incompatible changes, macros which can be used as
l-value in an assignment will not be converted.


Rationale
=========

The use of macros may have unintended adverse effects that are hard to
avoid, even for experienced C developers. Some issues have been known
for years, while others have been discovered recently in Python.
Working around macro pitfalls makes the macro code harder to read and
to maintain.

Converting macros to functions has multiple advantages:

* Functions don't suffer from macro pitfalls, for example the following
  ones described in `GCC documentation
  <https://gcc.gnu.org/onlinedocs/cpp/Macro-Pitfalls.html>`_:

  - Misnesting
  - Operator precedence problems
  - Swallowing the semicolon
  - Duplication of side effects
  - Self-referential macros
  - Argument prescan
  - Newlines in arguments

  Functions don't need the following workarounds for macro
  pitfalls, making them usually easier to read and to maintain than
  similar macro code:

  - Adding parentheses around arguments.
  - Using line continuation characters if the function is written on
    multiple lines.
  - Adding commas to execute multiple expressions.
  - Using ``do { ... } while (0)`` to write multiple statements.

* Argument types and the return type of functions are well defined.
* Debuggers and profilers can retrieve the name of inlined functions.
* Debuggers can put breakpoints on inlined functions.
* Variables have a well-defined scope.

Converting macros and static inline functions to regular functions makes
these regular functions accessible to projects which use Python but
cannot use macros and static inline functions.


Specification
=============

Convert macros to static inline functions
-----------------------------------------

Most macros will be converted to static inline functions.

The following macros will not be converted:

* Object-like macros (i.e. those which don't need parentheses and
  arguments). For example:

  * Empty macros. Example: ``#define Py_HAVE_CONDVAR``.
  * Macros only defining a value, even if a constant with a well defined
    type would be better. Example: ``#define METH_VARARGS 0x0001``.

* Compatibility layer for different C compilers, C language extensions,
  or recent C features.
  Example: ``Py_GCC_ATTRIBUTE()``, ``Py_ALWAYS_INLINE``, ``Py_MEMCPY()``.
* Macros used for definitions rather than behavior.
  Example: ``PyAPI_FUNC``, ``Py_DEPRECATED``, ``Py_PYTHON_H``.
* Macros that need C preprocessor features, like stringification and
  concatenation. Example: ``Py_STRINGIFY()``.
* Macros which cannot be converted to functions. Examples:
  ``Py_BEGIN_ALLOW_THREADS`` (contains an unpaired ``}``), ``Py_VISIT``
  (relies on specific variable names), Py_RETURN_RICHCOMPARE (returns
  from the calling function).
* Macros which can be used as an l-value in assignments. This would be
  an incompatible change and is out of the scope of this PEP.
  Example: ``PyBytes_AS_STRING()``.
* Macros which have different return types depending on the code path
  or arguments.


Convert static inline functions to regular functions
----------------------------------------------------

Static inline functions in the public C API may be converted to regular
functions, but only if there is no measurable performance impact of
changing the function.
The performance impact should be measured with benchmarks.


Cast pointer arguments
----------------------

Currently, most macros accepting pointers cast pointer arguments to
their expected types. For example, in Python 3.6, the ``Py_TYPE()``
macro casts its argument to ``PyObject*``:

.. code-block:: c

    #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type)

The ``Py_TYPE()`` macro accepts the ``PyObject*`` type, but also any
pointer types, such as ``PyLongObject*`` and ``PyDictObject*``.

Functions are strongly typed, and can only accept one type of argument.

To avoid compiler errors and warnings in existing code, when a macro is
converted to a function and the macro casts at least one of its arguments
a new macro will be added to keep the cast. The new macro
and the function will have the same name.

Example with the ``Py_TYPE()``
macro converted to a static inline function:

.. code-block:: c

    static inline PyTypeObject* Py_TYPE(PyObject *ob) {
        return ob->ob_type;
    }
    #define Py_TYPE(ob) Py_TYPE((PyObject*)(ob))

The cast is kept for all pointer types, not only ``PyObject*``.
This includes casts to ``void*``: removing a cast to ``void*`` would emit
a new warning if the function is called with a ``const void*`` variable.
For example, the ``PyUnicode_WRITE()`` macro casts its *data* argument to
``void*``, and so it currently accepts ``const void*`` type, even though
it writes into *data*.  This PEP will not change this.


Avoid the cast in the limited C API version 3.11
''''''''''''''''''''''''''''''''''''''''''''''''

The casts will be excluded from the limited C API version 3.11 and newer.
When an API user opts into the new limited API, they must pass the expected
type or perform the cast.

As an example, ``Py_TYPE()`` will be defined like this:

.. code-block:: c

    static inline PyTypeObject* Py_TYPE(PyObject *ob) {
        return ob->ob_type;
    }
    #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000
    #  define Py_TYPE(ob) Py_TYPE((PyObject*)(ob))
    #endif


Return type is not changed
--------------------------

When a macro is converted to a function, its return type must not change
to prevent emitting new compiler warnings.

For example, Python 3.7 changed the return type of ``PyUnicode_AsUTF8()``
from ``char*`` to ``const char*`` (`commit
<https://github.com/python/cpython/commit/2a404b63d48d73bbaa007d89efb7a01048475acd>`__).
The change emitted new compiler warnings when building C extensions
expecting ``char*``. This PEP doesn't change the return type to prevent
this issue.


Backwards Compatibility
=======================

The PEP is designed to avoid C API incompatible changes.

Only C extensions explicitly targeting the limited C API version 3.11
must now pass the expected types to functions: pointer arguments are no
longer cast to the expected types.

Function arguments of pointer types are still cast and return types are
not changed to prevent emitting new compiler warnings.

Macros which can be used as l-value in an assignment are not modified by
this PEP to avoid incompatible changes.


Examples of Macro Pitfalls
==========================

Duplication of side effects
---------------------------

Macros:

.. code-block:: c

    #define PySet_Check(ob) \
        (Py_IS_TYPE(ob, &PySet_Type) \
         || PyType_IsSubtype(Py_TYPE(ob), &PySet_Type))

    #define Py_IS_NAN(X) ((X) != (X))

If the *op* or 


id: pep-0102.rst
content:
PEP: 102
Title: Doing Python Micro Releases
Author: Anthony Baxter <anthony@interlink.com.au>,
        Barry Warsaw <barry@python.org>,
        Guido van Rossum <guido@python.org>
Status: Superseded
Type: Informational
Created: 09-Jan-2002
Post-History:
Superseded-By: 101


Replacement Note
================

Although the size of the to-do list in this PEP is much less scary
than that in :pep:`101`, it turns out not to be enough justification
for the duplication of information, and with it, the danger of one
of the copies to become out of date.  Therefore, this PEP is not
maintained anymore, and micro releases are fully covered by :pep:`101`.


Abstract
========

Making a Python release is an arduous process that takes a
minimum of half a day's work even for an experienced releaser.
Until recently, most -- if not all -- of that burden was borne by
Guido himself.  But several recent releases have been performed by
other folks, so this PEP attempts to collect, in one place, all
the steps needed to make a Python bugfix release.

The major Python release process is covered in :pep:`101` - this PEP
is just :pep:`101`, trimmed down to only include the bits that are
relevant for micro releases, a.k.a. patch, or bug fix releases.

It is organized as a recipe and you can actually print this out and
check items off as you complete them.


How to Make A Release
=====================

Here are the steps taken to make a Python release.  Some steps are
more fuzzy than others because there's little that can be
automated (e.g. writing the NEWS entries).  Where a step is
usually performed by An Expert, the name of that expert is given.
Otherwise, assume the step is done by the Release Manager (RM),
the designated person performing the release.  Almost every place
the RM is mentioned below, this step can also be done by the BDFL
of course!

XXX: We should include a dependency graph to illustrate the steps
that can be taken in parallel, or those that depend on other
steps.

We use the following conventions in the examples below.  Where a
release number is given, it is of the form X.Y.MaA, e.g. 2.1.2c1
for Python 2.1.2 release candidate 1, where "a" == alpha, "b" ==
beta, "c" == release candidate.  Final releases are tagged with
"releaseXYZ" in CVS.  The micro releases are made from the
maintenance branch of the major release, e.g. Python 2.1.2 is made
from the release21-maint branch.

1. Send an email to python-dev@python.org indicating the release is
   about to start.

2. Put a freeze on check ins into the maintenance branch.  At this
   point, nobody except the RM should make any commits to the branch
   (or his duly assigned agents, i.e. Guido the BDFL, Fred Drake for
   documentation, or Thomas Heller for Windows).  If the RM screwed up
   and some desperate last minute change to the branch is
   necessary, it can mean extra work for Fred and Thomas.  So try to
   avoid this!

3. On the branch, change Include/patchlevel.h in two places, to
   reflect the new version number you've just created.  You'll want
   to change the PY_VERSION macro, and one or several of the
   version subpart macros just above PY_VERSION, as appropriate.

4. Change the "%define version" line of Misc/RPM/python-2.3.spec to the
   same string as ``PY_VERSION`` was changed to above.  E.g::

       %define version 2.3.1

   You also probably want to reset the %define release line
   to '1pydotorg' if it's not already that.

5. If you're changing the version number for Python (e.g. from
   Python 2.1.1 to Python 2.1.2), you also need to update the
   README file, which has a big banner at the top proclaiming its
   identity.  Don't do this if you're just releasing a new alpha or
   beta release, but /do/ do this if you're release a new micro,
   minor or major release.

6. The LICENSE file also needs to be changed, due to several
   references to the release number.  As for the README file, changing
   these are necessary for a new micro, minor or major release.

   The LICENSE file contains a table that describes the legal
   heritage of Python; you should add an entry for the X.Y.Z
   release you are now making.  You should update this table in the
   LICENSE file on the CVS trunk too.

7. When the year changes, copyright legends need to be updated in
   many places, including the README and LICENSE files.

8. For the Windows build, additional files have to be updated.

   PCbuild/BUILDno.txt contains the Windows build number, see the
   instructions in this file how to change it.  Saving the project
   file PCbuild/pythoncore.dsp results in a change to
   PCbuild/pythoncore.dsp as well.

   PCbuild/python20.wse sets up the Windows installer version
   resource (displayed when you right-click on the installer .exe
   and select Properties), and also contains the Python version
   number.

   (Before version 2.3.2, it was required to manually edit
   PC/python_nt.rc, this step is now automated by the build
   process.)

9. After starting the process, the most important thing to do next
   is to update the Misc/NEWS file.  Thomas will need this in order to
   do the Windows release and he likes to stay up late.  This step
   can be pretty tedious, so it's best to get to it immediately
   after making the branch, or even before you've made the branch.
   The sooner the better (but again, watch for new checkins up
   until the release is made!)

   Add high level items new to this release.  E.g. if we're
   releasing 2.2a3, there must be a section at the top of the file
   explaining "What's new in Python 2.2a3".  It will be followed by
   a section entitled "What's new in Python 2.2a2".

   Note that you /hope/ that as developers add new features to the
   trunk, they've updated the NEWS file accordingly.  You can't be
   positive, so double check.  If you're a Unix weenie, it helps to
   verify with Thomas about changes on Windows, and Jack Jansen
   about changes on the Mac.

   This command should help you (but substitute the correct -r tag!)::

       % cvs log -rr22a1: | python Tools/scripts/logmerge.py > /tmp/news.txt

   IOW, you're printing out all the cvs log entries from the
   previous release until now.  You can then troll through the
   news.txt file looking for interesting things to add to NEWS.

10. Check your NEWS changes into the maintenance branch.  It's easy
    to forget to update the release date in this file!

11. Check in any changes to IDLE's NEWS.txt.  Update the header in
    Lib/idlelib/NEWS.txt to reflect its release version and date.
    Update the IDLE version in Lib/idlelib/idlever.py to match.

11. Once the release process has started, the documentation needs to
    be built and posted on python.org according to the instructions
    in :pep:`101`.

    Note that Fred is responsible both for merging doc changes from
    the trunk to the branch AND for merging any branch changes from
    the branch to the trunk during the cleaning up phase.
    Basically, if it's in Doc/ Fred will take care of it.

12. Thomas compiles everything with MSVC 6.0 SP5, and moves the
    python23.chm file into the src/chm directory.  The installer
    executable is then generated with Wise Installation System.

    The installer includes the MSVC 6.0 runtime in the files
    MSVCRT.DLL and MSVCIRT.DLL.  It leads to disaster if these files
    are taken from the system directory of the machine where the
    installer is built, instead it must be absolutely made sure that
    these files come from the VCREDIST.EXE redistributable package
    contained in the MSVC SP5 CD.  VCREDIST.EXE must be unpacked
    with winzip, and the Wise Installation System prompts for the
    directory.

    After building the installer, it should be opened with winzip,
    and the MS dlls extracted again and check for the same version
    number as those unpacked from VCREDIST.EXE.

    Thomas uploads this file to the starship.  He then sends the RM
    a notice which includes the location and MD5 checksum of 


id: pep-0664.rst
content:
PEP: 664
Title: Python 3.11 Release Schedule
Author: Pablo Galindo Salgado <pablogsal@python.org>
Status: Active
Type: Informational
Topic: Release
Created: 12-Jul-2021
Python-Version: 3.11


Abstract
========

This document describes the development and release schedule for
Python 3.11.  The schedule primarily concerns itself with PEP-sized
items.

.. Small features may be added up to the first beta
   release.  Bugs may be fixed until the final release,
   which is planned for October 2022.

Release Manager and Crew
========================

- 3.11 Release Manager: Pablo Galindo Salgado
- Windows installers: Steve Dower
- Mac installers: Ned Deily
- Documentation: Julien Palard


Release Schedule
================

3.11.0 schedule
---------------

Note: the dates below use a 17-month development period that results
in a 12-month release cadence between feature versions, as defined by
:pep:`602`.

Actual:

- 3.11 development begins: Monday, 2021-05-03
- 3.11.0 alpha 1: Tuesday, 2021-10-05
- 3.11.0 alpha 2: Tuesday, 2021-11-02
- 3.11.0 alpha 3: Wednesday, 2021-12-08
- 3.11.0 alpha 4: Friday, 2022-01-14
- 3.11.0 alpha 5: Thursday, 2022-02-03
- 3.11.0 alpha 6: Monday, 2022-03-07
- 3.11.0 alpha 7: Tuesday, 2022-04-05
- 3.11.0 beta 1: Sunday, 2022-05-08
  (No new features beyond this point.)
- 3.11.0 beta 2: Tuesday, 2022-05-31
- 3.11.0 beta 3: Wednesday, 2022-06-01
- 3.11.0 beta 4: Monday, 2022-07-11
- 3.11.0 beta 5: Tuesday, 2022-07-26
- 3.11.0 candidate 1: Monday, 2022-08-08
- 3.11.0 candidate 2: Monday, 2022-09-12
- 3.11.0 final:  Monday, 2022-10-24

Bugfix releases
---------------

Actual:

- 3.11.1: Tuesday, 2022-12-06
- 3.11.2: Wednesday, 2023-02-08
- 3.11.3: Wednesday, 2023-04-05
- 3.11.4: Tuesday, 2023-06-06
- 3.11.5: Thursday, 2023-08-24
- 3.11.6: Monday, 2023-10-02
- 3.11.7: Monday, 2023-12-04
- 3.11.8: Tuesday, 2024-02-06
- 3.11.9: Tuesday, 2024-04-02 (final regular bugfix release with binary
  installers)

Source-only security fix releases
---------------------------------

Provided irregularly on an "as-needed" basis until October 2027.

- 3.11.10: Saturday, 2024-09-07
- 3.11.11: Tuesday, 2024-12-03
- 3.11.12: Tuesday, 2025-04-08

3.11 Lifespan
-------------

3.11 received bugfix updates approximately every 2 months for
approximately 18 months.  Some time after the release of 3.12.0 final,
the ninth and final 3.11 bugfix update was released.  After that,
it is expected that security updates (source only) will be released
until 5 years after the release of 3.11.0 final, so until approximately
October 2027.


Features for 3.11
=================

Some of the notable features of Python 3.11 include:

* :pep:`654`, Exception Groups and ``except*``.
* :pep:`657`, Enhanced error locations in tracebacks.
* :pep:`680`, Support for parsing TOML in the standard library
* Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured
  a 1.25x speedup on the standard benchmark suite. See `Faster CPython
  <https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython>`__ for
  details.

Typing features:

* :pep:`646`, Variadic generics.
* :pep:`655`, Marking individual TypedDict items as required or potentially-missing.
* :pep:`673`, Self type.
* :pep:`675`, Arbitrary literal string type.
* :pep:`681`, Dataclass transforms

Copyright
=========

This document has been placed in the public domain.


id: pep-0328.rst
content:
PEP: 328
Title: Imports: Multi-Line and Absolute/Relative
Author: Aahz <aahz@pythoncraft.com>
Status: Final
Type: Standards Track
Created: 21-Dec-2003
Python-Version: 2.4, 2.5, 2.6
Post-History: 08-Mar-2004


Abstract
========

The ``import`` statement has two problems:

* Long ``import`` statements can be difficult to write, requiring
  various contortions to fit Pythonic style guidelines.

* Imports can be ambiguous in the face of packages; within a package,
  it's not clear whether ``import foo`` refers to a module within the
  package or some module outside the package.  (More precisely, a local
  module or package can shadow another hanging directly off
  ``sys.path``.)

For the first problem, it is proposed that parentheses be permitted to
enclose multiple names, thus allowing Python's standard mechanisms for
multi-line values to apply.  For the second problem, it is proposed that
all ``import`` statements be absolute by default (searching ``sys.path``
only) with special syntax (leading dots) for accessing package-relative
imports.


Timeline
========

In Python 2.5, you must enable the new absolute import behavior with ::

    from __future__ import absolute_import

You may use relative imports freely.  In Python 2.6, any ``import``
statement that results in an intra-package import will raise
``DeprecationWarning`` (this also applies to ``from <> import`` that
fails to use the relative import syntax).


Rationale for Parentheses
=========================

Currently, if you want to import a lot of names from a module or
package, you have to choose one of several unpalatable options:

* Write a long line with backslash continuations::

    from Tkinter import Tk, Frame, Button, Entry, Canvas, Text, \
        LEFT, DISABLED, NORMAL, RIDGE, END

* Write multiple ``import`` statements::

      from Tkinter import Tk, Frame, Button, Entry, Canvas, Text
      from Tkinter import LEFT, DISABLED, NORMAL, RIDGE, END

(``import *`` is *not* an option ;-)

Instead, it should be possible to use Python's standard grouping
mechanism (parentheses) to write the ``import`` statement::

    from Tkinter import (Tk, Frame, Button, Entry, Canvas, Text,
        LEFT, DISABLED, NORMAL, RIDGE, END)

This part of the proposal had BDFL approval from the beginning.

Parentheses support was added to Python 2.4.


Rationale for Absolute Imports
==============================

In Python 2.4 and earlier, if you're reading a module located inside a
package, it is not clear whether ::

    import foo

refers to a top-level module or to another module inside the package.
As Python's library expands, more and more existing package internal
modules suddenly shadow standard library modules by accident.  It's a
particularly difficult problem inside packages because there's no way to
specify which module is meant.  To resolve the ambiguity, it is proposed
that ``foo`` will always be a module or package reachable from
``sys.path``.  This is called an absolute import.

The python-dev community chose absolute imports as the default because
they're the more common use case and because absolute imports can provide
all the functionality of relative (intra-package) imports  -- albeit at
the cost of difficulty when renaming package pieces higher up in the
hierarchy or when moving one package inside another.

Because this represents a change in semantics, absolute imports will
be optional in Python 2.5 and 2.6 through the use of ::

    from __future__ import absolute_import

This part of the proposal had BDFL approval from the beginning.


Rationale for Relative Imports
==============================

With the shift to absolute imports, the question arose whether
relative imports should be allowed at all.  Several use cases were
presented, the most important of which is being able to rearrange the
structure of large packages without having to edit sub-packages.  In
addition, a module inside a package can't easily import itself without
relative imports.

Guido approved of the idea of relative imports, but there has been a
lot of disagreement on the spelling (syntax).  There does seem to be
agreement that relative imports will require listing specific names to
import (that is, ``import foo`` as a bare term will always be an
absolute import).

Here are the contenders:

* One from Guido::

      from .foo import bar

  and ::

      from ...foo import bar

  These two forms have a couple of different suggested semantics.  One
  semantic is to make each dot represent one level.  There have been
  many complaints about the difficulty of counting dots.  Another
  option is to only allow one level of relative import.  That misses a
  lot of functionality, and people still complained about missing the
  dot in the one-dot form.  The final option is to define an algorithm
  for finding relative modules and packages; the objection here is
  "Explicit is better than implicit".  (The algorithm proposed is
  "search up from current package directory until the ultimate package
  parent gets hit".)

  Some people have suggested other punctuation as the separator, such
  as "-" or "^".

  Some people have suggested using "*"::

      from *.foo import bar

* The next set of options is conflated from several posters::

      from __pkg__.__pkg__ import

  and ::

      from .__parent__.__parent__ import

  Many people (Guido included) think these look ugly, but they *are*
  clear and explicit.  Overall, more people prefer ``__pkg__`` as the
  shorter option.

* One suggestion was to allow only sibling references.  In other words,
  you would not be able to use relative imports to refer to modules
  higher in the package tree.  You would then be able to do either ::

      from .spam import eggs

  or ::

      import .spam.eggs

* Some people favor allowing indexed parents::

      from -2.spam import eggs

  In this scenario, importing from the current directory would be a
  simple ::

      from .spam import eggs


* Finally, some people dislike the way you have to change ``import``
  to ``from ... import`` when you want to dig inside a package.  They
  suggest completely rewriting the ``import`` syntax::

      from MODULE import NAMES as RENAME searching HOW

  or ::

      import NAMES as RENAME from MODULE searching HOW
          [from NAMES] [in WHERE] import ...

  However, this most likely could not be implemented for Python 2.5
  (too big a change), and allowing relative imports is sufficiently
  critical that we need something now (given that the standard
  ``import`` will change to absolute import).  More than that, this
  proposed syntax has several open questions:

  - What is the precise proposed syntax?  (Which clauses are optional
    under which circumstances?)

  - How strongly does the ``searching`` clause bind?  In other words,
    do you write::

        import foo as bar searching XXX, spam as ham searching XXX

    or::

        import foo as bar, spam as ham searching XXX


Guido's Decision
================

Guido has Pronounced [1]_ that relative imports will use leading dots.
A single leading dot indicates a relative import, starting with the
current package.  Two or more leading dots give a relative import to the
parent(s) of the current package, one level per dot after the first.
Here's a sample package layout::

    package/
        __init__.py
        subpackage1/
            __init__.py
            moduleX.py
            moduleY.py
        subpackage2/
            __init__.py
            moduleZ.py
        moduleA.py

Assuming that the current file is either ``moduleX.py`` or
``subpackage1/__init__.py``, following are correct usages of the new
syntax::

    from .moduleY import spam
    from .moduleY import spam as ham
    from . import moduleY
    from ..subpackage1 import moduleY
    from ..subpackage2.moduleZ import eggs
    from ..moduleA import foo
    from ...package import bar
    from ...sys import path

Note that while that last case is legal, it is certainly disco


id: pep-8012.rst
content:
PEP: 8012
Title: The Community Governance Model
Author: Åukasz Langa <lukasz@python.org>
Status: Rejected
Type: Informational
Topic: Governance
Created: 03-Oct-2018


PEP Rejection
=============

:pep:`8012` was rejected `by a core developer vote
<https://discuss.python.org/t/python-governance-vote-december-2018-results/546/>`__
described in :pep:`8001` on Monday, December 17, 2018.

:pep:`8016` and the governance model it describes were chosen instead.

Abstract
========

This PEP proposes a new model of Python governance based on consensus
and voting by the Python community. This model relies on workgroups to carry
out the governance of the Python language. This governance model works without
the role of a centralized singular leader or a governing council.

It describes how, when, and why votes are conducted for decisions affecting
the Python language. It also describes the criteria for voting eligibility.

Should this model be adopted, it will be codified in :pep:`13`.

This model can be affectionately called "The Least Worst Governance
Model" by its property that while far from ideal, it's still the most
robust one compared to the others.  Since avoiding issues inherent to
the other models is a paramount feature of the Community Governance
Model, we start the discussion a bit unusually: by rejecting the
other models.

Rejected Models
===============

Let's have another BDFL
-----------------------

This seems like a very attractive idea because it's a model we know.
One Dictator to rule us all.

Challenge: There is no other Guido
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

There is no other single person with the unique skillset of Guido van
Rossum.  Such a person would need to have the technical, communication, and
organizational experience to lead the project successfully. Specifically, the
person would need to:

* set and articulate a cohesive long-term vision for the project;

* possess deep technical understanding of the runtime, the standard library,
  and the wider third-party library context;

* negotiate and resolve contentious issues in ways acceptable to all
  parties involved;

* have free time and possess the energy to sustain continuous involvement
  over periods of years.

Risk: Malevolent Dictator For Life
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What if we got somebody who is not as well suited for the position as
our first Dictator?  There are possible scenarios in which this could
lead to severe consequences.

The Dictator could gather insufficient trust due to missing technical
depth, a "close" election, inconsistent vision, poor ability to deal
with conflict or burnout, and so on.  Given a controversial decision
decided by the Dictator in a specific way, a Dictator with
insufficient trust may cause a split within the project.

The Dictator setup invites lobbying concentrated on a single person.
Unless that person is immune to leverage due to wealth, health, and
a stable life situation, this poses risk of malicious actors steering
the project from behind the curtain.

Finally, the Dictator coming from a particular part of the community
may put more weight on the needs and interests of that particular part
of the user base, alienating others.

Observation: We don't actually need a Dictator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The irony of the Dictator model is that it requires an election.  Better
yet, we need an election to even decide on which governance model to
use.

If we are already able solve two problems of this gravity via the
community process, why not keep using it for all subsequent decisions?

Risk: The warm and fuzzy feeling of a vague proposal
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

One last thing worth mentioning is that when a BDFL model is suggested,
it's easy to bypass the criticism above by not mentioning *who* the BDFL
should be.  That way the hopeful reader can project their best
expectations and wants onto the abstract BDFL, making the idea appear
more attractive.  This is a mistake.

Without naming the BDFL in the model proposal we are not talking about
a concrete model.  We can avoid asking and answering the hard questions.
We can imagine our best-case scenario, a candidate we'd like to serve
the role.

Omitting a name for the BDFL also puts the Community Model at an unfair disadvantage.
We already know the good, the bad, and the ugly of our core developer
group.  It's no platonic ideal, no perfect sphere with no friction.  In
fact, we expect there to be a fair amount of friction and imperfections.

Thus, to fairly assess the BDFL model proposal, dear reader, you
should imagine the worst possible person within our team as that
BDFL.  A concrete human being.  Imagine it's me.

**Conclusion** While this has been our history, without Guido, this model
does not serve the best interests of the language into the future.

Let's have a Council
--------------------

This group of people roughly shares the responsibilities of a Dictator. The
group can also be called a Triumvirate, a Quorum, Elders, Steering Committee,
and so on.

Risk: Dilution and confusion
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This model favors a small group, between three and five people.
That way it shares most of the criticism with the Dictator model,
amplified.  Having not one but, say, three people in position of power
dilutes responsibility while still providing high risk of lobbying,
insufficient trust, or alienating parts of the community.

Risk: Internal Conflict
~~~~~~~~~~~~~~~~~~~~~~~

Additionally, having multiple people share the responsibility of
governance creates ample opportunity for internal conflict,
inconsistent long-term vision of the project, and multiplies the
required continuous time involvement by its members (it's no Quorum
if they can't "reach quorum" due to other time commitments).

Just like with a frictionless spherical BDFL, reject ideas of
Councils without considering how would it work for you if that
Council consisted of three people you find inadequate for the role.
Imagine if I had two friends.

Most importantly, just like with a Dictator, we don't need a Council.
By the time we had one, we would have already had two successful
elections.  Why not keep voting?

**Conclusion** This model has similar risks like a Dictator, only worse.

Motivation
==========

Now that we rejected the basics of other governance models, let's talk why we
even need a governance model on top of a loosely defined group of committers.

**Stability and Reliability** We want to prevent single committers from
making wide-reaching changes that impact the future of the language or its
usability.  Coherent vision and backwards compatibility are important in any
programming language, but they are doubly important for Python which is very
dynamic (e.g. has very complex backwards compatibility implications).

**Diverse Uses of Python** Moreover, Python is used by a
diverse group of users, from school children through scientists to
corporations with multi-million line codebases.  We want to include
all our varied audiences.

**Vitality** We want to avoid stagnation.  Python is a mature project but it
needs to keep evolving to stay relevant, both the runtime and the programming
language.  To do that, people interested in improving a particular part
of the project should be able to do so without needless friction.
But for substantial changes, we want some discourse and reflection to ensure
the changes are wise.


Rationale
=========

**Inclusive** The Community Model is the most inclusive model.  No single person
or a small group of people is in a distinguished position of power over
others.  Contributors and any workgroups in this model are self-selecting.

**Pragmatic** This model ensures no user group is put at a disadvantage due to
the interests of a single person or a small group of people.

**Proven** This model works.  There is a number of large open-source projects
run this way (two of which, Rust and Django, are described in :pep:


id: pep-0314.rst
content:
PEP: 314
Title: Metadata for Python Software Packages 1.1
Author: A.M. Kuchling, Richard Jones
Discussions-To: distutils-sig@python.org
Status: Superseded
Type: Standards Track
Topic: Packaging
Created: 12-Apr-2003
Python-Version: 2.5
Post-History: 29-Apr-2003
Replaces: 241
Superseded-By: 345

.. canonical-pypa-spec:: :ref:`packaging:core-metadata`


Introduction
============

This PEP describes a mechanism for adding metadata to Python
packages.  It includes specifics of the field names, and their
semantics and usage.

This document specifies version 1.1 of the metadata format.
Version 1.0 is specified in :pep:`241`.


Including Metadata in Packages
==============================

The Distutils ``sdist`` command will extract the metadata fields
from the arguments and write them to a file in the generated
zipfile or tarball.  This file will be named PKG-INFO and will be
placed in the top directory of the source distribution (where the
README, INSTALL, and other files usually go).

Developers may not provide their own PKG-INFO file.  The ``sdist``
command will, if it detects an existing PKG-INFO file, terminate
with an appropriate error message.  This should prevent confusion
caused by the PKG-INFO and setup.py files being out of sync.

The PKG-INFO file format is a single set of :rfc:`822` headers
parseable by the rfc822.py module.  The field names listed in the
following section are used as the header names.


Fields
======

This section specifies the names and semantics of each of the
supported metadata fields.

Fields marked with "(Multiple use)" may be specified multiple
times in a single PKG-INFO file.  Other fields may only occur
once in a PKG-INFO file.  Fields marked with "(optional)" are
not required to appear in a valid PKG-INFO file; all other
fields must be present.

Metadata-Version
----------------

Version of the file format; currently "1.0" and "1.1" are the
only legal values here.

Example::

   Metadata-Version: 1.1

Name
----

The name of the package.

Example::

   Name: BeagleVote

Version
-------

A string containing the package's version number.  This
field should be parseable by one of the Version classes
(StrictVersion or LooseVersion) in the distutils.version
module.

Example::

   Version: 1.0a2

Platform (multiple use)
-----------------------

A comma-separated list of platform specifications, summarizing
the operating systems supported by the package which are not
listed in the "Operating System" Trove classifiers. See
"Classifier" below.

Example::

   Platform: ObscureUnix, RareDOS

Supported-Platform (multiple use)
---------------------------------

Binary distributions containing a PKG-INFO file will use the
Supported-Platform field in their metadata to specify the OS and
CPU for which the binary package was compiled.  The semantics of
the Supported-Platform field are not specified in this PEP.

Example::

   Supported-Platform: RedHat 7.2
   Supported-Platform: i386-win32-2791

Summary
-------

A one-line summary of what the package does.

Example::

   Summary: A module for collecting votes from beagles.

Description (optional)
----------------------

A longer description of the package that can run to several
paragraphs.  Software that deals with metadata should not assume
any maximum size for this field, though people shouldn't include
their instruction manual as the description.

The contents of this field can be written using reStructuredText
markup [1]_.  For programs that work with the metadata,
supporting markup is optional; programs can also display the
contents of the field as-is.  This means that authors should be
conservative in the markup they use.

Example::

   Description: This module collects votes from beagles
                in order to determine their electoral wishes.
                Do *not* try to use this module with basset hounds;
                it makes them grumpy.

Keywords (optional)
-------------------

A list of additional keywords to be used to assist searching
for the package in a larger catalog.

Example::

   Keywords: dog puppy voting election

Home-page (optional)
--------------------

A string containing the URL for the package's home page.

Example::

   Home-page: http://www.example.com/~cschultz/bvote/

Download-URL
------------

A string containing the URL from which this version of the package
can be downloaded.  (This means that the URL can't be something like
".../package-latest.tgz", but instead must be "../package-0.45.tgz".)

Author (optional)
-----------------

A string containing the author's name at a minimum; additional
contact information may be provided.

Example::

   Author: C. Schultz, Universal Features Syndicate,
           Los Angeles, CA <cschultz@peanuts.example.com>

Author-email
------------

A string containing the author's e-mail address.  It can contain
a name and e-mail address in the legal forms for a :rfc:`822`
'From:' header.  It's not optional because cataloging systems
can use the e-mail portion of this field as a unique key
representing the author.  A catalog might provide authors the
ability to store their GPG key, personal home page, and other
additional metadata *about the author*, and optionally the
ability to associate several e-mail addresses with the same
person.  Author-related metadata fields are not covered by this
PEP.

Example::

   Author-email: "C. Schultz" <cschultz@example.com>

License
-------

Text indicating the license covering the package where the license
is not a selection from the "License" Trove classifiers. See
"Classifier" below.

Example::

   License: This software may only be obtained by sending the
            author a postcard, and then the user promises not
            to redistribute it.

Classifier (multiple use)
-------------------------

Each entry is a string giving a single classification value
for the package.  Classifiers are described in :pep:`301`.

Examples::

   Classifier: Development Status :: 4 - Beta
   Classifier: Environment :: Console (Text Based)


Requires (multiple use)
-----------------------

Each entry contains a string describing some other module or
package required by this package.

The format of a requirement string is identical to that of a
module or package name usable with the 'import' statement,
optionally followed by a version declaration within parentheses.

A version declaration is a series of conditional operators and
version numbers, separated by commas.  Conditional operators
must be one of "<", ">", "<=", ">=", "==", and "!=".  Version
numbers must be in the format accepted by the
distutils.version.StrictVersion class: two or three
dot-separated numeric components, with an optional "pre-release"
tag on the end consisting of the letter 'a' or 'b' followed by a
number.  Example version numbers are "1.0", "2.3a2", "1.3.99",

Any number of conditional operators can be specified, e.g.
the string ">1.0, !=1.3.4, <2.0" is a legal version declaration.

All of the following are possible requirement strings: "rfc822",
"zlib (>=1.1.4)", "zope".

There's no canonical list of what strings should be used; the
Python community is left to choose its own standards.

Example::

   Requires: re
   Requires: sys
   Requires: zlib
   Requires: xml.parsers.expat (>1.0)
   Requires: psycopg

Provides (multiple use)
-----------------------

Each entry contains a string describing a package or module that
will be provided by this package once it is installed.  These
strings should match the ones used in Requirements fields.  A
version declaration may be supplied (without a comparison
operator); the package's version number will be implied if none
is specified.

Example::

   Provides: xml
   Provides: xml.utils
   Provides: xml.utils.iso8601
   Provides: xml.dom
   Provides: xmltools (1.3)

Obsoletes (multiple use)
------------------------

Each entry contains a string describing a package or module
that this package renders obsolete, meaning that the two packages
should not be installed at the


id: pep-0472.rst
content:
PEP: 472
Title: Support for indexing with keyword arguments
Author: Stefano Borini, Joseph Martinot-Lagarde
Discussions-To: python-ideas@python.org
Status: Rejected
Type: Standards Track
Created: 24-Jun-2014
Python-Version: 3.6
Post-History: 02-Jul-2014
Resolution: https://mail.python.org/pipermail/python-dev/2019-March/156693.html

Abstract
========

This PEP proposes an extension of the indexing operation to support keyword
arguments. Notations in the form ``a[K=3,R=2]`` would become legal syntax.
For future-proofing considerations, ``a[1:2, K=3, R=4]`` are considered and
may be allowed as well, depending on the choice for implementation. In addition
to a change in the parser, the index protocol (``__getitem__``, ``__setitem__``
and ``__delitem__``) will also potentially require adaptation.

Motivation
==========

The indexing syntax carries a strong semantic content, differentiating it from
a method call: it implies referring to a subset of data. We believe this
semantic association to be important, and wish to expand the strategies allowed
to refer to this data.

As a general observation, the number of indices needed by an indexing operation
depends on the dimensionality of the data: one-dimensional data (e.g. a list)
requires one index (e.g. ``a[3]``), two-dimensional data (e.g. a matrix) requires
two indices (e.g. ``a[2,3]``) and so on. Each index is a selector along one of the
axes of the dimensionality, and the position in the index tuple is the
metainformation needed to associate each index to the corresponding axis.

The current python syntax focuses exclusively on position to express the
association to the axes, and also contains syntactic sugar to refer to
non-punctiform selection (slices)

::

    >>> a[3]       # returns the fourth element of a
    >>> a[1:10:2]  # slice notation (extract a non-trivial data subset)
    >>> a[3,2]     # multiple indexes (for multidimensional arrays)

The additional notation proposed in this PEP would allow notations involving
keyword arguments in the indexing operation, e.g.

::

    >>> a[K=3, R=2]

which would allow to refer to axes by conventional names.

One must additionally consider the extended form that allows both positional
and keyword specification

::

    >>> a[3,R=3,K=4]

This PEP will explore different strategies to enable the use of these notations.

Use cases
=========

The following practical use cases present two broad categories of usage of a
keyworded specification: Indexing and contextual option. For indexing:

1. To provide a more communicative meaning to the index, preventing e.g. accidental
   inversion of indexes

   ::

     >>> gridValues[x=3, y=5, z=8]
     >>> rain[time=0:12, location=location]

2. In some domain, such as computational physics and chemistry, the use of a
   notation such as ``Basis[Z=5]`` is a Domain Specific Language notation to represent
   a level of accuracy

   ::

     >>> low_accuracy_energy = computeEnergy(molecule, BasisSet[Z=3])

   In this case, the index operation would return a basis set at the chosen level
   of accuracy (represented by the parameter Z). The reason behind an indexing is that
   the BasisSet object could be internally represented as a numeric table, where
   rows (the "coefficient" axis, hidden to the user in this example) are associated
   to individual elements (e.g. row 0:5 contains coefficients for element 1,
   row 5:8 coefficients for element 2) and each column is associated to a given
   degree of accuracy ("accuracy" or "Z" axis) so that first column is low
   accuracy, second column is medium accuracy and so on. With that indexing,
   the user would obtain another object representing the contents of the column
   of the internal table for accuracy level 3.

Additionally, the keyword specification can be used as an option contextual to
the indexing. Specifically:

1. A "default" option allows to specify a default return value when the index
   is not present

   ::

     >>> lst = [1, 2, 3]
     >>> value = lst[5, default=0]  # value is 0

2. For a sparse dataset, to specify an interpolation strategy
   to infer a missing point from e.g. its surrounding data.

   ::

     >>> value = array[1, 3, interpolate=spline_interpolator]

3. A unit could be specified with the same mechanism

   ::

     >>> value = array[1, 3, unit="degrees"]

How the notation is interpreted is up to the implementing class.

Current implementation
======================

Currently, the indexing operation is handled by methods ``__getitem__``,
``__setitem__`` and ``__delitem__``. These methods' signature accept one argument
for the index (with ``__setitem__`` accepting an additional argument for the set
value). In the following, we will analyze ``__getitem__(self, idx)`` exclusively,
with the same considerations implied for the remaining two methods.

When an indexing operation is performed, ``__getitem__(self, idx)`` is called.
Traditionally, the full content between square brackets is turned into a single
object passed to argument ``idx``:

- When a single element is passed, e.g. ``a[2]``, ``idx`` will be ``2``.
- When multiple elements are passed, they must be separated by commas: ``a[2, 3]``.
  In this case, ``idx`` will be a tuple ``(2, 3)``. With ``a[2, 3, "hello", {}]``
  ``idx`` will be ``(2, 3, "hello", {})``.
- A slicing notation e.g. ``a[2:10]`` will produce a slice object, or a tuple
  containing slice objects if multiple values were passed.

Except for its unique ability to handle slice notation, the indexing operation
has similarities to a plain method call: it acts like one when invoked with
only one element; If the number of elements is greater than one, the ``idx``
argument behaves like a ``*args``. However, as stated in the Motivation section,
an indexing operation has the strong semantic implication of extraction of a
subset out of a larger set, which is not automatically associated to a regular
method call unless appropriate naming is chosen. Moreover, its different visual
style is important for readability.

Specifications
==============

The implementation should try to preserve the current signature for
``__getitem__``, or modify it in a backward-compatible way. We will present
different alternatives, taking into account the possible cases that need
to be addressed

::

    C0. a[1]; a[1,2]         # Traditional indexing
    C1. a[Z=3]
    C2. a[Z=3, R=4]
    C3. a[1, Z=3]
    C4. a[1, Z=3, R=4]
    C5. a[1, 2, Z=3]
    C6. a[1, 2, Z=3, R=4]
    C7. a[1, Z=3, 2, R=4]    # Interposed ordering

Strategy "Strict dictionary"
----------------------------

This strategy acknowledges that ``__getitem__`` is special in accepting only
one object, and the nature of that object must be non-ambiguous in its
specification of the axes: it can be either by order, or by name. As a result
of this assumption, in presence of keyword arguments, the passed entity is a
dictionary and all labels must be specified.

::

    C0. a[1]; a[1,2]      -> idx = 1; idx = (1, 2)
    C1. a[Z=3]            -> idx = {"Z": 3}
    C2. a[Z=3, R=4]       -> idx = {"Z": 3, "R": 4}
    C3. a[1, Z=3]         -> raise SyntaxError
    C4. a[1, Z=3, R=4]    -> raise SyntaxError
    C5. a[1, 2, Z=3]      -> raise SyntaxError
    C6. a[1, 2, Z=3, R=4] -> raise SyntaxError
    C7. a[1, Z=3, 2, R=4] -> raise SyntaxError

Pros
''''

- Strong conceptual similarity between the tuple case and the dictionary case.
  In the first case, we are specifying a tuple, so we are naturally defining
  a plain set of values separated by commas. In the second, we are specifying a
  dictionary, so we are specifying a homogeneous set of key/value pairs, as
  in ``dict(Z=3, R=4)``;
- Simple and easy to parse on the ``__getitem__`` side: if it gets a tuple,
  determine the axes using positioning. If it gets a dictionary, use
  the keywords.
- C interface does not need changes.

Neutral
'''''''

- Degeneracy of ``a[{"Z": 3, "R": 4}]`` with ``a[Z=3, R=4]`` means the notation
  is syntactic sugar


id: pep-0466.rst
content:
PEP: 466
Title: Network Security Enhancements for Python 2.7.x
Author: Alyssa Coghlan <ncoghlan@gmail.com>,
Status: Final
Type: Standards Track
Created: 23-Mar-2014
Python-Version: 2.7.9
Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014, 16-Apr-2014
Resolution: https://mail.python.org/pipermail/python-dev/2014-April/134163.html

Abstract
========

Most CPython tracker issues are classified as errors in behaviour or
proposed enhancements. Most patches to fix behavioural errors are
applied to all active maintenance branches.  Enhancement patches are
restricted to the default branch that becomes the next Python version.

This cadence works reasonably well during Python's normal 18-24 month
feature release cycle, which is still applicable to the Python 3 series.
However, the age of the standard library in Python 2 has now reached a point
where it is sufficiently far behind the state of the art in network security
protocols for it to be causing real problems in use cases where upgrading to
Python 3 in the near term may not be feasible.

In recognition of the additional practical considerations that have arisen
during the 4+ year maintenance cycle for Python 2.7, this PEP allows a
critical set of network security related features to be backported from
Python 3.4 to upcoming Python 2.7.x maintenance releases.

While this PEP does not make any changes to the core development team's
handling of security-fix-only branches that are no longer in active
maintenance, it *does* recommend that commercial redistributors providing
extended support periods for the Python standard library either backport
these features to their supported versions, or else explicitly disclaim
support for the use of older versions in roles that involve connecting
directly to the public internet.


New security related features in Python 2.7 maintenance releases
================================================================

Under this proposal, the following features will be backported from Python
3.4 to upcoming Python 2.7.x maintenance releases:

* in the ``os`` module:

  * persistent file descriptor for ``os.urandom()``.

* in the ``hmac`` module:

  * constant time comparison function (``hmac.compare_digest()``).

* in the ``hashlib`` module:

  * password hashing function (``hashlib.pbkdf2_hmac()``).
  * details of hash algorithm availability (``hashlib.algorithms_guaranteed``
    and ``hashlib.algorithms_available``).

* in the ``ssl`` module:

  * this module is almost entirely synchronised with its Python 3
    counterpart, bringing TLSv1.x settings, SSLContext manipulation, Server
    Name Indication, access to platform certificate stores, standard
    library support for peer hostname validation and more to the Python 2
    series.
  * the only ``ssl`` module features *not* backported under this policy are
    the ``ssl.RAND_*`` functions that provide access to OpenSSL's random
    number generation capabilities - use ``os.urandom()`` instead.

As a general change in maintenance policy, permission is also granted to
upgrade to newer feature releases of OpenSSL when preparing the binary
installers for new maintenance releases of Python 2.7.

This PEP does NOT propose a general exception for backporting new features
to Python 2.7 - every new feature proposed for backporting will still need
to be justified independently. In particular, it will need to be explained
why relying on an independently updated backport on the Python Package Index
instead is not an acceptable solution.


Implementation status
=====================

This PEP originally proposed adding all listed features to the Python 2.7.7
maintenance release. That approach proved to be too ambitious given the
limited time frame between the original creation and acceptance of the PEP
and the release of Python 2.7.7rc1. Instead, the progress of each individual
accepted feature backport is being tracked as an independent enhancement
targeting Python 2.7.

Implemented for Python 2.7.7:

* `Issue #21306`_: backport ``hmac.compare_digest``
* `Issue #21462`_: upgrade OpenSSL in the Python 2.7 Windows installers

.. _Issue #21306: http://bugs.python.org/issue21306
.. _Issue #21462: http://bugs.python.org/issue21462

Implemented for Python 2.7.8:

* `Issue #21304`_: backport ``hashlib.pbkdf2``

.. _Issue #21304: http://bugs.python.org/issue21304


Implemented for Python 2.7.9 (in development):

* `Issue #21308`_: backport specified ``ssl`` module features
* `Issue #21307`_: backport remaining specified ``hashlib`` module features
* `Issue #21305`_: backport ``os.urandom`` shared file descriptor change

.. _Issue #21308: http://bugs.python.org/issue21308
.. _Issue #21307: http://bugs.python.org/issue21307
.. _Issue #21305: http://bugs.python.org/issue21305

Backwards compatibility considerations
======================================

As in the Python 3 series, the backported ``ssl.create_default_context()``
API is granted a backwards compatibility exemption that permits the
protocol, options, cipher and other settings of the created SSL context to
be updated in maintenance releases to use higher default security settings.
This allows them to appropriately balance compatibility and security at the
time of the maintenance release, rather than at the time of the original
feature release.

This PEP does *not* grant any other exemptions to the usual backwards
compatibility policy for maintenance releases. Instead, by explicitly
encouraging the use of feature based checks, it is designed to make it easier
to write more secure cross-version compatible Python software, while still
limiting the risk of breaking currently working software when upgrading to
a new Python 2.7 maintenance release.

In all cases where this proposal allows new features to be backported to
the Python 2.7 release series, it is possible to write cross-version
compatible code that operates by "feature detection" (for example, checking
for particular attributes in a module), without needing to explicitly check
the Python version.

It is then up to library and framework code to provide an appropriate warning
and fallback behaviour if a desired feature is found to be missing. While
some especially security sensitive software MAY fail outright if a desired
security feature is unavailable, most software SHOULD instead emit a warning
and continue operating using a slightly degraded security configuration.

The backported APIs allow library and application code to perform the
following actions after detecting the presence of a relevant
network security related feature:

* explicitly opt in to more secure settings (to allow the use of enhanced
  security features in older maintenance releases of Python with less
  secure default behaviour)
* explicitly opt in to less secure settings (to allow the use of newer Python
  feature releases in lower security environments)
* determine the default setting for the feature (this MAY require explicit
  Python version checks to determine the Python feature release, but DOES
  NOT require checking for a specific maintenance release)

Security related changes to other modules (such as higher level networking
libraries and data format processing libraries) will continue to be made
available as backports and new modules on the Python Package Index, as
independent distribution remains the preferred approach to handling
software that must continue to evolve to handle changing development
requirements independently of the Python 2 standard library. Refer to
the `Motivation and Rationale`_ section for a review of the characteristics
that make the secure networking infrastructure worthy of special
consideration.


OpenSSL compatibility
---------------------

Under this proposal, OpenSSL may be upgraded to more recent feature releases
in Python 2.7 maintenance releases. On Linux and most other POSIX systems,
the specific version of OpenSSL used already varies, as CPython dynamically
links to the system provided OpenSSL library


id: pep-0499.rst
content:
PEP: 499
Title: ``python -m foo`` should also bind ``'foo'`` in ``sys.modules``
Author: Cameron Simpson <cs@cskk.id.au>, Chris Angelico <rosuav@gmail.com>, Joseph Jevnik <joejev@gmail.com>
BDFL-Delegate: Alyssa Coghlan
Status: Deferred
Type: Standards Track
Created: 07-Aug-2015
Python-Version: 3.10


PEP Deferral
============

The implementation of this PEP isn't currently expected to be ready for the
Python 3.9 feature freeze in April 2020, so it has been deferred 12 months to
Python 3.10.


Abstract
========

When a module is used as a main program on the Python command line,
such as by:

    python -m module.name ...

it is easy to accidentally end up with two independent instances
of the module if that module is again imported within the program.
This PEP proposes a way to fix this problem.

When a module is invoked via Python's -m option the module is bound
to ``sys.modules['__main__']`` and its ``.__name__`` attribute is set to
``'__main__'``.
This enables the standard "main program" boilerplate code at the
bottom of many modules, such as::

    if __name__ == '__main__':
        sys.exit(main(sys.argv))

However, when the above command line invocation is used it is a
natural inference to presume that the module is actually imported
under its official name ``module.name``,
and therefore that if the program again imports that name
then it will obtain the same module instance.

That actuality is that the module was imported only as ``'__main__'``.
Another import will obtain a distinct module instance, which can
lead to confusing bugs,
all stemming from having two instances of module global objects:
one in each module.

Examples include:

module level data structures
  Some modules provide features such as caches or registries
  as module level global variables,
  typically private.
  A second instance of a module creates a second data structure.
  If that structure is a cache
  such as in the ``re`` module
  then two caches exist leading to wasteful memory use.
  If that structure is a shared registry
  such as a mapping of values to handlers
  then it is possible to register a handler to one registry
  and to try to use it via the other registry, where it is unknown.

sentinels
  The standard test for a sentinel value provided by a module
  is the identity comparison using ``is``,
  as this avoids unreliable "looks like" comparisons
  such as equality which can both mismatch two values as "equal"
  (for example being zeroish)
  or raise a ``TypeError`` when the objects are incompatible.
  When there are two instances of a module
  there are two sentinel instances
  and only one will be recognised via ``is``.

classes
  With two modules
  there are duplicate class definitions of any classes provided.
  All operations which depend on recognising these classes
  and subclasses of these are prone to failure
  depending where the reference class
  (from one of the modules) is obtained
  and where the comparison class or instance is obtained.
  This impacts ``isinstance``, ``issubclass``
  and also ``try``/``except`` constructs.

Proposal
========

It is suggested that to fix this situation all that is needed is a
simple change to the way the ``-m`` option is implemented: in addition
to binding the module object to ``sys.modules['__main__']``, it is also
bound to ``sys.modules['module.name']``.

Alyssa (Nick) Coghlan has suggested that this is as simple as modifying the
``runpy`` module's ``_run_module_as_main`` function as follows::

    main_globals = sys.modules["__main__"].__dict__

to instead be::

    main_module = sys.modules["__main__"]
    sys.modules[mod_spec.name] = main_module
    main_globals = main_module.__dict__

Joseph Jevnik has pointed out that modules which are packages already
do something very similar to this proposal:
the __init__.py file is bound to the module's canonical name
and the __main__.py file is bound to "__main__".
As such, the double import issue does not occur.
Therefore, this PEP proposes to affect only simple non-package modules.


Considerations and Prerequisites
================================

Pickling Modules
----------------

Alyssa has mentioned `issue 19702`_ which proposes (quoted from the issue):

- runpy will ensure that when __main__ is executed via the import
  system, it will also be aliased in sys.modules as __spec__.name
- if __main__.__spec__ is set, pickle will use __spec__.name rather
  than __name__ to pickle classes, functions and methods defined in
  __main__
- multiprocessing is updated appropriately to skip creating __mp_main__
  in child processes when __main__.__spec__ is set in the parent
  process

The first point above covers this PEP's specific proposal.


A Normal Module's ``__name__`` Is No Longer Canonical
-----------------------------------------------------

Chris Angelico points out that it becomes possible to import a
module whose ``__name__`` is not what you gave to "import", since
"__main__" is now present at "module.name", so a subsequent
``import module.name`` finds it already present.
Therefore, ``__name__`` is no longer the canonical name for some normal imports.

Some counter arguments follow:

- As of :pep:`451` a module's canonical name is stored at ``__spec__.name``.
- Very little code should actually care about ``__name__`` being the canonical name
  and any that does should arguably be updated to consult ``__spec__.name``
  with fallback to ``__name__`` for older Pythons, should that be relevant.
  This is true even if this PEP is not approved.
- Should this PEP be approved,
  it becomes possible to introspect a module by its canonical name
  and ask "was this the main program?" by inferring from ``__name__``.
  This was not previously possible.

The glaring counter example is the standard "am I the main program?" boilerplate,
where ``__name__`` is expected to be "__main__".
This PEP explicitly preserves that semantic.


Reference Implementation
========================

`BPO 36375 <https://bugs.python.org/issue36375>`_ is the issue tracker entry
for the PEP's reference implementation, with the current draft PR being
available `on GitHub <https://github.com/python/cpython/pull/12490>`_.


Open Questions
==============

This proposal does raise some backwards compatibility concerns, and these will
need to be well understood, and either a deprecation process designed, or clear
porting guidelines provided.

Pickle compatibility
--------------------

If no changes are made to the pickle module, then pickles that were previously
being written with the correct module name (due to a dual import) may start
being written with ``__main__`` as their module name instead, and hence fail
to be loaded correctly by other projects.

Scenarios to be checked:

* ``python script.py`` writing, ``python -m script`` reading
* ``python -m script`` writing, ``python script.py`` reading
* ``python -m script`` writing, ``python some_other_app.py`` reading
* ``old_python -m script`` writing, ``new_python -m script`` reading
* ``new_python -m script`` writing, ``old_python -m script`` reading


Projects that special-case ``__main__``
---------------------------------------

In order to get the regression test suite to pass, the current reference
implementation had to patch ``pdb`` to avoid destroying its own global
namespace.

This suggests there may be a broader compatibility issue where some scripts are
relying on direct execution and import giving different namespaces (just as
package execution keeps the two separate by executing the ``__main__``
submodule in the ``__main__`` namespace, while the package name references
the ``__init__`` file as usual.


Background
==========

`I tripped over this issue`_ while debugging a main program via a
module which tried to monkey patch a named module, that being the
main program module.  Naturally, the monkey patching was ineffective
as it imported the main module by name and thus patched the second
module instance, not the running module instance.


id: pep-0498.rst
content:
PEP: 498
Title: Literal String Interpolation
Author: Eric V. Smith <eric@trueblade.com>
Status: Final
Type: Standards Track
Created: 01-Aug-2015
Python-Version: 3.6
Post-History: 07-Aug-2015, 30-Aug-2015, 04-Sep-2015, 19-Sep-2015, 06-Nov-2016
Resolution: https://mail.python.org/pipermail/python-dev/2015-September/141526.html

Abstract
========

Python supports multiple ways to format text strings. These include
%-formatting [#]_, ``str.format()`` [#]_, and ``string.Template``
[#]_. Each of these methods have their advantages, but in addition
have disadvantages that make them cumbersome to use in practice. This
PEP proposed to add a new string formatting mechanism: Literal String
Interpolation. In this PEP, such strings will be referred to as
"f-strings", taken from the leading character used to denote such
strings, and standing for "formatted strings".

This PEP does not propose to remove or deprecate any of the existing
string formatting mechanisms.

F-strings provide a way to embed expressions inside string literals,
using a minimal syntax. It should be noted that an f-string is really
an expression evaluated at run time, not a constant value. In Python
source code, an f-string is a literal string, prefixed with 'f', which
contains expressions inside braces. The expressions are replaced with
their values. Some examples are::

  >>> import datetime
  >>> name = 'Fred'
  >>> age = 50
  >>> anniversary = datetime.date(1991, 10, 12)
  >>> f'My name is {name}, my age next year is {age+1}, my anniversary is {anniversary:%A, %B %d, %Y}.'
  'My name is Fred, my age next year is 51, my anniversary is Saturday, October 12, 1991.'
  >>> f'He said his name is {name!r}.'
  "He said his name is 'Fred'."

A similar feature was proposed in :pep:`215`. :pep:`215` proposed to support
a subset of Python expressions, and did not support the type-specific
string formatting (the ``__format__()`` method) which was introduced
with :pep:`3101`.

Rationale
=========

This PEP is driven by the desire to have a simpler way to format
strings in Python. The existing ways of formatting are either error
prone, inflexible, or cumbersome.

%-formatting is limited as to the types it supports. Only ints, strs,
and doubles can be formatted. All other types are either not
supported, or converted to one of these types before formatting. In
addition, there's a well-known trap where a single value is passed::

  >>> msg = 'disk failure'
  >>> 'error: %s' % msg
  'error: disk failure'

But if msg were ever to be a tuple, the same code would fail::

  >>> msg = ('disk failure', 32)
  >>> 'error: %s' % msg
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: not all arguments converted during string formatting

To be defensive, the following code should be used::

  >>> 'error: %s' % (msg,)
  "error: ('disk failure', 32)"

``str.format()`` was added to address some of these problems with
%-formatting. In particular, it uses normal function call syntax (and
therefore supports multiple parameters) and it is extensible through
the ``__format__()`` method on the object being converted to a
string. See :pep:`3101` for a detailed rationale. This PEP reuses much of
the ``str.format()`` syntax and machinery, in order to provide
continuity with an existing Python string formatting mechanism.

However, ``str.format()`` is not without its issues. Chief among them
is its verbosity. For example, the text ``value`` is repeated here::

  >>> value = 4 * 20
  >>> 'The value is {value}.'.format(value=value)
  'The value is 80.'

Even in its simplest form there is a bit of boilerplate, and the value
that's inserted into the placeholder is sometimes far removed from
where the placeholder is situated::

  >>> 'The value is {}.'.format(value)
  'The value is 80.'

With an f-string, this becomes::

  >>> f'The value is {value}.'
  'The value is 80.'

F-strings provide a concise, readable way to include the value of
Python expressions inside strings.

In this sense, ``string.Template`` and %-formatting have similar
shortcomings to ``str.format()``, but also support fewer formatting
options. In particular, they do not support the ``__format__``
protocol, so that there is no way to control how a specific object is
converted to a string, nor can it be extended to additional types that
want to control how they are converted to strings (such as ``Decimal``
and ``datetime``). This example is not possible with
``string.Template``::

  >>> value = 1234
  >>> f'input={value:#06x}'
  'input=0x04d2'

And neither %-formatting nor ``string.Template`` can control
formatting such as::

  >>> date = datetime.date(1991, 10, 12)
  >>> f'{date} was on a {date:%A}'
  '1991-10-12 was on a Saturday'

No use of globals() or locals()
-------------------------------

In the discussions on python-dev [#]_, a number of solutions where
presented that used locals() and globals() or their equivalents. All
of these have various problems. Among these are referencing variables
that are not otherwise used in a closure. Consider::

  >>> def outer(x):
  ...     def inner():
  ...         return 'x={x}'.format_map(locals())
  ...     return inner
  ...
  >>> outer(42)()
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 3, in inner
  KeyError: 'x'

This returns an error because the compiler has not added a reference
to x inside the closure. You need to manually add a reference to x in
order for this to work::

  >>> def outer(x):
  ...     def inner():
  ...         x
  ...         return 'x={x}'.format_map(locals())
  ...     return inner
  ...
  >>> outer(42)()
  'x=42'

In addition, using locals() or globals() introduces an information
leak. A called routine that has access to the callers locals() or
globals() has access to far more information than needed to do the
string interpolation.

Guido stated [#]_ that any solution to better string interpolation
would not use locals() or globals() in its implementation.  (This does
not forbid users from passing locals() or globals() in, it just
doesn't require it, nor does it allow using these functions under the
hood.)

Specification
=============

In source code, f-strings are string literals that are prefixed by the
letter 'f' or 'F'. Everywhere this PEP uses 'f', 'F' may also be
used. 'f' may be combined with 'r' or 'R', in either order, to produce
raw f-string literals. 'f' may not be combined with 'b': this PEP does
not propose to add binary f-strings. 'f' may not be combined with 'u'.

When tokenizing source files, f-strings use the same rules as normal
strings, raw strings, binary strings, and triple quoted strings. That
is, the string must end with the same character that it started with:
if it starts with a single quote it must end with a single quote, etc.
This implies that any code that currently scans Python code looking
for strings should be trivially modifiable to recognize f-strings
(parsing within an f-string is another matter, of course).

Once tokenized, f-strings are parsed in to literal strings and
expressions.  Expressions appear within curly braces ``'{'`` and
``'}'``.  While scanning the string for expressions, any doubled
braces ``'{{'`` or ``'}}'`` inside literal portions of an f-string are
replaced by the corresponding single brace.  Doubled literal opening
braces do not signify the start of an expression.  A single closing
curly brace ``'}'`` in the literal portion of a string is an error:
literal closing curly braces must be doubled ``'}}'`` in order to
represent a single closing brace.

The parts of the f-string outside of braces are literal
strings.  These literal portions are then decoded.  For non-raw
f-strings, this includes converting backslash escapes such as
``'\n'``, ``'\"'``, ``"\'"``, ``'\xhh'``, ``'\uxxxx'``,
``'\Uxxxxxxxx'``, and named unicode characters ``'\N{name}'`` into
their associated Unicode characters [#]_.

Backslashes may not appear anywhere within expressions.  Co


id: pep-0467.rst
content:
PEP: 467
Title: Minor API improvements for binary sequences
Author: Alyssa Coghlan <ncoghlan@gmail.com>, Ethan Furman <ethan@stoneleaf.us>
Discussions-To: https://discuss.python.org/t/42001
Status: Draft
Type: Standards Track
Created: 30-Mar-2014
Python-Version: 3.15
Post-History: 30-Mar-2014, 15-Aug-2014, 16-Aug-2014, 07-Jun-2016, 01-Sep-2016,
              13-Apr-2021, 03-Nov-2021, 27-Dec-2023


Abstract
========

This PEP proposes small adjustments to the APIs of the ``bytes`` and
``bytearray`` types to make it easier to operate entirely in the binary domain:

* Add ``fromsize`` alternative constructor
* Add ``fromint`` alternative constructor
* Add ``getbyte`` byte retrieval method
* Add ``iterbytes`` alternative iterator

The last two (``getbyte`` and ``iterbytes``) will also be added to ``memoryview``.

Rationale
=========

During the initial development of the Python 3 language specification, the
core ``bytes`` type for arbitrary binary data started as the mutable type
that is now referred to as ``bytearray``. Other aspects of operating in
the binary domain in Python have also evolved over the course of the Python
3 series, for example with :pep:`461`.


Motivation
==========

With Python 3 and the split between ``str`` and ``bytes``, one small but
important area of programming became slightly more difficult, and much more
painful -- wire format protocols.

This area of programming is characterized by a mixture of binary data and
ASCII compatible segments of text (aka ASCII-encoded text).  The addition of
the new constructors, methods, and iterators will aid both in writing new
wire format code, and in updating existing code.

Common use-cases include ``dbf`` and ``pdf`` file formats, ``email``
formats, and ``FTP`` and ``HTTP`` communications, among many others.


Proposals
=========

Addition of explicit "count and byte initialised sequence" constructors
-----------------------------------------------------------------------

To replace the discouraged behavior of creating zero-filled ``bytes``-like
objects from the basic constructors (i.e. ``bytes(1)`` --> ``b'\x00'``), this
PEP proposes the addition of an explicit ``fromsize`` alternative constructor
as a class method on both ``bytes`` and ``bytearray`` whose first argument
is the count, and whose second argument is the fill byte to use (defaults
to ``\x00``)::

    >>> bytes.fromsize(3)
    b'\x00\x00\x00'
    >>> bytearray.fromsize(3)
    bytearray(b'\x00\x00\x00')
    >>> bytes.fromsize(5, b'\x0a')
    b'\x0a\x0a\x0a\x0a\x0a'
    >>> bytearray.fromsize(5, fill=b'\x0a')
    bytearray(b'\x0a\x0a\x0a\x0a\x0a')

``fromsize`` will behave just as the current constructors behave when passed a
single integer, while allowing for non-zero fill values when needed.


Addition of explicit "single byte" constructors
-----------------------------------------------

As binary counterparts to the text ``chr`` function, this PEP proposes
the addition of an explicit ``fromint`` alternative constructor as a class
method on both ``bytes`` and ``bytearray``::

    >>> bytes.fromint(65)
    b'A'
    >>> bytearray.fromint(65)
    bytearray(b'A')

These methods will only accept integers in the range 0 to 255 (inclusive)::

    >>> bytes.fromint(512)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: integer must be in range(0, 256)

    >>> bytes.fromint(1.0)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: 'float' object cannot be interpreted as an integer

The documentation of the ``ord`` builtin will be updated to explicitly note
that ``bytes.fromint`` is the primary inverse operation for binary data, while
``chr`` is the inverse operation for text data, and that ``bytearray.fromint``
also exists.

Behaviorally, ``bytes.fromint(x)`` will be equivalent to the current
``bytes([x])`` (and similarly for ``bytearray``). The new spelling is
expected to be easier to discover and easier to read (especially when used
in conjunction with indexing operations on binary sequence types).

As a separate method, the new spelling will also work better with higher
order functions like ``map``.

These new methods intentionally do NOT offer the same level of general integer
support as the existing ``int.to_bytes`` conversion method, which allows
arbitrarily large integers to be converted to arbitrarily long bytes objects. The
restriction to only accept positive integers that fit in a single byte means
that no byte order information is needed, and there is no need to handle
negative numbers. The documentation of the new methods will refer readers to
``int.to_bytes`` for use cases where handling of arbitrary integers is needed.


Addition of "getbyte" method to retrieve a single byte
------------------------------------------------------

This PEP proposes that ``bytes``, ``bytearray``, and ``memoryview`` gain the
method ``getbyte`` which will always return ``bytes``::

    >>> b'abc'.getbyte(0)
    b'a'

If an index is asked for that doesn't exist, ``IndexError`` is raised::

    >>> b'abc'.getbyte(9)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IndexError: index out of range


Addition of optimised iterator methods that produce ``bytes`` objects
---------------------------------------------------------------------

This PEP proposes that ``bytes``, ``bytearray``, and ``memoryview`` gain an
optimised ``iterbytes`` method that produces length 1 ``bytes`` objects rather
than integers::

    for x in data.iterbytes():
        # x is a length 1 ``bytes`` object, rather than an integer

For example::

    >>> tuple(b"ABC".iterbytes())
    (b'A', b'B', b'C')


Design discussion
=================

Why not rely on sequence repetition to create zero-initialised sequences?
-------------------------------------------------------------------------

Zero-initialised sequences can be created via sequence repetition::

    >>> b'\x00' * 3
    b'\x00\x00\x00'
    >>> bytearray(b'\x00') * 3
    bytearray(b'\x00\x00\x00')

However, this was also the case when the ``bytearray`` type was originally
designed, and the decision was made to add explicit support for it in the
type constructor. The immutable ``bytes`` type then inherited that feature
when it was introduced in :pep:`3137`.

This PEP isn't revisiting that original design decision, just changing the
spelling as users sometimes find the current behavior of the binary sequence
constructors surprising. In particular, there's a reasonable case to be made
that ``bytes(x)`` (where ``x`` is an integer) should behave like the
``bytes.fromint(x)`` proposal in this PEP. Providing both behaviors as separate
class methods avoids that ambiguity.

Current Workarounds
-------------------

After nearly a decade, there's seems to be no consensus on the best workarounds
for byte iteration, as demonstrated by
`Get single-byte bytes objects from bytes objects`_.


Omitting the originally proposed builtin function
-------------------------------------------------

When submitted to the Steering Council, this PEP proposed the introduction of
a ``bchr`` builtin (with the same behaviour as ``bytes.fromint``), recreating
the ``ord``/``chr``/``unichr`` trio from Python 2 under a different naming
scheme (``ord``/``bchr``/``chr``).

The SC indicated they didn't think this functionality was needed often enough
to justify offering two ways of doing the same thing, especially when one of
those ways was a new builtin function. That part of the proposal was therefore
dropped as being redundant with the ``bytes.fromint`` alternate constructor.

Developers that use this method frequently will instead have the option to
define their own ``bchr = bytes.fromint`` aliases.


References
==========

* `Initial March 2014 discussion thread on python-ideas <https://mail.python.org/pipermail/python-ideas/2014-March/027295.html>`_
* `Guido's initial feedback in that thread <https://mail.python.org/piper


id: pep-0301.rst
content:
PEP: 301
Title: Package Index and Metadata for Distutils
Author: Richard Jones <richard@python.org>
Status: Final
Type: Standards Track
Topic: Packaging
Created: 24-Oct-2002
Python-Version: 2.3
Post-History: 08-Nov-2002


Abstract
========

This PEP proposes several extensions to the Distutils packaging system
[1]_.  These enhancements include a central package index server,
tools for submitting package information to the index and extensions
to the package metadata to include Trove [2]_ information.

This PEP does not address issues of package dependency.  It also does
not address storage and download of packages as described in :pep:`243`.
Nor is it proposing a local database of packages as described
in :pep:`262`.

Existing package repositories such as the Vaults of Parnassus [3]_,
CPAN [4]_ and PAUSE [5]_ will be investigated as prior art in this
field.


Rationale
=========

Python programmers have long needed a simple method of discovering
existing modules and systems available for their use.  It is arguable
that the existence of these systems for other languages have been a
significant contribution to their popularity.  The existence of the
Catalog-SIG, and the many discussions there indicate that there is a
large population of users who recognise this need.

The introduction of the Distutils packaging system to Python
simplified the process of distributing shareable code, and included
mechanisms for the capture of package metadata, but did little with
the metadata save ship it with the package.

An interface to the index should be hosted in the python.org domain,
giving it an air of legitimacy that existing catalog efforts do not
have.

The interface for submitting information to the catalog should be as
simple as possible - hopefully just a one-line command for most users.

Issues of package dependency are not addressed due to the complexity
of such a system.  :pep:`262` proposes such a system, but as of this
writing the PEP is still unfinished.

Issues of package dissemination (storage on a central server) are
not addressed because they require assumptions about availability of
storage and bandwidth that I am not in a position to make.  :pep:`243`,
which is still being developed, is tackling these issues and many
more.  This proposal is considered compatible with, and adjunct to
the proposal in :pep:`243`.


Specification
=============

The specification takes three parts, the `web interface`_,  the
`Distutils register command`_ and the `Distutils Trove
classification`_.


Web Interface
-------------

A web interface is implemented over a simple store.  The interface is
available through the python.org domain, either directly or as
packages.python.org.

The store has columns for all metadata fields.  The (name, version)
double is used as a uniqueness key.  Additional submissions for an
existing (name, version) will result in an *update* operation.

The web interface implements the following commands/interfaces:

**index**
  Lists known packages, optionally filtered.  An additional HTML page,
  **search**, presents a form to the user which is used to customise
  the index view.  The index will include a browsing interface like
  that presented in the Trove interface design section 4.3.  The
  results will be paginated, sorted alphabetically and only showing
  the most recent version.  The most recent version information will
  be determined using the Distutils LooseVersion class.

**display**
  Displays information about the package.  All fields are displayed as
  plain text.  The "url" (or "home_page") field is hyperlinked.

**submit**
  Accepts a POST submission of metadata about a package.  The
  "name" and "version" fields are mandatory, as they uniquely identify
  an entry in the index.  **Submit** will automatically determine
  whether to create a new entry or update an existing entry.  The
  metadata is checked for correctness where appropriate - specifically
  the Trove discriminators are compared with the allowed set.  An
  update will update all information about the package based on the
  new submitted information.

  There will also be a submit/edit form that will allow manual
  submission and updating for those who do not use Distutils.

**submit_pkg_info**
  Accepts a POST submission of a PKG-INFO file and performs the same
  function as the **submit** interface.

**user**
  Registers a new user with the index.  Requires username, password
  and email address.  Passwords will be stored in the index database
  as SHA hashes.  If the username already exists in the database:

  1. If valid HTTP Basic authentication is provided, the password and
     email address are updated with the submission information, or
  2. If no valid authentication is provided, the user is informed that
     the login is already taken.

  Registration will be a three-step process, involving:

  1. User submission of details via the Distutils *register* command
     or through the web,
  2. Index server sending email to the user's email address with a URL
     to visit to confirm registration with a random one-time key, and
  3. User visits URL with the key and confirms registration.

**roles**
  An interface for changing user Role assignments.

**password_reset**
  Using a supplied email address as the key, this resets a user's
  password and sends an email with the new password to the user.

The **submit** command will require HTTP Basic authentication,
preferably over an HTTPS connection.

The server interface will indicate success or failure of the commands
through a subset of the standard HTTP response codes:

===== ============== ================================================
Code  Meaning        Register command implications
===== ============== ================================================
200   OK             Everything worked just fine
400   Bad request    Data provided for submission was malformed
401   Unauthorised   The username or password supplied were incorrect
403   Forbidden      User does not have permission to update the
                     package information (not Owner or Maintainer)
===== ============== ================================================

User Roles
----------

Three user Roles will be assignable to users:

Owner
  Owns a package name, may assign Maintainer Role for that name.  The
  first user to register information about a package is deemed Owner
  of the package name.  The Admin user may change this if necessary.
  May submit updates for the package name.

Maintainer
  Can submit and update info for a particular package name.

Admin
  Can assign Owner Role and edit user details. Not specific to a
  package name.


Index Storage (Schema)
----------------------

The index is stored in a set of relational database tables:

**packages**
  Lists package names and holds package-level metadata (currently
  just the stable release version)

**releases**
  Each package has an entry in **releases** for each version of the
  package that is released. A row holds the bulk of the information
  given in the package's PKG-INFO file. There is one row for each
  package (*name*, *version*).

**trove_discriminators**
  Lists the Trove discriminator text and assigns each one a unique
  ID.

**release_discriminators**
  Each entry maps a package (*name*, *version*) to a
  *discriminator_id*.  We map to releases instead of packages because
  the set of discriminators may change between releases.

**journals**
  Holds information about changes to package information in the
  index. Changes to the **packages**, **releases**, **roles**,
  and **release_discriminators** tables are listed here by
  package *name* and *version* if the change is release-specific.

**users**
  Holds our user database - user name, email address and password.

**roles**
  Maps *user_name* and *role_name* to a *package_name*.

An additional table, **rego_otk** holds the One Time Keys generated
during registration and is not interesting in the scope of the


id: pep-0315.rst
content:
PEP: 315
Title: Enhanced While Loop
Author: Raymond Hettinger <python@rcn.com>, W Isaac Carroll <icarroll@pobox.com>
Status: Rejected
Type: Standards Track
Created: 25-Apr-2003
Python-Version: 2.5
Post-History:


Abstract
========

This PEP proposes adding an optional "do" clause to the beginning
of the while loop to make loop code clearer and reduce errors
caused by code duplication.


Notice
======

Rejected; see [1]_.

This PEP has been deferred since 2006; see [2]_.

Subsequent efforts to revive the PEP in April 2009 did not
meet with success because no syntax emerged that could
compete with the following form::

    while True:
        <setup code>
        if not <condition>:
            break
        <loop body>

A syntax alternative to the one proposed in the PEP was found for
a basic do-while loop but it gained little support because the
condition was at the top::

    do ... while <cond>:
        <loop body>

Users of the language are advised to use the while-True form with
an inner if-break when a do-while loop would have been appropriate.


Motivation
==========

It is often necessary for some code to be executed before each
evaluation of the while loop condition.  This code is often
duplicated outside the loop, as setup code that executes once
before entering the loop::

    <setup code>
    while <condition>:
        <loop body>
        <setup code>

The problem is that duplicated code can be a source of errors if
one instance is changed but the other is not.  Also, the purpose
of the second instance of the setup code is not clear because it
comes at the end of the loop.

It is possible to prevent code duplication by moving the loop
condition into a helper function, or an if statement in the loop
body.  However, separating the loop condition from the while
keyword makes the behavior of the loop less clear::

    def helper(args):
        <setup code>
        return <condition>

    while helper(args):
        <loop body>

This last form has the additional drawback of requiring the loop's
else clause to be added to the body of the if statement, further
obscuring the loop's behavior::

    while True:
        <setup code>
        if not <condition>: break
        <loop body>

This PEP proposes to solve these problems by adding an optional
clause to the while loop, which allows the setup code to be
expressed in a natural way::

    do:
        <setup code>
    while <condition>:
        <loop body>

This keeps the loop condition with the while keyword where it
belongs, and does not require code to be duplicated.


Syntax
======

The syntax of the while statement ::

   while_stmt : "while" expression ":" suite
            ["else" ":" suite]

is extended as follows::

    while_stmt : ["do" ":" suite]
                "while" expression ":" suite
                ["else" ":" suite]


Semantics of break and continue
===============================

In the do-while loop the break statement will behave the same as
in the standard while loop: It will immediately terminate the loop
without evaluating the loop condition or executing the else
clause.

A continue statement in the do-while loop jumps to the while
condition check.

In general, when the while suite is empty (a pass statement),
the do-while loop and break and continue statements should match
the semantics of do-while in other languages.

Likewise, when the do suite is empty, the do-while loop and
break and continue statements should match behavior found
in regular while loops.


Future Statement
================

Because of the new keyword "do", the statement ::

   from __future__ import do_while

will initially be required to use the do-while form.


Implementation
==============

The first implementation of this PEP can compile the do-while loop
as an infinite loop with a test that exits the loop.


References
==========

.. [1] Guido van Rossum, PEP 315: do-while
       https://mail.python.org/pipermail/python-ideas/2013-June/021610.html

.. [2] Raymond Hettinger, release plan for 2.5 ?
       https://mail.python.org/pipermail/python-dev/2006-February/060718.html


Copyright
=========

This document is placed in the public domain.


id: pep-0473.rst
content:
PEP: 473
Title: Adding structured data to built-in exceptions
Author: Sebastian Kreft <skreft@deezer.com>
Status: Rejected
Type: Standards Track
Created: 29-Mar-2014
Post-History:
Resolution: https://mail.python.org/pipermail/python-dev/2019-March/156692.html


Abstract
========

Exceptions like ``AttributeError``, ``IndexError``, ``KeyError``,
``LookupError``, ``NameError``, ``TypeError``, and ``ValueError`` do not
provide all information required by programmers to debug and better understand
what caused them.
Furthermore, in some cases the messages even have slightly different formats,
which makes it really difficult for tools to automatically provide additional
information to diagnose the problem.
To tackle the former and to lay ground for the latter, it is proposed to expand
these exceptions so to hold both the offending and affected entities.


Rationale
=========

The main issue this PEP aims to solve is the fact that currently error messages
are not that expressive and lack some key information to resolve the exceptions.
Additionally, the information present on the error message is not always in the
same format, which makes it very difficult for third-party libraries to
provide automated diagnosis of the error.

These automated tools could, for example, detect typos or display or log extra
debug information. These could be particularly useful when running tests or in a
long running application.

Although it is in theory possible to have such libraries, they need to resort to
hacks in order to achieve the goal. One such example is
python-improved-exceptions [1]_, which modifies the byte-code to keep references
to the possibly interesting objects and also parses the error messages to
extract information like types or names. Unfortunately, such approach is
extremely fragile and not portable.

A similar proposal [2]_ has been implemented for ``ImportError`` and in the same
fashion this idea has received support [3]_. Additionally, almost 10 years ago
Guido asked in [11]_ to have a clean API to access the affected objects in
Exceptions like ``KeyError``, ``AttributeError``, ``NameError``, and
``IndexError``. Similar issues and proposals ideas have been written in the
last year. Some other issues have been created, but despite receiving support
they finally get abandoned. References to the created issues are listed below:

* ``AttributeError``: [11]_, [10]_, [5]_, [4]_, [3]_

* ``IndexError``: [11]_, [6]_, [3]_

* ``KeyError``: [11]_, [7]_, [3]_

* ``LookupError``: [11]_

* ``NameError``: [11]_, [10]_, [3]_

* ``TypeError``: [8]_

* ``ValueError``: [9]_


To move forward with the development and to centralize the information and
discussion, this PEP aims to be a meta-issue summarizing all the above
discussions and ideas.


Examples
========

IndexError
----------

The error message does not reference the list's length nor the index used.

::

  a = [1, 2, 3, 4, 5]
  a[5]
  IndexError: list index out of range


KeyError
--------

By convention the key is the first element of the error's argument, but there's
no other information regarding the affected dictionary (keys types, size, etc.)

::

  b = {'foo': 1}
  b['fo']
  KeyError: 'fo'


AttributeError
--------------

The object's type and the offending attribute are part of the error message.
However, there are some different formats and the information is not always
available. Furthermore, although the object type is useful in some cases, given
the dynamic nature of Python, it would be much more useful to have a reference
to the object itself. Additionally the reference to the type is not fully
qualified and in some cases the type is just too generic to provide useful
information, for example in case of accessing a module's attribute.

::

  c = object()
  c.foo
  AttributeError: 'object' object has no attribute 'foo'

  import string
  string.foo
  AttributeError: 'module' object has no attribute 'foo'

  a = string.Formatter()
  a.foo
  AttributeError: 'Formatter' object has no attribute 'foo'


NameError
---------

The error message provides typically the name.

::

  foo = 1
  fo
  NameError: global name 'fo' is not defined


Other Cases
-----------

Issues are even harder to debug when the target object is the result of
another expression, for example:

::

  a[b[c[0]]]

This issue is also related to the fact that opcodes only have line number
information and not the offset. This proposal would help in this case but not as
much as having offsets.


Proposal
========

Extend the exceptions ``AttributeError``, ``IndexError``, ``KeyError``,
``LookupError``, ``NameError``, ``TypeError``, and ``ValueError`` with the
following:

* ``AttributeError``: target :sup:`w`, attribute

* ``IndexError``: target :sup:`w`, key :sup:`w`, index (just an alias to
  key)

* ``KeyError``: target :sup:`w`, key :sup:`w`

* ``LookupError``: target :sup:`w`, key :sup:`w`

* ``NameError``: name, scope?

* ``TypeError``: unexpected_type

* ``ValueError``: unexpected_value :sup:`w`

Attributes with the superscript :sup:`w` may need to be weak references [12]_ to
prevent any memory cycles. However, this may add an unnecessary extra
complexity as noted by R. David Murray [13]_. This is specially true given that
builtin types do not support being weak referenced.

TODO(skreft): expand this with examples of corner cases.

To remain backwards compatible these new attributes will be optional and keyword
only.

It is proposed to add this information, rather than just improve the error, as
the former would allow new debugging frameworks and tools and also in the future
to switch to a lazy generated message. Generated messages are discussed in [2]_,
although they are not implemented at the moment. They would not only save some
resources, but also uniform the messages.

The stdlib will be then gradually changed so to start using these new
attributes.


Potential Uses
==============

An automated tool could for example search for similar keys within the object,
allowing to display the following:::

  a = {'foo': 1}
  a['fo']
  KeyError: 'fo'. Did you mean 'foo'?

  foo = 1
  fo
  NameError: global name 'fo' is not defined. Did you mean 'foo'?

See [3]_ for the output a TestRunner could display.


Performance
===========

Filling these new attributes would only require two extra parameters with data
already available so the impact should be marginal. However, it may need
special care for ``KeyError`` as the following pattern is already widespread.

::

  try:
    a[foo] = a[foo] + 1
  except:
    a[foo] = 0

Note as well that storing these objects into the error itself would allow the
lazy generation of the error message, as discussed in [2]_.


References
==========

.. [1] Python Exceptions Improved
   (https://www.github.com/sk-/python-exceptions-improved)

.. [2] ImportError needs attributes for module and file name
   (http://bugs.python.org/issue1559549)

.. [3] Enhance exceptions by attaching some more information to them
   (https://mail.python.org/pipermail/python-ideas/2014-February/025601.html)

.. [4] Specificity in AttributeError
   (https://mail.python.org/pipermail/python-ideas/2013-April/020308.html)

.. [5] Add an 'attr' attribute to AttributeError
   (http://bugs.python.org/issue18156)

.. [6] Add index attribute to IndexError
   (http://bugs.python.org/issue18162)

.. [7] Add a 'key' attribute to KeyError
   (http://bugs.python.org/issue18163)

.. [8] Add 'unexpected_type' to TypeError
   (http://bugs.python.org/issue18165)

.. [9] 'value' attribute for ValueError
   (http://bugs.python.org/issue18166)

.. [10] making builtin exceptions more informative
   (http://bugs.python.org/issue1182143)

.. [11] LookupError etc. need API to get the key
   (http://bugs.python.org/issue614557)

.. [12] weakref - Weak References
   (https://docs.python.org/3/library/weakref.html)

.. [13] Message by R.   David Murray: Weak refs on exceptions?
   (http://bugs.python.org/issue18163#msg190791)


Copyright
=========

This document


id: pep-0329.rst
content:
PEP: 329
Title: Treating Builtins as Constants in the Standard Library
Author: Raymond Hettinger <python@rcn.com>
Status: Rejected
Type: Standards Track
Created: 18-Apr-2004
Python-Version: 2.4
Post-History: 18-Apr-2004



Abstract
========

The proposal is to add a function for treating builtin references as
constants and to apply that function throughout the standard library.

Status
======

The PEP is self rejected by the author.  Though the ASPN recipe was
well received, there was less willingness to consider this for
inclusion in the core distribution.

The Jython implementation does not use byte codes, so its performance
would suffer if the current ``_len=len`` optimizations were removed.

Also, altering byte codes is one of the least clean ways to improve
performance and enable cleaner coding.  A more robust solution would
likely involve compiler pragma directives or metavariables indicating
what can be optimized (similar to const/volatile declarations).


Motivation
==========

The library contains code such as ``_len=len`` which is intended to
create fast local references instead of slower global lookups.  Though
necessary for performance, these constructs clutter the code and are
usually incomplete (missing many opportunities).

If the proposal is adopted, those constructs could be eliminated from
the code base and at the same time improve upon their results in terms
of performance.

There are currently over a hundred instances of ``while 1`` in the
library.  They were not replaced with the more readable ``while True``
because of performance reasons (the compiler cannot eliminate the test
because ``True`` is not known to always be a constant).  Conversion of
True to a constant will clarify the code while retaining performance.

Many other basic Python operations run much slower because of global
lookups.  In try/except statements, the trapped exceptions are
dynamically looked up before testing whether they match.
Similarly, simple identity tests such as ``while x is not None``
require the ``None`` variable to be re-looked up on every pass.
Builtin lookups are especially egregious because the enclosing global
scope must be checked first.  These lookup chains devour cache space
that is best used elsewhere.

In short, if the proposal is adopted, the code will become cleaner
and performance will improve across the board.


Proposal
========

Add a module called codetweaks.py which contains two functions,
``bind_constants()`` and ``bind_all()``.  The first function performs
constant binding and the second recursively applies it to every
function and class in a target module.

For most modules in the standard library, add a pair of lines near
the end of the script::

    import codetweaks, sys
    codetweaks.bind_all(sys.modules[__name__])

In addition to binding builtins, there are some modules (like
``sre_compile``) where it also makes sense to bind module variables
as well as builtins into constants.


Questions and Answers
=====================

1. Will this make everyone divert their attention to optimization
   issues?

   Because it is done automatically, it reduces the need to think
   about optimizations.

2. In a nutshell, how does it work?

   Every function has attributes with its bytecodes (the language of
   the Python virtual machine) and a table of constants.  The bind
   function scans the bytecodes for a ``LOAD_GLOBAL`` instruction and
   checks to see whether the value is already known.  If so, it adds
   that value to the constants table and replaces the opcode with
   ``LOAD_CONSTANT``.

3. When does it work?

   When a module is imported for the first time, python compiles the
   bytecode and runs the binding optimization.  Subsequent imports
   just re-use the previous work.  Each session repeats this process
   (the results are not saved in ``pyc`` files).

4. How do you know this works?

   I implemented it, applied it to every module in library, and the test
   suite ran without exception.

5. What if the module defines a variable shadowing a builtin?

   This does happen.  For instance, True can be redefined at the module
   level as ``True = (1==1)``.  The sample implementation below detects the
   shadowing and leaves the global lookup unchanged.

6. Are you the first person to recognize that most global lookups are for
   values that never change?

   No, this has long been known.  Skip Montanaro provides an eloquent
   explanation in :pep:`266`.

7. What if I want to replace the builtins module and supply my own
   implementations?

   Either do this before importing a module, or just reload the
   module, or disable ``codetweaks.py`` (it will have a disable flag).

8. How susceptible is this module to changes in Python's byte coding?

   It imports ``opcode.py`` to protect against renumbering.  Also, it
   uses ``LOAD_CONST`` and ``LOAD_GLOBAL`` which are fundamental and have
   been around forever.  That notwithstanding, the coding scheme could
   change and this implementation would have to change along with
   modules like ``dis`` which also rely on the current coding scheme.

9. What is the effect on startup time?

   I could not measure a difference.  None of the startup modules are
   bound except for warnings.py.  Also, the binding function is very
   fast, making just a single pass over the code string in search of
   the ``LOAD_GLOBAL`` opcode.


Sample Implementation
=====================

Here is a sample implementation for codetweaks.py::

    from types import ClassType, FunctionType
    from opcode import opmap, HAVE_ARGUMENT, EXTENDED_ARG
    LOAD_GLOBAL, LOAD_CONST = opmap['LOAD_GLOBAL'], opmap['LOAD_CONST']
    ABORT_CODES = (EXTENDED_ARG, opmap['STORE_GLOBAL'])

    def bind_constants(f, builtin_only=False, stoplist=[], verbose=False):
        """ Return a new function with optimized global references.

        Replaces global references with their currently defined values.
        If not defined, the dynamic (runtime) global lookup is left undisturbed.
        If builtin_only is True, then only builtins are optimized.
        Variable names in the stoplist are also left undisturbed.
        If verbose is True, prints each substitution as is occurs.

        """
        import __builtin__
        env = vars(__builtin__).copy()
        stoplist = dict.fromkeys(stoplist)
        if builtin_only:
            stoplist.update(f.func_globals)
        else:
            env.update(f.func_globals)

        co = f.func_code
        newcode = map(ord, co.co_code)
        newconsts = list(co.co_consts)
        codelen = len(newcode)

        i = 0
        while i < codelen:
            opcode = newcode[i]
            if opcode in ABORT_CODES:
                return f    # for simplicity, only optimize common cases
            if opcode == LOAD_GLOBAL:
                oparg = newcode[i+1] + (newcode[i+2] << 8)
                name = co.co_names[oparg]
                if name in env and name not in stoplist:
                    value = env[name]
                    try:
                        pos = newconsts.index(value)
                    except ValueError:
                        pos = len(newconsts)
                        newconsts.append(value)
                    newcode[i] = LOAD_CONST
                    newcode[i+1] = pos & 0xFF
                    newcode[i+2] = pos >> 8
                    if verbose:
                        print name, '-->', value
            i += 1
            if opcode >= HAVE_ARGUMENT:
                i += 2

        codestr = ''.join(map(chr, newcode))
        codeobj = type(co)(co.co_argcount, co.co_nlocals, co.co_stacksize,
                        co.co_flags, codestr, tuple(newconsts), co.co_names,
                        co.co_varnames, co.co_filename, co.co_name,
                        co.co_firstlineno, co.co_lnotab, co.co_freevars,
                        co.co_cellvars)
        return type(f)(codeobj, f.func_globals, f.func_name, f.func_defaults,
                  


id: pep-8013.rst
content:
PEP: 8013
Title: The External Council Governance Model
Author: Steve Dower <steve.dower@python.org>
Status: Rejected
Type: Informational
Topic: Governance
Created: 14-Sep-2018

Abstract
========

This PEP proposes a new model of Python governance based on a Council
of Auditors (CoA) tasked with making final decisions for the language.
It differs from :pep:`8010` by specifically not proposing a central
singular leader, and from :pep:`8011` by disallowing core committers from
being council members. It describes the size and role of the council,
how the initial group of council members will be chosen, any term
limits of the council members, and how successors will be elected.

It also spends significant time discussing the intended behaviour of
this model. By design, many processes are not specified here but are
left to the people involved. In order to select people who will make
the best decisions, it is important for those involved to understand
the expectations of the CoA but it is equally important to allow the
CoA the freedom to adjust process requirements for varying
circumstances. This only works when process is unspecified, but all
participants have similar expectations.

This PEP does *not* name the members of the CoA. Should this model be
adopted, it will be codified in :pep:`13` along with the names of all
officeholders described in this PEP.

PEP Rejection
=============

:pep:`8013` was rejected `by a core developer vote
<https://discuss.python.org/t/python-governance-vote-december-2018-results/546/>`__
described in :pep:`8001` on Monday, December 17, 2018.

:pep:`8016` and the governance model it describes were chosen instead.

The Importance of the Grey Area
===============================

In any actual decision-making process, there is going to be grey area.
This includes unexpected scenarios, and cases where there is no
"correct" answer.

Many process plans attempt to minimise grey area by defining processes
clearly enough that no flexibility is required.

This proposal deliberately goes the other way. The aim is to provide a
robust framework for choosing the best people to handle unexpected
situations, without defining how those people should handle those
situations.

Examples are provided of "good" responses to some situations as an
illustration. The hope is that the "best" people are the best because
they would live up to those examples. The process that is proposed has
been designed to minimise the damage that may be caused when those
people turn out not to be the best.

Grey area is guaranteed to exist. This proposal deliberately embraces
and works within that, rather than attempting to prevent it.

Model Overview
==============

Key people and their functions
------------------------------

The Council of Auditors (CoA) is a council of varying size, typically
two to four people, who are elected for the duration of a Python
release. One member of the CoA is considered the President, who has
some minor points of authority over the other members.

The CoA has responsibility for reviewing controversial decisions in
the form of PEPs written by members of the core development team. The
CoA may choose to accept a PEP exactly as presented, or may request
clarification or changes. These changes may be of any form and for any
reason. This flexibility is intentional, and allows the process to
change over time as different members are elected to the CoA. See the
later sections of this document for examples of the kinds of requests
that are expected.

The CoA only pronounces on PEPs submitted to python-committers. There
is no expectation that the CoA follows or participates on any other
mailing lists. (Note that this implies that only core developers may
submit PEPs. Non-core developers may write and discuss proposals on
other mailing lists, but without a core developer willing to support
the proposal by requesting pronouncement, it cannot proceed to
acceptance. This is essentially the same as the current system, but is
made explicit here to ensure that members of the CoA are not expected
to deal with proposals that are not supported by at least one core
developer.)

The CoA may not delegate authority to individuals who have not been
elected by the core developer team. (One relevant case here is that
this changes the implementation of the existing BDFL-Delegate system,
though without necessarily changing the spirit of that system. See the
later sections, particularly example scenario four, for more
discussion on this point.)

The Release Manager (RM) is also permitted the same ability to request
changes on any PEPs that specify the release they are responsible for.
After feature freeze, the RM retains this responsibility for their
release, while the CoA rotates and begins to focus on the subsequent
release. This is no different from the current process. The process
for selection of a RM is not changed in this proposal.

Core developers are responsible for electing members of the CoA, and
have the ability to call a "vote of no confidence" against a member of
the CoA. The details of these votes are discussed in a later section.

Where discussions between core developers and members of the CoA
appear to be ongoing but unfruitful, the President may step in to
overrule either party. Where the discussion involves the President, it
should be handled using a vote of no confidence.

Members of the CoA may choose to resign at any point. If at least two
members of the CoA remain, they may request a new election to refill
the group. If only one member remains, the election is triggered
automatically. (The scenario when the President resigns is described
in a later section.)

The intended balance of power is that the core developers will elect
members of the CoA who reflect the direction and have the trust of the
development team, and also have the ability to remove members who do
not honour commitments made prior to election.

Regular decision process
------------------------

Regular decisions continue to be made as at present.

For the sake of clarity, controversial decisions require a PEP, and
any decisions requiring a PEP are considered as controversial.

The CoA may be asked to advise on whether a decision would be better
made using the controversial decision process, or individual members
of the CoA may volunteer such a suggestion, but the core development
team is not bound by this advice.

Controversial decision process
------------------------------

Controversial decisions are always written up as PEPs, following the
existing process. The approver (formerly "BDFL-Delegate") is always
the CoA, and can no longer be delegated. Note that this does not
prevent the CoA from deciding to nominate a core developer to assess
the proposal and provide the CoA with a recommendation, which is
essentially the same as the current delegation process.

The CoA will pronounce on PEPs submitted to python-committers with a
request for pronouncement. Any member of the CoA, or the current RM,
may request changes to a PEP for any reason, provided they include
some indication of what additional work is required to meet their
expectations. See later sections for examples of expected reasons.

When all members of the CoA and the RM indicate that they have no
concerns with a PEP, it is formally accepted. When one or more members
of the CoA fail to respond in a reasonable time, the President of the
CoA may choose to interpret that as implied approval. Failure of the
President to respond should be handled using a vote of no confidence.

Election terms
--------------

Members of the CoA are elected for the duration of a release. The
members are elected prior to feature freeze for the previous release,
and hold their position until feature freeze for their release.

Members may seek re-election as many times as they like. There are no
term limits. It is up to the core developers to prevent re-election of
the CoA members where there is consensus that the individual should
not 


id: pep-0103.rst
content:
PEP: 103
Title: Collecting information about git
Author: Oleg Broytman <phd@phdru.name>
Status: Withdrawn
Type: Informational
Created: 01-Jun-2015
Post-History: 12-Sep-2015

.. withdrawn::

   It is too generic and doesn't really deal
   with Python development. It is no longer updated.

   The content was moved to `Python Wiki`_. Make further updates in the
   wiki.

   .. _`Python Wiki`: https://wiki.python.org/moin/Git

Abstract
========

This Informational PEP collects information about git. There is, of
course, a lot of documentation for git, so the PEP concentrates on
more complex (and more related to Python development) issues,
scenarios and examples.

The plan is to extend the PEP in the future collecting information
about equivalence of Mercurial and git scenarios to help migrating
Python development from Mercurial to git.

The author of the PEP doesn't currently plan to write a Process PEP on
migration Python development from Mercurial to git.


Documentation
=============

Git is accompanied with a lot of documentation, both online and
offline.


Documentation for starters
--------------------------

Git Tutorial: `part 1
<https://www.kernel.org/pub/software/scm/git/docs/gittutorial.html>`_,
`part 2
<https://www.kernel.org/pub/software/scm/git/docs/gittutorial-2.html>`_.

`Git User's manual
<https://www.kernel.org/pub/software/scm/git/docs/user-manual.html>`_.
`Everyday GIT With 20 Commands Or So
<https://www.kernel.org/pub/software/scm/git/docs/giteveryday.html>`_.
`Git workflows
<https://www.kernel.org/pub/software/scm/git/docs/gitworkflows.html>`_.


Advanced documentation
----------------------

`Git Magic
<http://www-cs-students.stanford.edu/~blynn/gitmagic/index.html>`_,
with a number of translations.

`Pro Git <https://git-scm.com/book>`_. The Book about git. Buy it at
Amazon or download in PDF, mobi, or ePub form. It has translations to
many different languages. Download Russian translation from `GArik
<https://github.com/GArik/progit/wiki>`_.

`Git Wiki <https://git.wiki.kernel.org/index.php/Main_Page>`_.

`Git Buch <http://gitbu.ch/index.html>`_ (German).


Offline documentation
---------------------

Git has builtin help: run ``git help $TOPIC``. For example, run
``git help git`` or ``git help help``.


Quick start
===========

Download and installation
-------------------------

Unix users: `download and install using your package manager
<https://git-scm.com/download/linux>`_.

Microsoft Windows: download `git-for-windows
<https://github.com/git-for-windows/git/releases>`_.

MacOS X: use git installed with `XCode
<https://developer.apple.com/xcode/>`_ or download from `MacPorts
<https://www.macports.org/ports.php?by=name&substr=git>`_ or
`git-osx-installer
<http://sourceforge.net/projects/git-osx-installer/files/>`_ or
install git with `Homebrew <http://brew.sh/>`_: ``brew install git``.

`git-cola <https://git-cola.github.io/index.html>`_ (`repository
<https://github.com/git-cola/git-cola>`__) is a Git GUI written in
Python and GPL licensed. Linux, Windows, MacOS X.

`TortoiseGit <https://tortoisegit.org/>`_ is a Windows Shell Interface
to Git based on TortoiseSVN; open source.


Initial configuration
---------------------

This simple code is often appears in documentation, but it is
important so let repeat it here. Git stores author and committer
names/emails in every commit, so configure your real name and
preferred email::

    $ git config --global user.name "User Name"
    $ git config --global user.email user.name@example.org


Examples in this PEP
====================

Examples of git commands in this PEP use the following approach. It is
supposed that you, the user, works with a local repository named
``python`` that has an upstream remote repo named ``origin``. Your
local repo has two branches ``v1`` and ``master``. For most examples
the currently checked out branch is ``master``. That is, it's assumed
you have done something like that::

    $ git clone https://git.python.org/python.git
    $ cd python
    $ git branch v1 origin/v1

The first command clones remote repository into local directory
``python``, creates a new local branch master, sets
remotes/origin/master as its upstream remote-tracking branch and
checks it out into the working directory.

The last command creates a new local branch v1 and sets
remotes/origin/v1 as its upstream remote-tracking branch.

The same result can be achieved with commands::

    $ git clone -b v1 https://git.python.org/python.git
    $ cd python
    $ git checkout --track origin/master

The last command creates a new local branch master, sets
remotes/origin/master as its upstream remote-tracking branch and
checks it out into the working directory.


Branches and branches
=====================

Git terminology can be a bit misleading. Take, for example, the term
"branch". In git it has two meanings. A branch is a directed line of
commits (possibly with merges). And a branch is a label or a pointer
assigned to a line of commits. It is important to distinguish when you
talk about commits and when about their labels. Lines of commits are
by itself unnamed and are usually only lengthening and merging.
Labels, on the other hand, can be created, moved, renamed and deleted
freely.


Remote repositories and remote branches
=======================================

Remote-tracking branches are branches (pointers to commits) in your
local repository. They are there for git (and for you) to remember
what branches and commits have been pulled from and pushed to what
remote repos (you can pull from and push to many remotes).
Remote-tracking branches live under ``remotes/$REMOTE`` namespaces,
e.g. ``remotes/origin/master``.

To see the status of remote-tracking branches run::

    $ git branch -rv

To see local and remote-tracking branches (and tags) pointing to
commits::

    $ git log --decorate

You never do your own development on remote-tracking branches. You
create a local branch that has a remote branch as upstream and do
development on that local branch. On push git pushes commits to the
remote repo and updates remote-tracking branches, on pull git fetches
commits from the remote repo, updates remote-tracking branches and
fast-forwards, merges or rebases local branches.

When you do an initial clone like this::

    $ git clone -b v1 https://git.python.org/python.git

git clones remote repository ``https://git.python.org/python.git`` to
directory ``python``, creates a remote named ``origin``, creates
remote-tracking branches, creates a local branch ``v1``, configure it
to track upstream remotes/origin/v1 branch and checks out ``v1`` into
the working directory.

Some commands, like ``git status --branch`` and ``git branch --verbose``,
report the difference between local and remote branches.
Please remember they only do comparison with remote-tracking branches
in your local repository, and the state of those remote-tracking
branches can be outdated. To update remote-tracking branches you
either fetch and merge (or rebase) commits from the remote repository
or update remote-tracking branches without updating local branches.


Updating local and remote-tracking branches
-------------------------------------------

To update remote-tracking branches without updating local branches run
``git remote update [$REMOTE...]``. For example::

    $ git remote update
    $ git remote update origin


Fetch and pull
''''''''''''''

There is a major difference between

::

    $ git fetch $REMOTE $BRANCH

and

::

    $ git fetch $REMOTE $BRANCH:$BRANCH

The first command fetches commits from the named $BRANCH in the
$REMOTE repository that are not in your repository, updates
remote-tracking branch and leaves the id (the hash) of the head commit
in file .git/FETCH_HEAD.

The second command fetches commits from the named $BRANCH in the
$REMOTE repository that are not in your repository and updates both
the local branch $BRANCH and its upstream remote-tracking branch. But
it refuses to update branches


id: pep-0665.rst
content:
PEP: 665
Title: A file format to list Python dependencies for reproducibility of an application
Author: Brett Cannon <brett@python.org>,
        Pradyun Gedam <pradyunsg@gmail.com>,
        Tzu-ping Chung <uranusjr@gmail.com>
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/9911
Status: Rejected
Type: Standards Track
Topic: Packaging
Created: 29-Jul-2021
Post-History: 29-Jul-2021, 03-Nov-2021, 25-Nov-2021
Superseded-By: 751
Resolution: https://discuss.python.org/t/pep-665-take-2-a-file-format-to-list-python-dependencies-for-reproducibility-of-an-application/11736/140

.. note::
   This PEP was rejected due to lukewarm reception from the community
   from the lack of source distribution support.

========
Abstract
========

This PEP specifies a file format to specify the list of Python package
installation requirements for an application, and the relation between
the specified requirements. The list of requirements is considered
exhaustive for the installation target, and thus not requiring any
information beyond the platform being installed for, and the file
itself. The file format is flexible enough to allow installing the
requirements across different platforms, which allows for
reproducibility on multiple platforms from the same file.


===========
Terminology
===========

There are several terms whose definition must be agreed upon in order
to facilitate a discussion on the topic of this PEP.

A *package* is something you install as a dependency and use via the
import system. The packages on PyPI are an example of this.

An *application* or *app* is an end product that other external code
does not directly rely on via the import system (i.e. they are
standalone). Desktop applications, command-line tools, etc. are
examples of applications.

A *lock file* records the packages that are to be installed for an
app. Traditionally, the exact version of the package to be installed
is specified by a lock file, but specified packages are not always
installed on a given platform (according a filtering logic described
in a later section), which enables the lock file to describe
reproducibility across multiple platforms. Examples of this are
``package-lock.json`` from npm_, ``Poetry.lock`` from Poetry_, etc.

*Locking* is the act of taking the input of the packages an app
depends on and producing a lock file from that.

A *locker* is a tool which produces a lock file.

An *installer* consumes a lock file to install what the lock file
specifies.


==========
Motivation
==========

Applications want reproducible installs for a few reasons (we are not
worrying about package development, integration into larger systems
that would handle locking dependencies external to the Python
application, or other situations where *flexible* installation
requirements are desired over strict, reproducible installations).

One, reproducibility eases development. When you and your fellow
developers all end up with the same files on a specific platform, you
make sure you are all developing towards the same experience for the
application. You also want your users to install the same files as
you expect to guarantee the experience is the same as you developed
for them.

Two, you want to be able to reproduce what gets installed across
multiple platforms. Thanks to Python's portability across operating
systems, CPUs, etc., it is very easy and often desirable to create
applications that are not restricted to a single platform. Thus, you
want to be flexible enough to allow for differences in your package
dependencies between platforms, while still having consistency
and reproducibility on any one specific platform.

Three, reproducibility is more secure. When you control exactly what
files are installed, you can make sure no malicious actor is
attempting to slip nefarious code into your application (i.e. some
supply chain attacks). By using a lock file which always leads to
reproducible installs, we can avoid certain risks entirely.

Four, relying on the `wheel file`_ format provides reproducibility
without requiring build tools to support reproducibility themselves.
Thanks to wheels being static and not executing code as part of
installation, wheels always lead to a reproducible result. Compare
this to source distributions (aka sdists) or source trees which only
lead to a reproducible install if their build tool supports
reproducibility due to inherent code execution. Unfortunately the vast
majority of build tools do not support reproducible builds, so this
PEP helps alleviate that issue by only supporting wheels as a package
format.

This PEP proposes a standard for a lock file, as the current solutions
don't meet the outlined goals. Today, the closest we come to a lock
file standard is the `requirements file format`_ from pip.
Unfortunately, that format does not lead to inherently reproducible
installs (it requires optional features both in the requirements file
and the installer itself, to be discussed later).

The community itself has also shown a need for lock files based on the
fact that multiple tools have independently created their own lock
file formats:

#. PDM_
#. `pip-tools`_
#. Pipenv_
#. Poetry_
#. Pyflow_

Unfortunately, those tools all use differing lock file formats. This
means tooling around these tools must be unique. This impacts tooling
such as code editors and hosting providers, which want to be as
flexible as possible when it comes to accepting a user's application
code, but also have a limit as to how much development resources they
can spend to add support for yet another lock file format. A
standardized format would allow tools to focus their work on a single
target, and make sure that workflow decisions made by developers
outside of the lock file format are of no concern to e.g. hosting
providers.

Other programming language communities have also shown the usefulness
of lock files by developing their own solution to this problem. Some
of those communities include:

#. Dart_
#. npm_/Node
#. Go
#. Rust_

The trend in programming languages in the past decade seems to have
been toward providing a lock file solution.


=========
Rationale
=========

-----------
File Format
-----------

We wanted the file format to be easy to read as a diff when auditing
a change to the lock file. As such, and thanks to :pep:`518` and
``pyproject.toml``, we decided to go with the TOML_ file format.


-----------------
Secure by Design
-----------------

Viewing the `requirements file format`_ as the closest we have to
a lock file standard, there are a few issues with the file format when
it comes to security. First is that the file format simply does not
require you to specify the exact version of a package. This is why
tools like `pip-tools`_ exist to help manage that users of
requirements files.

Second, you must opt into specifying what files are acceptable to be
installed by using the ``--hash`` argument for a specific dependency.
This is also optional with pip-tools as it requires specifying the
``--generate-hashes`` CLI argument. This requires ``--require-hashes``
for pip to make sure no dependencies lack a hash to check.


Third, even when you control what files may be installed, it does not
prevent other packages from being installed. If a dependency is not
listed in the requirements file, pip will happily go searching for a
file to meet that need. You must specify ``--no-deps`` as an
argument to pip to prevent unintended dependency resolution outside
of the requirements file.

Fourth, the format allows for installing a
`source distribution file`_ (aka "sdist"). By its very nature,
installing an sdist requires executing arbitrary Python code, meaning
that there is no control over what files may be installed. Only by
specifying ``--only-binary :all:`` can you guarantee pip to only use a
`wheel file`_ for each package.

To recap, in order for a requirements file to be as secure as what is
being proposed, a user should always do th


id: pep-0671.rst
content:
PEP: 671
Title: Syntax for late-bound function argument defaults
Author: Chris Angelico <rosuav@gmail.com>
Discussions-To: https://mail.python.org/archives/list/python-ideas@python.org/thread/UVOQEK7IRFSCBOH734T5GFJOEJXFCR6A/
Status: Draft
Type: Standards Track
Created: 24-Oct-2021
Python-Version: 3.12
Post-History: `24-Oct-2021 <https://mail.python.org/archives/list/python-ideas@python.org/thread/KR2TMLPFR7NHDZCDOS6VTNWDKZQQJN3V/>`__,
              `01-Dec-2021 <https://mail.python.org/archives/list/python-ideas@python.org/thread/UVOQEK7IRFSCBOH734T5GFJOEJXFCR6A/>`__

Abstract
========

Function parameters can have default values which are calculated during
function definition and saved. This proposal introduces a new form of
argument default, defined by an expression to be evaluated at function
call time.


Motivation
==========

Optional function arguments, if omitted, often have some sort of logical
default value. When this value depends on other arguments, or needs to be
reevaluated each function call, there is currently no clean way to state
this in the function header.

Currently-legal idioms for this include::

    # Very common: Use None and replace it in the function
    def bisect_right(a, x, lo=0, hi=None, *, key=None):
        if hi is None:
            hi = len(a)

    # Also well known: Use a unique custom sentinel object
    _USE_GLOBAL_DEFAULT = object()
    def connect(timeout=_USE_GLOBAL_DEFAULT):
        if timeout is _USE_GLOBAL_DEFAULT:
            timeout = default_timeout

    # Unusual: Accept star-args and then validate
    def add_item(item, *optional_target):
        if not optional_target:
            target = []
        else:
            target = optional_target[0]

In each form, ``help(function)`` fails to show the true default value. Each
one has additional problems, too; using ``None`` is only valid if None is not
itself a plausible function parameter, the custom sentinel requires a global
constant; and use of star-args implies that more than one argument could be
given.

Specification
=============

Function default arguments can be defined using the new ``=>`` notation::

    def bisect_right(a, x, lo=0, hi=>len(a), *, key=None):
    def connect(timeout=>default_timeout):
    def add_item(item, target=>[]):
    def format_time(fmt, time_t=>time.time()):

The expression is saved in its source code form for the purpose of inspection,
and bytecode to evaluate it is prepended to the function's body.

Notably, the expression is evaluated in the function's run-time scope, NOT the
scope in which the function was defined (as are early-bound defaults). This
allows the expression to refer to other arguments.

Multiple late-bound arguments are evaluated from left to right, and can refer
to previously-defined values. Order is defined by the function, regardless of
the order in which keyword arguments may be passed.

    def prevref(word="foo", a=>len(word), b=>a//2): # Valid
    def selfref(spam=>spam): # UnboundLocalError
    def spaminate(sausage=>eggs + 1, eggs=>sausage - 1): # Confusing, don't do this
    def frob(n=>len(items), items=[]): # See below

Evaluation order is left-to-right; however, implementations MAY choose to do so
in two separate passes, first for all passed arguments and early-bound defaults,
and then a second pass for late-bound defaults. Otherwise, all arguments will be
assigned strictly left-to-right.

Rejected choices of spelling
----------------------------

While this document specifies a single syntax ``name=>expression``, alternate
spellings are similarly plausible. The following spellings were considered::

    def bisect(a, hi=>len(a)):
    def bisect(a, hi:=len(a)):
    def bisect(a, hi?=len(a)):
    def bisect(a, @hi=len(a)):

Since default arguments behave largely the same whether they're early or late
bound, the chosen syntax ``hi=>len(a)`` is deliberately similar to the existing
early-bind syntax.

One reason for rejection of the ``:=`` syntax is its behaviour with annotations.
Annotations go before the default, so in all syntax options, it must be
unambiguous (both to the human and the parser) whether this is an annotation,
a default, or both. The alternate syntax ``target:=expr`` runs the risk of
being misinterpreted as ``target:int=expr`` with the annotation omitted in
error, and may thus mask bugs. The chosen syntax ``target=>expr`` does not
have this problem.


How to Teach This
=================

Early-bound default arguments should always be taught first, as they are the
simpler and more efficient way to evaluate arguments. Building on them, late
bound arguments are broadly equivalent to code at the top of the function::

    def add_item(item, target=>[]):

    # Equivalent pseudocode:
    def add_item(item, target=<OPTIONAL>):
        if target was omitted: target = []

A simple rule of thumb is: "target=expression" is evaluated when the function
is defined, and "target=>expression" is evaluated when the function is called.
Either way, if the argument is provided at call time, the default is ignored.
While this does not completely explain all the subtleties, it is sufficient to
cover the important distinction here (and the fact that they are similar).


Interaction with other proposals
================================

:pep:`661` attempts to solve one of the same problems as this does. It seeks to
improve the documentation of sentinel values in default arguments, where this
proposal seeks to remove the need for sentinels in many common cases. :pep:`661`
is able to improve documentation in arbitrarily complicated functions (it
cites ``traceback.print_exception`` as its primary motivation, which has two
arguments which must both-or-neither be specified); on the other hand, many
of the common cases would no longer need sentinels if the true default could
be defined by the function. Additionally, dedicated sentinel objects can be
used as dictionary lookup keys, where :pep:`671` does not apply.

A generic system for deferred evaluation has been proposed at times (not to be
confused with :pep:`563` and :pep:`649` which are specific to annotations).
While it may seem, on the surface, that late-bound argument defaults are of a
similar nature, they are in fact unrelated and orthogonal ideas, and both could
be of value to the language. The acceptance or rejection of this proposal would
not affect the viability of a deferred evaluation proposal, and vice versa. (A
key difference between generalized deferred evaluation and argument defaults is
that argument defaults will always and only be evaluated as the function begins
executing, whereas deferred expressions would only be realized upon reference.)


Implementation details
======================

The following relates to the reference implementation, and is not necessarily
part of the specification.

Argument defaults (positional or keyword) have both their values, as already
retained, and an extra piece of information. For positional arguments, the
extras are stored in a tuple in ``__defaults_extra__``, and for keyword-only,
a dict in ``__kwdefaults_extra__``. If this attribute is ``None``, it is
equivalent to having ``None`` for every argument default.

For each parameter with a late-bound default, the special value ``Ellipsis``
is stored as the value placeholder, and the corresponding extra information
needs to be queried. If it is ``None``, then the default is indeed the value
``Ellipsis``; otherwise, it is a descriptive string and the true value is
calculated as the function begins.

When a parameter with a late-bound default is omitted, the function will begin
with the parameter unbound. The function begins by testing for each parameter
with a late-bound default using a new opcode QUERY_FAST/QUERY_DEREF, and if
unbound, evaluates the original expression. This opcode (available only for
fast locals and closure variables) pushes True onto the stack if the given
local has a value, and False if not - meaning that it pushes False if LOAD_F


id: pep-0659.rst
content:
PEP: 659
Title: Specializing Adaptive Interpreter
Author: Mark Shannon <mark@hotpy.org>
Status: Final
Type: Informational
Created: 13-Apr-2021
Post-History: 11-May-2021

.. canonical-doc:: :ref:`Specializing Adaptive Interpreter <whatsnew311-pep659>`


Abstract
========

In order to perform well, virtual machines for dynamic languages must
specialize the code that they execute to the types and values in the
program being run. This specialization is often associated with "JIT"
compilers, but is beneficial even without machine code generation.

A specializing, adaptive interpreter is one that speculatively specializes
on the types or values it is currently operating on, and adapts to changes
in those types and values.

Specialization gives us improved performance, and adaptation allows the
interpreter to rapidly change when the pattern of usage in a program alters,
limiting the amount of additional work caused by mis-specialization.

This PEP proposes using a specializing, adaptive interpreter that specializes
code aggressively, but over a very small region, and is able to adjust to
mis-specialization rapidly and at low cost.

Adding a specializing, adaptive interpreter to CPython will bring significant
performance improvements. It is hard to come up with meaningful numbers,
as it depends very much on the benchmarks and on work that has not yet happened.
Extensive experimentation suggests speedups of up to 50%.
Even if the speedup were only 25%, this would still be a worthwhile enhancement.

Motivation
==========

Python is widely acknowledged as slow.
Whilst Python will never attain the performance of low-level languages like C,
Fortran, or even Java, we would like it to be competitive with fast
implementations of scripting languages, like V8 for Javascript or luajit for
lua.
Specifically, we want to achieve these performance goals with CPython to
benefit all users of Python including those unable to use PyPy or
other alternative virtual machines.

Achieving these performance goals is a long way off, and will require a lot of
engineering effort, but we can make a significant step towards those goals by
speeding up the interpreter.
Both academic research and practical implementations have shown that a fast
interpreter is a key part of a fast virtual machine.

Typical optimizations for virtual machines are expensive, so a long "warm up"
time is required to gain confidence that the cost of optimization is justified.
In order to get speed-ups rapidly, without noticeable warmup times,
the VM should speculate that specialization is justified even after a few
executions of a function. To do that effectively, the interpreter must be able
to optimize and de-optimize continually and very cheaply.

By using adaptive and speculative specialization at the granularity of
individual virtual machine instructions,
we get a faster interpreter that also generates profiling information
for more sophisticated optimizations in the future.

Rationale
=========

There are many practical ways to speed-up a virtual machine for a dynamic
language.
However, specialization is the most important, both in itself and as an
enabler of other optimizations.
Therefore it makes sense to focus our efforts on specialization first,
if we want to improve the performance of CPython.

Specialization is typically done in the context of a JIT compiler,
but research shows specialization in an interpreter can boost performance
significantly, even outperforming a naive compiler [1]_.

There have been several ways of doing this proposed in the academic
literature, but most attempt to optimize regions larger than a
single bytecode [1]_ [2]_.
Using larger regions than a single instruction requires code to handle
de-optimization in the middle of a region.
Specialization at the level of individual bytecodes makes de-optimization
trivial, as it cannot occur in the middle of a region.

By speculatively specializing individual bytecodes, we can gain significant
performance improvements without anything but the most local,
and trivial to implement, de-optimizations.

The closest approach to this PEP in the literature is
"Inline Caching meets Quickening" [3]_.
This PEP has the advantages of inline caching,
but adds the ability to quickly de-optimize making the performance
more robust in cases where specialization fails or is not stable.

Performance
-----------

The speedup from specialization is hard to determine, as many specializations
depend on other optimizations. Speedups seem to be in the range 10% - 60%.

* Most of the speedup comes directly from specialization. The largest
  contributors are speedups to attribute lookup, global variables, and calls.
* A small, but useful, fraction is from improved dispatch such as
  super-instructions and other optimizations enabled by quickening.

Implementation
==============

Overview
--------

Any instruction that would benefit from specialization will be replaced by an
"adaptive" form of that instruction. When executed, the adaptive instructions
will specialize themselves in response to the types and values that they see.
This process is known as "quickening".

Once an instruction in a code object has executed enough times,
that instruction will be "specialized" by replacing it with a new instruction
that is expected to execute faster for that operation.

Quickening
----------

Quickening is the process of replacing slow instructions with faster variants.

Quickened code has a number of advantages over immutable bytecode:

* It can be changed at runtime.
* It can use super-instructions that span lines and take multiple operands.
* It does not need to handle tracing as it can fallback to the original
  bytecode for that.

In order that tracing can be supported, the quickened instruction format
should match the immutable, user visible, bytecode format:
16-bit instructions of 8-bit opcode followed by 8-bit operand.

Adaptive instructions
---------------------

Each instruction that would benefit from specialization is replaced by an
adaptive version during quickening. For example,
the ``LOAD_ATTR`` instruction would be replaced with ``LOAD_ATTR_ADAPTIVE``.

Each adaptive instruction periodically attempts to specialize itself.

Specialization
--------------

CPython bytecode contains many instructions that represent high-level
operations, and would benefit from specialization. Examples include ``CALL``,
``LOAD_ATTR``, ``LOAD_GLOBAL`` and ``BINARY_ADD``.

By introducing a "family" of specialized instructions for each of these
instructions allows effective specialization,
since each new instruction is specialized to a single task.
Each family will include an "adaptive" instruction, that maintains a counter
and attempts to specialize itself when that counter reaches zero.

Each family will also include one or more specialized instructions that
perform the equivalent of the generic operation much faster provided their
inputs are as expected.
Each specialized instruction will maintain a saturating counter which will
be incremented whenever the inputs are as expected. Should the inputs not
be as expected, the counter will be decremented and the generic operation
will be performed.
If the counter reaches the minimum value, the instruction is de-optimized by
simply replacing its opcode with the adaptive version.

Ancillary data
--------------

Most families of specialized instructions will require more information than
can fit in an 8-bit operand. To do this, a number of 16 bit entries immediately
following the instruction are used to store this data. This is a form of inline
cache, an "inline data cache". Unspecialized, or adaptive, instructions will
use the first entry of this cache as a counter, and simply skip over the others.

Example families of instructions
--------------------------------

LOAD_ATTR
'''''''''

The ``LOAD_ATTR`` instruction loads the named attribute of the object on top of the stack,
then replaces the object on top of the stack with the attribute.


id: pep-0711.rst
content:
PEP: 711
Title: PyBI: a standard format for distributing Python Binaries
Author: Nathaniel J. Smith <njs@pobox.com>
PEP-Delegate: TODO
Discussions-To: https://discuss.python.org/t/pep-711-pybi-a-standard-format-for-distributing-python-binaries/25547
Status: Draft
Type: Standards Track
Topic: Packaging
Created: 06-Apr-2023
Post-History: `06-Apr-2023 <https://discuss.python.org/t/pep-711-pybi-a-standard-format-for-distributing-python-binaries/25547>`__


Abstract
========

âLike wheels, but instead of a pre-built python package, itâs a
pre-built python interpreterâ


Motivation
==========

End goal: Pypi.org has pre-built packages for all Python versions on all
popular platforms, so automated tools can easily grab any of them and
set it up. It becomes quick and easy to try Python prereleases, pin
Python versions in CI, make a temporary environment to reproduce a bug
report that only happens on a specific Python point release, etc.

First step (this PEP): define a standard packaging file format to hold pre-built
Python interpreters, that reuses existing Python packaging standards as much as
possible.


Examples
========

Example pybi builds are available at `pybi.vorpus.org
<https://pybi.vorpus.org>`__. They're zip files, so you can unpack them and poke
around inside if you want to get a feel for how they're laid out.

You can also look at the `tooling I used to create them
<https://github.com/njsmith/pybi-tools>`__.


Specification
=============

Filename
--------

Filename: ``{distribution}-{version}[-{build tag}]-{platform tag}.pybi``

This matches the wheel file format defined in :pep:`427`, except dropping the
``{python tag}`` and ``{abi tag}`` and changing the extension from ``.whl`` â
``.pybi``.

For example:

-  ``cpython-3.9.3-manylinux_2014.pybi``
-  ``cpython-3.10b2-win_amd64.pybi``

Just like for wheels, if a pybi supports multiple platforms, you can
separate them by dots to make a âcompressed tag setâ:

-  ``cpython-3.9.5-macosx_11_0_x86_64.macosx_11_0_arm64.pybi``

(Though in practice this probably wonât be used much, e.g.Â the above
filename is more idiomatically written as
``cpython-3.9.5-macosx_11_0_universal2.pybi``.)


File contents
-------------

A ``.pybi`` file is a zip file, that can be unpacked directly into an
arbitrary location and then used as a self-contained Python environment.
Thereâs no ``.data`` directory or install scheme keys, because the
Python environment knows which install scheme itâs using, so it can just
put things in the right places to start with.

The âarbitrary locationâ part is important: the pybi canât contain any
hardcoded absolute paths. In particular, any preinstalled scripts MUST
NOT embed absolute paths in their shebang lines.

Similar to wheelsâ ``<package>-<version>.dist-info`` directory, the pybi archive
must contain a top-level directory named ``pybi-info/``. (Rationale: calling it
``pybi-info`` instead ``dist-info`` makes sure that tools donât get confused
about which kind of metadata theyâre looking at; leaving off the
``{name}-{version}`` part is fine because only one pybi can be installed into a
given directory.) The ``pybi-info/`` directory contains at least the following
files:

-  ``.../PYBI``: metadata about the archive itself, in the same
   RFC822-ish format as ``METADATA`` and ``WHEEL`` files:

   ::

      Pybi-Version: 1.0
      Generator: {name} {version}
      Tag: {platform tag}
      Tag: {another platform tag}
      Tag: {...and so on...}
      Build: 1   # optional

-  ``.../RECORD``: same as in wheels, except see the note about
   symlinks, below.

-  ``.../METADATA``: In the same format as described in the current core
   metadata spec, except that the following keys are forbidden because
   they donât make sense:

   -  ``Requires-Dist``
   -  ``Provides-Extra``
   -  ``Requires-Python``

   And also there are some new, required keys described below.

Pybi-specific core metadata
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Here's an example of the new ``METADATA`` fields, before we give the full details::

   Pybi-Environment-Marker-Variables: {"implementation_name": "cpython", "implementation_version": "3.10.8", "os_name": "posix", "platform_machine": "x86_64", "platform_system": "Linux", "python_full_version": "3.10.8", "platform_python_implementation": "CPython", "python_version": "3.10", "sys_platform": "linux"}
   Pybi-Paths: {"stdlib": "lib/python3.10", "platstdlib": "lib/python3.10", "purelib": "lib/python3.10/site-packages", "platlib": "lib/python3.10/site-packages", "include": "include/python3.10", "platinclude": "include/python3.10", "scripts": "bin", "data": "."}
   Pybi-Wheel-Tag: cp310-cp310-PLATFORM
   Pybi-Wheel-Tag: cp310-abi3-PLATFORM
   Pybi-Wheel-Tag: cp310-none-PLATFORM
   Pybi-Wheel-Tag: cp39-abi3-PLATFORM
   Pybi-Wheel-Tag: cp38-abi3-PLATFORM
   Pybi-Wheel-Tag: cp37-abi3-PLATFORM
   Pybi-Wheel-Tag: cp36-abi3-PLATFORM
   Pybi-Wheel-Tag: cp35-abi3-PLATFORM
   Pybi-Wheel-Tag: cp34-abi3-PLATFORM
   Pybi-Wheel-Tag: cp33-abi3-PLATFORM
   Pybi-Wheel-Tag: cp32-abi3-PLATFORM
   Pybi-Wheel-Tag: py310-none-PLATFORM
   Pybi-Wheel-Tag: py3-none-PLATFORM
   Pybi-Wheel-Tag: py39-none-PLATFORM
   Pybi-Wheel-Tag: py38-none-PLATFORM
   Pybi-Wheel-Tag: py37-none-PLATFORM
   Pybi-Wheel-Tag: py36-none-PLATFORM
   Pybi-Wheel-Tag: py35-none-PLATFORM
   Pybi-Wheel-Tag: py34-none-PLATFORM
   Pybi-Wheel-Tag: py33-none-PLATFORM
   Pybi-Wheel-Tag: py32-none-PLATFORM
   Pybi-Wheel-Tag: py31-none-PLATFORM
   Pybi-Wheel-Tag: py30-none-PLATFORM
   Pybi-Wheel-Tag: py310-none-any
   Pybi-Wheel-Tag: py3-none-any
   Pybi-Wheel-Tag: py39-none-any
   Pybi-Wheel-Tag: py38-none-any
   Pybi-Wheel-Tag: py37-none-any
   Pybi-Wheel-Tag: py36-none-any
   Pybi-Wheel-Tag: py35-none-any
   Pybi-Wheel-Tag: py34-none-any
   Pybi-Wheel-Tag: py33-none-any
   Pybi-Wheel-Tag: py32-none-any
   Pybi-Wheel-Tag: py31-none-any
   Pybi-Wheel-Tag: py30-none-any

Specification:

-  ``Pybi-Environment-Marker-Variables``: The value of all PEP 508
   environment marker variables that are static across installs of this
   Pybi, as a JSON dict. So for example:

   - ``python_version`` will always be present, because a Python 3.10 package
     always has ``python_version == "3.10"``.

   - ``platform_version`` will generally not be present, because it gives
     detailed information about the OS where Python is running, for example::

       #60-Ubuntu SMP Thu May 6 07:46:32 UTC 2021

     ``platform_release`` has similar issues.

   - ``platform_machine`` will *usually* be present, except for macOS universal2
     pybis: these can potentially be run in either x86-64 or arm64 mode, and we
     don't know which until the interpreter is actually invoked, so we can't
     record it in static metadata.

   **Rationale:** In many cases, this should allow a resolver running on Linux
   to compute package pins for a Python environment on Windows, or vice-versa,
   so long as the resolver has access to the target platformâs .pybi file. (Note
   that ``Requires-Python`` constraints can be checked by using the
   ``python_full_version`` value.) While we have to leave out a few keys
   sometimes, they're either fairly useless (``platform_version``,
   ``platform_release``) or can be reconstructed by the resolver
   (``platform_machine``).

   The markers are also just generally useful information to have
   accessible. For example, if you have a ``pypy3-7.3.2`` pybi, and you
   want to know what version of the Python language that supports, then
   thatâs recorded in the ``python_version`` marker.

   (Note: we may want to deprecate/remove ``platform_version`` and
   ``platform_release``? They're problematic and I can't figure out any cases
   where they're useful. But that's out of scope of this particular PEP.)

-  ``Pybi-Paths``: The install paths needed to install wheels (same keys
   as ``sysconfig.get_paths()``), as relative paths starting at the root
   of the zip file, as a JSON dict.

   These paths MUS


id: pep-0705.rst
content:
PEP: 705
Title: TypedDict: Read-only items
Author: Alice Purcell <alicederyn@gmail.com>
Sponsor: Pablo Galindo <pablogsal@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-705-read-only-typeddict-items/37867
Status: Final
Type: Standards Track
Topic: Typing
Created: 07-Nov-2022
Python-Version: 3.13
Post-History: `30-Sep-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/6FR6RKNUZU4UY6B6RXC2H4IAHKBU3UKV/>`__,
              `02-Nov-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/2P26R4VH2ZCNNNOQCBZWEM4RNF35OXOW/>`__,
              `14-Mar-2023 <https://discuss.python.org/t/pep-705-typedmapping/24827>`__,
              `17-Oct-2023 <https://discuss.python.org/t/pep-705-typeddict-read-only-and-other-keys/36457>`__,
              `04-Nov-2023 <https://discuss.python.org/t/pep-705-read-only-typeddict-items/37867>`__,
Resolution: `29-Feb-2024 <https://discuss.python.org/t/pep-705-read-only-typeddict-items/37867/39>`__

.. canonical-typing-spec:: :ref:`typing:readonly` and
                           :external+py3.13:data:`typing.ReadOnly`

Abstract
========

:pep:`589` defines the structural type :class:`~typing.TypedDict` for dictionaries with a fixed set of keys.
As ``TypedDict`` is a mutable type, it is difficult to correctly annotate methods which accept read-only parameters in a way that doesn't prevent valid inputs.

This PEP proposes a new type qualifier, ``typing.ReadOnly``, to support these usages. It makes no Python grammar changes. Correct usage of read-only keys of TypedDicts is intended to be enforced only by static type checkers, and will not be enforced by Python itself at runtime.

Motivation
==========

Representing structured data using (potentially nested) dictionaries with string keys is a common pattern in Python programs. :pep:`589` allows these values to be type checked when the exact type is known up-front, but it is hard to write read-only code that accepts more specific variants: for instance, where values may be subtypes or restrict a union of possible types. This is an especially common issue when writing APIs for services, which may support a wide range of input structures, and typically do not need to modify their input.

Pure functions
--------------

Consider trying to add type hints to a function ``movie_string``::

    def movie_string(movie: Movie) -> str:
        if movie.get("year") is None:
            return movie["name"]
        else:
            return f'{movie["name"]} ({movie["year"]})'

We could define this ``Movie`` type using a ``TypedDict``::

    from typing import NotRequired, TypedDict

    class Movie(TypedDict):
        name: str
        year: NotRequired[int | None]

But suppose we have another type where year is required::

    class MovieRecord(TypedDict):
        name: str
        year: int

Attempting to pass a ``MovieRecord`` into ``movie_string`` results in the error (using mypy):

.. code-block:: text

    Argument 1 to "movie_string" has incompatible type "MovieRecord"; expected "Movie"

This particular use case should be type-safe, but the type checker correctly stops the
user from passing a ``MovieRecord`` into a ``Movie`` parameter in the general case, because
the ``Movie`` class has mutator methods that could potentially allow the function to break
the type constraints in ``MovieRecord`` (e.g. with ``movie["year"] = None`` or ``del movie["year"]``).
The problem disappears if we don't have mutator methods in ``Movie``. This could be achieved by defining an immutable interface using a :pep:`544` :class:`~typing.Protocol`::

    from typing import Literal, Protocol, overload

    class Movie(Protocol):
        @overload
        def get(self, key: Literal["name"]) -> str: ...

        @overload
        def get(self, key: Literal["year"]) -> int | None: ...

        @overload
        def __getitem__(self, key: Literal["name"]) -> str: ...

        @overload
        def __getitem__(self, key: Literal["year"]) -> int | None: ...

This is very repetitive, easy to get wrong, and is still missing important method definitions like ``__contains__()`` and ``keys()``.

Updating nested dicts
---------------------

The structural typing of ``TypedDict`` is supposed to permit writing update functions that only constrain the types of items they modify::

    class HasTimestamp(TypedDict):
        timestamp: float

    class Logs(TypedDict):
        timestamp: float
        loglines: list[str]

    def update_timestamp(d: HasTimestamp) -> None:
        d["timestamp"] = now()

    def add_logline(logs: Logs, logline: str) -> None:
        logs["loglines"].append(logline)
        update_timestamp(logs)  # Accepted by type checker

However, this no longer works once you start nesting dictionaries::

    class HasTimestampedMetadata(TypedDict):
        metadata: HasTimestamp

    class UserAudit(TypedDict):
        name: str
        metadata: Logs

    def update_metadata_timestamp(d: HasTimestampedMetadata) -> None:
        d["metadata"]["timestamp"] = now()

    def rename_user(d: UserAudit, name: str) -> None:
        d["name"] = name
        update_metadata_timestamp(d)  # Type check error: "metadata" is not of type HasTimestamp

This looks like an error, but is simply due to the (unwanted) ability to overwrite the ``metadata`` item held by the ``HasTimestampedMetadata`` instance with a different ``HasTimestamp`` instance, that may no longer be a ``Logs`` instance.

It is possible to work around this issue with generics (as of Python 3.11), but it is very complicated, requiring a type parameter for every nested dict.


Rationale
=========

These problems can be resolved by removing the ability to update one or more of the items in a ``TypedDict``. This does not mean the items are immutable: a reference to the underlying dictionary could still exist with a different but compatible type in which those items have mutator operations. These items are "read-only", and we introduce a new ``typing.ReadOnly`` type qualifier for this purpose.

The ``movie_string`` function in the first motivating example can then be typed as follows::

    from typing import NotRequired, ReadOnly, TypedDict

    class Movie(TypedDict):
        name: ReadOnly[str]
        year: ReadOnly[NotRequired[int | None]]

    def movie_string(movie: Movie) -> str:
        if movie.get("year") is None:
            return movie["name"]
        else:
            return f'{movie["name"]} ({movie["year"]})'

A mixture of read-only and non-read-only items is permitted, allowing the second motivating example to be correctly annotated::

    class HasTimestamp(TypedDict):
        timestamp: float

    class HasTimestampedMetadata(TypedDict):
        metadata: ReadOnly[HasTimestamp]

    def update_metadata_timestamp(d: HasTimestampedMetadata) -> None:
        d["metadata"]["timestamp"] = now()

    class Logs(HasTimestamp):
        loglines: list[str]

    class UserAudit(TypedDict):
        name: str
        metadata: Logs

    def rename_user(d: UserAudit, name: str) -> None:
        d["name"] = name
        update_metadata_timestamp(d)  # Now OK

In addition to these benefits, by flagging arguments of a function as read-only (by using a ``TypedDict`` like ``Movie`` with read-only items), it makes explicit not just to typecheckers but also to users that the function is not going to modify its inputs, which is usually a desirable property of a function interface.

This PEP proposes making ``ReadOnly`` valid only in a ``TypedDict``. A possible future extension would be to support it in additional contexts, such as in protocols.


Specification
=============

A new ``typing.ReadOnly`` type qualifier is added.

``typing.ReadOnly`` type qualifier
----------------------------------

The ``typing.ReadOnly`` type qualifier is used to indicate that an item declared in a ``TypedDict`` definition may not be mutated (added, modified, or removed)::

    from typing import ReadOnly

    class Band(TypedDict):
  


id: pep-0739.rst
content:
PEP: 739
Title: ``build-details.json`` 1.0 â a static description file for Python build details
Author: Filipe LaÃ­ns <lains@python.org>
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-739-static-description-file-for-build-details-of-python-installations/44572
Status: Accepted
Type: Standards Track
Topic: Packaging
Created: 19-Dec-2023
Python-Version: 3.14
Resolution: https://discuss.python.org/t/44572/90


Abstract
========

This PEP introduces ``build-details.json``, a static description file containing
build details of Python installations.

It includes the definition of version 1.0 of the file format, and defines the
standard location for this file.


Rationale
=========

When introspecting a Python installation, running code is often undesirable or
impossible. Having a static description file makes various build details of the
Python installation available without having to run the interpreter.

This is helpful for use-cases such as cross-compilation, Python launchers, etc.


Scope
=====

``build-details.json`` is an installation-wide file, meaning that it **MUST**
only contain information that is constant across all environments of the Python
installation.

Information specific to a Python environment, such as the ``site-packages``
path, is outside the scope for this file, and the PEP authors expect that a
static environment description file will be introduced via a future PEP.


Specification
=============

Starting from Python 3.14, a file named ``build-details.json`` following the
format specified in this PEP, or a future version, **MUST** be installed in the
platform-independent standard library directory (``stdlib``, eg.
``/usr/lib/python3.14/build-details.json``), **UNLESS** unfeasible due to
technical limitations.


.. attention::

  In addition to the standard location specified by this PEP, the
  ``build-details.json`` file **MAY** also be installed into **additional**
  locations, and under a different name. Notwithstanding, the file **SHOULD**
  still be available at the standard location.

  In actuality, the PEP authors expect future PEPs to define additional
  install locations with better discoverability.


Format
======

The format specification is defined by the JSON Schema definition provided
below, which is rendered in an human-readable format here.

..
    Rendered with https://gist.github.com/FFY00/eb02d9da2870aae547bc579b7e17a145

.. _spec-start:

.. list-table::
    :widths: 25 75

    * - ``$schema``
      - https://json-schema.org/draft/2020-12/schema
    * - ``$id``
      - https://github.com/python/peps/blob/main/peps/pep-0739/python-build-info-v1.0.schema.json
    * - Title
      - build-details.json â a static description file with build
        details of Python installations
    * - Type
      - ``object``
    * - Additional properties
      - **Not allowed**

``schema_version``
------------------

.. list-table::
    :widths: 25 75

    * - Type
      - ``string`` (constant â ``1.0``)
    * - Description
      - Schema version.

        This is a string following the format ``<MAJOR>.<MINOR>``, where
        ``<MAJOR>`` and ``<MINOR>`` are unpadded numbers and represent
        the **major** and **minor** components of the version. Versions
        may be arithmetically compared by interpreting the version
        string as a decimal number.

        For this specification version, this value is constant and
        **MUST** be ``1.0``.

        Future versions of this schema **MUST** use a higher version
        number. Future versions of this schema **MUST NOT** use the same
        **major** version component as other schema version unless its
        specification is deemed backwards-compatible with them â it
        can't change, or extend, any parts of the current specification
        in such a way as the semantics of the interpreted data differ,
        or that data valid under the new specification is invalid under
        the older specification, with the exception of additional
        properties (errors caused by ``additionalProperties``).
    * - Required
      - **True**

``base_prefix``
---------------

.. list-table::
    :widths: 25 75

    * - Type
      - ``string``
    * - Description
      - Base prefix of the Python installation.

        Either an absolute path, or a path relative to directory where
        this file is contained.
    * - Examples
      - ``/usr``, ``../..``, etc.
    * - Required
      - **True**

``base_interpreter``
--------------------

.. list-table::
    :widths: 25 75

    * - Type
      - ``string``
    * - Description
      - The path to the Python interpreter of the base installation.

        Either an absolute path, or a path relative to ``base_prefix``.

        This field **MUST** be present if the installation provides an
        interpreter executable.
    * - Examples
      - - ``/usr/bin/python``
        - ``bin/python``
        - etc.
    * - Required
      - **False**

``platform``
------------

.. list-table::
    :widths: 25 75

    * - Type
      - ``string``
    * - Description
      - System platform string.

        This field **SHOULD** be equivalent to
        ``sysconfig.get_platform()``.
    * - Examples
      - - ``linux-x86_64``
        - etc.
    * - Required
      - **True**

``language``
------------

.. list-table::
    :widths: 25 75

    * - Type
      - ``object``
    * - Description
      - Object containing details related to the Python language
        specification.
    * - Required
      - **True**
    * - Additional properties
      - **Not allowed**

``language.version``
~~~~~~~~~~~~~~~~~~~~

.. list-table::
    :widths: 25 75

    * - Type
      - ``string``
    * - Description
      - String representation the Python language version â a version
        string consisting only of the *major* and *minor* components.

        This field **SHOULD** be equivalent to
        ``sysconfig.get_python_version()``.
    * - Examples
      - ``3.14``, etc.
    * - Required
      - **True**

``language.version_info``
~~~~~~~~~~~~~~~~~~~~~~~~~

.. list-table::
    :widths: 25 75

    * - Type
      - ``object``
    * - Description
      - Object in the format of :py:data:`sys.version_info`.

        This section **SHOULD** be equivalent to
        :py:data:`sys.version_info`.
    * - Examples
      - - ``{'major': 3, 'minor': 14, 'micro': 1, 'releaselevel': 'final', 'serial': 0}``
        - etc.
    * - Required
      - **False**
    * - Additional properties
      - **Not allowed**

``language.version_info.major``
+++++++++++++++++++++++++++++++

.. list-table::
    :widths: 25 75

    * - Type
      - ``number``
    * - Required
      - **True**

``language.version_info.minor``
+++++++++++++++++++++++++++++++

.. list-table::
    :widths: 25 75

    * - Type
      - ``number``
    * - Required
      - **True**

``language.version_info.micro``
+++++++++++++++++++++++++++++++

.. list-table::
    :widths: 25 75

    * - Type
      - ``number``
    * - Required
      - **True**

``language.version_info.releaselevel``
++++++++++++++++++++++++++++++++++++++

.. list-table::
    :widths: 25 75

    * - Type
      - ``string`` (enum â ``alpha``, ``beta``, ``candidate``, ``final``)
    * - Required
      - **True**

``language.version_info.serial``
++++++++++++++++++++++++++++++++

.. list-table::
    :widths: 25 75

    * - Type
      - ``number``
    * - Required
      - **True**

``implementation``
------------------

.. list-table::
    :widths: 25 75

    * - Type
      - ``object``
    * - Description
      - Object containing details related to the Python implementation.

        This section **SHOULD** be equivalent to
        :py:data:`sys.implementation`. It follows the specification defined
        in :pep:`421`, meaning that in addition to the required keys,
        implementation-specific keys can also exist, but must be
        prefixed with an underscore.
    * - Required


id: pep-0513.rst
content:
PEP: 513
Title: A Platform Tag for Portable Linux Built Distributions
Author: Robert T. McGibbon <rmcgibbo@gmail.com>, Nathaniel J. Smith <njs@pobox.com>
BDFL-Delegate: Alyssa Coghlan <ncoghlan@gmail.com>
Discussions-To: distutils-sig@python.org
Status: Superseded
Type: Informational
Topic: Packaging
Created: 19-Jan-2016
Post-History: 19-Jan-2016, 25-Jan-2016, 29-Jan-2016
Superseded-By: 600
Resolution: https://mail.python.org/pipermail/distutils-sig/2016-January/028211.html


Abstract
========

This PEP proposes the creation of a new platform tag for Python package built
distributions, such as wheels, called ``manylinux1_{x86_64,i686}`` with
external dependencies limited to a standardized, restricted subset of
the Linux kernel and core userspace ABI. It proposes that PyPI support
uploading and distributing wheels with this platform tag, and that ``pip``
support downloading and installing these packages on compatible platforms.


Rationale
=========

Currently, distribution of binary Python extensions for Windows and OS X is
straightforward. Developers and packagers build wheels (:pep:`427`, :pep:`491`),
which are
assigned platform tags such as ``win32`` or ``macosx_10_6_intel``, and upload
these wheels to PyPI. Users can download and install these wheels using tools
such as ``pip``.

For Linux, the situation is much more delicate. In general, compiled Python
extension modules built on one Linux distribution will not work on other Linux
distributions, or even on different machines running the same Linux
distribution with different system libraries installed.

Build tools using :pep:`425` platform tags do not track information about the
particular Linux distribution or installed system libraries, and instead assign
all wheels the too-vague ``linux_i686`` or ``linux_x86_64`` tags. Because of
this ambiguity, there is no expectation that ``linux``-tagged built
distributions compiled on one machine will work properly on another, and for
this reason, PyPI has not permitted the uploading of wheels for Linux.

It would be ideal if wheel packages could be compiled that would work on *any*
linux system. But, because of the incredible diversity of Linux systems -- from
PCs to Android to embedded systems with custom libcs -- this cannot
be guaranteed in general.

Instead, we define a standard subset of the kernel+core userspace ABI that,
in practice, is compatible enough that packages conforming to this standard
will work on *many* linux systems, including essentially all of the desktop
and server distributions in common use. We know this because there are
companies who have been distributing such widely-portable pre-compiled Python
extension modules for Linux -- e.g. Enthought with Canopy [4]_ and Continuum
Analytics with Anaconda [5]_.

Building on the compatibility lessons learned from these companies, we thus
define a baseline ``manylinux1`` platform tag for use by binary Python
wheels, and introduce the implementation of preliminary tools to aid in the
construction of these ``manylinux1`` wheels.


Key Causes of Inter-Linux Binary Incompatibility
================================================

To properly define a standard that will guarantee that wheel packages meeting
this specification will operate on *many* linux platforms, it is necessary to
understand the root causes which often prevent portability of pre-compiled
binaries on Linux. The two key causes are dependencies on shared libraries
which are not present on users' systems, and dependencies on particular
versions of certain core libraries like ``glibc``.


External Shared Libraries
-------------------------

Most desktop and server linux distributions come with a system package manager
(examples include ``APT`` on Debian-based systems, ``yum`` on
``RPM``-based systems, and ``pacman`` on Arch linux) that manages, among other
responsibilities, the installation of shared libraries installed to system
directories such as ``/usr/lib``. Most non-trivial Python extensions will depend
on one or more of these shared libraries, and thus function properly only on
systems where the user has the proper libraries (and the proper
versions thereof), either installed using their package manager, or installed
manually by setting certain environment variables such as ``LD_LIBRARY_PATH``
to notify the runtime linker of the location of the depended-upon shared
libraries.


Versioning of Core Shared Libraries
-----------------------------------

Even if the developers a Python extension module wish to use no
external shared libraries, the modules will generally have a dynamic runtime
dependency on the GNU C library, ``glibc``. While it is possible, statically
linking ``glibc`` is usually a bad idea because certain important C functions
like ``dlopen()`` cannot be called from code that statically links ``glibc``. A
runtime shared library dependency on a system-provided ``glibc`` is unavoidable
in practice.

The maintainers of the GNU C library follow a strict symbol versioning scheme
for backward compatibility. This ensures that binaries compiled against an older
version of ``glibc`` can run on systems that have a newer ``glibc``. The
opposite is generally not true -- binaries compiled on newer Linux
distributions tend to rely upon versioned functions in ``glibc`` that are not
available on older systems.

This generally prevents wheels compiled on the latest Linux distributions
from being portable.


The ``manylinux1`` policy
=========================

For these reasons, to achieve broad portability, Python wheels

* should depend only on an extremely limited set of external shared
  libraries; and
* should depend only on "old" symbol versions in those external shared
  libraries; and
* should depend only on a widely-compatible kernel ABI.

To be eligible for the ``manylinux1`` platform tag, a Python wheel must
therefore both (a) contain binary executables and compiled code that links
*only* to libraries with SONAMEs
included in the following list: ::

    libpanelw.so.5
    libncursesw.so.5
    libgcc_s.so.1
    libstdc++.so.6
    libm.so.6
    libdl.so.2
    librt.so.1
    libc.so.6
    libnsl.so.1
    libutil.so.1
    libpthread.so.0
    libresolv.so.2
    libX11.so.6
    libXext.so.6
    libXrender.so.1
    libICE.so.6
    libSM.so.6
    libGL.so.1
    libgobject-2.0.so.0
    libgthread-2.0.so.0
    libglib-2.0.so.0

and, (b) work on a stock CentOS 5.11 [6]_ system that contains the system
package manager's provided versions of these libraries.

``libcrypt.so.1`` was retrospectively removed from the whitelist after
Fedora 30 was released with ``libcrypt.so.2`` instead.

Because CentOS 5 is only available for x86_64 and i686 architectures,
these are the only architectures currently supported by the ``manylinux1``
policy.

On Debian-based systems, these libraries are provided by the packages ::

    libncurses5 libgcc1 libstdc++6 libc6 libx11-6 libxext6
    libxrender1 libice6 libsm6 libgl1-mesa-glx libglib2.0-0

On RPM-based systems, these libraries are provided by the packages ::

    ncurses libgcc libstdc++ glibc libXext libXrender
    libICE libSM mesa-libGL glib2

This list was compiled by checking the external shared library dependencies of
the Canopy [4]_ and Anaconda [5]_ distributions, which both include a wide array
of the most popular Python modules and have been confirmed in practice to work
across a wide swath of Linux systems in the wild.

Many of the permitted system libraries listed above use symbol versioning
schemes for backward compatibility. The latest symbol versions provided with
the CentOS 5.11 versions of these libraries are: ::

    GLIBC_2.5
    CXXABI_3.4.8
    GLIBCXX_3.4.9
    GCC_4.2.0

Therefore, as a consequence of requirement (b), any wheel that depends on
versioned symbols from the above shared libraries may depend only on symbols
with the following versions: ::

    GLIBC <= 2.5
    CXXABI <= 3.4.8
    GLIBCXX <= 3.4.9
    GCC <= 4.2.0

These recommendations are the outcome of th


id: pep-0275.rst
content:
PEP: 275
Title: Switching on Multiple Values
Author: Marc-AndrÃ© Lemburg <mal@lemburg.com>
Status: Rejected
Type: Standards Track
Created: 10-Nov-2001
Python-Version: 2.6
Post-History:

Rejection Notice
================

A similar PEP for Python 3000, :pep:`3103`, was already rejected,
so this proposal has no chance of being accepted either.

Abstract
========

This PEP proposes strategies to enhance Python's performance
with respect to handling switching on a single variable having
one of multiple possible values.

Problem
=======

Up to Python 2.5, the typical way of writing multi-value switches
has been to use long switch constructs of the following type::

    if x == 'first state':
        ...
    elif x == 'second state':
        ...
    elif x == 'third state':
        ...
    elif x == 'fourth state':
        ...
    else:
        # default handling
        ...

This works fine for short switch constructs, since the overhead of
repeated loading of a local (the variable x in this case) and
comparing it to some constant is low (it has a complexity of O(n)
on average). However, when using such a construct to write a state
machine such as is needed for writing parsers the number of
possible states can easily reach 10 or more cases.

The current solution to this problem lies in using a dispatch
table to find the case implementing method to execute depending on
the value of the switch variable (this can be tuned to have a
complexity of O(1) on average, e.g. by using perfect hash
tables). This works well for state machines which require complex
and lengthy processing in the different case methods. It does not
perform well for ones which only process one or two instructions
per case, e.g.

::

    def handle_data(self, data):
        self.stack.append(data)

A nice example of this is the state machine implemented in
pickle.py which is used to serialize Python objects. Other
prominent cases include XML SAX parsers and Internet protocol
handlers.

Proposed Solutions
==================

This PEP proposes two different but not necessarily conflicting
solutions:

1. Adding an optimization to the Python compiler and VM
   which detects the above if-elif-else construct and
   generates special opcodes for it which use a read-only
   dictionary for storing jump offsets.

2. Adding new syntax to Python which mimics the C style
   switch statement.

The first solution has the benefit of not relying on adding new
keywords to the language, while the second looks cleaner. Both
involve some run-time overhead to assure that the switching
variable is immutable and hashable.

Both solutions use a dictionary lookup to find the right
jump location, so they both share the same problem space in
terms of requiring that both the switch variable and the
constants need to be compatible to the dictionary implementation
(hashable, comparable, a==b => hash(a)==hash(b)).

Solution 1: Optimizing if-elif-else
-----------------------------------

Implementation:

It should be possible for the compiler to detect an
if-elif-else construct which has the following signature::

    if x == 'first':...
    elif x == 'second':...
    else:...

i.e. the left hand side always references the same variable,
the right hand side a hashable immutable builtin type.  The
right hand sides need not be all of the same type, but they
should be comparable to the type of the left hand switch
variable.

The compiler could then setup a read-only (perfect) hash
table, store it in the constants and add an opcode SWITCH in
front of the standard if-elif-else byte code stream which
triggers the following run-time behaviour:

At runtime, SWITCH would check x for being one of the
well-known immutable types (strings, unicode, numbers) and
use the hash table for finding the right opcode snippet. If
this condition is not met, the interpreter should revert to
the standard if-elif-else processing by simply skipping the
SWITCH opcode and proceeding with the usual if-elif-else byte
code stream.


Issues:

The new optimization should not change the current Python
semantics (by reducing the number of ``__cmp__`` calls and adding
``__hash__`` calls in if-elif-else constructs which are affected
by the optimization). To assure this, switching can only
safely be implemented either if a "from __future__" style
flag is used, or the switching variable is one of the builtin
immutable types: int, float, string, unicode, etc. (not
subtypes, since it's not clear whether these are still
immutable or not)

To prevent post-modifications of the jump-table dictionary
(which could be used to reach protected code), the jump-table
will have to be a read-only type (e.g. a read-only
dictionary).

The optimization should only be used for if-elif-else
constructs which have a minimum number of n cases (where n is
a number which has yet to be defined depending on performance
tests).

Solution 2: Adding a switch statement to Python
-----------------------------------------------

New Syntax
''''''''''
::

    switch EXPR:
        case CONSTANT:
            SUITE
        case CONSTANT:
            SUITE
        ...
        else:
            SUITE

(modulo indentation variations)

The "else" part is optional. If no else part is given and
none of the defined cases matches, no action is taken and
the switch statement is ignored. This is in line with the
current if-behaviour. A user who wants to signal this
situation using an exception can define an else-branch
which then implements the intended action.

Note that the constants need not be all of the same type, but
they should be comparable to the type of the switch variable.

Implementation
''''''''''''''

The compiler would have to compile this into byte code
similar to this::

    def whatis(x):
        switch(x):
            case 'one':
                print '1'
            case 'two':
                print '2'
            case 'three':
                print '3'
            else:
                print "D'oh!"

into (omitting POP_TOP's and SET_LINENO's)::

       6  LOAD_FAST         0 (x)
       9  LOAD_CONST        1 (switch-table-1)
      12  SWITCH            26 (to 38)

      14  LOAD_CONST        2 ('1')
      17  PRINT_ITEM
      18  PRINT_NEWLINE
      19  JUMP 43

      22  LOAD_CONST        3 ('2')
      25  PRINT_ITEM
      26  PRINT_NEWLINE
      27  JUMP 43

      30  LOAD_CONST        4 ('3')
      33  PRINT_ITEM
      34  PRINT_NEWLINE
      35  JUMP 43

      38  LOAD_CONST        5 ("D'oh!")
      41  PRINT_ITEM
      42  PRINT_NEWLINE

    >>43  LOAD_CONST        0 (None)
      46  RETURN_VALUE

Where the 'SWITCH' opcode would jump to 14, 22, 30 or 38
depending on 'x'.

Thomas Wouters has written a patch which demonstrates the
above. You can download it from [1]_.

Issues
''''''

The switch statement should not implement fall-through
behaviour (as does the switch statement in C). Each case
defines a complete and independent suite; much like in an
if-elif-else statement. This also enables using break in
switch statements inside loops.

If the interpreter finds that the switch variable x is
not hashable, it should raise a TypeError at run-time
pointing out the problem.

There have been other proposals for the syntax which reuse
existing keywords and avoid adding two new ones ("switch" and
"case"). Others have argued that the keywords should use new
terms to avoid confusion with the C keywords of the same name
but slightly different semantics (e.g. fall-through without
break). Some of the proposed variants::

    case EXPR:
        of CONSTANT:
            SUITE
        of CONSTANT:
            SUITE
        else:
            SUITE

    case EXPR:
        if CONSTANT:
             SUITE
        if CONSTANT:
            SUITE
        else:
            SUITE

    when EXPR:
        in CONSTANT_TUPLE:
            SUITE
        in CONSTANT_TUPLE:
            SUITE
        ...
    else:
         SUITE

The switch statement could be extended to allow multiple
values for one section (e.g.


id: pep-0261.rst
content:
PEP: 261
Title: Support for "wide" Unicode characters
Author: Paul Prescod <paul@prescod.net>
Status: Final
Type: Standards Track
Created: 27-Jun-2001
Python-Version: 2.2
Post-History: 27-Jun-2001


Abstract
========

Python 2.1 unicode characters can have ordinals only up to ``2**16 - 1``.
This range corresponds to a range in Unicode known as the Basic
Multilingual Plane. There are now characters in Unicode that live
on other "planes". The largest addressable character in Unicode
has the ordinal ``17 * 2**16 - 1`` (``0x10ffff``). For readability, we
will call this TOPCHAR and call characters in this range "wide
characters".


Glossary
========

Character
   Used by itself, means the addressable units of a Python
   Unicode string.

Code point
   A code point is an integer between 0 and TOPCHAR.
   If you imagine Unicode as a mapping from integers to
   characters, each integer is a code point. But the
   integers between 0 and TOPCHAR that do not map to
   characters are also code points. Some will someday
   be used for characters. Some are guaranteed never
   to be used for characters.

Codec
   A set of functions for translating between physical
   encodings (e.g. on disk or coming in from a network)
   into logical Python objects.

Encoding
   Mechanism for representing abstract characters in terms of
   physical bits and bytes. Encodings allow us to store
   Unicode characters on disk and transmit them over networks
   in a manner that is compatible with other Unicode software.

Surrogate pair
   Two physical characters that represent a single logical
   character. Part of a convention for representing 32-bit
   code points in terms of two 16-bit code points.

Unicode string
   A Python type representing a sequence of code points with
   "string semantics" (e.g. case conversions, regular
   expression compatibility, etc.) Constructed with the
   ``unicode()`` function.


Proposed Solution
=================

One solution would be to merely increase the maximum ordinal
to a larger value. Unfortunately the only straightforward
implementation of this idea is to use 4 bytes per character.
This has the effect of doubling the size of most Unicode
strings. In order to avoid imposing this cost on every
user, Python 2.2 will allow the 4-byte implementation as a
build-time option. Users can choose whether they care about
wide characters or prefer to preserve memory.

The 4-byte option is called "wide ``Py_UNICODE``". The 2-byte option
is called "narrow ``Py_UNICODE``".

Most things will behave identically in the wide and narrow worlds.

* ``unichr(i)`` for 0 <= i < ``2**16`` (``0x10000``) always returns a
  length-one string.

* ``unichr(i)`` for ``2**16`` <= i <= TOPCHAR will return a
  length-one string on wide Python builds. On narrow builds it will
  raise ``ValueError``.

  **ISSUE**

     Python currently allows ``\U`` literals that cannot be
     represented as a single Python character. It generates two
     Python characters known as a "surrogate pair". Should this
     be disallowed on future narrow Python builds?

  **Pro:**

     Python already the construction of a surrogate pair
     for a large unicode literal character escape sequence.
     This is basically designed as a simple way to construct
     "wide characters" even in a narrow Python build. It is also
     somewhat logical considering that the Unicode-literal syntax
     is basically a short-form way of invoking the unicode-escape
     codec.

  **Con:**

     Surrogates could be easily created this way but the user
     still needs to be careful about slicing, indexing, printing
     etc. Therefore, some have suggested that Unicode
     literals should not support surrogates.


  **ISSUE**

     Should Python allow the construction of characters that do
     not correspond to Unicode code points?  Unassigned Unicode
     code points should obviously be legal (because they could
     be assigned at any time). But code points above TOPCHAR are
     guaranteed never to be used by Unicode. Should we allow access
     to them anyhow?

  **Pro:**

     If a Python user thinks they know what they're doing why
     should we try to prevent them from violating the Unicode
     spec? After all, we don't stop 8-bit strings from
     containing non-ASCII characters.

  **Con:**

     Codecs and other Unicode-consuming code will have to be
     careful of these characters which are disallowed by the
     Unicode specification.

* ``ord()`` is always the inverse of ``unichr()``

* There is an integer value in the sys module that describes the
  largest ordinal for a character in a Unicode string on the current
  interpreter. ``sys.maxunicode`` is ``2**16-1`` (``0xffff``) on narrow builds
  of Python and TOPCHAR on wide builds.

  **ISSUE:**

     Should there be distinct constants for accessing
     TOPCHAR and the real upper bound for the domain of
     ``unichr`` (if they differ)? There has also been a
     suggestion of ``sys.unicodewidth`` which can take the
     values ``'wide'`` and ``'narrow'``.

* every Python Unicode character represents exactly one Unicode code
  point (i.e. Python Unicode Character = Abstract Unicode character).

* codecs will be upgraded to support "wide characters"
  (represented directly in UCS-4, and as variable-length sequences
  in UTF-8 and UTF-16). This is the main part of the implementation
  left to be done.

* There is a convention in the Unicode world for encoding a 32-bit
  code point in terms of two 16-bit code points. These are known
  as "surrogate pairs". Python's codecs will adopt this convention
  and encode 32-bit code points as surrogate pairs on narrow Python
  builds.

  **ISSUE**

     Should there be a way to tell codecs not to generate
     surrogates and instead treat wide characters as
     errors?

  **Pro:**

     I might want to write code that works only with
     fixed-width characters and does not have to worry about
     surrogates.

  **Con:**

     No clear proposal of how to communicate this to codecs.

* there are no restrictions on constructing strings that use
  code points "reserved for surrogates" improperly. These are
  called "isolated surrogates". The codecs should disallow reading
  these from files, but you could construct them using string
  literals or ``unichr()``.


Implementation
==============

There is a new define::

    #define Py_UNICODE_SIZE 2

To test whether UCS2 or UCS4 is in use, the derived macro
``Py_UNICODE_WIDE`` should be used, which is defined when UCS-4 is in
use.

There is a new configure option:

=====================  ============================================
--enable-unicode=ucs2  configures a narrow ``Py_UNICODE``, and uses
                       wchar_t if it fits
--enable-unicode=ucs4  configures a wide ``Py_UNICODE``, and uses
                       wchar_t if it fits
--enable-unicode       same as "=ucs2"
--disable-unicode      entirely remove the Unicode functionality.
=====================  ============================================

It is also proposed that one day ``--enable-unicode`` will just
default to the width of your platforms ``wchar_t``.

Windows builds will be narrow for a while based on the fact that
there have been few requests for wide characters, those requests
are mostly from hard-core programmers with the ability to buy
their own Python and Windows itself is strongly biased towards
16-bit characters.


Notes
=====

This PEP does NOT imply that people using Unicode need to use a
4-byte encoding for their files on disk or sent over the network.
It only allows them to do so. For example, ASCII is still a
legitimate (7-bit) Unicode-encoding.

It has been proposed that there should be a module that handles
surrogates in narrow Python builds for programmers. If someone
wants to implement that, it will be another PEP. It might also be
combined with features that allow other kinds of character-,
word- and line- based indexing.


Rejected Suggestions
====================


id: pep-0507.rst
content:
PEP: 507
Title: Migrate CPython to Git and GitLab
Author: Barry Warsaw <barry@python.org>
Status: Rejected
Type: Process
Created: 30-Sep-2015
Post-History:
Resolution: https://mail.python.org/pipermail/core-workflow/2016-January/000345.html


Abstract
========

This PEP proposes migrating the repository hosting of CPython and the
supporting repositories to Git.  Further, it proposes adopting a
hosted GitLab instance as the primary way of handling merge requests,
code reviews, and code hosting.  It is similar in intent to :pep:`481`
but proposes an open source alternative to GitHub and omits the
proposal to run Phabricator.  As with :pep:`481`, this particular PEP is
offered as an alternative to :pep:`474` and :pep:`462`.


Rationale
=========

CPython is an open source project which relies on a number of
volunteers donating their time.  As with any healthy, vibrant open
source project, it relies on attracting new volunteers as well as
retaining existing developers.  Given that volunteer time is the most
scarce resource, providing a process that maximizes the efficiency of
contributors and reduces the friction for contributions, is of vital
importance for the long-term health of the project.

The current tool chain of the CPython project is a custom and unique
combination of tools.  This has two critical implications:

* The unique nature of the tool chain means that contributors must
  remember or relearn, the process, workflow, and tools whenever they
  contribute to CPython, without the advantage of leveraging long-term
  memory and familiarity they retain by working with other projects in
  the FLOSS ecosystem.  The knowledge they gain in working with
  CPython is unlikely to be applicable to other projects.

* The burden on the Python/PSF infrastructure team is much greater in
  order to continue to maintain custom tools, improve them over time,
  fix bugs, address security issues, and more generally adapt to new
  standards in online software development with global collaboration.

These limitations act as a barrier to contribution both for highly
engaged contributors (e.g. core Python developers) and especially for
more casual "drive-by" contributors, who care more about getting their
bug fix than learning a new suite of tools and workflows.

By proposing the adoption of both a different version control system
and a modern, well-maintained hosting solution, this PEP addresses
these limitations.  It aims to enable a modern, well-understood
process that will carry CPython development for many years.


Version Control System
----------------------

Currently the CPython and supporting repositories use Mercurial.  As a
modern distributed version control system, it has served us well since
the migration from Subversion.  However, when evaluating the VCS we
must consider the capabilities of the VCS itself as well as the
network effect and mindshare of the community around that VCS.

There are really only two real options for this, Mercurial and Git.
The technical capabilities of the two systems are largely equivalent,
therefore this PEP instead focuses on their social aspects.

It is not possible to get exact numbers for the number of projects or
people which are using a particular VCS, however we can infer this by
looking at several sources of information for what VCS projects are
using.

The Open Hub (previously Ohloh) statistics [#openhub-stats]_ show that
37% of the repositories indexed by The Open Hub are using Git (second
only to Subversion which has 48%) while Mercurial has just 2%, beating
only Bazaar which has 1%.  This has Git being just over 18 times as
popular as Mercurial on The Open Hub.

Another source of information on VCS popularity is PyPI itself. This
source is more targeted at the Python community itself since it
represents projects developed for Python.  Unfortunately PyPI does not
have a standard location for representing this information, so this
requires manual processing.  If we limit our search to the top 100
projects on PyPI (ordered by download counts) we can see that 62% of
them use Git, while 22% of them use Mercurial, and 13% use something
else.  This has Git being just under 3 times as popular as Mercurial
for the top 100 projects on PyPI.

These numbers back up the anecdotal evidence for Git as the far more
popular DVCS for open source projects.  Choosing the more popular VCS
has a number of positive benefits.

For new contributors it increases the likelihood that they will have already
learned the basics of Git as part of working with another project or if they
are just now learning Git, that they'll be able to take that knowledge and
apply it to other projects.  Additionally a larger community means more people
writing how to guides, answering questions, and writing articles about Git
which makes it easier for a new user to find answers and information about the
tool they are trying to learn and use.  Given its popularity, there may also
be more auxiliary tooling written *around* Git.  This increases options for
everything from GUI clients, helper scripts, repository hosting, etc.

Further, the adoption of Git as the proposed back-end repository
format doesn't prohibit the use of Mercurial by fans of that VCS!
Mercurial users have the [#hg-git]_ plugin which allows them to push
and pull from a Git server using the Mercurial front-end.  It's a
well-maintained and highly functional plugin that seems to be
well-liked by Mercurial users.


Repository Hosting
------------------

Where and how the official repositories for CPython are hosted is in
someways determined by the choice of VCS.  With Git there are several
options.  In fact, once the repository is hosted in Git, branches can
be mirrored in many locations, within many free, open, and proprietary
code hosting sites.

It's still important for CPython to adopt a single, official
repository, with a web front-end that allows for many convenient and
common interactions entirely through the web, without always requiring
local VCS manipulations.  These interactions include as a minimum,
code review with inline comments, branch diffing, CI integration, and
auto-merging.

This PEP proposes to adopt a [#GitLab]_ instance, run within the
python.org domain, accessible to and with ultimate control from the
PSF and the Python infrastructure team, but donated, hosted, and
primarily maintained by GitLab, Inc.

Why GitLab?  Because it is a fully functional Git hosting system, that
sports modern web interactions, software workflows, and CI
integration.  GitLab's Community Edition (CE) is open source software,
and thus is closely aligned with the principles of the CPython
community.


Code Review
-----------

Currently CPython uses a custom fork of Rietveld modified to not run
on Google App Engine and which is currently only really maintained by
one person.  It is missing common features present in many modern code
review tools.

This PEP proposes to utilize GitLab's built-in merge requests and
online code review features to facilitate reviews of all proposed
changes.


GitLab merge requests
---------------------

The normal workflow for a GitLab hosted project is to submit a *merge request*
asking that a feature or bug fix branch be merged into a target branch,
usually one or more of the stable maintenance branches or the next-version
master branch for new features.  GitLab's merge requests are similar in form
and function to GitHub's pull requests, so anybody who is already familiar
with the latter should be able to immediately utilize the former.

Once submitted, a conversation about the change can be had between the
submitter and reviewer.  This includes both general comments, and inline
comments attached to a particular line of the diff between the source and
target branches.  Projects can also be configured to automatically run
continuous integration on the submitted branch, the results of which are
readily visible from the merge request page.  Thus both the reviewer and
sub


id: pep-0249.rst
content:
PEP: 249
Title: Python Database API Specification v2.0
Author: Marc-AndrÃ© Lemburg <mal@lemburg.com>
Discussions-To: db-sig@python.org
Status: Final
Type: Informational
Created: 12-Apr-1999
Post-History:
Replaces: 248


Introduction
============

This API has been defined to encourage similarity between the Python
modules that are used to access databases.  By doing this, we hope to
achieve a consistency leading to more easily understood modules, code
that is generally more portable across databases, and a broader reach
of database connectivity from Python.

Comments and questions about this specification may be directed to the
`SIG for Database Interfacing with Python <db-sig@python.org>`__.

For more information on database interfacing with Python and available
packages see the `Database Topic Guide
<http://www.python.org/topics/database/>`__.

This document describes the Python Database API Specification 2.0 and
a set of common optional extensions.  The previous version 1.0 version
is still available as reference, in :PEP:`248`. Package writers are
encouraged to use this version of the specification as basis for new
interfaces.


Module Interface
=================

Constructors
------------

Access to the database is made available through connection
objects. The module must provide the following constructor for these:

.. _connect:

`connect`_\ ( *parameters...* )
    Constructor for creating a connection to the database.

    Returns a Connection_ Object. It takes a number of parameters
    which are database dependent. [1]_


Globals
-------

These module globals must be defined:

.. _apilevel:

`apilevel`_
    String constant stating the supported DB API level.

    Currently only the strings "``1.0``" and "``2.0``" are allowed.
    If not given, a DB-API 1.0 level interface should be assumed.


.. _threadsafety:

`threadsafety`_
    Integer constant stating the level of thread safety the interface
    supports.  Possible values are:

    ============ =======================================================
    threadsafety Meaning
    ============ =======================================================
               0 Threads may not share the module.
               1 Threads may share the module, but not connections.
               2 Threads may share the module and connections.
               3 Threads may share the module, connections and cursors.
    ============ =======================================================

    Sharing in the above context means that two threads may use a
    resource without wrapping it using a mutex semaphore to implement
    resource locking.  Note that you cannot always make external
    resources thread safe by managing access using a mutex: the
    resource may rely on global variables or other external sources
    that are beyond your control.


.. _paramstyle:

`paramstyle`_
    String constant stating the type of parameter marker formatting
    expected by the interface. Possible values are [2]_:

    ============ ==============================================================
    paramstyle   Meaning
    ============ ==============================================================
    ``qmark``    Question mark style, e.g. ``...WHERE name=?``
    ``numeric``  Numeric, positional style, e.g. ``...WHERE name=:1``
    ``named``    Named style, e.g. ``...WHERE name=:name``
    ``format``   ANSI C printf format codes, e.g. ``...WHERE name=%s``
    ``pyformat`` Python extended format codes, e.g.  ``...WHERE name=%(name)s``
    ============ ==============================================================


Exceptions
----------

The module should make all error information available through these
exceptions or subclasses thereof:

.. _Warning:

`Warning`_
    Exception raised for important warnings like data truncations
    while inserting, etc. It must be a subclass of the Python
    ``Exception`` class [10]_ [11]_.


.. _Error:

`Error`_
    Exception that is the base class of all other error
    exceptions. You can use this to catch all errors with one single
    ``except`` statement. Warnings are not considered errors and thus
    should not use this class as base. It must be a subclass of the
    Python ``Exception`` class [10]_.


.. _InterfaceError:

`InterfaceError`_
    Exception raised for errors that are related to the database
    interface rather than the database itself.  It must be a subclass
    of Error_.


.. _DatabaseError:

`DatabaseError`_
    Exception raised for errors that are related to the database.  It
    must be a subclass of Error_.


.. _DataError:

`DataError`_
    Exception raised for errors that are due to problems with the
    processed data like division by zero, numeric value out of range,
    etc. It must be a subclass of DatabaseError_.


.. _OperationalError:

`OperationalError`_
    Exception raised for errors that are related to the database's
    operation and not necessarily under the control of the programmer,
    e.g. an unexpected disconnect occurs, the data source name is not
    found, a transaction could not be processed, a memory allocation
    error occurred during processing, etc.  It must be a subclass of
    DatabaseError_.


.. _IntegrityError:

`IntegrityError`_
    Exception raised when the relational integrity of the database is
    affected, e.g. a foreign key check fails.  It must be a subclass
    of DatabaseError_.


.. _InternalError:

`InternalError`_
    Exception raised when the database encounters an internal error,
    e.g. the cursor is not valid anymore, the transaction is out of
    sync, etc.  It must be a subclass of DatabaseError_.


.. _ProgrammingError:

`ProgrammingError`_
    Exception raised for programming errors, e.g. table not found or
    already exists, syntax error in the SQL statement, wrong number of
    parameters specified, etc.  It must be a subclass of
    DatabaseError_.


.. _NotSupportedError:

`NotSupportedError`_
    Exception raised in case a method or database API was used which
    is not supported by the database, e.g. requesting a
    `.rollback()`_ on a connection that does not support transaction
    or has transactions turned off.  It must be a subclass of
    DatabaseError_.

This is the exception inheritance layout [10]_ [11]_:

.. code-block:: text

    Exception
    |__Warning
    |__Error
       |__InterfaceError
       |__DatabaseError
          |__DataError
          |__OperationalError
          |__IntegrityError
          |__InternalError
          |__ProgrammingError
          |__NotSupportedError

.. Note::
    The values of these exceptions are not defined. They should give the user
    a fairly good idea of what went wrong, though.


.. _Connection:

Connection Objects
==================

Connection objects should respond to the following methods.


Connection methods
------------------

.. .close():
.. _Connection.close:

`.close() <#Connection.close>`_
    Close the connection now (rather than whenever ``.__del__()`` is
    called).

    The connection will be unusable from this point forward; an Error_
    (or subclass) exception will be raised if any operation is
    attempted with the connection. The same applies to all cursor
    objects trying to use the connection.  Note that closing a
    connection without committing the changes first will cause an
    implicit rollback to be performed.


.. _.commit:
.. _.commit():

`.commit`_\ ()
    Commit any pending transaction to the database.

    Note that if the database supports an auto-commit feature, this must be
    initially off. An interface method may be provided to turn it back on.

    Database modules that do not support transactions should implement this
    method with void functionality.


.. _.rollback:
.. _.rollback():

`.rollback`_\ ()
    This method is optional since not all databases provide transaction
    support. [3]_

    In case a database does provide transactions this method causes the
    database to roll back to th


id: pep-0539.rst
content:
PEP: 539
Title: A New C-API for Thread-Local Storage in CPython
Author: Erik M. Bray, Masayuki Yamamoto
BDFL-Delegate: Alyssa Coghlan
Status: Final
Type: Standards Track
Created: 20-Dec-2016
Python-Version: 3.7
Post-History: 16-Dec-2016, 31-Aug-2017, 08-Sep-2017
Resolution: https://mail.python.org/pipermail/python-dev/2017-September/149358.html

Abstract
========

The proposal is to add a new Thread Local Storage (TLS) API to CPython which
would supersede use of the existing TLS API within the CPython interpreter,
while deprecating the existing API.  The new API is named the "Thread
Specific Storage (TSS) API" (see `Rationale for Proposed Solution`_ for the
origin of the name).

Because the existing TLS API is only used internally (it is not mentioned in
the documentation, and the header that defines it, ``pythread.h``, is not
included in ``Python.h`` either directly or indirectly), this proposal
probably only affects CPython, but might also affect other interpreter
implementations (PyPy?) that implement parts of the CPython API.

This is motivated primarily by the fact that the old API uses ``int`` to
represent TLS keys across all platforms, which is neither POSIX-compliant,
nor portable in any practical sense [1]_.

.. note::

    Throughout this document the acronym "TLS" refers to Thread Local
    Storage and should not be confused with "Transportation Layer Security"
    protocols.


Specification
=============

The current API for TLS used inside the CPython interpreter consists of 6
functions::

    PyAPI_FUNC(int) PyThread_create_key(void)
    PyAPI_FUNC(void) PyThread_delete_key(int key)
    PyAPI_FUNC(int) PyThread_set_key_value(int key, void *value)
    PyAPI_FUNC(void *) PyThread_get_key_value(int key)
    PyAPI_FUNC(void) PyThread_delete_key_value(int key)
    PyAPI_FUNC(void) PyThread_ReInitTLS(void)

These would be superseded by a new set of analogous functions::

    PyAPI_FUNC(int) PyThread_tss_create(Py_tss_t *key)
    PyAPI_FUNC(void) PyThread_tss_delete(Py_tss_t *key)
    PyAPI_FUNC(int) PyThread_tss_set(Py_tss_t *key, void *value)
    PyAPI_FUNC(void *) PyThread_tss_get(Py_tss_t *key)

The specification also adds a few new features:

* A new type ``Py_tss_t``--an opaque type the definition of which may
  depend on the underlying TLS implementation.  It is defined::

      typedef struct {
          int _is_initialized;
          NATIVE_TSS_KEY_T _key;
      } Py_tss_t;

  where ``NATIVE_TSS_KEY_T`` is a macro whose value depends on the
  underlying native TLS implementation (e.g. ``pthread_key_t``).

* An initializer for ``Py_tss_t`` variables, ``Py_tss_NEEDS_INIT``.

* Three new functions::

      PyAPI_FUNC(Py_tss_t *) PyThread_tss_alloc(void)
      PyAPI_FUNC(void) PyThread_tss_free(Py_tss_t *key)
      PyAPI_FUNC(int) PyThread_tss_is_created(Py_tss_t *key)

  The first two are needed for dynamic (de-)allocation of a ``Py_tss_t``,
  particularly in extension modules built with ``Py_LIMITED_API``, where
  static allocation of this type is not possible due to its implementation
  being opaque at build time.  A value returned by ``PyThread_tss_alloc`` is
  in the same state as a value initialized with ``Py_tss_NEEDS_INIT``, or
  ``NULL`` in the case of dynamic allocation failure.  The behavior of
  ``PyThread_tss_free`` involves calling ``PyThread_tss_delete``
  preventively, or is a no-op if the value pointed to by the ``key``
  argument is ``NULL``.  ``PyThread_tss_is_created`` returns non-zero if the
  given ``Py_tss_t`` has been initialized (i.e. by ``PyThread_tss_create``).

The new TSS API does not provide functions which correspond to
``PyThread_delete_key_value`` and ``PyThread_ReInitTLS``, because these
functions were needed only for CPython's now defunct built-in TLS
implementation; that is the existing behavior of these functions is treated
as follows: ``PyThread_delete_key_value(key)`` is equivalent to
``PyThread_set_key_value(key, NULL)``, and ``PyThread_ReInitTLS()`` is a
no-op [8]_.

The new ``PyThread_tss_`` functions are almost exactly analogous to their
original counterparts with a few minor differences:  Whereas
``PyThread_create_key`` takes no arguments and returns a TLS key as an
``int``, ``PyThread_tss_create`` takes a ``Py_tss_t*`` as an argument and
returns an ``int`` status code. The behavior of ``PyThread_tss_create`` is
undefined if the value pointed to by the ``key`` argument is not initialized
by ``Py_tss_NEEDS_INIT``. The returned status code is zero on success
and non-zero on failure.  The meanings of non-zero status codes are not
otherwise defined by this specification.

Similarly the other ``PyThread_tss_`` functions are passed a ``Py_tss_t*``
whereas previously the key was passed by value.  This change is necessary, as
being an opaque type, the ``Py_tss_t`` type could hypothetically be almost
any size.  This is especially necessary for extension modules built with
``Py_LIMITED_API``, where the size of the type is not known.  Except for
``PyThread_tss_free``, the behaviors of ``PyThread_tss_`` are undefined if the
value pointed to by the ``key`` argument is ``NULL``.

Moreover, because of the use of ``Py_tss_t`` instead of ``int``, there are
behaviors in the new API which differ from the existing API with regard to
key creation and deletion.  ``PyThread_tss_create`` can be called repeatedly
on the same key--calling it on an already initialized key is a no-op and
immediately returns success. Similarly for calling ``PyThread_tss_delete``
with an uninitialized key.

The behavior of ``PyThread_tss_delete`` is defined to change the key's
initialization state to "uninitialized"--this allows, for example,
statically allocated keys to be reset to a sensible state when restarting
the CPython interpreter without terminating the process (e.g. embedding
Python in an application) [12]_.

The old ``PyThread_*_key*`` functions will be marked as deprecated in the
documentation, but will not generate runtime deprecation warnings.

Additionally, on platforms where ``sizeof(pthread_key_t) != sizeof(int)``,
``PyThread_create_key`` will return immediately with a failure status, and
the other TLS functions will all be no-ops on such platforms.

Comparison of API Specification
-------------------------------

=================  =============================  =============================
API                Thread Local Storage (TLS)     Thread Specific Storage (TSS)
=================  =============================  =============================
Version            Existing                       New
Key Type           ``int``                        ``Py_tss_t`` (opaque type)
Handle Native Key  cast to ``int``                conceal into internal field
Function Argument  ``int``                        ``Py_tss_t *``
Features           - create key                   - create key
                   - delete key                   - delete key
                   - set value                    - set value
                   - get value                    - get value
                   - delete value                 - (set ``NULL`` instead) [8]_
                   - reinitialize keys (after     - (unnecessary) [8]_
                     fork)
                                                  - dynamically (de-)allocate
                                                    key
                                                  - check key's initialization
                                                    state
Key Initializer    (``-1`` as key creation        ``Py_tss_NEEDS_INIT``
                   failure)
Requirement        native threads                 native threads
                   (since CPython 3.7 [9]_)
Restriction        No support for platforms       Unable to statically allocate
                   where native TLS key is        keys when ``Py_LIMITED_API``
                   defined in a way that cannot   is defined.
                   be safely cast to ``int``.
=================  =============================  ===


id: pep-0277.rst
content:
PEP: 277
Title: Unicode file name support for Windows NT
Author: Neil Hodgson <neilh@scintilla.org>
Status: Final
Type: Standards Track
Created: 11-Jan-2002
Python-Version: 2.3
Post-History:


Abstract
========

This PEP discusses supporting access to all files possible on
Windows NT by passing Unicode file names directly to the system's
wide-character functions.


Rationale
=========

Python 2.2 on Win32 platforms converts Unicode file names passed
to open and to functions in the ``os`` module into the 'mbcs' encoding
before passing the result to the operating system.  This is often
successful in the common case where the script is operating with
the locale set to the same value as when the file was created.
Most machines are set up as one locale and rarely if ever changed
from this locale.  For some users, locale is changed more often
and on servers there are often files saved by users using
different locales.

On Windows NT and descendent operating systems, including Windows
2000 and Windows XP, wide-character APIs are available that
provide direct access to all file names, including those that are
not representable using the current locale.  The purpose of this
proposal is to provide access to these wide-character APIs through
the standard Python file object and posix module and so provide
access to all files on Windows NT.


Specification
=============

On Windows platforms which provide wide-character file APIs, when
Unicode arguments are provided to file APIs, wide-character calls
are made instead of the standard C library and posix calls.

The Python file object is extended to use a Unicode file name
argument directly rather than converting it.  This affects the
file object constructor ``file(filename[, mode[, bufsize]])`` and also
the ``open`` function which is an alias of this constructor.  When a
Unicode filename argument is used here then the ``name`` attribute of
the file object will be Unicode.  The representation of a file
object, ``repr(f)`` will display Unicode file names as an escaped
string in a similar manner to the representation of Unicode
strings.

The ``posix`` module contains functions that take file or directory
names: ``chdir``, ``listdir``, ``mkdir``, ``open``, ``remove``, ``rename``,
``rmdir``, ``stat``, and ``_getfullpathname``.  These will use Unicode
arguments directly rather than converting them.  For the ``rename`` function, this
behaviour is triggered when either of the arguments is Unicode and
the other argument converted to Unicode using the default
encoding.

The ``listdir`` function currently returns a list of strings.  Under
this proposal, it will return a list of Unicode strings when its
path argument is Unicode.


Restrictions
============

On the consumer Windows operating systems, Windows 95, Windows 98,
and Windows ME, there are no wide-character file APIs so behaviour
is unchanged under this proposal.  It may be possible in the
future to extend this proposal to cover these operating systems as
the VFAT-32 file system used by them does support Unicode file
names but access is difficult and so implementing this would
require much work.  The "Microsoft Layer for Unicode" could be a
starting point for implementing this.

Python can be compiled with the size of Unicode characters set to
4 bytes rather than 2 by defining ``PY_UNICODE_TYPE`` to be a 4 byte
type and ``Py_UNICODE_SIZE`` to be 4.  As the Windows API does not
accept 4 byte characters, the features described in this proposal
will not work in this mode so the implementation falls back to the
current 'mbcs' encoding technique. This restriction could be lifted
in the future by performing extra conversions using
``PyUnicode_AsWideChar`` but for now that would add too much
complexity for a very rarely used feature.


Reference Implementation
========================

The implementation is available at [2]_.


References
==========

[1] Microsoft Windows APIs
\   https://msdn.microsoft.com/

.. [2] https://github.com/python/cpython/issues/37017


Copyright
=========

This document has been placed in the public domain.


id: pep-0511.rst
content:
PEP: 511
Title: API for code transformers
Author: Victor Stinner <vstinner@python.org>
Status: Rejected
Type: Standards Track
Created: 04-Jan-2016
Python-Version: 3.6

Rejection Notice
================

This PEP was rejected by its author.

This PEP was seen as blessing new Python-like programming languages
which are close but incompatible with the regular Python language. It
was decided to not promote syntaxes incompatible with Python.

This PEP was also seen as a nice tool to experiment new Python features,
but it is already possible to experiment them without the PEP, only with
importlib hooks. If a feature becomes useful, it should be directly part
of Python, instead of depending on an third party Python module.

Finally, this PEP was driven was the FAT Python optimization project
which was abandoned in 2016, since it was not possible to show any
significant speedup, but also because of the lack of time to implement
the most advanced and complex optimizations.


Abstract
========

Propose an API to register bytecode and AST transformers. Add also ``-o
OPTIM_TAG`` command line option to change ``.pyc`` filenames, ``-o
noopt`` disables the peephole optimizer. Raise an ``ImportError``
exception on import if the ``.pyc`` file is missing and the code
transformers required to transform the code are missing.  code
transformers are not needed code transformed ahead of time (loaded from
``.pyc`` files).


Rationale
=========

Python does not provide a standard way to transform the code. Projects
transforming the code use various hooks. The MacroPy project uses an
import hook: it adds its own module finder in ``sys.meta_path`` to
hook its AST transformer. Another option is to monkey-patch the
builtin ``compile()`` function. There are even more options to
hook a code transformer.

Python 3.4 added a ``compile_source()`` method to
``importlib.abc.SourceLoader``. But code transformation is wider than
just importing modules, see described use cases below.

Writing an optimizer or a preprocessor is out of the scope of this PEP.

Usage 1: AST optimizer
----------------------

Transforming an Abstract Syntax Tree (AST) is a convenient
way to implement an optimizer. It's easier to work on the AST than
working on the bytecode, AST contains more information and is more high
level.

Since the optimization can done ahead of time, complex but slow
optimizations can be implemented.

Example of optimizations which can be implemented with an AST optimizer:

* `Copy propagation
  <https://en.wikipedia.org/wiki/Copy_propagation>`_:
  replace ``x=1; y=x`` with ``x=1; y=1``
* `Constant folding
  <https://en.wikipedia.org/wiki/Constant_folding>`_:
  replace ``1+1`` with ``2``
* `Dead code elimination
  <https://en.wikipedia.org/wiki/Dead_code_elimination>`_

Using guards (see :pep:`510`), it is possible to
implement a much wider choice of optimizations. Examples:

* Simplify iterable: replace ``range(3)`` with ``(0, 1, 2)`` when used
  as iterable
* `Loop unrolling <https://en.wikipedia.org/wiki/Loop_unrolling>`_
* Call pure builtins: replace ``len("abc")`` with ``3``
* Copy used builtin symbols to constants
* See also `optimizations implemented in fatoptimizer
  <https://fatoptimizer.readthedocs.org/en/latest/optimizations.html>`_,
  a static optimizer for Python 3.6.

The following issues can be implemented with an AST optimizer:

* `Issue #1346238
  <https://bugs.python.org/issue1346238>`_: A constant folding
  optimization pass for the AST
* `Issue #2181 <http://bugs.python.org/issue2181>`_:
  optimize out local variables at end of function
* `Issue #2499 <http://bugs.python.org/issue2499>`_:
  Fold unary + and not on constants
* `Issue #4264 <http://bugs.python.org/issue4264>`_:
  Patch: optimize code to use LIST_APPEND instead of calling list.append
* `Issue #7682 <http://bugs.python.org/issue7682>`_:
  Optimisation of if with constant expression
* `Issue #10399 <https://bugs.python.org/issue10399>`_: AST
  Optimization: inlining of function calls
* `Issue #11549 <http://bugs.python.org/issue11549>`_:
  Build-out an AST optimizer, moving some functionality out of the
  peephole optimizer
* `Issue #17068 <http://bugs.python.org/issue17068>`_:
  peephole optimization for constant strings
* `Issue #17430 <http://bugs.python.org/issue17430>`_:
  missed peephole optimization


Usage 2: Preprocessor
---------------------

A preprocessor can be easily implemented with an AST transformer. A
preprocessor has various and different usages.

Some examples:

* Remove debug code like assertions and logs to make the code faster to
  run it for production.
* `Tail-call Optimization <https://en.wikipedia.org/wiki/Tail_call>`_
* Add profiling code
* `Lazy evaluation <https://en.wikipedia.org/wiki/Lazy_evaluation>`_:
  see `lazy_python <https://github.com/llllllllll/lazy_python>`_
  (bytecode transformer) and `lazy macro of MacroPy
  <https://github.com/lihaoyi/macropy#lazy>`_ (AST transformer)
* Change dictionary literals into collection.OrderedDict instances
* Declare constants: see `@asconstants of codetransformer
  <https://pypi.python.org/pypi/codetransformer>`_
* Domain Specific Language (DSL) like SQL queries. The
  Python language itself doesn't need to be modified. Previous attempts
  to implement DSL for SQL like :pep:`PEP 335 - Overloadable Boolean
  Operators <335>` was rejected.
* Pattern Matching of functional languages
* String Interpolation, but :pep:`498`
  was merged into Python
  3.6.

`MacroPy <https://github.com/lihaoyi/macropy>`_ has a long list of
examples and use cases.

This PEP does not add any new code transformer. Using a code transformer
will require an external module and to register it manually.

See also `PyXfuscator <https://bitbucket.org/namn/pyxfuscator>`_: Python
obfuscator, deobfuscator, and user-assisted decompiler.


Usage 3: Disable all optimization
---------------------------------

Ned Batchelder asked to add an option to disable the peephole optimizer
because it makes code coverage more difficult to implement. See the
discussion on the python-ideas mailing list: `Disable all peephole
optimizations
<https://mail.python.org/pipermail/python-ideas/2014-May/027893.html>`_.

This PEP adds a new ``-o noopt`` command line option to disable the
peephole optimizer. In Python, it's as easy as::

    sys.set_code_transformers([])

It will fix the `Issue #2506 <https://bugs.python.org/issue2506>`_: Add
mechanism to disable optimizations.


Usage 4: Write new bytecode optimizers in Python
------------------------------------------------

Python 3.6 optimizes the code using a peephole optimizer. By
definition, a peephole optimizer has a narrow view of the code and so
can only implement basic optimizations. The optimizer rewrites the
bytecode. It is difficult to enhance it, because it written in C.

With this PEP, it becomes possible to implement a new bytecode optimizer
in pure Python and experiment new optimizations.

Some optimizations are easier to implement on the AST like constant
folding, but optimizations on the bytecode are still useful. For
example, when the AST is compiled to bytecode, useless jumps can be
emitted because the compiler is naive and does not try to optimize
anything.


Use Cases
=========

This section give examples of use cases explaining when and how code
transformers will be used.

Interactive interpreter
-----------------------

It will be possible to use code transformers with the interactive
interpreter which is popular in Python and commonly used to demonstrate
Python.

The code is transformed at runtime and so the interpreter can be slower
when expensive code transformers are used.

Build a transformed package
---------------------------

It will be possible to build a package of the transformed code.

A transformer can have a configuration. The configuration is not stored
in the package.

All ``.pyc`` files of the package must be transformed with the same code
transformers and the same transformers configuration.

It is possi


id: pep-0505.rst
content:
PEP: 505
Title: None-aware operators
Author: Mark E. Haase <mehaase@gmail.com>, Steve Dower <steve.dower@python.org>
Status: Deferred
Type: Standards Track
Created: 18-Sep-2015
Python-Version: 3.8

Abstract
========

Several modern programming languages have so-called "``null``-coalescing" or
"``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift, and PHP
(starting in version 7). There are also stage 3 draft proposals for their
addition to ECMAScript (a.k.a. JavaScript) [3]_ [4]_. These operators provide
syntactic sugar for common patterns involving null references.

* The "``null``-coalescing" operator is a binary operator that returns its left
  operand if it is not ``null``. Otherwise it returns its right operand.
* The "``null``-aware member access" operator accesses an instance member only
  if that instance is non-``null``. Otherwise it returns ``null``. (This is also
  called a "safe navigation" operator.)
* The "``null``-aware index access" operator accesses an element of a collection
  only if that collection is non-``null``. Otherwise it returns ``null``. (This
  is another type of "safe navigation" operator.)

This PEP proposes three ``None``-aware operators for Python, based on the
definitions and other language's implementations of those above. Specifically:

* The "``None`` coalescing" binary operator ``??`` returns the left hand side
  if it evaluates to a value that is not ``None``, or else it evaluates and
  returns the right hand side. A coalescing ``??=`` augmented assignment
  operator is included.
* The "``None``-aware attribute access" operator ``?.`` ("maybe dot") evaluates
  the complete expression if the left hand side evaluates to a value that is
  not ``None``
* The "``None``-aware indexing" operator ``?[]`` ("maybe subscript") evaluates
  the complete expression if the left hand site evaluates to a value that is
  not ``None``

See the `Grammar changes`_ section for specifics and examples of the required
grammar changes.

See the `Examples`_ section for more realistic examples of code that could be
updated to use the new operators.

Syntax and Semantics
====================

Specialness of ``None``
-----------------------

The ``None`` object denotes the lack of a value. For the purposes of these
operators, the lack of a value indicates that the remainder of the expression
also lacks a value and should not be evaluated.

A rejected proposal was to treat any value that evaluates as "false" in a
Boolean context as not having a value. However, the purpose of these operators
is to propagate the "lack of value" state, rather than the "false" state.

Some argue that this makes ``None`` special. We contend that ``None`` is
already special, and that using it as both the test and the result of these
operators does not change the existing semantics in any way.

See the `Rejected Ideas`_ section for discussions on alternate approaches.

Grammar changes
---------------

The following rules of the Python grammar are updated to read::

    augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' | '^=' |
                '<<=' | '>>=' | '**=' | '//=' | '??=')

    power: coalesce ['**' factor]
    coalesce: atom_expr ['??' factor]
    atom_expr: ['await'] atom trailer*
    trailer: ('(' [arglist] ')' |
              '[' subscriptlist ']' |
              '?[' subscriptlist ']' |
              '.' NAME |
              '?.' NAME)

The coalesce rule
*****************

The ``coalesce`` rule provides the ``??`` binary operator. Unlike most binary
operators, the right-hand side is not evaluated until the left-hand side is
determined to be ``None``.

The ``??`` operator binds more tightly than other binary operators as most
existing implementations of these do not propagate ``None`` values (they will
typically raise ``TypeError``). Expressions that are known to potentially
result in ``None`` can be substituted for a default value without needing
additional parentheses.

Some examples of how implicit parentheses are placed when evaluating operator
precedence in the presence of the ``??`` operator::

    a, b = None, None
    def c(): return None
    def ex(): raise Exception()

    (a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3))
    (a * b ?? c // d) == a * (b ?? c) // d
    (a ?? True and b ?? False) == (a ?? True) and (b ?? False)
    (c() ?? c() ?? True) == True
    (True ?? ex()) == True
    (c ?? ex)() == c()

Particularly for cases such as ``a ?? 2 ** b ?? 3``, parenthesizing the
sub-expressions any other way would result in ``TypeError``, as ``int.__pow__``
cannot be called with ``None`` (and the fact that the ``??`` operator is used
at all implies that ``a`` or ``b`` may be ``None``). However, as usual,
while parentheses are not required they should be added if it helps improve
readability.

An augmented assignment for the ``??`` operator is also added. Augmented
coalescing assignment only rebinds the name if its current value is ``None``.
If the target name already has a value, the right-hand side is not evaluated.
For example::

    a = None
    b = ''
    c = 0

    a ??= 'value'
    b ??= undefined_name
    c ??= shutil.rmtree('/')    # don't try this at home, kids

    assert a == 'value'
    assert b == ''
    assert c == 0 and any(os.scandir('/'))

The maybe-dot and maybe-subscript operators
*******************************************

The maybe-dot and maybe-subscript operators are added as trailers for atoms,
so that they may be used in all the same locations as the regular operators,
including as part of an assignment target (more details below). As the
existing evaluation rules are not directly embedded in the grammar, we specify
the required changes below.

Assume that the ``atom`` is always successfully evaluated. Each ``trailer`` is
then evaluated from left to right, applying its own parameter (either its
arguments, subscripts or attribute name) to produce the value for the next
``trailer``. Finally, if present, ``await`` is applied.

For example, ``await a.b(c).d[e]`` is currently parsed as
``['await', 'a', '.b', '(c)', '.d', '[e]']`` and evaluated::

    _v = a
    _v = _v.b
    _v = _v(c)
    _v = _v.d
    _v = _v[e]
    await _v

When a ``None``-aware operator is present, the left-to-right evaluation may be
short-circuited. For example, ``await a?.b(c).d?[e]`` is evaluated::

    _v = a
    if _v is not None:
        _v = _v.b
        _v = _v(c)
        _v = _v.d
        if _v is not None:
            _v = _v[e]
    await _v

.. note::
    ``await`` will almost certainly fail in this context, as it would in
    the case where code attempts ``await None``. We are not proposing to add a
    ``None``-aware ``await`` keyword here, and merely include it in this
    example for completeness of the specification, since the ``atom_expr``
    grammar rule includes the keyword. If it were in its own rule, we would have
    never mentioned it.

Parenthesised expressions are handled by the ``atom`` rule (not shown above),
which will implicitly terminate the short-circuiting behaviour of the above
transformation. For example, ``(a?.b ?? c).d?.e`` is evaluated as::

    # a?.b
    _v = a
    if _v is not None:
        _v = _v.b

    # ... ?? c
    if _v is None:
        _v = c

    # (...).d?.e
    _v = _v.d
    if _v is not None:
        _v = _v.e

When used as an assignment target, the ``None``-aware operations may only be
used in a "load" context. That is, ``a?.b = 1`` and ``a?[b] = 1`` will raise
``SyntaxError``. Use earlier in the expression (``a?.b.c = 1``) is permitted,
though unlikely to be useful unless combined with a coalescing operation::

    (a?.b ?? d).c = 1

Reading expressions
-------------------

For the maybe-dot and maybe-subscript operators, the intention is that
expressions including these operators should be read and interpreted as for the
regular versions of these operators. In "normal" cases, the end results are
going to be identical between an expression such as ``a?.b?[c]`` and
``a.b[c]


id: pep-0263.rst
content:
PEP: 263
Title: Defining Python Source Code Encodings
Author: Marc-AndrÃ© Lemburg <mal@lemburg.com>,
        Martin von LÃ¶wis <martin@v.loewis.de>
Status: Final
Type: Standards Track
Created: 06-Jun-2001
Python-Version: 2.3
Post-History:


Abstract
========

This PEP proposes to introduce a syntax to declare the encoding of
a Python source file. The encoding information is then used by the
Python parser to interpret the file using the given encoding. Most
notably this enhances the interpretation of Unicode literals in
the source code and makes it possible to write Unicode literals
using e.g. UTF-8 directly in an Unicode aware editor.


Problem
=======

In Python 2.1, Unicode literals can only be written using the
Latin-1 based encoding "unicode-escape". This makes the
programming environment rather unfriendly to Python users who live
and work in non-Latin-1 locales such as many of the Asian
countries. Programmers can write their 8-bit strings using the
favorite encoding, but are bound to the "unicode-escape" encoding
for Unicode literals.


Proposed Solution
=================

I propose to make the Python source code encoding both visible and
changeable on a per-source file basis by using a special comment
at the top of the file to declare the encoding.

To make Python aware of this encoding declaration a number of
concept changes are necessary with respect to the handling of
Python source code data.


Defining the Encoding
=====================

Python will default to ASCII as standard encoding if no other
encoding hints are given.

To define a source code encoding, a magic comment must
be placed into the source files either as first or second
line in the file, such as::

    # coding=<encoding name>

or (using formats recognized by popular editors)::

    #!/usr/bin/python
    # -*- coding: <encoding name> -*-

or::

    #!/usr/bin/python
    # vim: set fileencoding=<encoding name> :

More precisely, the first or second line must match the following
regular expression::

    ^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+)

The first group of this
expression is then interpreted as encoding name. If the encoding
is unknown to Python, an error is raised during compilation. There
must not be any Python statement on the line that contains the
encoding declaration.  If the first line matches the second line
is ignored.

To aid with platforms such as Windows, which add Unicode BOM marks
to the beginning of Unicode files, the UTF-8 signature
``\xef\xbb\xbf`` will be interpreted as 'utf-8' encoding as well
(even if no magic encoding comment is given).

If a source file uses both the UTF-8 BOM mark signature and a
magic encoding comment, the only allowed encoding for the comment
is 'utf-8'.  Any other encoding will cause an error.


Examples
========

These are some examples to clarify the different styles for
defining the source code encoding at the top of a Python source
file:

1. With interpreter binary and using Emacs style file encoding
   comment::

       #!/usr/bin/python
       # -*- coding: latin-1 -*-
       import os, sys
       ...

       #!/usr/bin/python
       # -*- coding: iso-8859-15 -*-
       import os, sys
       ...

       #!/usr/bin/python
       # -*- coding: ascii -*-
       import os, sys
       ...

2. Without interpreter line, using plain text::

       # This Python file uses the following encoding: utf-8
       import os, sys
       ...

3. Text editors might have different ways of defining the file's
   encoding, e.g.::

       #!/usr/local/bin/python
       # coding: latin-1
       import os, sys
       ...

4. Without encoding comment, Python's parser will assume ASCII
   text::

       #!/usr/local/bin/python
       import os, sys
       ...

5. Encoding comments which don't work:

   1. Missing "coding:" prefix::

          #!/usr/local/bin/python
          # latin-1
          import os, sys
          ...

   2. Encoding comment not on line 1 or 2::

          #!/usr/local/bin/python
          #
          # -*- coding: latin-1 -*-
          import os, sys
          ...

   3. Unsupported encoding::

          #!/usr/local/bin/python
          # -*- coding: utf-42 -*-
          import os, sys
          ...


Concepts
========

The PEP is based on the following concepts which would have to be
implemented to enable usage of such a magic comment:

1. The complete Python source file should use a single encoding.
   Embedding of differently encoded data is not allowed and will
   result in a decoding error during compilation of the Python
   source code.

   Any encoding which allows processing the first two lines in the
   way indicated above is allowed as source code encoding, this
   includes ASCII compatible encodings as well as certain
   multi-byte encodings such as Shift_JIS. It does not include
   encodings which use two or more bytes for all characters like
   e.g. UTF-16. The reason for this is to keep the encoding
   detection algorithm in the tokenizer simple.

2. Handling of escape sequences should continue to work as it does
   now, but with all possible source code encodings, that is
   standard string literals (both 8-bit and Unicode) are subject to
   escape sequence expansion while raw string literals only expand
   a very small subset of escape sequences.

3. Python's tokenizer/compiler combo will need to be updated to
   work as follows:

   1. read the file

   2. decode it into Unicode assuming a fixed per-file encoding

   3. convert it into a UTF-8 byte string

   4. tokenize the UTF-8 content

   5. compile it, creating Unicode objects from the given Unicode data
      and creating string objects from the Unicode literal data
      by first reencoding the UTF-8 data into 8-bit string data
      using the given file encoding

Note that Python identifiers are restricted to the ASCII
subset of the encoding, and thus need no further conversion
after step 4.


Implementation
==============

For backwards-compatibility with existing code which currently
uses non-ASCII in string literals without declaring an encoding,
the implementation will be introduced in two phases:

1. Allow non-ASCII in string literals and comments, by internally
   treating a missing encoding declaration as a declaration of
   "iso-8859-1". This will cause arbitrary byte strings to
   correctly round-trip between step 2 and step 5 of the
   processing, and provide compatibility with Python 2.2 for
   Unicode literals that contain non-ASCII bytes.

   A warning will be issued if non-ASCII bytes are found in the
   input, once per improperly encoded input file.

2. Remove the warning, and change the default encoding to "ascii".

The builtin ``compile()`` API will be enhanced to accept Unicode as
input.  8-bit string input is subject to the standard procedure for
encoding detection as described above.

If a Unicode string with a coding declaration is passed to ``compile()``,
a ``SyntaxError`` will be raised.

SUZUKI Hisao is working on a patch; see [2]_ for details. A patch
implementing only phase 1 is available at [1]_.


Phases
======

Implementation of steps 1 and 2 above were completed in 2.3,
except for changing the default encoding to "ascii".

The default encoding was set to "ascii" in version 2.5.


Scope
=====

This PEP intends to provide an upgrade path from the current
(more-or-less) undefined source code encoding situation to a more
robust and portable definition.


References
==========

.. [1] Phase 1 implementation:
       https://bugs.python.org/issue526840

.. [2] Phase 2 implementation:
       https://bugs.python.org/issue534304

History
=======

- 1.10 and above: see CVS history
- 1.8: Added '.' to the coding RE.
- 1.7: Added warnings to phase 1 implementation. Replaced the
  Latin-1 default encoding with the interpreter's default
  encoding. Added tweaks to ``compile()``.
- 1.4 - 1.6: Minor tweaks
- 1.3: Worked in comments by Martin v. Loewis:
  UTF-8 BOM mark detection, Emacs style magic comment,
  two phase approach


id: pep-0288.rst
content:
PEP: 288
Title: Generators Attributes and Exceptions
Author: Raymond Hettinger <python@rcn.com>
Status: Withdrawn
Type: Standards Track
Created: 21-Mar-2002
Python-Version: 2.5
Post-History:


Abstract
========

This PEP proposes to enhance generators by providing mechanisms for
raising exceptions and sharing data with running generators.


Status
======

This PEP is withdrawn.  The exception raising mechanism was extended
and subsumed into :pep:`343`.  The attribute passing capability
never built a following, did not have a clear implementation,
and did not have a clean way for the running generator to access
its own namespace.


Rationale
=========

Currently, only class based iterators can provide attributes and
exception handling.  However, class based iterators are harder to
write, less compact, less readable, and slower.  A better solution
is to enable these capabilities for generators.

Enabling attribute assignments allows data to be passed to and from
running generators.  The approach of sharing data using attributes
pervades Python.  Other approaches exist but are somewhat hackish
in comparison.

Another evolutionary step is to add a generator method to allow
exceptions to be passed to a generator.  Currently, there is no
clean method for triggering exceptions from outside the generator.
Also, generator exception passing helps mitigate the try/finally
prohibition for generators.  The need is especially acute for
generators needing to flush buffers or close resources upon termination.

The two proposals are backwards compatible and require no new
keywords.  They are being recommended for Python version 2.5.



Specification for Generator Attributes
======================================

Essentially, the proposal is to emulate attribute writing for classes.
The only wrinkle is that generators lack a way to refer to instances of
themselves.  So, the proposal is to provide a function for discovering
the reference.  For example::

    def mygen(filename):
        self = sys.get_generator()
        myfile = open(filename)
        for line in myfile:
            if len(line) < 10:
                continue
            self.pos = myfile.tell()
            yield line.upper()

    g = mygen('sample.txt')
    line1 = g.next()
    print 'Position', g.pos

Uses for generator attributes include:

1. Providing generator clients with extra information (as shown
   above).
2. Externally setting control flags governing generator operation
   (possibly telling a generator when to step in or step over
   data groups).
3. Writing lazy consumers with complex execution states
   (an arithmetic encoder output stream for example).
4. Writing co-routines (as demonstrated in Dr. Mertz's articles [1]_).

The control flow of 'yield' and 'next' is unchanged by this
proposal.  The only change is that data can passed to and from the
generator.  Most of the underlying machinery is already in place,
only the access function needs to be added.



Specification for Generator Exception Passing
=============================================

Add a ``.throw(exception)`` method to the generator interface::

    def logger():
        start = time.time()
        log = []
        try:
            while True:
                log.append(time.time() - start)
                yield log[-1]
        except WriteLog:
            writelog(log)

    g = logger()
    for i in [10,20,40,80,160]:
        testsuite(i)
        g.next()
    g.throw(WriteLog)

There is no existing work-around for triggering an exception
inside a generator.  It is the only case in Python where active
code cannot be excepted to or through.

Generator exception passing also helps address an intrinsic
limitation on generators, the prohibition against their using
try/finally to trigger clean-up code (:pep:`255`).

Note A: The name of the throw method was selected for several
reasons.  Raise is a keyword and so cannot be used as a method
name.  Unlike raise which immediately raises an exception from the
current execution point, throw will first return to the generator
and then raise the exception.  The word throw is suggestive of
putting the exception in another location.  The word throw is
already associated with exceptions in other languages.

Alternative method names were considered: ``resolve()``, ``signal()``,
``genraise()``, ``raiseinto()``, and ``flush()``.  None of these fit as well
as ``throw()``.

Note B:  To keep the ``throw()`` syntax simple only the instance
version of the raise syntax would be supported (no variants for
"``raise string``" or "``raise class, instance``").

Calling ``g.throw(instance)`` would correspond to writing
``raise instance`` immediately after the most recent yield.



References
==========

.. [1] Dr. David Mertz's draft columns for Charming Python
       http://gnosis.cx/publish/programming/charming_python_b5.txt
       http://gnosis.cx/publish/programming/charming_python_b7.txt



Copyright
=========

This document has been placed in the public domain.


id: pep-0713.rst
content:
PEP: 713
Title: Callable Modules
Author: Amethyst Reese <amethyst at n7.gg>
Sponsor: Åukasz Langa <lukasz at python.org>
Discussions-To: https://discuss.python.org/t/pep-713-callable-modules/26127
Status: Rejected
Type: Standards Track
Created: 20-Apr-2023
Python-Version: 3.12
Post-History: `23-Apr-2023 <https://discuss.python.org/t/pep-713-callable-modules/26127>`__
Resolution: https://discuss.python.org/t/26127/86


Rejection Notice
================

The Steering Council didn't feel that there was a compelling reason to
have this PEP, even though it clearly could be done from a consistency
point of view.
If this idea comes up again in the future, this is a useful prior
discussion to refer to.


Abstract
========

Modules are currently not directly callable. Classes can define a ``__call__``
method that makes instance objects callable, but defining a similarly named
function in the global module scope has no effect, and that function can
only be called by importing or referencing it directly as ``module.__call__``.
:pep:`562` added support for :meth:`~object.__getattr__` and :meth:`~object.__dir__` for modules, but
defining ``__getattr__`` to return a value for ``__call__`` still does not
make a module callable.

This PEP proposes support for making modules directly callable by defining
a ``__call__`` object in the module's global namespace, either as a standard
function, or an arbitrary callable object.


Motivation
==========

Many modules have only a single primary interface to their functionality.
In many cases, that interface is a single callable object, where being able
to import and use the module directly as a callable provides a more "Pythonic"
interface for users::

    # user.py

    import fancy

    @fancy
    def func(...):
        ...

Currently, providing this style of interface requires modifying the module
object at runtime to make it callable.

This is commonly done by replacing the module object in ``sys.modules`` with
a callable alternative (such as a function or class instance)::

    # fancy.py

    def fancy(...):
        ...

    sys.modules[__name__] = fancy

This has the effect of making the original module effectively unreachable
without further hooks from the author, even with ``from module import member``.
It also results in a "module" object that is missing all of the special module
attributes, including ``__doc__``, ``__package__``, ``__path__``, etc.

Alternatively, a module author can choose to override the module's ``__class__``
property with a custom type that provides a callable interface::

    # fancy.py

    def fancy(...):
        ...

    class FancyModule(types.ModuleType):
        def __call__(self, ...):
            return fancy(...)

    sys.modules[__name__].__class__ = FancyModule

The downside of either approach is that it not only results in extra
boilerplate, but also results in type checker failures because they don't
recognize that the module is callable at runtime:

.. code-block:: console

    $ mypy user.py
    user.py:3: error: Module not callable  [operator]
    Found 1 error in 1 file (checked 1 source file)


Specification
=============

When a module object is called, and a ``__call__`` object is found (either
as the result of a ``__getattr__`` or ``__dict__`` lookup), then that object
will be called with the given arguments.

If a ``__call__`` object is not found, then a ``TypeError`` will be raised,
matching the existing behavior.

All of these examples would be considered valid, callable modules:

.. code-block:: python

    # hello.py

    def __call__(...):
        pass

.. code-block:: python

    # hello.py

    class Hello:
        pass

    __call__ = Hello

.. code-block:: python

    # hello.py

    def hello():
        pass

    def __getattr__(name):
        if name == "__call__":
            return hello

The first two styles should generally be preferred, as it allows for easier
static analysis from tools like type checkers, though the third form would be
allowed in order to make the implementation more consistent.

The intent is to allow arbitrary callable object to be assigned to the module's
``__call__`` property or returned by the module's ``__getattr__`` method,
enabling module authors to pick the most suitable mechanism for making their
module callable by users.


Backwards Compatibility and Impact on Performance
=================================================

This PEP is not expected to cause any backwards incompatibility. Any modules
that already contain a ``__call__`` object will continue to function the same
as before, though with the additional ability to be called directly. It is
considered unlikely that modules with an existing ``__call__`` object would
depend on the existing behavior of raising ``TypeError`` when called.

Performance implications of this PEP are minimal, as it defines a new interface.
Calling a module would trigger a lookup for the name ``__call__`` on a module
object. Existing workarounds for creating callable modules already depend on
this behavior for generic objects, resulting in similar performance for these
callable modules.

Type checkers will likely need to be updated accordingly to treat modules with
a ``__call__`` object as callable. This should be possible to support in type
checkers when checking code targeted at older Python versions that do not
support callable modules, with the expectation that these modules would also
include one of the workarounds mentioned earlier to make the module callable.


How to Teach This
=================

The documentation for :external+python:ref:`callable types <types>` will
include modules in the list, with a link to :meth:`~object.__call__`.
The :external+python:ref:`callable-types` documentation will include a section
covering callable modules, with example code, similar to the section for
`customizing module attribute access`__.

__ https://docs.python.org/3/reference/datamodel.html#customizing-module-attribute-access


Reference Implementation
========================

The proposed implementation of callable modules is available in
`CPython PR #103742 <https://github.com/python/cpython/pull/103742>`_.


Rejected Ideas
==============

Given the introduction of ``__getattr__`` and ``__dir__``, and the proposal
to enable use of ``__call__``, it was considered if it was worth allowing use
of *all* :external+python:ref:`specialnames` for modules, such as ``__or__``
and ``__iter__``. While this would not be completely undesired, it increases
the potential for backward compatibility concerns, and these other special
methods are likely to provide less utility to library authors in comparison
to ``__call__``.


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0707.rst
content:
PEP: 707
Title: A simplified signature for __exit__ and __aexit__
Author: Irit Katriel <irit@python.org>
Discussions-To: https://discuss.python.org/t/24402
Status: Rejected
Type: Standards Track
Created: 18-Feb-2023
Python-Version: 3.12
Post-History: `02-Mar-2023 <https://discuss.python.org/t/24402/>`__,
Resolution: https://discuss.python.org/t/pep-707-a-simplified-signature-for-exit-and-aexit/24402/46

Rejection Notice
================

`Per the SC <https://discuss.python.org/t/24402/46>`__:

    We discussed the PEP and have decided to reject it. Our thinking was the
    magic and risk of potential breakage didnât warrant the benefits. We are
    totally supportive, though, of exploring a potential context manager v2
    API or ``__leave__``.

Abstract
========

This PEP proposes to make the interpreter accept context managers whose
:meth:`~py3.11:object.__exit__` / :meth:`~py3.11:object.__aexit__` method
takes only a single exception instance,
while continuing to also support the current ``(typ, exc, tb)`` signature
for backwards compatibility.

This proposal is part of an ongoing effort to remove the redundancy of
the 3-item exception representation from the language, a relic of earlier
Python versions which now confuses language users while adding complexity
and overhead to the interpreter.

The proposed implementation uses introspection, which is tailored to the
requirements of this use case. The solution ensures the safety of the new
feature by supporting it only in non-ambiguous cases. In particular, any
signature that *could* accept three arguments is assumed to expect them.

Because reliable introspection of callables is not currently possible in
Python, the solution proposed here is limited in that only the common types
of single-arg callables will be identified as such, while some of the more
esoteric ones will continue to be called with three arguments. This
imperfect solution was chosen among several imperfect alternatives in the
spirit of practicality.  It is my hope that the discussion about this PEP
will explore the other options and lead us to the best way forward, which
may well be to remain with our imperfect status quo.


Motivation
==========

In the past, an exception was represented in many parts of Python by a
tuple of three elements: the type of the exception, its value, and its
traceback.  While there were good reasons for this design at the time,
they no longer hold because the type and traceback can now be reliably
deduced from the exception instance. Over the last few years we saw
several efforts to simplify the representation of exceptions.

Since 3.10 in `CPython PR #70577 <https://github.com/python/cpython/issues/70577>`_,
the :mod:`py3.11:traceback` module's functions accept either a 3-tuple
as described above, or just an exception instance as a single argument.

Internally, the interpreter no longer represents exceptions as a triplet.
This was `removed for the handled exception in 3.11
<https://github.com/python/cpython/pull/30122>`_ and
`for the raised exception in 3.12
<https://github.com/python/cpython/pull/101607>`_. As a consequence,
several APIs that expose the triplet can now be replaced by
simpler alternatives:

.. list-table::
   :header-rows: 1
   :widths: auto

   * -
     - Legacy API
     - Alternative
   * - Get handled exception (Python)
     - :func:`py3.12:sys.exc_info`
     - :func:`py3.12:sys.exception`
   * - Get handled exception (C)
     - :external+py3.12:c:func:`PyErr_GetExcInfo`
     - :external+py3.12:c:func:`PyErr_GetHandledException`
   * - Set handled exception (C)
     - :external+py3.12:c:func:`PyErr_SetExcInfo`
     - :external+py3.12:c:func:`PyErr_SetHandledException`
   * - Get raised exception (C)
     - :external+py3.12:c:func:`PyErr_Fetch`
     - :external+py3.12:c:func:`PyErr_GetRaisedException`
   * - Set raised exception (C)
     - :external+py3.12:c:func:`PyErr_Restore`
     - :external+py3.12:c:func:`PyErr_SetRaisedException`
   * - Construct an exception instance from the 3-tuple (C)
     - :external+py3.12:c:func:`PyErr_NormalizeException`
     - N/A


The current proposal is a step in this process, and considers the way
forward for one more case in which the 3-tuple representation has
leaked to the language. The motivation for all this work is twofold.

Simplify the implementation of the language
-------------------------------------------

The simplification gained by reducing the interpreter's internal
representation of the handled exception to a single object was significant.
Previously, the interpreter needed to push onto/pop
from the stack three items whenever it did anything with exceptions.
This increased stack depth (adding pressure on caches and registers) and
complicated some of the bytecodes. Reducing this to one item
`removed about 100 lines of code <https://github.com/python/cpython/pull/30122>`_
from ``ceval.c`` (the interpreter's eval loop implementation), and it was later
followed by the removal of the ``POP_EXCEPT_AND_RERAISE`` opcode which has
become simple enough to be `replaced by generic stack manipulation instructions
<https://github.com/python/cpython/issues/90360>`_.  Micro-benchmarks showed
`a speedup of about 10% for catching and raising an exception, as well as
for creating generators
<https://github.com/faster-cpython/ideas/issues/106#issuecomment-990172363>`_.
To summarize, removing this redundancy in Python's internals simplified the
interpreter and made it faster.

The performance of invoking ``__exit__``/``__aexit__`` when leaving
a context manager can be also improved by replacing a multi-arg function
call with a single-arg one. Micro-benchmarks showed that entering and exiting
a context manager with single-arg ``__exit__`` is about 13% faster.

Simplify the language itself
----------------------------

One of the reasons for the popularity of Python is its simplicity. The
:func:`py3.11:sys.exc_info` triplet is cryptic for new learners,
and the redundancy in it is confusing for those who do understand it.

It will take multiple releases to get to a point where we can think of
deprecating ``sys.exc_info()``. However, we can relatively quickly reach a
stage where new learners do not need to know about it, or about the 3-tuple
representation, at least until they are maintaining legacy code.

Rationale
=========

The only reason to object today to the removal of the last remaining
appearances of the 3-tuple from the language is the concerns about
disruption that such changes can bring. The goal of this PEP is to propose
a safe, gradual and minimally disruptive way to make this change in the
case of ``__exit__``, and with this to initiate a discussion of our options
for evolving its method signature.

In the case of the :mod:`py3.11:traceback` module's API, evolving the
functions to have a hybrid signature is relatively straightforward and
safe. The functions take one positional and two optional arguments, and
interpret them according to their types. This is safe when sentinels
are used for default values.  The signatures of callbacks, which are
defined by the user's program, are harder to evolve.

The safest option is to make the user explicitly indicate which signature
the callback is expecting, by marking it with an additional attribute or
giving it a different name. For example, we could make the interpreter
look for a ``__leave__`` method on the context manager, and call it with
a single arg if it exists (otherwise, it looks for ``__exit__`` and
continues as it does now). The introspection-based alternative proposed
here intends to make it more convenient for users to write new code,
because they can just use the single-arg version and remain unaware of
the legacy API. However, if the limitations of introspection are found
to be too severe, we should consider an explicit option. Having both
``__exit__`` and ``__leave__`` around for 5-10 years with similar
functionality is not ideal, but it is an option.

Let us now examine the limitati


id: pep-0667.rst
content:
PEP: 667
Title: Consistent views of namespaces
Author: Mark Shannon <mark@hotpy.org>,
        Tian Gao <gaogaotiantian@hotmail.com>
Discussions-To: https://discuss.python.org/t/46631
Status: Final
Type: Standards Track
Created: 30-Jul-2021
Python-Version: 3.13
Post-History: 20-Aug-2021, 22-Feb-2024
Resolution:  `25-Apr-2024 <https://discuss.python.org/t/46631/25>`__

.. canonical-doc:: :external+py3.13:func:`locals`

Abstract
========

In early versions of Python all namespaces, whether in functions,
classes or modules, were all implemented the same way: as a dictionary.

For performance reasons, the implementation of function namespaces was
changed. Unfortunately this meant that accessing these namespaces through
``locals()`` and ``frame.f_locals`` ceased to be consistent and some
odd bugs crept in over the years as threads, generators and coroutines
were added.

This PEP proposes making these namespaces consistent once more.
Modifications to ``frame.f_locals`` will always be visible in
the underlying variables. Modifications to local variables will
immediately be visible in ``frame.f_locals``, and they will be
consistent regardless of threading or coroutines.

The ``locals()`` function will act the same as it does now for class
and modules scopes. For function scopes it will return an instantaneous
snapshot of the underlying ``frame.f_locals`` rather than implicitly
refreshing a single shared dictionary cached on the frame object.

.. _pep-667-motivation:

Motivation
==========

The implementation of ``locals()`` and ``frame.f_locals`` in releases up to and
including Python 3.12 is slow, inconsistent and buggy.
We want to make it faster, consistent, and most importantly fix the bugs.

For example, when attempting to manipulate local variables via frame objects::

    class C:
        x = 1
        sys._getframe().f_locals['x'] = 2
        print(x)

prints ``2``, but::

    def f():
        x = 1
        sys._getframe().f_locals['x'] = 2
        print(x)
    f()

prints ``1``.

This is inconsistent, and confusing. Worse than that, the Python 3.12 behavior can
result in strange `bugs <https://github.com/python/cpython/issues/74929>`__.

With this PEP both examples would print ``2`` as the function level
change would be written directly to the optimized local variables in
the frame rather than to a cached dictionary snapshot.

There are no compensating advantages for the Python 3.12 behavior;
it is unreliable and slow.

The ``locals()`` builtin has its own undesirable behaviours. Refer to :pep:`558`
for additional details on those concerns.


.. _pep-667-rationale:

Rationale
=========

Making the ``frame.f_locals`` attribute a write-through proxy
-------------------------------------------------------------

The Python 3.12 implementation of ``frame.f_locals`` returns a dictionary
that is created on the fly from the array of local variables. The
``PyFrame_LocalsToFast()`` C API is then called by debuggers and trace
functions that want to write their changes back to the array (until
Python 3.11, this API was called implicitly after every trace function
invocation rather than being called explicitly by the trace functions).

This can result in the array and dictionary getting out of sync with
each other. Writes to the ``f_locals`` frame attribute may not show up as
modifications to local variables if ``PyFrame_LocalsToFast()`` is never
called. Writes to local variables can get lost if a dictionary snapshot
created before the variables were modified is written back to the frame
(since *every* known variable stored in the snapshot is written back to
the frame, even if the value stored on the frame had changed since the
snapshot was taken).

By making ``frame.f_locals`` return a view on the
underlying frame, these problems go away. ``frame.f_locals`` is always in
sync with the frame because it is a view of it, not a copy of it.

Making the ``locals()`` builtin return independent snapshots
------------------------------------------------------------

:pep:`558` considered three potential options for standardising the behavior of the
``locals()`` builtin in :term:`optimized scopes <py3.13:optimized scope>`:

* retain the historical behaviour of having each call to ``locals()`` on a given frame
  update a single shared snapshot of the local variables
* make ``locals()`` return write-through proxy instances (similar
  to ``frame.f_locals``)
* make ``locals()`` return genuinely independent snapshots so that
  attempts to change the values of local variables via ``exec()``
  would be *consistently* ignored rather than being accepted in some circumstances

The last option was chosen as the one which could most easily be explained in the
language reference, and memorised by users:

* the ``locals()`` builtin gives an instantaneous snapshot of the local variables in
  optimized scopes, and read/write access in other scopes; and
* ``frame.f_locals`` gives read/write access to the local variables in all scopes,
  including optimized scopes

This approach allows the intent of a piece of code to be clearer than it would be if both
APIs granted full read/write access in optimized scopes, even when write access wasn't
needed or desired. For additional details on this design decision, refer to :pep:`558`,
especially the :ref:`pep-558-motivation` section and :ref:`pep-558-exec-eval-impact`.

This approach is not without its drawbacks, which are covered
in the Backwards Compatibility section below.

Specification
=============

Python API
----------

.. _pep-667-f_locals-spec:

The ``frame.f_locals`` attribute
''''''''''''''''''''''''''''''''

For module and class scopes (including ``exec()`` and ``eval()``
invocations), ``frame.f_locals`` is a direct
reference to the local variable namespace used in code execution.

For function scopes (and other :term:`optimized scopes <py3.13:optimized scope>`)
it will be an instance of a new write-through proxy type that can directly modify
the optimized local variable storage array in the underlying frame, as well as the
contents of any cell references to non-local variables.

The view objects fully implement the ``collections.abc.Mapping`` interface,
and also implement the following mutable mapping operations:

* using assignment to add new key/value pairs
* using assignment to update the value associated with a key
* conditional assignment via the ``setdefault()`` method
* bulk updates via the ``update()`` method

Views of different frames compare unequal even if they have the same contents.

All writes to the ``f_locals`` mapping will be immediately visible
in the underlying variables. All changes to the underlying variables
will be immediately visible in the mapping.

The ``f_locals`` object will be a full mapping, and can have arbitrary
key-value pairs added to it. New names added via the proxies
will be stored in a dedicated shared dictionary stored on the
underlying frame object (so all proxy instances for a given frame
will be able to access any names added this way).

Extra keys (which do not correspond to local variables on the underlying
frame) may be removed as usual with ``del`` statements or the ``pop()``
method.

Using ``del``, or the ``pop()`` method, to remove keys that correspond to local
variables on the underlying frame is NOT supported, and attempting to do so
will raise ``ValueError``.
Local variables can only be set to ``None`` (or some other value) via the proxy,
they cannot be unbound completely.

The ``clear()`` method is NOT implemented on the write-through proxies, as it
is unclear how it should handle the inability to delete entries corresponding
to local variables.

To maintain backwards compatibility, proxy APIs that need to produce a
new mapping (such as ``copy()``) will produce regular builtin ``dict``
instances, rather than write-through proxy instances.

To avoid introducing a circular reference between frame objects and the
write-through proxies, each access to ``frame.f_locals`` returns a *new*
w


id: pep-0101.rst
content:
PEP: 101
Title: Doing Python Releases 101
Author: Barry Warsaw <barry@python.org>, Guido van Rossum <guido@python.org>
Status: Active
Type: Informational
Created: 22-Aug-2001
Post-History:
Replaces: 102

.. highlight:: shell

Abstract
========

Making a Python release is a thrilling and crazy process.  You've heard
the expression "herding cats"?  Imagine trying to also saddle those
purring little creatures up, and ride them into town, with some of their
buddies firmly attached to your bare back, anchored by newly sharpened
claws.  At least they're cute, you remind yourself.

Actually, no, that's a slight exaggeration ð  The Python release
process has steadily improved over the years and now, with the help of our
amazing community, is really not too difficult.  This PEP attempts to
collect, in one place, all the steps needed to make a Python release.
Most of the steps are now automated or guided by automation, so manually
following this list is no longer necessary.

Things You'll Need
==================

As a release manager there are a lot of resources you'll need to access.
Here's a hopefully-complete list.

* A GPG key.

  Python releases before 3.14 are digitally signed with GPG; for these you'll
  need a key, which hopefully will be on the "web of trust" with at least one of
  the other release managers.

  .. note:: GPG instructions in this PEP can be ignored for Python 3.14 and
            later. See :pep:`761` for details.

* A bunch of software:

  * A checkout of the `python/release-tools`_ repo.
    It contains a `requirements.txt
    <https://github.com/python/release-tools/blob/master/requirements.txt>`_
    file that you need to install
    dependencies from first. Afterwards, you can fire up scripts in the
    repo, covered later in this PEP.

  * `blurb <https://github.com/python/blurb>`__, the
    `Misc/NEWS <https://github.com/python/cpython/tree/main/Misc/NEWS.d>`_
    management tool. You can pip install it.

* Access to servers where you will upload files:

  * ``downloads.nyc1.psf.io``, the server that hosts download files; and
  * ``docs.nyc1.psf.io``, the server that hosts the documentation.

* Administrator access to `python/cpython`_.

* An administrator account on `www.python.org`_, including an "API key".

* Write access to the `python/peps`_ repository.

  If you're reading this, you probably already have this--the first
  task of any release manager is to draft the release schedule.  But
  in case you just signed up... sucker!  I mean, uh, congratulations!

* Posting access to `blog.python.org`_, a Blogger-hosted weblog.
  The RSS feed from this blog is used for the 'Python News' section
  on `www.python.org`_.

* A subscription to the super secret release manager mailing list, which may
  or may not be called ``python-cabal``. Bug Barry about this.

* A ``@python.org`` email address that you will use to sign your releases
  with. Ask ``postmaster@`` for an address; you can either get a full
  account, or a redirecting alias + SMTP credentials to send email from
  this address that looks legit to major email providers.

* Be added to the `Python Security Response Team
  <https://www.python.org/dev/security/>`__.

Types of Releases
=================

There are several types of releases you will need to make.  These include:

* ``alpha``
* ``begin beta``, also known as ``beta 1``, also known as ``new branch``
* ``beta 2+``
* ``release candidate 1``
* ``release candidate 2+``
* ``final``
* ``new branch``
* ``begin bugfix mode``
* ``begin security-only mode``
* ``end-of-life``

Some of these release types actually involve more than
one release branch. In particular, a **new branch** is that point in the
release cycle when a new feature release cycle begins.  Under the current
organization of the CPython Git repository, the *main* branch is always
the target for new features.  At some point in the release cycle of the
next feature release, a **new branch** release is made which creates a
new separate branch for stabilization and later maintenance of the
current in-progress feature release (``3.n.0``) and the *main* branch is modified
to build a new version (which will eventually be released as ``3.n+1.0``).
While the **new branch** release step could occur at one of several points
in the release cycle, current practice is for it to occur at feature code
cutoff for the release which is scheduled for the first beta release.

In the descriptions that follow, steps specific to release types are
labeled accordingly, for now, **new branch** and **final**.

How To Make A Release
=====================

Here are the steps taken to make a Python release.  Some steps are more
fuzzy than others because there's little that can be automated (e.g.
writing the NEWS entries).  Where a step is usually performed by An
Expert, the role of that expert is given.  Otherwise, assume the step is
done by the Release Manager (RM), the designated person performing the
release.  The roles and their current experts are:

* RM = Release Manager

  - Hugo van Kemenade <hugo@python.org> (FI)
  - Thomas Wouters <thomas@python.org> (NL)
  - Pablo Galindo Salgado <pablogsal@python.org> (UK)
  - Åukasz Langa <lukasz@python.org> (PL)

* WE = Windows - Steve Dower <steve.dower@python.org>
* ME = Mac - Ned Deily <nad@python.org> (US)
* DE = Docs - Julien Palard <julien@python.org> (Central Europe)

.. note:: It is highly recommended that the RM contact the Experts the day
  before the release.  Because the world is round and everyone lives
  in different timezones, the RM must ensure that the release tag is
  created in enough time for the Experts to cut binary releases.

  You should not make the release public (by updating the website and
  sending announcements) before all experts have updated their bits.
  In rare cases where the expert for Windows or Mac is MIA, you may add
  a message "(Platform) binaries will be provided shortly" and proceed.

As much as possible, the release steps are automated and guided by the
release script, which is available in a separate repository:
`python/release-tools`_.

We use the following conventions in the examples below.  Where a release
number is given, it is of the form ``3.X.YaN``, e.g. 3.13.0a3 for Python 3.13.0
alpha 3, where "a" == alpha, "b" == beta, "rc" == release candidate.

Release tags are named ``v3.X.YaN``.  The branch name for minor release
maintenance branches is ``3.X``.

This helps by performing several automatic editing steps, and guides you
to perform some manual editing steps.

- Log into Discord and join the Python Core Devs server. Ask Thomas
  or Åukasz for an invite.

  You probably need to coordinate with other people around the world.
  This communication channel is where we've arranged to meet.

- Check to see if there are any showstopper bugs.

  Go to https://github.com/python/cpython/issues and look for any open
  bugs that can block this release.  You're looking at two relevant labels:

  `release-blocker`_
      Stops the release dead in its tracks.  You may not
      make any release with any open release blocker bugs.

  `deferred-blocker`_
      Doesn't block this release, but it will block a
      future release.  You may not make a final or
      candidate release with any open deferred blocker
      bugs.

  Review the release blockers and either resolve them, bump them down to
  deferred, or stop the release and ask for community assistance.  If
  you're making a final or candidate release, do the same with any open
  deferred.

- Check the stable buildbots.

  Go to https://buildbot.python.org/all/#/release_status

  Look at the buildbots for the release
  you're making.  Ignore any that are offline (or inform the community so
  they can be restarted).  If what remains are (mostly) green buildbots,
  you're good to go.  If you have non-offline red buildbots, you may want
  to hold up the release until they are fixed.  Review the problems and
  use your judgement, taking into acco


id: pep-0673.rst
content:
PEP: 673
Title: Self Type
Author: Pradeep Kumar Srinivasan <gohanpra@gmail.com>,
        James Hilton-Balfe <gobot1234yt@gmail.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: typing-sig@python.org
Status: Final
Type: Standards Track
Topic: Typing
Created: 10-Nov-2021
Python-Version: 3.11
Post-History: 17-Nov-2021
Resolution: https://mail.python.org/archives/list/python-dev@python.org/thread/J7BWL5KWOPQQK5KFWKENVLXW6UGSPTGI/

.. canonical-typing-spec:: :ref:`typing:self` and
                           :py:data:`typing.Self`

Abstract
========

This PEP introduces a simple and intuitive way to annotate methods that return
an instance of their class. This behaves the same as the ``TypeVar``-based
approach specified in :pep:`484`
but is more concise and easier to follow.

Motivation
==========

A common use case is to write a method that returns an instance of the same
class, usually by returning ``self``.

::

    class Shape:
        def set_scale(self, scale: float):
            self.scale = scale
            return self

    Shape().set_scale(0.5)  # => should be Shape


One way to denote the return type is to specify it as the current class, say,
``Shape``. Using the method makes the type checker infer the type ``Shape``,
as expected.

::

    class Shape:
        def set_scale(self, scale: float) -> Shape:
            self.scale = scale
            return self

    Shape().set_scale(0.5)  # => Shape


However, when we call ``set_scale`` on a subclass of ``Shape``, the type
checker still infers the return type to be ``Shape``. This is problematic in
situations such as the one shown below, where the type checker will return an
error because we are trying to use attributes or methods not present on the
base class.

::

    class Circle(Shape):
        def set_radius(self, r: float) -> Circle:
            self.radius = r
            return self

    Circle().set_scale(0.5)  # *Shape*, not Circle
    Circle().set_scale(0.5).set_radius(2.7)
    # => Error: Shape has no attribute set_radius


The present workaround for such instances is to define a ``TypeVar`` with the
base class as the bound and use it as the annotation for the ``self``
parameter and the return type:

::

    from typing import TypeVar

    TShape = TypeVar("TShape", bound="Shape")

    class Shape:
        def set_scale(self: TShape, scale: float) -> TShape:
            self.scale = scale
            return self


    class Circle(Shape):
        def set_radius(self, radius: float) -> Circle:
            self.radius = radius
            return self

    Circle().set_scale(0.5).set_radius(2.7)  # => Circle

Unfortunately, this is verbose and unintuitive. Because ``self`` is usually
not explicitly annotated, the above solution doesn't immediately come to mind,
and even if it does, it is very easy to go wrong by forgetting either the
bound on the ``TypeVar(bound="Shape")`` or the annotation for ``self``.

This difficulty means that users often give up and either use fallback types
like ``Any`` or just omit the type annotation completely, both of which make
the code less safe.

We propose a more intuitive and succinct way of expressing the above
intention. We introduce a special form ``Self`` that stands for a type
variable bound to the encapsulating class. For situations such as the one
above, the user simply has to annotate the return type as ``Self``:

::

    from typing import Self

    class Shape:
        def set_scale(self, scale: float) -> Self:
            self.scale = scale
            return self


    class Circle(Shape):
        def set_radius(self, radius: float) -> Self:
            self.radius = radius
            return self

By annotating the return type as ``Self``, we no longer have to declare a
``TypeVar`` with an explicit bound on the base class. The return type ``Self``
mirrors the fact that the function returns ``self`` and is easier to
understand.

As in the above example, the type checker will correctly infer the type of
``Circle().set_scale(0.5)`` to be ``Circle``, as expected.

Usage statistics
----------------

We `analyzed
<https://github.com/pradeep90/annotation_collector/#self-type-stats>`_ popular
open-source projects and found that patterns like the above were used about
**40%** as often as popular types like ``dict`` or ``Callable``. For example,
in typeshed alone, such âSelfâ types are used 523 times, compared to 1286 uses
of ``dict`` and 1314 uses of ``Callable`` `as of October 2021
<https://github.com/pradeep90/annotation_collector/#overall-stats-in-typeshed>`_.
This suggests that a ``Self`` type will be used quite often and users will
benefit a lot from the simpler approach above.

Users of Python types have also frequently requested this feature,
both on the `proposal doc
<https://docs.google.com/document/d/1ujuSMXDmSIOJpiZyV7mvBEC8P-y55AgSzXcvhrZciuI/edit?disco=AAAARP_cNdc>`_
and on `GitHub <https://github.com/python/mypy/issues/11871>`_.

Specification
=============

Use in Method Signatures
------------------------

``Self`` used in the signature of a method is treated as if it were a
``TypeVar`` bound to the class.

::

    from typing import Self

    class Shape:
        def set_scale(self, scale: float) -> Self:
            self.scale = scale
            return self

is treated equivalently to:

::

    from typing import TypeVar

    SelfShape = TypeVar("SelfShape", bound="Shape")

    class Shape:
        def set_scale(self: SelfShape, scale: float) -> SelfShape:
            self.scale = scale
            return self

This works the same for a subclass too:

::

    class Circle(Shape):
        def set_radius(self, radius: float) -> Self:
            self.radius = radius
            return self

which is treated equivalently to:

::

    SelfCircle = TypeVar("SelfCircle", bound="Circle")

    class Circle(Shape):
        def set_radius(self: SelfCircle, radius: float) -> SelfCircle:
            self.radius = radius
            return self

One implementation strategy is to simply desugar the former to the latter in a
preprocessing step. If a method uses ``Self`` in its signature, the type of
``self`` within a method will be ``Self``. In other cases, the type of
``self`` will remain the enclosing class.


Use in Classmethod Signatures
-----------------------------

The ``Self`` type annotation is also useful for classmethods that return
an instance of the class that they operate on. For example, ``from_config`` in
the following snippet builds a ``Shape`` object from a given ``config``.

::

    class Shape:
        def __init__(self, scale: float) -> None: ...

        @classmethod
        def from_config(cls, config: dict[str, float]) -> Shape:
            return cls(config["scale"])


However, this means that ``Circle.from_config(...)`` is inferred to return a
value of type ``Shape``, when in fact it should be ``Circle``:

::

    class Circle(Shape):
        def circumference(self) -> float: ...

    shape = Shape.from_config({"scale": 7.0})
    # => Shape

    circle = Circle.from_config({"scale": 7.0})
    # => *Shape*, not Circle

    circle.circumference()
    # Error: `Shape` has no attribute `circumference`


The current workaround for this is unintuitive and error-prone:

::

    Self = TypeVar("Self", bound="Shape")

    class Shape:
        @classmethod
        def from_config(
            cls: type[Self], config: dict[str, float]
        ) -> Self:
            return cls(config["scale"])

We propose using ``Self`` directly:

::

    from typing import Self

    class Shape:
        @classmethod
        def from_config(cls, config: dict[str, float]) -> Self:
            return cls(config["scale"])

This avoids the complicated ``cls: type[Self]`` annotation and the ``TypeVar``
declaration with a ``bound``. Once again, the latter code behaves equivalently
to the former code.

Use in Parameter Types
----------------------

Another use for ``Self`` is to annotate parameters that expect instances of
the cu


id: pep-0698.rst
content:
PEP: 698
Title: Override Decorator for Static Typing
Author: Steven Troxler <steven.troxler@gmail.com>,
        Joshua Xu <jxu425@fb.com>,
        Shannon Zhu <szhu@fb.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra at gmail.com>
Discussions-To: https://discuss.python.org/t/pep-698-a-typing-override-decorator/20839
Status: Final
Type: Standards Track
Topic: Typing
Created: 05-Sep-2022
Python-Version: 3.12
Post-History: `20-May-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/V23I4D6DEOFW4BBPWBMYTHZUOMKR7KQE/>`__,
              `17-Aug-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/7JDW2PKGF6YTERUJGWM3BRP3GDHRFP4O/>`__,
              `11-Oct-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/TOIYZ3SNPBJZDBRU3ZSBREXV2NNHF4KW/>`__,
              `07-Nov-2022 <https://discuss.python.org/t/pep-698-a-typing-override-decorator/20839>`__,
Resolution: https://discuss.python.org/t/pep-698-a-typing-override-decorator/20839/11

.. canonical-typing-spec:: :ref:`typing:override` and
                           :py:func:`@typing.override <typing.override>`

Abstract
========

This PEP proposes adding an ``@override`` decorator to the Python type system.
This will allow type checkers to prevent a class of bugs that occur when a base
class changes methods that are inherited by derived classes.


Motivation
==========

A primary purpose of type checkers is to flag when refactors or changes break
pre-existing semantic structures in the code, so users can identify and make
fixes across their project without doing a manual audit of their code.


Safe Refactoring
----------------

Pythonâs type system does not provide a way to identify call sites that need to
be changed to stay consistent when an overridden function API changes. This
makes refactoring and transforming code more dangerous.

Consider this simple inheritance structure:

.. code-block:: python

    class Parent:
        def foo(self, x: int) -> int:
            return x

    class Child(Parent):
        def foo(self, x: int) -> int:
            return x + 1

    def parent_callsite(parent: Parent) -> None:
        parent.foo(1)

    def child_callsite(child: Child) -> None:
        child.foo(1)


If the overridden method on the superclass is renamed or deleted, type checkers
will only alert us to update call sites that deal with the base type directly.
But the type checker can only see the new code, not the change we made, so it
has no way of knowing that we probably also needed to rename the same method on
child classes.

A type checker will happily accept this code, even though we are likely
introducing bugs:

.. code-block:: python

    class Parent:
        # Rename this method
        def new_foo(self, x: int) -> int:
            return x

    class Child(Parent):
        # This (unchanged) method used to override `foo` but is unrelated to `new_foo`
        def foo(self, x: int) -> int:
            return x + 1

    def parent_callsite(parent: Parent) -> None:
        # If we pass a Child instance weâll now run Parent.new_foo - likely a bug
        parent.new_foo(1)

    def child_callsite(child: Child) -> None:
        # We probably wanted to invoke new_foo here. Instead, we forked the method
        child.foo(1)


This code will type check, but there are two potential sources of bugs:

- If we pass a ``Child`` instance to the ``parent_callsite`` function, it will
  invoke the implementation in ``Parent.new_foo``. rather than ``Child.foo``.
  This is probably a bug - we presumably would not have written ``Child.foo`` in
  the first place if we didnât need custom behavior.
- Our system was likely relying on ``Child.foo``  behaving in a similar way to
  ``Parent.foo``. But unless we catch this early, we have now forked the
  methods, and in future refactors it is likely no one will realize that major
  changes to the behavior of ``new_foo`` likely require updating ``Child.foo`` as
  well, which could lead to major bugs later.

The incorrectly-refactored code is type-safe, but is probably not what we
intended and could cause our system to behave incorrectly. The bug can be
difficult to track down because our new code likely does execute without
throwing exceptions. Tests are less likely to catch the problem, and silent
errors can take longer to track down in production.

We are aware of several production outages in multiple typed codebases caused by
such incorrect refactors. This is our primary motivation for adding an ``@override``
decorator to the type system, which lets developers express the relationship
between ``Parent.foo`` and ``Child.foo`` so that type checkers can detect the problem.


Rationale
=========


Subclass Implementations Become More Explicit
---------------------------------------------

We believe that explicit overrides will make unfamiliar code easier to read than
implicit overrides. A developer reading the implementation of a subclass that
uses ``@override``  can immediately see which methods are overriding
functionality in some base class; without this decorator, the only way to
quickly find out is using a static analysis tool.


Precedent in Other Languages and Runtime Libraries
--------------------------------------------------

Static Override Checks in Other Languages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Many popular programming languages support override checks. For example:

- `C++ has <https://en.cppreference.com/w/cpp/language/override>`_ ``override``.
- `C# has <https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/override/>`_ ``override``.
- `Hack has <https://docs.hhvm.com/hack/attributes/predefined-attributes#__override>`_ ``<<__Override>>``.
- `Java has <https://docs.oracle.com/javase/tutorial/java/IandI/override.html>`_ ``@Override``.
- `Kotlin has <https://kotlinlang.org/docs/inheritance.html#overriding-methods>`_ ``override``.
- `Scala has <https://www.javatpoint.com/scala-method-overriding>`_ ``override``.
- `Swift has <https://docs.swift.org/swift-book/documentation/the-swift-programming-language/inheritance/#Overriding>`_ ``override``.
- `TypeScript has <https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-3.html#override-and-the---noimplicitoverride-flag>`_ ``override``.

Runtime Override Checks in Python
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Today, there is an `Overrides library <https://pypi.org/project/overrides/>`_
that provides decorators ``@overrides`` [sic] and ``@final`` and will enforce
them at runtime.

:pep:`591` added a ``@final`` decorator with the same semantics as those in the
Overrides library. But the override component of the runtime library is not
supported statically at all, which has added some confusion around the
mix/matched support.

Providing support for ``@override`` in static checks would add value because:

- Bugs can be caught earlier, often in-editor.
- Static checks come with no performance overhead, unlike runtime checks.
- Bugs will be caught quickly even in rarely-used modules, whereas with runtime
  checks these might go undetected for a time without automated tests of all
  imports.


Disadvantages
-------------

Using ``@override`` will make code more verbose.


Specification
=============

When type checkers encounter a method decorated with ``@typing.override`` they
should treat it as a type error unless that method is overriding a compatible
method or attribute in some ancestor class.


.. code-block:: python

    from typing import override

    class Parent:
        def foo(self) -> int:
            return 1

        def bar(self, x: str) -> str:
            return x

    class Child(Parent):
        @override
        def foo(self) -> int:
            return 2

        @override
        def baz(self) -> int:  # Type check error: no matching signature in ancestor
            return 1


The ``@override`` decorator should be permitted anywhere a type checker
considers a method to be a valid override, whi


id: pep-0459.rst
content:
PEP: 459
Title: Standard Metadata Extensions for Python Software Packages
Author: Alyssa Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: Alyssa Coghlan <ncoghlan@gmail.com>
Discussions-To: distutils-sig@python.org
Status: Withdrawn
Type: Standards Track
Topic: Packaging
Requires: 426
Created: 11-Nov-2013
Post-History: 21-Dec-2013


PEP Withdrawal
==============

This PEP depends on :pep:`426`, which has itself been withdrawn. See the
PEP Withdrawal section in that PEP for details.

In the meantime, metadata extensions will continue to be handled as they
have been for past examples like ``entry_points.txt``: as additional files
installed into metadata directories alongside the main ``METADATA`` file.


Abstract
========

This PEP describes several standard extensions to the Python metadata.

Like all metadata extensions, each standard extension format is
independently versioned. Changing any of the formats requires an update
to this PEP, but does not require an update to the core packaging metadata.


Standard Extension Namespace
============================

The ``python`` project on the Python Package Index refers to the CPython
reference interpreter. This namespace is used as the namespace for the
standard metadata extensions.

The currently defined standard extensions are:

* ``python.details``
* ``python.project``
* ``python.integrator``
* ``python.exports``
* ``python.commands``
* ``python.constraints``

All standard extensions are currently at version ``1.0``, and thus the
``extension_metadata`` field may be omitted without losing access to any
functionality.


The ``python.details`` extension
================================

The ``python.details`` extension allows for more information to be provided
regarding the software distribution.

The ``python.details`` extension contains four custom subfields:

* ``license``: the copyright license for the distribution
* ``keywords``: package index keywords for the distribution
* ``classifiers``: package index Trove classifiers for the distribution
* ``document_names``: the names of additional metadata files

All of these fields are optional. Automated tools MUST operate correctly if
a distribution does not provide them, including failing cleanly when an
operation depending on one of these fields is requested.


License
-------

A short string summarising the license used for this distribution.

Note that distributions that provide this field should still specify any
applicable license Trove classifiers in the `Classifiers`_ field. Even
when an appropriate Trove classifier is available, the license summary can
be a good way to specify a particular version of that license, or to
indicate any variations or exception to the license.

This field SHOULD contain fewer than 512 characters and MUST contain fewer
than 2048.

This field SHOULD NOT contain any line breaks.

The full license text SHOULD be included as a separate file in the source
archive for the distribution. See `Document names`_ for details.

Example::

    "license": "GPL version 3, excluding DRM provisions"


Keywords
--------

A list of additional keywords to be used to assist searching for the
distribution in a larger catalog.

Example::

    "keywords": ["comfy", "chair", "cushions", "too silly", "monty python"]


Classifiers
-----------

A list of strings, with each giving a single classification value
for the distribution.  Classifiers are described in :pep:`301` [2].

Example::

    "classifiers": [
      "Development Status :: 4 - Beta",
      "Environment :: Console (Text Based)",
      "License :: OSI Approved :: GNU General Public License v3 (GPLv3)"
    ]


Document names
--------------

Filenames for supporting documents included in the distribution's
``dist-info`` metadata directory.

The following supporting documents can be named:

* ``description``: a file containing a long description of the distribution
* ``license``: a file with the full text of the distribution's license
* ``changelog``: a file describing changes made to the distribution

Supporting documents MUST be included directly in the ``dist-info``
directory. Directory separators are NOT permitted in document names.

The markup format (if any) for the file is indicated by the file extension.
This allows index servers and other automated tools to render included
text documents correctly and provide feedback on rendering errors, rather
than having to guess the intended format.

If the filename has no extension, or the extension is not recognised, the
default rendering format MUST be plain text.

The following markup renderers SHOULD be used for the specified file
extensions:

* Plain text: ``.txt``, no extension, unknown extension
* reStructured Text: ``.rst``
* Markdown: ``.md``
* AsciiDoc: ``.adoc``, ``.asc``, ``.asciidoc``
* HTML: ``.html``, ``.htm``

Automated tools MAY render one or more of the specified formats as plain
text and MAY render other markup formats beyond those listed.

Automated tools SHOULD NOT make any assumptions regarding the maximum length
of supporting document content, except as necessary to protect the
integrity of a service.

Example::

    "document_names": {
        "description": "README.rst",
        "license": "LICENSE.rst",
        "changelog": "NEWS"
    }


The ``python.project`` extension
================================

The ``python.project`` extension allows for more information to be provided
regarding the creation and maintenance of the distribution.

The ``python.project`` extension contains three custom subfields:

* ``contacts``: key contact points for the distribution
* ``contributors``: other contributors to the distribution
* ``project_urls``: relevant URLs for the distribution


Contact information
-------------------

Details on individuals and organisations are recorded as mappings with
the following subfields:

* ``name``: the name of an individual or group
* ``email``: an email address (this may be a mailing list)
* ``url``: a URL (such as a profile page on a source code hosting service)
* ``role``: one of ``"author"``, ``"maintainer"`` or ``"contributor"``

The ``name`` subfield is required, the other subfields are optional.

If no specific role is stated, the default is ``contributor``.

Email addresses must be in the form ``local-part@domain`` where the
local-part may be up to 64 characters long and the entire email address
contains no more than 254 characters. The formal specification of the
format is in :rfc:`5322` (sections 3.2.3 and 3.4.1) and :rfc:`5321`, with a more
readable form given in the informational :rfc:`3696` and the associated errata.

The defined contributor roles are as follows:

* ``author``: the original creator of a distribution
* ``maintainer``: the current lead contributor for a distribution, when
  they are not the original creator
* ``contributor``: any other individuals or organizations involved in the
  creation of the distribution

Contact and contributor metadata is optional. Automated tools MUST operate
correctly if a distribution does not provide it, including failing cleanly
when an operation depending on one of these fields is requested.


Contacts
--------

A list of contributor entries giving the recommended contact points for
getting more information about the project.

The example below would be suitable for a project that was in the process
of handing over from the original author to a new lead maintainer, while
operating as part of a larger development group.

Example::

    "contacts": [
      {
        "name": "Python Packaging Authority/Distutils-SIG",
        "email": "distutils-sig@python.org",
        "url": "https://bitbucket.org/pypa/"
      },
      {
        "name": "Samantha C.",
        "role": "maintainer",
        "email": "dontblameme@example.org"
      },
      {
        "name": "Charlotte C.",
        "role": "author",
        "email": "iambecomingasketchcomedian@example.com"
      }
    ]


Contributors
------------

A list of contributor entries for other con


id: pep-8011.rst
content:
PEP: 8011
Title: Python Governance Model Lead by Trio of Pythonistas
Author: Mariatta <mariatta@python.org>, Barry Warsaw <barry@python.org>
Status: Rejected
Type: Informational
Topic: Governance
Created: 24-Aug-2018


Abstract
========

This PEP proposes a governance model for the Core Python development community,
led by a trio of equally authoritative leaders. The Trio of Pythonistas
(ToP, or simply Trio) is tasked with making final decisions for the language.
It differs from :pep:`8010` by specifically not proposing a central singular leader,
but instead a group of three people as the leaders.

This PEP also proposes a formation of specialized workgroups to assist the leadership
trio in making decisions.

This PEP does *not* name the members of the Trio.  Should this model be adopted,
it will be codified in :pep:`13` along with the names of all officeholders
described in this PEP.

This PEP describes:

- The role and responsibilities of the Trio
- Guidelines of how trio members should be formed
- Reasoning of the group of three, instead of a singular leader
- Role and responsibilities of Python core developers to the trio
- Sustainability considerations
- Diversity and inclusivity considerations

PEP Rejection
=============

:pep:`8011` was rejected `by a core developer vote
<https://discuss.python.org/t/python-governance-vote-december-2018-results/546/>`__
described in :pep:`8001` on Monday, December 17, 2018.

:pep:`8016` and the governance model it describes were chosen instead.

Open discussion points
======================

Various tweaks to the parameters of this PEP are allowed during the governance
discussion process, such as the exact responsibilities of the Trio, term lengths
of service, voting procedures, and trio disbandment.
These will be codified by the time the PEP is ready to be voted on.

It is allowed, and perhaps even expected, that as experience is gained with this
model, these parameters may be tweaked in order to provide for a smoother
governing process. The process for tweaking these parameters will generally
be the same voting process as described in :pep:`8001`.


Roles and responsibilities of the leadership trio
=================================================

- Be open, considerate, respectful. In other words, adhering to The PSF's code of conduct.
- Pronounce on PEPs, either as a team, or individually if the other trio members agree.
- Provide vision and leadership for Python, the programming language and the community.
- Understand their own limitation, and seek advice whenever necessary.
- Provide mentorship to the next generation of leaders.
- Be a Python core developer
- Be a voting member of The PSF (one of Contributing / Manager / Fellow / Supporter). [2]_
- Understand that Python is not just a language but also a community. They need
  to be aware of issues in Python not just the technical aspects, but also
  other issues in the community.
- Facilitate the formation of specialized working groups within Core Python.
  See "formation of specialized working groups" section below.
- Set good example of behavior, culture, and tone to Python community.
  Just as Python looks at and learn from other communities for inspiration, other
  communities will look at Python and learn from us.


Authority of the trio
=====================

To be clear, in case any dispute arises: the trio has the final
authority to pronounce on PEPs (except for the governance PEP), to
decide whether a particular decision requires a PEP, and to resolve
technical disputes in general. The trio's authority does not include
changing the governance itself, or other non-technical disputes that
may arise; these should be handled through the process described in
:pep:`8001`.


What are NOT considered as the role responsibilities of the trio
================================================================

The following are not the expected out of the trio, however they can do these if they wish.

- They are not always the ones coming up with all the ideas, vision, problems to
  solve, and what not. The trio will be open and accepting suggestions from core developers
  and community.
- Day to day bug reports do not require the trio to intervene. Any core devs are able
  to make decisions, but will defer to the respective focused workgroups, and
  will eventually defer to the trio when there are major disagreements among core developers.
- Does not run / decide on Python language summit and its logistics.
- Does not run / decide on Python core sprint and its logistics.
- Does not handle CoC cases. Those are responsibilities of the PSF CoC workgroup,
  but will speak out if they witness those cases.
- Does not make decisions about other Python implementations (Cython, IronPython, etc).
- Does not run / decide on Python conferences and its logistics.
- Not an evangelist of Python. The trio is not expected to preach/advertise for
  Python. They can if they want to, but not expected.
- Not an educator of Python. The trio is not expected to be the ones teaching/writing
  about Python. They can if they want to, but not expected.
- The trio is not expected to be available 24/7, 365 days a year. They are free
  to decide for themselves their availability for Python.
- Not a PEP editor.


Guidelines for the formation of the trio
========================================

The success of this governance model relies on the members of the trio, and the
ability of the trio members to collaborate and work well together.

The three people need to have similar vision to Python, and each can have
different skills that complement one another.

With such a team, disagreements and conflict should be rare, but can still happen.
We will need to trust the people we select that they are able to resolve this among
themselves.

When it comes to select the members of the trio, instead of nominating various
individuals and choosing the top three, core developers will nominate trios
and vote for groups of threes who they believe can form this united trio. There
is no restriction that an individual can only be nominated in one slate.

This PEP will not name or nominate anyone into the trio.

Only once this PEP is accepted, any active core developers (who are eligible to vote)
can submit nomination of groups of three.

Once this PEP is accepted and core devs have submitted their nominations, voting
can begin, and the voting mechanism described in :pep:`8001` will be used.

Qualities desired out of the trio:

- Be a Python core developer.
- Be a voting PSF member (one of Contributing / Manager / Fellow / Supporter). [2]_
- Be a member of the community with good standing.
- Adhere to The PSF's code of conduct (Be open, considerate, and respectful). [1]_
- Be willing to accept the said roles and responsibilities.
- Able to communicate and articulate their thoughts effectively.

The following are not requirements when considering someone into the trio:

- "Experience being a BDFL of something" is not a requirement.
- "Be a genius" is not a requirement.

Diversity and inclusivity
-------------------------

The core Python development team fully supports the Python Software Foundationâs
diversity statement, and welcomes participation and contribution from people
from diverse backgrounds. When nominating people to be part of the trio,
Python core developers will take every effort into including members from
underrepresented group into consideration.

Ideally, nomination should include and reflect the diversity of core Python
contributors.

Sustainability
--------------

Lack of employer support or lack of luxury of free time should not be a factor
when identifying who should be in a trio. If there are individuals who the core
devs have identified as having the necessary skills for being a member of the
trio, but they are unable to do it because of lack of time, lack of financial
support, then we should open discussion with The PSF or other parties into
providing the needed support.


Ad


id: pep-0303.rst
content:
PEP: 303
Title: Extend divmod() for Multiple Divisors
Author: Thomas Bellman <bellman+pep-divmod@lysator.liu.se>
Status: Rejected
Type: Standards Track
Created: 31-Dec-2002
Python-Version: 2.3
Post-History:


Abstract
========

This PEP describes an extension to the built-in ``divmod()`` function,
allowing it to take multiple divisors, chaining several calls to
``divmod()`` into one.


Pronouncement
=============

This PEP is rejected.  Most uses for chained ``divmod()`` involve a
constant modulus (in radix conversions for example) and are more
properly coded as a loop.  The example of splitting seconds
into days/hours/minutes/seconds does not generalize to months
and years; rather, the whole use case is handled more flexibly and
robustly by date and time modules.  The other use cases mentioned
in the PEP are somewhat rare in real code.  The proposal is also
problematic in terms of clarity and obviousness.  In the examples,
it is not immediately clear that the argument order is correct or
that the target tuple is of the right length.  Users from other
languages are more likely to understand the standard two argument
form without having to re-read the documentation.  See python-dev
discussion on 17 June 2005 [1]_.


Specification
=============

The built-in ``divmod()`` function would be changed to accept multiple
divisors, changing its signature from ``divmod(dividend, divisor)`` to
``divmod(dividend, *divisors)``.  The dividend is divided by the last
divisor, giving a quotient and a remainder.  The quotient is then
divided by the second to last divisor, giving a new quotient and
remainder.  This is repeated until all divisors have been used,
and ``divmod()`` then returns a tuple consisting of the quotient from
the last step, and the remainders from all the steps.

A Python implementation of the new ``divmod()`` behaviour could look
like::

    def divmod(dividend, *divisors):
        modulos = ()
        q = dividend
        while divisors:
            q, r = q.__divmod__(divisors[-1])
            modulos = (r,) + modulos
            divisors = divisors[:-1]
        return (q,) + modulos


Motivation
==========

Occasionally one wants to perform a chain of ``divmod()`` operations,
calling ``divmod()`` on the quotient from the previous step, with
varying divisors.  The most common case is probably converting a
number of seconds into weeks, days, hours, minutes and seconds.
This would today be written as::

    def secs_to_wdhms(seconds):
        m, s = divmod(seconds, 60)
        h, m = divmod(m, 60)
        d, h = divmod(h, 24)
        w, d = divmod(d, 7)
        return (w, d, h, m, s)

This is tedious and easy to get wrong each time you need it.

If instead the ``divmod()`` built-in is changed according the proposal,
the code for converting seconds to weeks, days, hours, minutes and
seconds then become ::

    def secs_to_wdhms(seconds):
        w, d, h, m, s = divmod(seconds, 7, 24, 60, 60)
        return (w, d, h, m, s)

which is easier to type, easier to type correctly, and easier to
read.

Other applications are:

- Astronomical angles (declination is measured in degrees, minutes
  and seconds, right ascension is measured in hours, minutes and
  seconds).
- Old British currency (1 pound = 20 shilling, 1 shilling = 12 pence).
- Anglo-Saxon length units: 1 mile = 1760 yards, 1 yard = 3 feet,
  1 foot = 12 inches.
- Anglo-Saxon weight units: 1 long ton = 160 stone, 1 stone = 14
  pounds, 1 pound = 16 ounce, 1 ounce = 16 dram.
- British volumes: 1 gallon = 4 quart, 1 quart = 2 pint, 1 pint
  = 20 fluid ounces.


Rationale
=========

The idea comes from APL, which has an operator that does this.  (I
don't remember what the operator looks like, and it would probably
be impossible to render in ASCII anyway.)

The APL operator takes a list as its second operand, while this
PEP proposes that each divisor should be a separate argument to
the ``divmod()`` function.  This is mainly because it is expected that
the most common uses will have the divisors as constants right in
the call (as the 7, 24, 60, 60 above), and adding a set of
parentheses or brackets would just clutter the call.

Requiring an explicit sequence as the second argument to ``divmod()``
would seriously break backwards compatibility.  Making ``divmod()``
check its second argument for being a sequence is deemed to be too
ugly to contemplate.  And in the case where one *does* have a
sequence that is computed other-where, it is easy enough to write
``divmod(x, *divs)`` instead.

Requiring at least one divisor, i.e rejecting ``divmod(x)``, has been
considered, but no good reason to do so has come to mind, and is
thus allowed in the name of generality.

Calling ``divmod()`` with no divisors should still return a tuple (of
one element).  Code that calls ``divmod()`` with a varying number of
divisors, and thus gets a return value with an "unknown" number of
elements, would otherwise have to special case that case.  Code
that *knows* it is calling ``divmod()`` with no divisors is considered
to be too silly to warrant a special case.

Processing the divisors in the other direction, i.e dividing with
the first divisor first, instead of dividing with the last divisor
first, has been considered.  However, the result comes with the
most significant part first and the least significant part last
(think of the chained divmod as a way of splitting a number into
"digits", with varying weights), and it is reasonable to specify
the divisors (weights) in the same order as the result.

The inverse operation::

    def inverse_divmod(seq, *factors):
        product = seq[0]
        for x, y in zip(factors, seq[1:]):
            product = product * x + y
        return product

could also be useful.  However, writing ::

    seconds = (((((w * 7) + d) * 24 + h) * 60 + m) * 60 + s)

is less cumbersome both to write and to read than the chained
divmods.  It is therefore deemed to be less important, and its
introduction can be deferred to its own PEP.  Also, such a
function needs a good name, and the PEP author has not managed to
come up with one yet.

Calling ``divmod("spam")`` does not raise an error, despite strings
supporting neither division nor modulo.  However, unless we know
the other object too, we can't determine whether ``divmod()`` would
work or not, and thus it seems silly to forbid it.


Backwards Compatibility
=======================

Any module that replaces the ``divmod()`` function in the ``__builtin__``
module, may cause other modules using the new syntax to break.  It
is expected that this is very uncommon.

Code that expects a ``TypeError`` exception when calling ``divmod()`` with
anything but two arguments will break.  This is also expected to
be very uncommon.

No other issues regarding backwards compatibility are known.


Reference Implementation
========================

Not finished yet, but it seems a rather straightforward
new implementation of the function ``builtin_divmod()`` in
``Python/bltinmodule.c``.


References
==========

.. [1] Raymond Hettinger, "Propose rejection of PEP 303 -- Extend divmod() for
       Multiple Divisors" https://mail.python.org/pipermail/python-dev/2005-June/054283.html


Copyright
=========

This document has been placed in the public domain.


id: pep-0465.rst
content:
PEP: 465
Title: A dedicated infix operator for matrix multiplication
Author: Nathaniel J. Smith <njs@pobox.com>
Status: Final
Type: Standards Track
Created: 20-Feb-2014
Python-Version: 3.5
Post-History: 13-Mar-2014
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/D63NDWHPF7OC2Z455MPHOW6QLLSNQUJ5/


Abstract
========

This PEP proposes a new binary operator to be used for matrix
multiplication, called ``@``.  (Mnemonic: ``@`` is ``*`` for
mATrices.)


Specification
=============

A new binary operator is added to the Python language, together
with the corresponding in-place version:

=======  ========================= ===============================
 Op      Precedence/associativity     Methods
=======  ========================= ===============================
``@``    Same as ``*``             ``__matmul__``, ``__rmatmul__``
``@=``   n/a                       ``__imatmul__``
=======  ========================= ===============================

No implementations of these methods are added to the builtin or
standard library types.  However, a number of projects have reached
consensus on the recommended semantics for these operations; see
`Intended usage details`_ below for details.

For details on how this operator will be implemented in CPython, see
`Implementation details`_.


Motivation
==========

Executive summary
-----------------

In numerical code, there are two important operations which compete
for use of Python's ``*`` operator: elementwise multiplication, and
matrix multiplication.  In the nearly twenty years since the Numeric
library was first proposed, there have been many attempts to resolve
this tension [#hugunin]_; none have been really satisfactory.
Currently, most numerical Python code uses ``*`` for elementwise
multiplication, and function/method syntax for matrix multiplication;
however, this leads to ugly and unreadable code in common
circumstances.  The problem is bad enough that significant amounts of
code continue to use the opposite convention (which has the virtue of
producing ugly and unreadable code in *different* circumstances), and
this API fragmentation across codebases then creates yet more
problems.  There does not seem to be any *good* solution to the
problem of designing a numerical API within current Python syntax --
only a landscape of options that are bad in different ways.  The
minimal change to Python syntax which is sufficient to resolve these
problems is the addition of a single new infix operator for matrix
multiplication.

Matrix multiplication has a singular combination of features which
distinguish it from other binary operations, which together provide a
uniquely compelling case for the addition of a dedicated infix
operator:

* Just as for the existing numerical operators, there exists a vast
  body of prior art supporting the use of infix notation for matrix
  multiplication across all fields of mathematics, science, and
  engineering; ``@`` harmoniously fills a hole in Python's existing
  operator system.

* ``@`` greatly clarifies real-world code.

* ``@`` provides a smoother onramp for less experienced users, who are
  particularly harmed by hard-to-read code and API fragmentation.

* ``@`` benefits a substantial and growing portion of the Python user
  community.

* ``@`` will be used frequently -- in fact, evidence suggests it may
  be used more frequently than ``//`` or the bitwise operators.

* ``@`` allows the Python numerical community to reduce fragmentation,
  and finally standardize on a single consensus duck type for all
  numerical array objects.


Background: What's wrong with the status quo?
---------------------------------------------

When we crunch numbers on a computer, we usually have lots and lots of
numbers to deal with.  Trying to deal with them one at a time is
cumbersome and slow -- especially when using an interpreted language.
Instead, we want the ability to write down simple operations that
apply to large collections of numbers all at once.  The *n-dimensional
array* is the basic object that all popular numeric computing
environments use to make this possible.  Python has several libraries
that provide such arrays, with numpy being at present the most
prominent.

When working with n-dimensional arrays, there are two different ways
we might want to define multiplication.  One is elementwise
multiplication::

  [[1, 2],     [[11, 12],     [[1 * 11, 2 * 12],
   [3, 4]]  x   [13, 14]]  =   [3 * 13, 4 * 14]]

and the other is `matrix multiplication`_:

.. _matrix multiplication: https://en.wikipedia.org/wiki/Matrix_multiplication

::

  [[1, 2],     [[11, 12],     [[1 * 11 + 2 * 13, 1 * 12 + 2 * 14],
   [3, 4]]  x   [13, 14]]  =   [3 * 11 + 4 * 13, 3 * 12 + 4 * 14]]

Elementwise multiplication is useful because it lets us easily and
quickly perform many multiplications on a large collection of values,
without writing a slow and cumbersome ``for`` loop.  And this works as
part of a very general schema: when using the array objects provided
by numpy or other numerical libraries, all Python operators work
elementwise on arrays of all dimensionalities.  The result is that one
can write functions using straightforward code like ``a * b + c / d``,
treating the variables as if they were simple values, but then
immediately use this function to efficiently perform this calculation
on large collections of values, while keeping them organized using
whatever arbitrarily complex array layout works best for the problem
at hand.

Matrix multiplication is more of a special case.  It's only defined on
2d arrays (also known as "matrices"), and multiplication is the only
operation that has an important "matrix" version -- "matrix addition"
is the same as elementwise addition; there is no such thing as "matrix
bitwise-or" or "matrix floordiv"; "matrix division" and "matrix
to-the-power-of" can be defined but are not very useful, etc.
However, matrix multiplication is still used very heavily across all
numerical application areas; mathematically, it's one of the most
fundamental operations there is.

Because Python syntax currently allows for only a single
multiplication operator ``*``, libraries providing array-like objects
must decide: either use ``*`` for elementwise multiplication, or use
``*`` for matrix multiplication.  And, unfortunately, it turns out
that when doing general-purpose number crunching, both operations are
used frequently, and there are major advantages to using infix rather
than function call syntax in both cases.  Thus it is not at all clear
which convention is optimal, or even acceptable; often it varies on a
case-by-case basis.

Nonetheless, network effects mean that it is very important that we
pick *just one* convention.  In numpy, for example, it is technically
possible to switch between the conventions, because numpy provides two
different types with different ``__mul__`` methods.  For
``numpy.ndarray`` objects, ``*`` performs elementwise multiplication,
and matrix multiplication must use a function call (``numpy.dot``).
For ``numpy.matrix`` objects, ``*`` performs matrix multiplication,
and elementwise multiplication requires function syntax.  Writing code
using ``numpy.ndarray`` works fine.  Writing code using
``numpy.matrix`` also works fine.  But trouble begins as soon as we
try to integrate these two pieces of code together.  Code that expects
an ``ndarray`` and gets a ``matrix``, or vice-versa, may crash or
return incorrect results.  Keeping track of which functions expect
which types as inputs, and return which types as outputs, and then
converting back and forth all the time, is incredibly cumbersome and
impossible to get right at any scale.  Functions that defensively try
to handle both types as input and DTRT, find themselves floundering
into a swamp of ``isinstance`` and ``if`` statements.

:pep:`238` split ``/`` into two operators: ``/`` and ``//``.  Imagine the
chaos that would have resulted if it had instead split ``int`` int


id: pep-0471.rst
content:
PEP: 471
Title: os.scandir() function -- a better and faster directory iterator
Author: Ben Hoyt <benhoyt@gmail.com>
BDFL-Delegate: Victor Stinner <vstinner@python.org>
Status: Final
Type: Standards Track
Created: 30-May-2014
Python-Version: 3.5
Post-History: 27-Jun-2014, 08-Jul-2014, 14-Jul-2014


Abstract
========

This PEP proposes including a new directory iteration function,
``os.scandir()``, in the standard library. This new function adds
useful functionality and increases the speed of ``os.walk()`` by 2-20
times (depending on the platform and file system) by avoiding calls to
``os.stat()`` in most cases.


Rationale
=========

Python's built-in ``os.walk()`` is significantly slower than it needs
to be, because -- in addition to calling ``os.listdir()`` on each
directory -- it executes the ``stat()`` system call or
``GetFileAttributes()`` on each file to determine whether the entry is
a directory or not.

But the underlying system calls -- ``FindFirstFile`` /
``FindNextFile`` on Windows and ``readdir`` on POSIX systems --
already tell you whether the files returned are directories or not, so
no further system calls are needed. Further, the Windows system calls
return all the information for a ``stat_result`` object on the directory
entry, such as file size and last modification time.

In short, you can reduce the number of system calls required for a
tree function like ``os.walk()`` from approximately 2N to N, where N
is the total number of files and directories in the tree. (And because
directory trees are usually wider than they are deep, it's often much
better than this.)

In practice, removing all those extra system calls makes ``os.walk()``
about **8-9 times as fast on Windows**, and about **2-3 times as fast
on POSIX systems**. So we're not talking about
micro-optimizations. See more `benchmarks here`_.

.. _`benchmarks here`: https://github.com/benhoyt/scandir#benchmarks

Somewhat relatedly, many people (see Python `Issue 11406`_) are also
keen on a version of ``os.listdir()`` that yields filenames as it
iterates instead of returning them as one big list. This improves
memory efficiency for iterating very large directories.

So, as well as providing a ``scandir()`` iterator function for calling
directly, Python's existing ``os.walk()`` function can be sped up a
huge amount.

.. _`Issue 11406`: http://bugs.python.org/issue11406


Implementation
==============

The implementation of this proposal was written by Ben Hoyt (initial
version) and Tim Golden (who helped a lot with the C extension
module). It lives on GitHub at `benhoyt/scandir`_. (The implementation
may lag behind the updates to this PEP a little.)

.. _`benhoyt/scandir`: https://github.com/benhoyt/scandir

Note that this module has been used and tested (see "Use in the wild"
section in this PEP), so it's more than a proof-of-concept. However,
it is marked as beta software and is not extensively battle-tested.
It will need some cleanup and more thorough testing before going into
the standard library, as well as integration into ``posixmodule.c``.



Specifics of proposal
=====================

os.scandir()
------------

Specifically, this PEP proposes adding a single function to the ``os``
module in the standard library, ``scandir``, that takes a single,
optional string as its argument::

    scandir(path='.') -> generator of DirEntry objects

Like ``listdir``, ``scandir`` calls the operating system's directory
iteration system calls to get the names of the files in the given
``path``, but it's different from ``listdir`` in two ways:

* Instead of returning bare filename strings, it returns lightweight
  ``DirEntry`` objects that hold the filename string and provide
  simple methods that allow access to the additional data the
  operating system may have returned.

* It returns a generator instead of a list, so that ``scandir`` acts
  as a true iterator instead of returning the full list immediately.

``scandir()`` yields a ``DirEntry`` object for each file and
sub-directory in ``path``. Just like ``listdir``, the ``'.'``
and ``'..'`` pseudo-directories are skipped, and the entries are
yielded in system-dependent order. Each ``DirEntry`` object has the
following attributes and methods:

* ``name``: the entry's filename, relative to the scandir ``path``
  argument (corresponds to the return values of ``os.listdir``)

* ``path``: the entry's full path name (not necessarily an absolute
  path) -- the equivalent of ``os.path.join(scandir_path,
  entry.name)``

* ``inode()``: return the inode number of the entry. The result is cached on
  the ``DirEntry`` object, use ``os.stat(entry.path,
  follow_symlinks=False).st_ino`` to fetch up-to-date information.
  On Unix, no system call is required.

* ``is_dir(*, follow_symlinks=True)``: similar to
  ``pathlib.Path.is_dir()``, but the return value is cached on the
  ``DirEntry`` object; doesn't require a system call in most cases;
  don't follow symbolic links if ``follow_symlinks`` is False

* ``is_file(*, follow_symlinks=True)``: similar to
  ``pathlib.Path.is_file()``, but the return value is cached on the
  ``DirEntry`` object; doesn't require a system call in most cases;
  don't follow symbolic links if ``follow_symlinks`` is False

* ``is_symlink()``: similar to ``pathlib.Path.is_symlink()``, but the
  return value is cached on the ``DirEntry`` object; doesn't require a
  system call in most cases

* ``stat(*, follow_symlinks=True)``: like ``os.stat()``, but the
  return value is cached on the ``DirEntry`` object; does not require a
  system call on Windows (except for symlinks); don't follow symbolic links
  (like ``os.lstat()``) if ``follow_symlinks`` is False

All *methods* may perform system calls in some cases and therefore
possibly raise ``OSError`` -- see the "Notes on exception handling"
section for more details.

The ``DirEntry`` attribute and method names were chosen to be the same
as those in the new ``pathlib`` module where possible, for
consistency. The only difference in functionality is that the
``DirEntry`` methods cache their values on the entry object after the
first call.

Like the other functions in the ``os`` module, ``scandir()`` accepts
either a bytes or str object for the ``path`` parameter, and
returns the ``DirEntry.name`` and ``DirEntry.path`` attributes with
the same type as ``path``. However, it is *strongly recommended*
to use the str type, as this ensures cross-platform support for
Unicode filenames. (On Windows, bytes filenames have been deprecated
since Python 3.3).

os.walk()
---------

As part of this proposal, ``os.walk()`` will also be modified to use
``scandir()`` rather than ``listdir()`` and ``os.path.isdir()``. This
will increase the speed of ``os.walk()`` very significantly (as
mentioned above, by 2-20 times, depending on the system).


Examples
========

First, a very simple example of ``scandir()`` showing use of the
``DirEntry.name`` attribute and the ``DirEntry.is_dir()`` method::

    def subdirs(path):
        """Yield directory names not starting with '.' under given path."""
        for entry in os.scandir(path):
            if not entry.name.startswith('.') and entry.is_dir():
                yield entry.name

This ``subdirs()`` function will be significantly faster with scandir
than ``os.listdir()`` and ``os.path.isdir()`` on both Windows and POSIX
systems, especially on medium-sized or large directories.

Or, for getting the total size of files in a directory tree, showing
use of the ``DirEntry.stat()`` method and ``DirEntry.path``
attribute::

    def get_tree_size(path):
        """Return total size of files in given path and subdirs."""
        total = 0
        for entry in os.scandir(path):
            if entry.is_dir(follow_symlinks=False):
                total += get_tree_size(entry.path)
            else:
                total += entry.stat(follow_symlinks=False).st_size
        return total

This also shows the use of the ``follow_symlinks`` parameter to
``is_dir()`` --


id: pep-0317.rst
content:
PEP: 317
Title: Eliminate Implicit Exception Instantiation
Author: Steven Taschuk <staschuk@telusplanet.net>
Status: Rejected
Type: Standards Track
Created: 06-May-2003
Python-Version: 2.4
Post-History: 09-Jun-2003


Abstract
========

    "For clarity in new code, the form ``raise class(argument, ...)``
    is recommended (i.e. make an explicit call to the constructor)."

    -- Guido van Rossum, in 1997 [1]_

This PEP proposes the formal deprecation and eventual elimination of
forms of the ``raise`` statement which implicitly instantiate an
exception.  For example, statements such as ::

    raise HullBreachError
    raise KitchenError, 'all out of baked beans'

must under this proposal be replaced with their synonyms ::

    raise HullBreachError()
    raise KitchenError('all out of baked beans')

Note that these latter statements are already legal, and that this PEP
does not change their meaning.

Eliminating these forms of ``raise`` makes it impossible to use string
exceptions; accordingly, this PEP also proposes the formal deprecation
and eventual elimination of string exceptions.

Adoption of this proposal breaks backwards compatibility.  Under the
proposed implementation schedule, Python 2.4 will introduce warnings
about uses of ``raise`` which will eventually become incorrect, and
Python 3.0 will eliminate them entirely.  (It is assumed that this
transition period -- 2.4 to 3.0 -- will be at least one year long, to
comply with the guidelines of :pep:`5`.)


Motivation
==========

String Exceptions
-----------------

It is assumed that removing string exceptions will be uncontroversial,
since it has been intended since at least Python 1.5, when the
standard exception types were changed to classes [1]_.

For the record: string exceptions should be removed because the
presence of two kinds of exception complicates the language without
any compensation.  Instance exceptions are superior because, for
example,

* the class-instance relationship more naturally expresses the
  relationship between the exception type and value,

* they can be organized naturally using superclass-subclass
  relationships, and

* they can encapsulate error-reporting behaviour (for example).


Implicit Instantiation
----------------------

Guido's 1997 essay [1]_ on changing the standard exceptions into
classes makes clear why ``raise`` can instantiate implicitly:

    "The raise statement has been extended to allow raising a class
    exception without explicit instantiation. The following forms,
    called the "compatibility forms" of the raise statement [...]  The
    motivation for introducing the compatibility forms was to allow
    backward compatibility with old code that raised a standard
    exception."

For example, it was desired that pre-1.5 code which used string
exception syntax such as ::

    raise TypeError, 'not an int'

would work both on versions of Python in which ``TypeError`` was a
string, and on versions in which it was a class.

When no such consideration obtains -- that is, when the desired
exception type is not a string in any version of the software which
the code must support -- there is no good reason to instantiate
implicitly, and it is clearer not to.  For example:

1. In the code ::

       try:
           raise MyError, raised
       except MyError, caught:
           pass

   the syntactic parallel between the ``raise`` and ``except``
   statements strongly suggests that ``raised`` and ``caught`` refer
   to the same object.  For string exceptions this actually is the
   case, but for instance exceptions it is not.

2. When instantiation is implicit, it is not obvious when it occurs,
   for example, whether it occurs when the exception is raised or when
   it is caught.  Since it actually happens at the ``raise``, the code
   should say so.

   (Note that at the level of the C API, an exception can be "raised"
   and "caught" without being instantiated; this is used as an
   optimization by, for example, ``PyIter_Next``.  But in Python, no
   such optimization is or should be available.)

3. An implicitly instantiating ``raise`` statement with no arguments,
   such as ::

       raise MyError

   simply does not do what it says: it does not raise the named
   object.

4. The equivalence of ::

       raise MyError
       raise MyError()

   conflates classes and instances, creating a possible source of
   confusion for beginners.  (Moreover, it is not clear that the
   interpreter could distinguish between a new-style class and an
   instance of such a class, so implicit instantiation may be an
   obstacle to any future plan to let exceptions be new-style
   objects.)

In short, implicit instantiation has no advantages other than
backwards compatibility, and so should be phased out along with what
it exists to ensure compatibility with, namely, string exceptions.


Specification
=============

The syntax of ``raise_stmt`` [3]_ is to be changed from ::

    raise_stmt ::= "raise" [expression ["," expression ["," expression]]]

to ::

    raise_stmt ::= "raise" [expression ["," expression]]

If no expressions are present, the ``raise`` statement behaves as it
does presently: it re-raises the last exception that was active in the
current scope, and if no exception has been active in the current
scope, a ``TypeError`` is raised indicating that this is the problem.

Otherwise, the first expression is evaluated, producing the *raised
object*.  Then the second expression is evaluated, if present,
producing the *substituted traceback*.  If no second expression is
present, the substituted traceback is ``None``.

The raised object must be an instance.  The class of the instance is
the exception type, and the instance itself is the exception value.
If the raised object is not an instance -- for example, if it is a
class or string -- a ``TypeError`` is raised.

If the substituted traceback is not ``None``, it must be a traceback
object, and it is substituted instead of the current location as the
place where the exception occurred.  If it is neither a traceback
object nor ``None``, a ``TypeError`` is raised.


Backwards Compatibility
=======================

Migration Plan
--------------

Future Statement
''''''''''''''''

Under the :pep:`236` future statement::

    from __future__ import raise_with_two_args

the syntax and semantics of the ``raise`` statement will be as
described above.  This future feature is to appear in Python 2.4; its
effect is to become standard in Python 3.0.

As the examples below illustrate, this future statement is only needed
for code which uses the substituted traceback argument to ``raise``;
simple exception raising does not require it.


Warnings
''''''''

Three new :pep:`warnings <230>`, all of category ``DeprecationWarning``, are
to be issued to point out uses of ``raise`` which will become
incorrect under the proposed changes.

The first warning is issued when a ``raise`` statement is executed in
which the first expression evaluates to a string.  The message for
this warning is::

    raising strings will be impossible in the future

The second warning is issued when a ``raise`` statement is executed in
which the first expression evaluates to a class.  The message for this
warning is::

    raising classes will be impossible in the future

The third warning is issued when a ``raise`` statement with three
expressions is compiled.  (Not, note, when it is executed; this is
important because the ``SyntaxError`` which this warning presages will
occur at compile-time.)  The message for this warning is::

    raising with three arguments will be impossible in the future

These warnings are to appear in Python 2.4, and disappear in Python
3.0, when the conditions which cause them are simply errors.


Examples
--------

Code Using Implicit Instantiation
'''''''''''''''''''''''''''''''''

Code such as ::

    class MyError(Exception):
        pass

    raise MyError, 'spam'

will issue a warning when the ``raise`` statem


