# embeddings

id: pep-3152.rst
content:
PEP: 3152
Title: Cofunctions
Author: Gregory Ewing <greg.ewing@canterbury.ac.nz>
Status: Rejected
Type: Standards Track
Created: 13-Feb-2009
Python-Version: 3.3
Post-History:


Abstract
========

A syntax is proposed for defining and calling a special type of
generator called a 'cofunction'.  It is designed to provide a
streamlined way of writing generator-based coroutines, and allow the
early detection of certain kinds of error that are easily made when
writing such code, which otherwise tend to cause hard-to-diagnose
symptoms.

This proposal builds on the 'yield from' mechanism described in PEP
380, and describes some of the semantics of cofunctions in terms of
it.  However, it would be possible to define and implement cofunctions
independently of :pep:`380` if so desired.

Rejection
---------

See https://mail.python.org/pipermail/python-dev/2015-April/139503.html


Specification
=============

Cofunction definitions
----------------------

A new keyword ``codef`` is introduced which is used in place of
``def`` to define a cofunction.  A cofunction is a special kind of
generator having the following characteristics:

1. A cofunction is always a generator, even if it does not contain any
   ``yield`` or ``yield from`` expressions.

2. A cofunction cannot be called the same way as an ordinary function.
   An exception is raised if an ordinary call to a cofunction is
   attempted.

Cocalls
-------

Calls from one cofunction to another are made by marking the call with
a new keyword ``cocall``.  The expression ::

    cocall f(*args, **kwds)

is semantically equivalent to ::

    yield from f.__cocall__(*args, **kwds)

except that the object returned by __cocall__ is expected to be an
iterator, so the step of calling iter() on it is skipped.

The full syntax of a cocall expression is described by the following
grammar lines::

    atom: cocall | <existing alternatives for atom>
    cocall: 'cocall' atom cotrailer* '(' [arglist] ')'
    cotrailer: '[' subscriptlist ']' | '.' NAME

The ``cocall`` keyword is syntactically valid only inside a
cofunction.  A SyntaxError will result if it is used in any other
context.

Objects which implement __cocall__ are expected to return an object
obeying the iterator protocol.  Cofunctions respond to __cocall__ the
same way as ordinary generator functions respond to __call__, i.e. by
returning a generator-iterator.

Certain objects that wrap other callable objects, notably bound
methods, will be given __cocall__ implementations that delegate to the
underlying object.

New builtins, attributes and C API functions
--------------------------------------------

To facilitate interfacing cofunctions with non-coroutine code, there will
be a built-in function ``costart`` whose definition is equivalent to ::

    def costart(obj, *args, **kwds):
        return obj.__cocall__(*args, **kwds)

There will also be a corresponding C API function ::

    PyObject *PyObject_CoCall(PyObject *obj, PyObject *args, PyObject *kwds)

It is left unspecified for now whether a cofunction is a distinct type
of object or, like a generator function, is simply a specially-marked
function instance.  If the latter, a read-only boolean attribute
``__iscofunction__`` should be provided to allow testing whether a
given function object is a cofunction.


Motivation and Rationale
========================

The ``yield from`` syntax is reasonably self-explanatory when used for
the purpose of delegating part of the work of a generator to another
function.  It can also be used to good effect in the implementation of
generator-based coroutines, but it reads somewhat awkwardly when used
for that purpose, and tends to obscure the true intent of the code.

Furthermore, using generators as coroutines is somewhat error-prone.
If one forgets to use ``yield from`` when it should have been used, or
uses it when it shouldn't have, the symptoms that result can be
obscure and confusing.

Finally, sometimes there is a need for a function to be a coroutine
even though it does not yield anything, and in these cases it is
necessary to resort to kludges such as ``if 0: yield`` to force it to
be a generator.

The ``codef`` and ``cocall`` constructs address the first issue by
making the syntax directly reflect the intent, that is, that the
function forms part of a coroutine.

The second issue is addressed by making it impossible to mix coroutine
and non-coroutine code in ways that don't make sense.  If the rules
are violated, an exception is raised that points out exactly what and
where the problem is.

Lastly, the need for dummy yields is eliminated by making the form of
definition determine whether the function is a coroutine, rather than
what it contains.


Prototype Implementation
========================

An implementation in the form of patches to Python 3.1.2 can be found
here:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cofunctions.html


Copyright
=========

This document has been placed in the public domain.


id: pep-3146.rst
content:
PEP: 3146
Title: Merging Unladen Swallow into CPython
Author: Collin Winter <collinwinter@google.com>,
        Jeffrey Yasskin <jyasskin@google.com>,
        Reid Kleckner <rnk@mit.edu>
Status: Withdrawn
Type: Standards Track
Created: 01-Jan-2010
Python-Version: 3.3
Post-History:


PEP Withdrawal
==============

With Unladen Swallow going the way of the Norwegian Blue [#us-post-mortem]_
[#dead-parrot]_, this PEP has been deemed to have been withdrawn.


Abstract
========

This PEP proposes the merger of the Unladen Swallow project [#us]_ into
CPython's source tree. Unladen Swallow is an open-source branch of CPython
focused on performance. Unladen Swallow is source-compatible with valid Python
2.6.4 applications and C extension modules.

Unladen Swallow adds a just-in-time (JIT) compiler to CPython, allowing for the
compilation of selected Python code to optimized machine code. Beyond classical
static compiler optimizations, Unladen Swallow's JIT compiler takes advantage of
data collected at runtime to make checked assumptions about code behaviour,
allowing the production of faster machine code.

This PEP proposes to integrate Unladen Swallow into CPython's development tree
in a separate ``py3k-jit`` branch, targeted for eventual merger with the main
``py3k`` branch. While Unladen Swallow is by no means finished or perfect, we
feel that Unladen Swallow has reached sufficient maturity to warrant
incorporation into CPython's roadmap. We have sought to create a stable platform
that the wider CPython development team can build upon, a platform that will
yield increasing performance for years to come.

This PEP will detail Unladen Swallow's implementation and how it differs from
CPython 2.6.4; the benchmarks used to measure performance; the tools used to
ensure correctness and compatibility; the impact on CPython's current platform
support; and the impact on the CPython core development process. The PEP
concludes with a proposed merger plan and brief notes on possible directions
for future work.

We seek the following from the BDFL:

- Approval for the overall concept of adding a just-in-time compiler to CPython,
  following the design laid out below.
- Permission to continue working on the just-in-time compiler in the CPython
  source tree.
- Permission to eventually merge the just-in-time compiler into the ``py3k``
  branch once all blocking issues [#us-punchlist]_ have been addressed.
- A pony.


Rationale, Implementation
=========================

Many companies and individuals would like Python to be faster, to enable its
use in more projects. Google is one such company.

Unladen Swallow is a Google-sponsored branch of CPython, initiated to improve
the performance of Google's numerous Python libraries, tools and applications.
To make the adoption of Unladen Swallow as easy as possible, the project
initially aimed at four goals:

- A performance improvement of 5x over the baseline of CPython 2.6.4 for
  single-threaded code.
- 100% source compatibility with valid CPython 2.6 applications.
- 100% source compatibility with valid CPython 2.6 C extension modules.
- Design for eventual merger back into CPython.

We chose 2.6.4 as our baseline because Google uses CPython 2.4 internally, and
jumping directly from CPython 2.4 to CPython 3.x was considered infeasible.

To achieve the desired performance, Unladen Swallow has implemented a
just-in-time (JIT) compiler [#jit]_ in the tradition of Urs Hoelzle's work on
Self [#urs-self]_, gathering feedback at runtime and using that to inform
compile-time optimizations. This is similar to the approach taken by the current
breed of JavaScript engines [#v8]_, [#squirrelfishextreme]_; most Java virtual
machines [#hotspot]_; Rubinius [#rubinius]_, MacRuby [#macruby]_, and other Ruby
implementations; Psyco [#psyco]_; and others.

We explicitly reject any suggestion that our ideas are original. We have sought
to reuse the published work of other researchers wherever possible. If we have
done any original work, it is by accident. We have tried, as much as possible,
to take good ideas from all corners of the academic and industrial community. A
partial list of the research papers that have informed Unladen Swallow is
available on the Unladen Swallow wiki [#us-relevantpapers]_.

The key observation about optimizing dynamic languages is that they are only
dynamic in theory; in practice, each individual function or snippet of code is
relatively static, using a stable set of types and child functions. The current
CPython bytecode interpreter assumes the worst about the code it is running,
that at any moment the user might override the ``len()`` function or pass a
never-before-seen type into a function. In practice this never happens, but user
code pays for that support. Unladen Swallow takes advantage of the relatively
static nature of user code to improve performance.

At a high level, the Unladen Swallow JIT compiler works by translating a
function's CPython bytecode to platform-specific machine code, using data
collected at runtime, as well as classical compiler optimizations, to improve
the quality of the generated machine code. Because we only want to spend
resources compiling Python code that will actually benefit the runtime of the
program, an online heuristic is used to assess how hot a given function is. Once
the hotness value for a function crosses a given threshold, it is selected for
compilation and optimization. Until a function is judged hot, however, it runs
in the standard CPython eval loop, which in Unladen Swallow has been
instrumented to record interesting data about each bytecode executed. This
runtime data is used to reduce the flexibility of the generated machine code,
allowing us to optimize for the common case. For example, we collect data on

- Whether a branch was taken/not taken. If a branch is never taken, we will not
  compile it to machine code.
- Types used by operators. If we find that ``a + b`` is only ever adding
  integers, the generated machine code for that snippet will not support adding
  floats.
- Functions called at each callsite. If we find that a particular ``foo()``
  callsite is always calling the same ``foo`` function, we can optimize the
  call or inline it away

Refer to [#us-llvm-notes]_ for a complete list of data points gathered and how
they are used.

However, if by chance the historically-untaken branch is now taken, or some
integer-optimized ``a + b`` snippet receives two strings, we must support this.
We cannot change Python semantics. Each of these sections of optimized machine
code is preceded by a ``guard``, which checks whether the simplifying
assumptions we made when optimizing still hold. If the assumptions are still
valid, we run the optimized machine code; if they are not, we revert back to
the interpreter and pick up where we left off.

We have chosen to reuse a set of existing compiler libraries called LLVM
[#llvm]_ for code generation and code optimization. This has saved our small
team from needing to understand and debug code generation on multiple machine
instruction sets and from needing to implement a large set of classical compiler
optimizations. The project would not have been possible without such code reuse.
We have found LLVM easy to modify and its community receptive to our suggestions
and modifications.

In somewhat more depth, Unladen Swallow's JIT works by compiling CPython
bytecode to LLVM's own intermediate representation (IR) [#llvm-langref]_, taking
into account any runtime data from the CPython eval loop. We then run a set of
LLVM's built-in optimization passes, producing a smaller, optimized version of
the original LLVM IR. LLVM then lowers the IR to platform-specific machine code,
performing register allocation, instruction scheduling, and any necessary
relocations. This arrangement of the compilation pipeline allows the LLVM-based
JIT to be easily omitted from a compiled ``python`` binary by passing
``--without-llvm`` to ``./configure``; v


id: pep-3150.rst
content:
PEP: 3150
Title: Statement local namespaces (aka "given" clause)
Author: Alyssa Coghlan <ncoghlan@gmail.com>
Status: Deferred
Type: Standards Track
Created: 09-Jul-2010
Python-Version: 3.4
Post-History: 14-Jul-2010, 21-Apr-2011, 13-Jun-2011


Abstract
========

This PEP proposes the addition of an optional ``given`` clause to several
Python statements that do not currently have an associated code suite. This
clause will create a statement local namespace for additional names that are
accessible in the associated statement, but do not become part of the
containing namespace.

Adoption of a new symbol, ``?``, is proposed to denote a forward reference
to the namespace created by running the associated code suite. It will be
a reference to a ``types.SimpleNamespace`` object.

The primary motivation is to enable a more declarative style of programming,
where the operation to be performed is presented to the reader first, and the
details of the necessary subcalculations are presented in the following
indented suite. As a key example, this would elevate ordinary assignment
statements to be on par with ``class`` and ``def`` statements where the name
of the item to be defined is presented to the reader in advance of the
details of how the value of that item is calculated. It also allows named
functions to be used in a "multi-line lambda" fashion, where the name is used
solely as a placeholder in the current expression and then defined in the
following suite.

A secondary motivation is to simplify interim calculations in module and
class level code without polluting the resulting namespaces.

The intent is that the relationship between a given clause and a separate
function definition that performs the specified operation will be similar to
the existing relationship between an explicit while loop and a generator that
produces the same sequence of operations as that while loop.

The specific proposal in this PEP has been informed by various explorations
of this and related concepts over the years (e.g. [1]_, [2]_, [3]_, [6]_,
[8]_), and is inspired to some degree by the ``where`` and ``let`` clauses in
Haskell. It avoids some problems that have been identified in past proposals,
but has not yet itself been subject to the test of implementation.


Proposal
========

This PEP proposes the addition of an optional ``given`` clause to the
syntax for simple statements which may contain an expression, or may
substitute for such a statement for purely syntactic purposes. The
current list of simple statements that would be affected by this
addition is as follows:

* expression statement
* assignment statement
* augmented assignment statement
* del statement
* return statement
* yield statement
* raise statement
* assert statement
* pass statement

The ``given`` clause would allow subexpressions to be referenced by
name in the header line, with the actual definitions following in
the indented clause. As a simple example::

   sorted_data = sorted(data, key=?.sort_key) given:
       def sort_key(item):
           return item.attr1, item.attr2

The new symbol ``?`` is used to refer to the given namespace. It would be a
``types.SimpleNamespace`` instance, so ``?.sort_key`` functions as
a forward reference to a name defined in the ``given`` clause.

A docstring would be permitted in the given clause, and would be attached
to the result namespace as its ``__doc__`` attribute.

The ``pass`` statement is included to provide a consistent way to skip
inclusion of a meaningful expression in the header line. While this is not
an intended use case, it isn't one that can be prevented as multiple
alternatives (such as ``...`` and ``()``) remain available even if ``pass``
itself is disallowed.

The body of the given clause will execute in a new scope, using normal
function closure semantics. To support early binding of loop variables
and global references, as well as to allow access to other names defined at
class scope, the ``given`` clause will also allow explicit
binding operations in the header line::

   # Explicit early binding via given clause
   seq = []
   for i in range(10):
       seq.append(?.f) given i=i in:
           def f():
               return i
   assert [f() for f in seq] == list(range(10))


Semantics
---------

The following statement::

   op(?.f, ?.g) given bound_a=a, bound_b=b in:
       def f():
           return bound_a + bound_b
       def g():
           return bound_a - bound_b

Would be roughly equivalent to the following code (``__var`` denotes a
hidden compiler variable or simply an entry on the interpreter stack)::

   __arg1 = a
   __arg2 = b
   def __scope(bound_a, bound_b):
       def f():
           return bound_a + bound_b
       def g():
           return bound_a - bound_b
      return types.SimpleNamespace(**locals())
   __ref = __scope(__arg1, __arg2)
   __ref.__doc__ = __scope.__doc__
   op(__ref.f, __ref.g)

A ``given`` clause is essentially a nested function which is created and
then immediately executed. Unless explicitly passed in, names are looked
up using normal scoping rules, and thus names defined at class scope will
not be visible. Names declared as forward references are returned and
used in the header statement, without being bound locally in the
surrounding namespace.


Syntax Change
-------------

Current::

   expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
                ('=' (yield_expr|testlist_star_expr))*)
   del_stmt: 'del' exprlist
   pass_stmt: 'pass'
   return_stmt: 'return' [testlist]
   yield_stmt: yield_expr
   raise_stmt: 'raise' [test ['from' test]]
   assert_stmt: 'assert' test [',' test]


New::

   expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
                ('=' (yield_expr|testlist_star_expr))*) [given_clause]
   del_stmt: 'del' exprlist [given_clause]
   pass_stmt: 'pass' [given_clause]
   return_stmt: 'return' [testlist] [given_clause]
   yield_stmt: yield_expr [given_clause]
   raise_stmt: 'raise' [test ['from' test]] [given_clause]
   assert_stmt: 'assert' test [',' test] [given_clause]
   given_clause: "given" [(NAME '=' test)+ "in"]":" suite

(Note that ``expr_stmt`` in the grammar is a slight misnomer, as it covers
assignment and augmented assignment in addition to simple expression
statements)

.. note::
   These proposed grammar changes don't yet cover the forward reference
   expression syntax for accessing names defined in the statement local
   namespace.


The new clause is added as an optional element of the existing statements
rather than as a new kind of compound statement in order to avoid creating
an ambiguity in the grammar. It is applied only to the specific elements
listed so that nonsense like the following is disallowed::

   break given:
       a = b = 1

   import sys given:
       a = b = 1

However, the precise Grammar change described above is inadequate, as it
creates problems for the definition of simple_stmt (which allows chaining of
multiple single line statements with ";" rather than "\\n").

So the above syntax change should instead be taken as a statement of intent.
Any actual proposal would need to resolve the simple_stmt parsing problem
before it could be seriously considered. This would likely require a
non-trivial restructuring of the grammar, breaking up small_stmt and
flow_stmt to separate the statements that potentially contain arbitrary
subexpressions and then allowing a single one of those statements with
a ``given`` clause at the simple_stmt level. Something along the lines of::

   stmt: simple_stmt | given_stmt | compound_stmt
   simple_stmt: small_stmt (';' (small_stmt | subexpr_stmt))* [';'] NEWLINE
   small_stmt: (pass_stmt | flow_stmt | import_stmt |
                global_stmt | nonlocal_stmt)
   flow_stmt: break_stmt | continue_stmt
   given_stmt: subexpr_stmt (given_clause |
                 (';' (small_stmt | subexpr_stmt))* [';']) NEWLINE
   subexpr_stmt: expr_stmt | del_stmt | flow_subexpr_stmt | assert_stmt
   flo


id: pep-3144.rst
content:
PEP: 3144
Title: IP Address Manipulation Library for the Python Standard Library
Author: Peter Moody <pmoody@google.com>
BDFL-Delegate: Alyssa Coghlan
Discussions-To: ipaddr-py-dev@googlegroups.com
Status: Final
Type: Standards Track
Created: 06-Feb-2012
Python-Version: 3.3
Resolution: https://mail.python.org/pipermail/python-dev/2012-May/119474.html


Abstract
========

This PEP proposes a design and for an IP address manipulation module for
python.


PEP Acceptance
==============

This PEP was accepted by Alyssa Coghlan on the 15th of May, 2012.


Motivation
==========

Several very good IP address modules for python already exist.
The truth is that all of them struggle with the balance between
adherence to Pythonic principals and the shorthand upon which
network engineers and administrators rely.  ``ipaddress`` aims to
strike the right balance.


Rationale
=========

The existence of several Python IP address manipulation modules is
evidence of an outstanding need for the functionality this module
seeks to provide.


Background
==========

:pep:`3144` and ``ipaddr`` have been up for inclusion before.  The
version of the library specified here is backwards incompatible
with the version on PyPI and the one which was discussed before.
In order to avoid confusing users of the current ``ipaddr``, I've
renamed this version of the library ``ipaddress``.

The main differences between ipaddr and ipaddress are:

* ``ipaddress`` \*Network classes are equivalent to the ``ipaddr`` \*Network
  class counterparts with the ``strict`` flag set to ``True``.

* ``ipaddress`` \*Interface classes are equivalent to the ``ipaddr``
  \*Network class counterparts with the ``strict`` flag set to ``False``.

* The factory functions in ``ipaddress`` were renamed to disambiguate
  them from classes.

* A few attributes were renamed to disambiguate their purpose as
  well. (eg. ``network``, ``network_address``)

* A number of methods and functions which returned containers in ``ipaddr`` now
  return iterators. This includes ``subnets``, ``address_exclude``,
  ``summarize_address_range`` and ``collapse_address_list``.


Due to the backwards incompatible API changes between ``ipaddress`` and ``ipaddr``,
the proposal is to add the module using the new provisional API status:

* http://docs.python.org/dev/glossary.html#term-provisional-package


Relevant messages on python-dev:

* https://mail.python.org/pipermail/python-dev/2012-January/116016.html
* https://mail.python.org/pipermail/python-dev/2012-February/116656.html
* https://mail.python.org/pipermail/python-dev/2012-February/116688.html


Specification
=============

The ``ipaddr`` module defines a total of 6 new public classes, 3 for
manipulating IPv4 objects and 3 for manipulating IPv6 objects.
The classes are as follows:

- ``IPv4Address``/``IPv6Address`` - These define individual addresses, for
  example the IPv4 address returned by an A record query for
  www.google.com (74.125.224.84) or the IPv6 address returned by a
  AAAA record query for ipv6.google.com (2001:4860:4001:801::1011).

- ``IPv4Network``/``IPv6Network`` - These define networks or groups of
  addresses, for example the IPv4 network reserved for multicast use
  (224.0.0.0/4) or the IPv6 network reserved for multicast
  (ff00::/8, wow, that's big).

- ``IPv4Interface``/``IPv6Interface`` - These hybrid classes refer to an
  individual address on a given network.  For example, the IPV4
  address 192.0.2.1 on the network 192.0.2.0/24 could be referred to
  as 192.0.2.1/24.  Likewise, the IPv6 address 2001:DB8::1 on the
  network 2001:DB8::/96 could be referred to as 2001:DB8::1/96.
  It's very common to refer to addresses assigned to computer
  network interfaces like this, hence the Interface name.

All IPv4 classes share certain characteristics and methods; the
number of bits needed to represent them, whether or not they
belong to certain special IPv4 network ranges, etc.  Similarly,
all IPv6 classes share characteristics and methods.

``ipaddr`` makes extensive use of inheritance to avoid code
duplication as much as possible.  The parent classes are private,
but they are outlined here:

- ``_IPAddrBase`` - Provides methods common to all ``ipaddr`` objects.

- ``_BaseAddress`` - Provides methods common to ``IPv4Address`` and
  ``IPv6Address``.

- ``_BaseInterface`` - Provides methods common to ``IPv4Interface`` and
  ``IPv6Interface``, as well as ``IPv4Network`` and ``IPv6Network`` (``ipaddr``
  treats the Network classes as a special case of Interface).

- ``_BaseV4`` - Provides methods and variables (eg, ``_max_prefixlen``)
  common to all IPv4 classes.

- ``_BaseV6`` - Provides methods and variables common to all IPv6 classes.

Comparisons between objects of differing IP versions results in a
``TypeError`` [1]_.  Additionally, comparisons of objects with
different _Base parent classes results in a ``TypeError``.  The effect
of the _Base parent class limitation is that ``IPv4Interface``'s can
be compared to ``IPv4Network``'s and ``IPv6Interface``'s can be compared
to ``IPv6Network``'s.


Reference Implementation
========================

The current reference implementation can be found at:

http://code.google.com/p/ipaddress-py/source/browse/ipaddress.py

Or see the tarball to include the README and unittests.
http://code.google.com/p/ipaddress-py/downloads/detail?name=ipaddress-1.0.tar.gz

More information about using the reference implementation can be
found at: http://code.google.com/p/ipaddr-py/wiki/Using3144


References
==========


.. [1] Appealing to authority is a logical fallacy, but Vint Cerf is an
       authority who can't be ignored. Full text of the email
       follows:

          I have seen a substantial amount of traffic about IPv4 and
          IPv6 comparisons and the general consensus is that these are
          not comparable.

          If we were to take a very simple minded view, we might treat
          these as pure integers in which case there is an ordering but
          not a useful one.

          In the IPv4 world, "length" is important because we take
          longest (most specific) address first for routing.  Length is
          determine by the mask, as you know.

          Assuming that the same style of argument works in IPv6, we
          would have to conclude that treating an IPv6 value purely as
          an integer for comparison with IPv4 would lead to some really
          strange results.

          All of IPv4 space would lie in the host space of 0::0/96
          prefix of IPv6. For any useful interpretation of IPv4, this is
          a non-starter.

          I think the only sensible conclusion is that IPv4 values and
          IPv6 values should be treated as non-comparable.

          Vint


Copyright
=========

This document has been placed in the public domain.


id: pep-3145.rst
content:
PEP: 3145
Title: Asynchronous I/O For subprocess.Popen
Author: Eric Pruitt, Charles R. McCreary, Josiah Carlson
Status: Withdrawn
Type: Standards Track
Created: 04-Aug-2009
Python-Version: 3.2
Post-History:


Abstract
========

In its present form, the ``subprocess.Popen`` implementation is prone to
dead-locking and blocking of the parent Python script while waiting on data
from the child process. This PEP proposes to make
``subprocess.Popen`` more asynchronous to help alleviate these
problems.


PEP Deferral
============

Further exploration of the concepts covered in this PEP has been deferred
at least until after :pep:`3156` has been resolved.


PEP Withdrawal
==============

This can be dealt with in the bug tracker.  A specific proposal is
attached to [11]_.


Motivation
==========

A search for "python asynchronous subprocess" will turn up numerous
accounts of people wanting to execute a child process and communicate with
it from time to time reading only the data that is available instead of
blocking to wait for the program to produce data [1]_ [2]_ [3]_.  The current
behavior of the ``subprocess`` module is that when a user sends or receives
data via the stdin, stderr and stdout file objects, dead locks are common
and documented [4]_ [5]_.  While communicate can be used to alleviate some of
the buffering issues, it will still cause the parent process to block while
attempting to read data when none is available to be read from the child
process.


Rationale
=========

There is a documented need for asynchronous, non-blocking functionality in
``subprocess.Popen`` [6]_ [7]_ [2]_ [3]_.  Inclusion of the code would improve the
utility of the Python standard library that can be used on Unix based and
Windows builds of Python.  Practically every I/O object in Python has a
file-like wrapper of some sort.  Sockets already act as such and for
strings there is ``StringIO``.  Popen can be made to act like a file by simply
using the methods attached to the ``subprocess.Popen.stderr``, stdout and
stdin file-like objects.  But when using the read and write methods of
those options, you do not have the benefit of asynchronous I/O.  In the
proposed solution the wrapper wraps the asynchronous methods to mimic a
file object.


Reference Implementation
========================

I have been maintaining a Google Code repository that contains all of my
changes including tests and documentation [9]_ as well as blog detailing
the problems I have come across in the development process [10]_.

I have been working on implementing non-blocking asynchronous I/O in the
``subprocess`` module as well as a wrapper class for ``subprocess.Popen``
that makes it so that an executed process can take the place of a file by
duplicating all of the methods and attributes that file objects have.

There are two base functions that have been added to the ``subprocess.Popen``
class: ``Popen.send`` and ``Popen._recv``, each with two separate implementations,
one for Windows and one for Unix-based systems.  The Windows
implementation uses ctypes to access the functions needed to control pipes
in the kernel 32 DLL in an asynchronous manner.  On Unix based systems,
the Python interface for file control serves the same purpose.  The
different implementations of ``Popen.send`` and ``Popen._recv`` have identical
arguments to make code that uses these functions work across multiple
platforms.

When calling the ``Popen._recv`` function, it requires the pipe name be
passed as an argument so there exists the ``Popen.recv`` function that passes
selects stdout as the pipe for ``Popen._recv`` by default. ``Popen.recv_err``
selects stderr as the pipe by default. ``Popen.recv`` and ``Popen.recv_err``
are much easier to read and understand than ``Popen._recv('stdout' ...)`` and
``Popen._recv('stderr' ...)`` respectively.

Since the ``Popen._recv`` function does not wait on data to be produced
before returning a value, it may return empty bytes. ``Popen.asyncread``
handles this issue by returning all data read over a given time
interval.

The ``ProcessIOWrapper`` class uses the ``asyncread`` and ``asyncwrite`` functions to
allow a process to act like a file so that there are no blocking issues
that can arise from using the stdout and stdin file objects produced from
a ``subprocess.Popen`` call.


References
==========

.. [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess
       https://mail.python.org/pipermail/python-bugs-list/2006-December/036524.html

.. [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess
       http://ivory.idyll.org/blog/problems-with-subprocess.html

.. [3] How can I run an external command asynchronously from Python? - Stack
       Overflow
       https://stackoverflow.com/q/636561

.. [4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation
       https://docs.python.org/2.6/library/subprocess.html#subprocess.Popen.wait

.. [5] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation
       https://docs.python.org/2.6/library/subprocess.html#subprocess.Popen.kill

.. [6] Issue 1191964: asynchronous Subprocess - Python tracker
       https://github.com/python/cpython/issues/41922

.. [7] Module to allow Asynchronous subprocess use on Windows and Posix
       platforms - ActiveState Code
       https://code.activestate.com/recipes/440554/

[8] subprocess.rst - subprocdev - Project Hosting on Google Code
\   https://web.archive.org/web/20130306074135/http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=svn2c925e935cad0166d5da85e37c742d8e7f609de5&r=2c925e935cad0166d5da85e37c742d8e7f609de5

.. [9] subprocdev - Project Hosting on Google Code
       https://code.google.com/archive/p/subprocdev/

.. [10] Python Subprocess Dev
        https://subdev.blogspot.com/

.. [11] https://github.com/python/cpython/issues/63023 -- Idle: use pipes instead of
        sockets to talk with user subprocess

Copyright
=========

This P.E.P. is licensed under the Open Publication License;
http://www.opencontent.org/openpub/.


id: pep-3151.rst
content:
PEP: 3151
Title: Reworking the OS and IO exception hierarchy
Author: Antoine Pitrou <solipsis@pitrou.net>
BDFL-Delegate: Barry Warsaw
Status: Final
Type: Standards Track
Created: 21-Jul-2010
Python-Version: 3.3
Post-History:
Resolution: https://mail.python.org/pipermail/python-dev/2011-October/114033.html

Abstract
========

The standard exception hierarchy is an important part of the Python
language.  It has two defining qualities: it is both generic and
selective.  Generic in that the same exception type can be raised
- and handled - regardless of the context (for example, whether you are
trying to add something to an integer, to call a string method, or to write
an object on a socket, a TypeError will be raised for bad argument types).
Selective in that it allows the user to easily handle (silence, examine,
process, store or encapsulate...) specific kinds of error conditions
while letting other errors bubble up to higher calling contexts.  For
example, you can choose to catch ZeroDivisionErrors without affecting
the default handling of other ArithmeticErrors (such as OverflowErrors).

This PEP proposes changes to a part of the exception hierarchy in
order to better embody the qualities mentioned above: the errors
related to operating system calls (OSError, IOError, mmap.error,
select.error, and all their subclasses).


Rationale
=========

Confusing set of OS-related exceptions
--------------------------------------

OS-related (or system call-related) exceptions are currently a diversity
of classes, arranged in the following sub-hierarchies::

    +-- EnvironmentError
        +-- IOError
            +-- io.BlockingIOError
            +-- io.UnsupportedOperation (also inherits from ValueError)
            +-- socket.error
                +-- socket.gaierror
                +-- socket.herror
                +-- socket.timeout
        +-- OSError
            +-- VMSError
            +-- WindowsError
        +-- mmap.error
    +-- select.error

While some of these distinctions can be explained by implementation
considerations, they are often not very logical at a higher level.  The
line separating OSError and IOError, for example, is often blurry.  Consider
the following::

    >>> os.remove("fff")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    OSError: [Errno 2] No such file or directory: 'fff'
    >>> open("fff")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: [Errno 2] No such file or directory: 'fff'

The same error condition (a non-existing file) gets cast as two different
exceptions depending on which library function was called.  The reason
for this is that the ``os`` module exclusively raises OSError (or its
subclass WindowsError) while the ``io`` module mostly raises IOError.
However, the user is interested in the nature of the error, not in which
part of the interpreter it comes from (since the latter is obvious from
reading the traceback message or application source code).

In fact, it is hard to think of any situation where OSError should be
caught but not IOError, or the reverse.

A further proof of the ambiguity of this segmentation is that the standard
library itself sometimes has problems deciding.  For example, in the
``select`` module, similar failures will raise ``select.error``, ``OSError``
or ``IOError`` depending on whether you are using select(), a poll object,
a kqueue object, or an epoll object.  This makes user code uselessly
complicated since it has to be prepared to catch various exception types,
depending on which exact implementation of a single primitive it chooses
to use at runtime.

As for WindowsError, it seems to be a pointless distinction.  First, it
only exists on Windows systems, which requires tedious compatibility code
in cross-platform applications (such code can be found in ``Lib/shutil.py``).
Second, it inherits from OSError and is raised for similar errors as OSError
is raised for on other systems. Third, the user wanting access to low-level
exception specifics has to examine the ``errno`` or ``winerror`` attribute
anyway.

.. note::
    `Appendix B <PEP 3151 Appendix B_>`_ surveys the use of the
    various exception types across the interpreter and the standard library.


Lack of fine-grained exceptions
-------------------------------

The current variety of OS-related exceptions doesn't allow the user to filter
easily for the desired kinds of failures.  As an example, consider the task
of deleting a file if it exists.  The Look Before You Leap (LBYL) idiom
suffers from an obvious race condition::

    if os.path.exists(filename):
        os.remove(filename)

If a file named as ``filename`` is created by another thread or process
between the calls to ``os.path.exists`` and ``os.remove``, it won't be
deleted.  This can produce bugs in the application, or even security issues.

Therefore, the solution is to try to remove the file, and ignore the error
if the file doesn't exist (an idiom known as Easier to Ask Forgiveness
than to get Permission, or EAFP).  Careful code will read like the following
(which works under both POSIX and Windows systems)::

    try:
        os.remove(filename)
    except OSError as e:
        if e.errno != errno.ENOENT:
            raise

or even::

    try:
        os.remove(filename)
    except EnvironmentError as e:
        if e.errno != errno.ENOENT:
            raise

This is a lot more to type, and also forces the user to remember the various
cryptic mnemonics from the ``errno`` module.  It imposes an additional
cognitive burden and gets tiresome rather quickly.  Consequently, many
programmers will instead write the following code, which silences exceptions
too broadly::

    try:
        os.remove(filename)
    except OSError:
        pass

``os.remove`` can raise an OSError not only when the file doesn't exist,
but in other possible situations (for example, the filename points to a
directory, or the current process doesn't have permission to remove
the file), which all indicate bugs in the application logic and therefore
shouldn't be silenced.  What the programmer would like to write instead is
something such as::

    try:
        os.remove(filename)
    except FileNotFoundError:
        pass


Compatibility strategy
======================

Reworking the exception hierarchy will obviously change the exact semantics
of at least some existing code.  While it is not possible to improve on the
current situation without changing exact semantics, it is possible to define
a narrower type of compatibility, which we will call *useful compatibility*.

For this we first must explain what we will call *careful* and *careless*
exception handling.  *Careless* (or "naÃ¯ve") code is defined as code which
blindly catches any of ``OSError``, ``IOError``, ``socket.error``,
``mmap.error``, ``WindowsError``, ``select.error`` without checking the ``errno``
attribute.  This is because such exception types are much too broad to signify
anything.  Any of them can be raised for error conditions as diverse as: a
bad file descriptor (which will usually indicate a programming error), an
unconnected socket (ditto), a socket timeout, a file type mismatch, an invalid
argument, a transmission failure, insufficient permissions, a non-existent
directory, a full filesystem, etc.

(moreover, the use of certain of these exceptions is irregular; `Appendix B
<PEP 3151 Appendix B_>`_ exposes the case of the `select`_ module,
which raises different exceptions depending on the implementation)

*Careful* code is defined as code which, when catching any of the above
exceptions, examines the ``errno`` attribute to determine the actual error
condition and takes action depending on it.

Then we can define *useful compatibility* as follows:

* useful compatibility doesn't make exception catching any narrower, but
  it can be broader for *careless* exception-catching code.  Given the following
  kind of snippet, all exceptions caught be


id: pep-0535.rst
content:
PEP: 535
Title: Rich comparison chaining
Author: Alyssa Coghlan <ncoghlan@gmail.com>
Status: Deferred
Type: Standards Track
Requires: 532
Created: 12-Nov-2016
Python-Version: 3.8

PEP Deferral
============

Further consideration of this PEP has been deferred until Python 3.8 at the
earliest.


Abstract
========

Inspired by :pep:`335`, and building on the circuit breaking protocol described
in :pep:`532`, this PEP proposes a change to the definition of chained comparisons,
where the comparison chaining will be updated to use the left-associative
circuit breaking operator (``else``) rather than the logical disjunction
operator  (``and``) if the left hand comparison returns a circuit breaker as
its result.

While there are some practical complexities arising from the current handling
of single-valued arrays in NumPy, this change should be sufficient to allow
elementwise chained comparison operations for matrices, where the result
is a matrix of boolean values, rather than raising ``ValueError``
or tautologically returning ``True`` (indicating a non-empty matrix).


Relationship with other PEPs
============================

This PEP has been extracted from earlier iterations of :pep:`532`, as a
follow-on use case for the circuit breaking protocol, rather than an essential
part of its introduction.

The specific proposal in this PEP to handle the element-wise comparison use
case by changing the semantic definition of comparison chaining is drawn
directly from Guido's rejection of :pep:`335`.


Specification
=============

A chained comparison like ``0 < x < 10`` written as::

    LEFT_BOUND LEFT_OP EXPR RIGHT_OP RIGHT_BOUND

is currently roughly semantically equivalent to::

    _expr = EXPR
    _lhs_result = LEFT_BOUND LEFT_OP _expr
    _expr_result = _lhs_result and (_expr RIGHT_OP RIGHT_BOUND)

Using the circuit breaking concepts introduced in :pep:`532`, this PEP proposes
that comparison chaining be changed to explicitly check if the left comparison
returns a circuit breaker, and if so, use ``else`` rather than ``and`` to
implement the comparison chaining::

    _expr = EXPR
    _lhs_result = LEFT_BOUND LEFT_OP _expr
    if hasattr(type(_lhs_result), "__else__"):
        _expr_result = _lhs_result else (_expr RIGHT_OP RIGHT_BOUND)
    else:
        _expr_result = _lhs_result and (_expr RIGHT_OP RIGHT_BOUND)

This allows types like NumPy arrays to control the behaviour of chained
comparisons by returning suitably defined circuit breakers from comparison
operations.

The expansion of this logic to an arbitrary number of chained comparison
operations would be the same as the existing expansion for ``and``.

Rationale
=========

In ultimately rejecting :pep:`335`, Guido van Rossum noted [1]_:

    The NumPy folks brought up a somewhat separate issue: for them,
    the most common use case is chained comparisons (e.g. A < B < C).

To understand this observation, we first need to look at how comparisons work
with NumPy arrays::

    >>> import numpy as np
    >>> increasing = np.arange(5)
    >>> increasing
    array([0, 1, 2, 3, 4])
    >>> decreasing = np.arange(4, -1, -1)
    >>> decreasing
    array([4, 3, 2, 1, 0])
    >>> increasing < decreasing
    array([ True,  True, False, False, False], dtype=bool)

Here we see that NumPy array comparisons are element-wise by default, comparing
each element in the left hand array to the corresponding element in the right
hand array, and producing a matrix of boolean results.

If either side of the comparison is a scalar value, then it is broadcast across
the array and compared to each individual element::

    >>> 0 < increasing
    array([False,  True,  True,  True,  True], dtype=bool)
    >>> increasing < 4
    array([ True,  True,  True,  True, False], dtype=bool)

However, this broadcasting idiom breaks down if we attempt to use chained
comparisons::

    >>> 0 < increasing < 4
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

The problem is that internally, Python implicitly expands this chained
comparison into the form::

    >>> 0 < increasing and increasing < 4
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

And NumPy only permits implicit coercion to a boolean value for single-element
arrays where ``a.any()`` and ``a.all()`` can be assured of having the same
result::

    >>> np.array([False]) and np.array([False])
    array([False], dtype=bool)
    >>> np.array([False]) and np.array([True])
    array([False], dtype=bool)
    >>> np.array([True]) and np.array([False])
    array([False], dtype=bool)
    >>> np.array([True]) and np.array([True])
    array([ True], dtype=bool)

The proposal in this PEP would allow this situation to be changed by updating
the definition of element-wise comparison operations in NumPy to return a
dedicated subclass that implements the new circuit breaking protocol and also
changes the result array's interpretation in a boolean context to always
return ``False`` and hence never trigger the short-circuiting behaviour::

    class ComparisonResultArray(np.ndarray):
        def __bool__(self):
            # Element-wise comparison chaining never short-circuits
            return False
        def _raise_NotImplementedError(self):
            msg = ("Comparison array truth values are ambiguous outside "
                   "chained comparisons. Use a.any() or a.all()")
            raise NotImplementedError(msg)
        def __not__(self):
            self._raise_NotImplementedError()
        def __then__(self, result):
            self._raise_NotImplementedError()
        def __else__(self, result):
            return np.logical_and(self, other.view(ComparisonResultArray))

With this change, the chained comparison example above would be able to return::

    >>> 0 < increasing < 4
    ComparisonResultArray([ False,  True,  True,  True, False], dtype=bool)


Implementation
==============

Actual implementation has been deferred pending in-principle interest in the
idea of making the changes proposed in :pep:`532`.

...TBD...


References
==========

.. [1] PEP 335 rejection notification
   (https://mail.python.org/pipermail/python-dev/2012-March/117510.html)

Copyright
=========

This document has been placed in the public domain under the terms of the
CC0 1.0 license: https://creativecommons.org/publicdomain/zero/1.0/


id: pep-0253.rst
content:
PEP: 253
Title: Subtyping Built-in Types
Author: Guido van Rossum <guido@python.org>
Status: Final
Type: Standards Track
Created: 14-May-2001
Python-Version: 2.2
Post-History:

Abstract
========

This PEP proposes additions to the type object API that will allow
the creation of subtypes of built-in types, in C and in Python.

[Editor's note: the ideas described in this PEP have been incorporated
into Python.  The PEP no longer accurately describes the implementation.]


Introduction
============

Traditionally, types in Python have been created statically, by
declaring a global variable of type PyTypeObject and initializing
it with a static initializer.  The slots in the type object
describe all aspects of a Python type that are relevant to the
Python interpreter.  A few slots contain dimensional information
(like the basic allocation size of instances), others contain
various flags, but most slots are pointers to functions to
implement various kinds of behaviors.  A NULL pointer means that
the type does not implement the specific behavior; in that case
the system may provide a default behavior or raise an exception
when the behavior is invoked for an instance of the type.  Some
collections of function pointers that are usually defined together
are obtained indirectly via a pointer to an additional structure
containing more function pointers.

While the details of initializing a PyTypeObject structure haven't
been documented as such, they are easily gleaned from the examples
in the source code, and I am assuming that the reader is
sufficiently familiar with the traditional way of creating new
Python types in C.

This PEP will introduce the following features:

- a type can be a factory function for its instances

- types can be subtyped in C

- types can be subtyped in Python with the class statement

- multiple inheritance from types is supported (insofar as
  practical -- you still can't multiply inherit from list and
  dictionary)

- the standard coercion functions (int, tuple, str etc.) will
  be redefined to be the corresponding type objects, which serve
  as their own factory functions

- a class statement can contain a ``__metaclass__`` declaration,
  specifying the metaclass to be used to create the new class

- a class statement can contain a ``__slots__`` declaration,
  specifying the specific names of the instance variables
  supported

This PEP builds on :pep:`252`, which adds standard introspection to
types; for example, when a particular type object initializes the
``tp_hash`` slot, that type object has a ``__hash__`` method when
introspected.  :pep:`252` also adds a dictionary to type objects
which contains all methods.  At the Python level, this dictionary
is read-only for built-in types; at the C level, it is accessible
directly (but it should not be modified except as part of
initialization).

For binary compatibility, a flag bit in the tp_flags slot
indicates the existence of the various new slots in the type
object introduced below.  Types that don't have the
``Py_TPFLAGS_HAVE_CLASS`` bit set in their ``tp_flags`` slot are assumed
to have NULL values for all the subtyping slots.  (Warning: the
current implementation prototype is not yet consistent in its
checking of this flag bit.  This should be fixed before the final
release.)

In current Python, a distinction is made between types and
classes.  This PEP together with :pep:`254` will remove that
distinction.  However, for backwards compatibility the distinction
will probably remain for years to come, and without :pep:`254`, the
distinction is still large: types ultimately have a built-in type
as a base class, while classes ultimately derive from a
user-defined class.  Therefore, in the rest of this PEP, I will
use the word type whenever I can -- including base type or
supertype, derived type or subtype, and metatype.  However,
sometimes the terminology necessarily blends, for example an
object's type is given by its ``__class__`` attribute, and subtyping
in Python is spelled with a class statement.  If further
distinction is necessary, user-defined classes can be referred to
as "classic" classes.


About metatypes
===============

Inevitably the discussion comes to metatypes (or metaclasses).
Metatypes are nothing new in Python: Python has always been able
to talk about the type of a type::

    >>> a = 0
    >>> type(a)
    <type 'int'>
    >>> type(type(a))
    <type 'type'>
    >>> type(type(type(a)))
    <type 'type'>
    >>>

In this example, ``type(a)`` is a "regular" type, and ``type(type(a))`` is
a metatype.  While as distributed all types have the same metatype
(``PyType_Type``, which is also its own metatype), this is not a
requirement, and in fact a useful and relevant 3rd party extension
(ExtensionClasses by Jim Fulton) creates an additional metatype.
The type of classic classes, known as ``types.ClassType``, can also be
considered a distinct metatype.

A feature closely connected to metatypes is the "Don Beaudry
hook", which says that if a metatype is callable, its instances
(which are regular types) can be subclassed (really subtyped)
using a Python class statement.  I will use this rule to support
subtyping of built-in types, and in fact it greatly simplifies the
logic of class creation to always simply call the metatype.  When
no base class is specified, a default metatype is called -- the
default metatype is the "ClassType" object, so the class statement
will behave as before in the normal case.  (This default can be
changed per module by setting the global variable ``__metaclass__``.)

Python uses the concept of metatypes or metaclasses in a different
way than Smalltalk.  In Smalltalk-80, there is a hierarchy of
metaclasses that mirrors the hierarchy of regular classes,
metaclasses map 1-1 to classes (except for some funny business at
the root of the hierarchy), and each class statement creates both
a regular class and its metaclass, putting class methods in the
metaclass and instance methods in the regular class.

Nice though this may be in the context of Smalltalk, it's not
compatible with the traditional use of metatypes in Python, and I
prefer to continue in the Python way.  This means that Python
metatypes are typically written in C, and may be shared between
many regular types. (It will be possible to subtype metatypes in
Python, so it won't be absolutely necessary to write C to use
metatypes; but the power of Python metatypes will be limited.  For
example, Python code will never be allowed to allocate raw memory
and initialize it at will.)

Metatypes determine various **policies** for types, such as what
happens when a type is called, how dynamic types are (whether a
type's ``__dict__`` can be modified after it is created), what the
method resolution order is, how instance attributes are looked
up, and so on.

I'll argue that left-to-right depth-first is not the best
solution when you want to get the most use from multiple
inheritance.

I'll argue that with multiple inheritance, the metatype of the
subtype must be a descendant of the metatypes of all base types.

I'll come back to metatypes later.


Making a type a factory for its instances
=========================================

Traditionally, for each type there is at least one C factory
function that creates instances of the type (``PyTuple_New()``,
``PyInt_FromLong()`` and so on).  These factory functions take care of
both allocating memory for the object and initializing that
memory.  As of Python 2.0, they also have to interface with the
garbage collection subsystem, if the type chooses to participate
in garbage collection (which is optional, but strongly recommended
for so-called "container" types: types that may contain references
to other objects, and hence may participate in reference cycles).

In this proposal, type objects can be factory functions for their
instances, making the types directly callable from Python.  This
mimics the way classes are instantiated.  The C APIs for creating
i


id: pep-0247.rst
content:
PEP: 247
Title: API for Cryptographic Hash Functions
Author: A.M. Kuchling <amk@amk.ca>
Status: Final
Type: Informational
Created: 23-Mar-2001
Post-History: 20-Sep-2001


Abstract
========

There are several different modules available that implement cryptographic
hashing algorithms such as MD5 or SHA.  This document specifies a standard API
for such algorithms, to make it easier to switch between different
implementations.


Specification
=============

All hashing modules should present the same interface.  Additional methods or
variables can be added, but those described in this document should always be
present.

Hash function modules define one function:

| ``new([string])            (unkeyed hashes)``
| ``new([key] , [string])    (keyed hashes)``

   Create a new hashing object and return it.  The first form is for hashes
   that are unkeyed, such as MD5 or SHA.  For keyed hashes such as HMAC, *key*
   is a required parameter containing a string giving the key to use.  In both
   cases, the optional *string* parameter, if supplied, will be immediately
   hashed into the object's starting state, as if ``obj.update(string)``
   was called.

   After creating a hashing object, arbitrary strings can be fed into the
   object using its ``update()`` method, and the hash value can be obtained at
   any time by calling the object's ``digest()`` method.

   Arbitrary additional keyword arguments can be added to this function, but if
   they're not supplied, sensible default values should be used.  For example,
   ``rounds`` and ``digest_size`` keywords could be added for a hash function
   which supports a variable number of rounds and several different output
   sizes, and they should default to values believed to be secure.

Hash function modules define one variable:

| ``digest_size``

   An integer value; the size of the digest produced by the hashing objects
   created by this module, measured in bytes. You could also obtain this value
   by creating a sample object and accessing its ``digest_size`` attribute, but
   it can be convenient to have this value available from the module. Hashes
   with a variable output size will set this variable to ``None``.

Hashing objects require a single attribute:

| ``digest_size``

   This attribute is identical to the module-level ``digest_size`` variable,
   measuring the size of the digest produced by the hashing object, measured in
   bytes.  If the hash has a variable output size, this output size must be
   chosen when the hashing object is created, and this attribute must contain
   the selected size.  Therefore, ``None`` is *not* a legal value for this
   attribute.


Hashing objects require the following methods:

| ``copy()``

   Return a separate copy of this hashing object.  An update to this copy won't
   affect the original object.

| ``digest()``

   Return the hash value of this hashing object as a string containing 8-bit
   data.  The object is not altered in any way by this function; you can
   continue updating the object after calling this function.

| ``hexdigest()``

   Return the hash value of this hashing object as a string containing
   hexadecimal digits.  Lowercase letters should be used for the digits ``a``
   through ``f``.  Like the ``.digest()`` method, this method mustn't alter the
   object.

| ``update(string)``

   Hash *string* into the current state of the hashing object. ``update()`` can
   be called any number of times during a hashing object's lifetime.

Hashing modules can define additional module-level functions or object methods
and still be compliant with this specification.

Here's an example, using a module named ``MD5``::

    >>> from Crypto.Hash import MD5
    >>> m = MD5.new()
    >>> m.digest_size
    16
    >>> m.update('abc')
    >>> m.digest()
    '\x90\x01P\x98<\xd2O\xb0\xd6\x96?}(\xe1\x7fr'
    >>> m.hexdigest()
    '900150983cd24fb0d6963f7d28e17f72'
    >>> MD5.new('abc').digest()
    '\x90\x01P\x98<\xd2O\xb0\xd6\x96?}(\xe1\x7fr'


Rationale
=========

The digest size is measured in bytes, not bits, even though hash algorithm
sizes are usually quoted in bits; MD5 is a 128-bit algorithm and not a 16-byte
one, for example.  This is because, in the sample code I looked at, the length
in bytes is often needed (to seek ahead or behind in a file; to compute the
length of an output string) while the length in bits is rarely used. Therefore,
the burden will fall on the few people actually needing the size in bits, who
will have to multiply ``digest_size`` by 8.

It's been suggested that the ``update()`` method would be better named
``append()``.  However, that method is really causing the current state of the
hashing object to be updated, and ``update()`` is already used by the md5 and
sha modules included with Python, so it seems simplest to leave the name
``update()`` alone.

The order of the constructor's arguments for keyed hashes was a sticky issue.
It wasn't clear whether the *key* should come first or second.  It's a required
parameter, and the usual convention is to place required parameters first, but
that also means that the *string* parameter moves from the first position to
the second. It would be possible to get confused and pass a single argument to
a keyed hash, thinking that you're passing an initial string to an unkeyed
hash, but it doesn't seem worth making the interface for keyed hashes more
obscure to avoid this potential error.


Changes
=======

2001-09-17: Renamed ``clear()`` to ``reset()``; added ``digest_size`` attribute
to objects; added ``.hexdigest()`` method.

2001-09-20: Removed ``reset()`` method completely.

2001-09-28: Set ``digest_size`` to ``None`` for variable-size hashes.


Acknowledgements
================

Thanks to Aahz, Andrew Archibald, Rich Salz, Itamar Shtull-Trauring, and the
readers of the python-crypto list for their comments on this PEP.


Copyright
=========

This document has been placed in the public domain.


id: pep-0521.rst
content:
PEP: 521
Title: Managing global context via 'with' blocks in generators and coroutines
Author: Nathaniel J. Smith <njs@pobox.com>
Status: Withdrawn
Type: Standards Track
Created: 27-Apr-2015
Python-Version: 3.6
Post-History: 29-Apr-2015

PEP Withdrawal
==============

Withdrawn in favor of :pep:`567`.


Abstract
========

While we generally try to avoid global state when possible, there
nonetheless exist a number of situations where it is agreed to be the
best approach.  In Python, a standard pattern for handling such cases
is to store the global state in global or thread-local storage, and
then use ``with`` blocks to limit modifications of this global state
to a single dynamic scope. Examples where this pattern is used include
the standard library's ``warnings.catch_warnings`` and
``decimal.localcontext``, NumPy's ``numpy.errstate`` (which exposes
the error-handling settings provided by the IEEE 754 floating point
standard), and the handling of logging context or HTTP request context
in many server application frameworks.

However, there is currently no ergonomic way to manage such local
changes to global state when writing a generator or coroutine. For
example, this code::

  def f():
      with warnings.catch_warnings():
          for x in g():
              yield x

may or may not successfully catch warnings raised by ``g()``, and may
or may not inadvertently swallow warnings triggered elsewhere in the
code.  The context manager, which was intended to apply only to ``f``
and its callees, ends up having a dynamic scope that encompasses
arbitrary and unpredictable parts of its call\ **ers**. This problem
becomes particularly acute when writing asynchronous code, where
essentially all functions become coroutines.

Here, we propose to solve this problem by notifying context managers
whenever execution is suspended or resumed within their scope,
allowing them to restrict their effects appropriately.


Specification
=============

Two new, optional, methods are added to the context manager protocol:
``__suspend__`` and ``__resume__``.  If present, these methods will be
called whenever a frame's execution is suspended or resumed from
within the context of the ``with`` block.

More formally, consider the following code::

  with EXPR as VAR:
      PARTIAL-BLOCK-1
      f((yield foo))
      PARTIAL-BLOCK-2

Currently this is equivalent to the following code (copied from :pep:`343`)::

  mgr = (EXPR)
  exit = type(mgr).__exit__  # Not calling it yet
  value = type(mgr).__enter__(mgr)
  exc = True
  try:
      try:
          VAR = value  # Only if "as VAR" is present
          PARTIAL-BLOCK-1
          f((yield foo))
          PARTIAL-BLOCK-2
      except:
          exc = False
          if not exit(mgr, *sys.exc_info()):
              raise
  finally:
      if exc:
          exit(mgr, None, None, None)

This PEP proposes to modify ``with`` block handling to instead become::

  mgr = (EXPR)
  exit = type(mgr).__exit__  # Not calling it yet
  ### --- NEW STUFF ---
  if the_block_contains_yield_points:  # known statically at compile time
      suspend = getattr(type(mgr), "__suspend__", lambda: None)
      resume = getattr(type(mgr), "__resume__", lambda: None)
  ### --- END OF NEW STUFF ---
  value = type(mgr).__enter__(mgr)
  exc = True
  try:
      try:
          VAR = value  # Only if "as VAR" is present
          PARTIAL-BLOCK-1
          ### --- NEW STUFF ---
          suspend(mgr)
          tmp = yield foo
          resume(mgr)
          f(tmp)
          ### --- END OF NEW STUFF ---
          PARTIAL-BLOCK-2
      except:
          exc = False
          if not exit(mgr, *sys.exc_info()):
              raise
  finally:
      if exc:
          exit(mgr, None, None, None)

Analogous suspend/resume calls are also wrapped around the ``yield``
points embedded inside the ``yield from``, ``await``, ``async with``,
and ``async for`` constructs.


Nested blocks
-------------

Given this code::

  def f():
      with OUTER:
          with INNER:
              yield VALUE

then we perform the following operations in the following sequence::

  INNER.__suspend__()
  OUTER.__suspend__()
  yield VALUE
  OUTER.__resume__()
  INNER.__resume__()

Note that this ensures that the following is a valid refactoring::

  def f():
      with OUTER:
          yield from g()

  def g():
      with INNER
          yield VALUE

Similarly, ``with`` statements with multiple context managers suspend
from right to left, and resume from left to right.


Other changes
-------------

Appropriate ``__suspend__`` and ``__resume__`` methods are added to
``warnings.catch_warnings`` and ``decimal.localcontext``.


Rationale
=========

In the abstract, we gave an example of plausible but incorrect code::

  def f():
      with warnings.catch_warnings():
          for x in g():
              yield x

To make this correct in current Python, we need to instead write
something like::

  def f():
      with warnings.catch_warnings():
          it = iter(g())
      while True:
          with warnings.catch_warnings():
              try:
                  x = next(it)
              except StopIteration:
                  break
          yield x

OTOH, if this PEP is accepted then the original code will become
correct as-is.  Or if this isn't convincing, then here's another
example of broken code; fixing it requires even greater gyrations, and
these are left as an exercise for the reader::

  async def test_foo_emits_warning():
      with warnings.catch_warnings(record=True) as w:
          await foo()
      assert len(w) == 1
      assert "xyzzy" in w[0].message

And notice that this last example isn't artificial at all -- this is
exactly how you write a test that an async/await-using coroutine
correctly raises a warning.  Similar issues arise for pretty much any
use of ``warnings.catch_warnings``, ``decimal.localcontext``, or
``numpy.errstate`` in async/await-using code.  So there's clearly a
real problem to solve here, and the growing prominence of async code
makes it increasingly urgent.


Alternative approaches
----------------------

The main alternative that has been proposed is to create some kind of
"task-local storage", analogous to "thread-local storage"
[#yury-task-local-proposal]_. In essence, the idea would be that the
event loop would take care to allocate a new "task namespace" for each
task it schedules, and provide an API to at any given time fetch the
namespace corresponding to the currently executing task.  While there
are many details to be worked out [#task-local-challenges]_, the basic
idea seems doable, and it is an especially natural way to handle the
kind of global context that arises at the top-level of async
application frameworks (e.g., setting up context objects in a web
framework).  But it also has a number of flaws:

* It only solves the problem of managing global state for coroutines
  that ``yield`` back to an asynchronous event loop.  But there
  actually isn't anything about this problem that's specific to
  asyncio -- as shown in the examples above, simple generators run
  into exactly the same issue.

* It creates an unnecessary coupling between event loops and code that
  needs to manage global state. Obviously an async web framework needs
  to interact with some event loop API anyway, so it's not a big deal
  in that case. But it's weird that ``warnings`` or ``decimal`` or
  NumPy should have to call into an async library's API to access
  their internal state when they themselves involve no async code.
  Worse, since there are multiple event loop APIs in common use, it
  isn't clear how to choose which to integrate with.  (This could be
  somewhat mitigated by CPython providing a standard API for creating
  and switching "task-local domains" that asyncio, Twisted, tornado,
  etc. could then work with.)

* It's not at all clear that this can be made acceptably fast.  NumPy
  has to check the floating point error settings on every single
  arithmetic operation.


id: pep-0509.rst
content:
PEP: 509
Title: Add a private version to dict
Author: Victor Stinner <vstinner@python.org>
Status: Superseded
Type: Standards Track
Created: 04-Jan-2016
Python-Version: 3.6
Post-History: `08-Jan-2016 <https://mail.python.org/archives/list/python-ideas@python.org/thread/FPB7MLWMRSCYQVOUYN2SUV4NR47TPPG3/>`__,
              `11-Jan-2016 <https://mail.python.org/archives/list/python-dev@python.org/thread/AY42HER5DFKA7DJV25AL7YX5DQPF6RMV/>`__,
              `14-Apr-2016 <https://mail.python.org/archives/list/python-dev@python.org/thread/UXEQIDSTVLAHJVDQJMJPCU2QZGYMBV2H/>`__,
              `19-Apr-2016 <https://mail.python.org/archives/list/python-dev@python.org/thread/2S562SVRK5S2QKP3SFONG357Z72I6KLE/>`__
Superseded-By: 699
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/QFVJV6YQOUSWIYY4FBORY647YCBSCIMQ/


Abstract
========

Add a new private version to the builtin ``dict`` type, incremented at
each dictionary creation and at each dictionary change, to implement
fast guards on namespaces.


Rationale
=========

In Python, the builtin ``dict`` type is used by many instructions. For
example, the ``LOAD_GLOBAL`` instruction looks up a variable in the
global namespace, or in the builtins namespace (two dict lookups).
Python uses ``dict`` for the builtins namespace, globals namespace, type
namespaces, instance namespaces, etc. The local namespace (function
namespace) is usually optimized to an array, but it can be a dict too.

Python is hard to optimize because almost everything is mutable: builtin
functions, function code, global variables, local variables, ... can be
modified at runtime. Implementing optimizations respecting the Python
semantics requires to detect when "something changes": we will call
these checks "guards".

The speedup of optimizations depends on the speed of guard checks. This
PEP proposes to add a private version to dictionaries to implement fast
guards on namespaces.

Dictionary lookups can be skipped if the version does not change, which
is the common case for most namespaces. The version is globally unique,
so checking the version is also enough to verify that the namespace
dictionary was not replaced with a new dictionary.

When the dictionary version does not change, the performance of a guard
does not depend on the number of watched dictionary entries: the
complexity is O(1).

Example of optimization: copy the value of a global variable to function
constants.  This optimization requires a guard on the global variable to
check if it was modified after it was copied. If the global variable is
not modified, the function uses the cached copy. If the global variable
is modified, the function uses a regular lookup, and maybe also
deoptimizes the function (to remove the overhead of the guard check for
next function calls).

See the :pep:`510 -- Specialized functions with guards <510>`
for concrete usage of
guards to specialize functions and for a more general rationale on
Python static optimizers.


Guard example
=============

Pseudo-code of a fast guard to check if a dictionary entry was modified
(created, updated or deleted) using a hypothetical
``dict_get_version(dict)`` function::

    UNSET = object()

    class GuardDictKey:
        def __init__(self, dict, key):
            self.dict = dict
            self.key = key
            self.value = dict.get(key, UNSET)
            self.version = dict_get_version(dict)

        def check(self):
            """Return True if the dictionary entry did not change
            and the dictionary was not replaced."""

            # read the version of the dictionary
            version = dict_get_version(self.dict)
            if version == self.version:
                # Fast-path: dictionary lookup avoided
                return True

            # lookup in the dictionary
            value = self.dict.get(self.key, UNSET)
            if value is self.value:
                # another key was modified:
                # cache the new dictionary version
                self.version = version
                return True

            # the key was modified
            return False


Usage of the dict version
=========================

Speedup method calls
--------------------

Yury Selivanov wrote a `patch to optimize method calls
<https://bugs.python.org/issue26110>`_. The patch depends on the
`"implement per-opcode cache in ceval"
<https://bugs.python.org/issue26219>`_ patch which requires dictionary
versions to invalidate the cache if the globals dictionary or the
builtins dictionary has been modified.

The cache also requires that the dictionary version is globally unique.
It is possible to define a function in a namespace and call it in a
different namespace, using ``exec()`` with the *globals* parameter for
example. In this case, the globals dictionary was replaced and the cache
must also be invalidated.


Specialized functions using guards
----------------------------------

:pep:`510` proposes an API to support
specialized functions with guards. It allows to implement static
optimizers for Python without breaking the Python semantics.

The `fatoptimizer <http://fatoptimizer.readthedocs.org/>`_ of the `FAT
Python <http://faster-cpython.readthedocs.org/fat_python.html>`_ project
is an example of a static Python optimizer. It implements many
optimizations which require guards on namespaces:

* Call pure builtins: to replace ``len("abc")`` with ``3``, guards on
  ``builtins.__dict__['len']`` and ``globals()['len']`` are required
* Loop unrolling: to unroll the loop ``for i in range(...): ...``,
  guards on ``builtins.__dict__['range']`` and ``globals()['range']``
  are required
* etc.


Pyjion
------

According of Brett Cannon, one of the two main developers of Pyjion,
Pyjion can benefit from dictionary version to implement optimizations.

`Pyjion <https://github.com/Microsoft/Pyjion>`_ is a JIT compiler for
Python based upon CoreCLR (Microsoft .NET Core runtime).


Cython
------

Cython can benefit from dictionary version to implement optimizations.

`Cython <http://cython.org/>`_ is an optimising static compiler for both
the Python programming language and the extended Cython programming
language.


Unladen Swallow
---------------

Even if dictionary version was not explicitly mentioned, optimizing
globals and builtins lookup was part of the Unladen Swallow plan:
"Implement one of the several proposed schemes for speeding lookups of
globals and builtins." (source: `Unladen Swallow ProjectPlan
<https://code.google.com/p/unladen-swallow/wiki/ProjectPlan>`_).

Unladen Swallow is a fork of CPython 2.6.1 adding a JIT compiler
implemented with LLVM. The project stopped in 2011: `Unladen Swallow
Retrospective
<http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html>`_.


Changes
=======

Add a ``ma_version_tag`` field to the ``PyDictObject`` structure with
the C type ``PY_UINT64_T``, 64-bit unsigned integer. Add also a global
dictionary version.

Each time a dictionary is created, the global version is incremented and
the dictionary version is initialized to the global version.

Each time the dictionary content is modified, the global version must be
incremented and copied to the dictionary version. Dictionary methods
which can modify its content:

* ``clear()``
* ``pop(key)``
* ``popitem()``
* ``setdefault(key, value)``
* ``__delitem__(key)``
* ``__setitem__(key, value)``
* ``update(...)``

The choice of increasing or not the version when a dictionary method
does not change its content is left to the Python implementation. A
Python implementation can decide to not increase the version to avoid
dictionary lookups in guards. Examples of cases when dictionary methods
don't modify its content:

* ``clear()`` if the dict is already empty
* ``pop(key)`` if the key does not exist
* ``popitem()`` if the dict is empty
* ``setdefault(key, value)`` if the key already exists
* ``__delitem__(key)`` if the key does not exist
* ``__setitem__(key, value)`` if the 


id: pep-0290.rst
content:
PEP: 290
Title: Code Migration and Modernization
Author: Raymond Hettinger <python@rcn.com>
Status: Active
Type: Informational
Created: 06-Jun-2002
Post-History:


Abstract
========

This PEP is a collection of procedures and ideas for updating Python
applications when newer versions of Python are installed.

The migration tips highlight possible areas of incompatibility and
make suggestions on how to find and resolve those differences.  The
modernization procedures show how older code can be updated to take
advantage of new language features.


Rationale
=========

This repository of procedures serves as a catalog or checklist of
known migration issues and procedures for addressing those issues.

Migration issues can arise for several reasons.  Some obsolete
features are slowly deprecated according to the guidelines in :pep:`4`.
Also, some code relies on undocumented behaviors which are
subject to change between versions.  Some code may rely on behavior
which was subsequently shown to be a bug and that behavior changes
when the bug is fixed.

Modernization options arise when new versions of Python add features
that allow improved clarity or higher performance than previously
available.


Guidelines for New Entries
==========================

Developers with commit access may update this PEP directly.  Others
can send their ideas to a developer for possible inclusion.

While a consistent format makes the repository easier to use, feel
free to add or subtract sections to improve clarity.

Grep patterns may be supplied as tool to help maintainers locate code
for possible updates.  However, fully automated search/replace style
regular expressions are not recommended.  Instead, each code fragment
should be evaluated individually.

The contra-indications section is the most important part of a new
entry.  It lists known situations where the update SHOULD NOT be
applied.


Migration Issues
================

Comparison Operators Not a Shortcut for Producing 0 or 1
--------------------------------------------------------

Prior to Python 2.3, comparison operations returned 0 or 1 rather
than True or False.  Some code may have used this as a shortcut for
producing zero or one in places where their boolean counterparts are
not appropriate.  For example::

    def identity(m=1):
        """Create and m-by-m identity matrix"""
        return [[i==j for i in range(m)] for j in range(m)]

In Python 2.2, a call to identity(2) would produce::

    [[1, 0], [0, 1]]

In Python 2.3, the same call would produce::

    [[True, False], [False, True]]

Since booleans are a subclass of integers, the matrix would continue
to calculate normally, but it will not print as expected.  The list
comprehension should be changed to read::

    return [[int(i==j) for i in range(m)] for j in range(m)]

There are similar concerns when storing data to be used by other
applications which may expect a number instead of True or False.


Modernization Procedures
========================

Procedures are grouped by the Python version required to be able to
take advantage of the modernization.

Python 2.4 or Later
-------------------

Inserting and Popping at the Beginning of Lists
'''''''''''''''''''''''''''''''''''''''''''''''

Python's lists are implemented to perform best with appends and pops on
the right.  Use of ``pop(0)`` or ``insert(0, x)`` triggers O(n) data
movement for the entire list.  To help address this need, Python 2.4
introduces a new container, ``collections.deque()`` which has efficient
append and pop operations on the both the left and right (the trade-off
is much slower getitem/setitem access).  The new container is especially
helpful for implementing data queues:

Pattern::

    c = list(data)   -->   c = collections.deque(data)
    c.pop(0)         -->   c.popleft()
    c.insert(0, x)   -->   c.appendleft()

Locating::

    grep pop(0 or
    grep insert(0

Simplifying Custom Sorts
''''''''''''''''''''''''

In Python 2.4, the ``sort`` method for lists and the new ``sorted``
built-in function both accept a ``key`` function for computing sort
keys.  Unlike the ``cmp`` function which gets applied to every
comparison, the key function gets applied only once to each record.
It is much faster than cmp and typically more readable while using
less code.  The key function also maintains the stability of the
sort (records with the same key are left in their original order.

Original code using a comparison function::

    names.sort(lambda x,y: cmp(x.lower(), y.lower()))

Alternative original code with explicit decoration::

    tempnames = [(n.lower(), n) for n in names]
    tempnames.sort()
    names = [original for decorated, original in tempnames]

Revised code using a key function::

    names.sort(key=str.lower)       # case-insensitive sort


Locating: ``grep sort *.py``

Replacing Common Uses of Lambda
'''''''''''''''''''''''''''''''

In Python 2.4, the ``operator`` module gained two new functions,
itemgetter() and attrgetter() that can replace common uses of
the ``lambda`` keyword.  The new functions run faster and
are considered by some to improve readability.

Pattern::

    lambda r: r[2]      -->  itemgetter(2)
    lambda r: r.myattr  -->  attrgetter('myattr')

Typical contexts::

    sort(studentrecords, key=attrgetter('gpa'))   # set a sort field
    map(attrgetter('lastname'), studentrecords)   # extract a field

Locating: ``grep lambda *.py``

Simplified Reverse Iteration
''''''''''''''''''''''''''''

Python 2.4 introduced the ``reversed`` builtin function for reverse
iteration.  The existing approaches to reverse iteration suffered
from wordiness, performance issues (speed and memory consumption),
and/or lack of clarity.  A preferred style is to express the
sequence in a forwards direction, apply ``reversed`` to the result,
and then loop over the resulting fast, memory friendly iterator.

Original code expressed with half-open intervals::

    for i in range(n-1, -1, -1):
        print seqn[i]

Alternative original code reversed in multiple steps::

    rseqn = list(seqn)
    rseqn.reverse()
    for value in rseqn:
        print value

Alternative original code expressed with extending slicing::

    for value in seqn[::-1]:
        print value

Revised code using the ``reversed`` function::

    for value in reversed(seqn):
        print value

Python 2.3 or Later
-------------------

Testing String Membership
'''''''''''''''''''''''''

In Python 2.3, for ``string2 in string1``, the length restriction on
``string2`` is lifted; it can now be a string of any length.  When
searching for a substring, where you don't care about the position of
the substring in the original string, using the ``in`` operator makes
the meaning clear.

Pattern::

    string1.find(string2) >= 0   -->  string2 in string1
    string1.find(string2) != -1  -->  string2 in string1

Replace apply() with a Direct Function Call
'''''''''''''''''''''''''''''''''''''''''''

In Python 2.3, apply() was marked for Pending Deprecation because it
was made obsolete by Python 1.6's introduction of * and ** in
function calls.  Using a direct function call was always a little
faster than apply() because it saved the lookup for the builtin.
Now, apply() is even slower due to its use of the warnings module.

Pattern::

    apply(f, args, kwds)  -->  f(*args, **kwds)

Note: The Pending Deprecation was removed from apply() in Python 2.3.3
since it creates pain for people who need to maintain code that works
with Python versions as far back as 1.5.2, where there was no
alternative to apply().  The function remains deprecated, however.


Python 2.2 or Later
-------------------

Testing Dictionary Membership
'''''''''''''''''''''''''''''

For testing dictionary membership, use the 'in' keyword instead of the
'has_key()' method.  The result is shorter and more readable.  The
style becomes consistent with tests for membership in lists.  The
result is slightly faster because ``has_key`` requires an attribute
search 


id: pep-0284.rst
content:
PEP: 284
Title: Integer for-loops
Author: David Eppstein <eppstein@ics.uci.edu>,
        Gregory Ewing <greg.ewing@canterbury.ac.nz>
Status: Rejected
Type: Standards Track
Created: 01-Mar-2002
Python-Version: 2.3
Post-History:


Abstract
========

This PEP proposes to simplify iteration over intervals of
integers, by extending the range of expressions allowed after a
"for" keyword to allow three-way comparisons such as ::

   for lower <= var < upper:

in place of the current ::

   for item in list:

syntax.  The resulting loop or list iteration will loop over all
values of var that make the comparison true, starting from the
left endpoint of the given interval.


Pronouncement
=============

This PEP is rejected.  There were a number of fixable issues with
the proposal (see the fixups listed in Raymond Hettinger's
python-dev post on 18 June 2005 [1]_).  However, even with the fixups the
proposal did not garner support.  Specifically, Guido did not buy
the premise that the ``range()`` format needed fixing, "The whole point
(15 years ago) of ``range()`` was to *avoid* needing syntax to specify a
loop over numbers. I think it's worked out well and there's nothing
that needs to be fixed (except ``range()`` needs to become an iterator,
which it will in Python 3.0)."


Rationale
=========

One of the most common uses of for-loops in Python is to iterate
over an interval of integers.  Python provides functions ``range()``
and ``xrange()`` to generate lists and iterators for such intervals,
which work best for the most frequent case: half-open intervals
increasing from zero.  However, the ``range()`` syntax is more awkward
for open or closed intervals, and lacks symmetry when reversing
the order of iteration.  In addition, the call to an unfamiliar
function makes it difficult for newcomers to Python to understand
code that uses ``range()`` or ``xrange()``.

The perceived lack of a natural, intuitive integer iteration
syntax has led to heated debate on python-list, and spawned at
least four PEPs before this one.  :pep:`204` (rejected) proposed
to re-use Python's slice syntax for integer ranges, leading to a
terser syntax but not solving the readability problem of
multi-argument ``range()``.  :pep:`212` (deferred) proposed several
syntaxes for directly converting a list to a sequence of integer
indices, in place of the current idiom ::

   range(len(list))

for such conversion, and :pep:`281` proposes to simplify the same
idiom by allowing it to be written as ::

   range(list).

:pep:`276` proposes to allow automatic conversion of integers to
iterators, simplifying the most common half-open case but not
addressing the complexities of other types of interval.
Additional alternatives have been discussed on python-list.

The solution described here is to allow a three-way comparison
after a "for" keyword, both in the context of a for-loop and of a
list comprehension::

    for lower <= var < upper:

This would cause iteration over an interval of consecutive
integers, beginning at the left bound in the comparison and ending
at the right bound.  The exact comparison operations used would
determine whether the interval is open or closed at either end and
whether the integers are considered in ascending or descending
order.

This syntax closely matches standard mathematical notation, so is
likely to be more familiar to Python novices than the current
``range()`` syntax.  Open and closed interval endpoints are equally
easy to express, and the reversal of an integer interval can be
formed simply by swapping the two endpoints and reversing the
comparisons.  In addition, the semantics of such a loop would
closely resemble one way of interpreting the existing Python
for-loops::

    for item in list

iterates over exactly those values of item that cause the
expression ::

   item in list

to be true.  Similarly, the new format ::

   for lower <= var < upper:

would iterate over exactly those integer values of var that cause
the expression ::

   lower <= var < upper

to be true.


Specification
=============

We propose to extend the syntax of a for statement, currently ::

   for_stmt: "for" target_list "in" expression_list ":" suite
         ["else" ":" suite]

as described below::

    for_stmt: "for" for_test ":" suite ["else" ":" suite]
    for_test: target_list "in" expression_list |
            or_expr less_comp or_expr less_comp or_expr |
            or_expr greater_comp or_expr greater_comp or_expr
    less_comp: "<" | "<="
    greater_comp: ">" | ">="

Similarly, we propose to extend the syntax of list comprehensions,
currently ::

   list_for: "for" expression_list "in" testlist [list_iter]

by replacing it with::

    list_for: "for" for_test [list_iter]

In all cases the expression formed by for_test would be subject to
the same precedence rules as comparisons in expressions.  The two
comp_operators in a for_test must be required to be both of
similar types, unlike chained comparisons in expressions which do
not have such a restriction.

We refer to the two or_expr's occurring on the left and right
sides of the for-loop syntax as the bounds of the loop, and the
middle or_expr as the variable of the loop.  When a for-loop using
the new syntax is executed, the expressions for both bounds will
be evaluated, and an iterator object created that iterates through
all integers between the two bounds according to the comparison
operations used.  The iterator will begin with an integer equal or
near to the left bound, and then step through the remaining
integers with a step size of +1 or -1 if the comparison operation
is in the set described by less_comp or greater_comp respectively.
The execution will then proceed as if the expression had been ::

   for variable in iterator

where "variable" refers to the variable of the loop and "iterator"
refers to the iterator created for the given integer interval.

The values taken by the loop variable in an integer for-loop may
be either plain integers or long integers, according to the
magnitude of the bounds.  Both bounds of an integer for-loop must
evaluate to a real numeric type (integer, long, or float).  Any
other value will cause the for-loop statement to raise a ``TypeError``
exception.


Issues
======

The following issues were raised in discussion of this and related
proposals on the Python list.

- Should the right bound be evaluated once, or every time through
  the loop?  Clearly, it only makes sense to evaluate the left
  bound once.  For reasons of consistency and efficiency, we have
  chosen the same convention for the right bound.

- Although the new syntax considerably simplifies integer
  for-loops, list comprehensions using the new syntax are not as
  simple.  We feel that this is appropriate since for-loops are
  more frequent than comprehensions.

- The proposal does not allow access to integer iterator objects
  such as would be created by ``xrange``.  True, but we see this as a
  shortcoming in the general list-comprehension syntax, beyond the
  scope of this proposal.  In addition, ``xrange()`` will still be
  available.

- The proposal does not allow increments other than 1 and -1.
  More general arithmetic progressions would need to be created by
  ``range()`` or ``xrange()``, or by a list comprehension syntax such as ::

   [2*x for 0 <= x <= 100]

- The position of the loop variable in the middle of a three-way
  comparison is not as apparent as the variable in the present ::

   for item in list

  syntax, leading to a possible loss of readability.  We feel that
  this loss is outweighed by the increase in readability from a
  natural integer iteration syntax.

- To some extent, this PEP addresses the same issues as :pep:`276`.
  We feel that the two PEPs are not in conflict since :pep:`276`
  is primarily concerned with half-open ranges starting in 0
  (the easy case of ``range()``) while this PEP is primarily concerned
  with simplifying all other cases.  However, if this PEP is
  approved,


id: pep-0737.rst
content:
PEP: 737
Title: C API to format a type fully qualified name
Author: Victor Stinner <vstinner@python.org>
Discussions-To: https://discuss.python.org/t/pep-737-unify-type-name-formatting/39872
Status: Final
Type: Standards Track
Created: 29-Nov-2023
Python-Version: 3.13
Post-History: `29-Nov-2023 <https://discuss.python.org/t/pep-737-unify-type-name-formatting/39872>`__
Resolution: https://discuss.python.org/t/pep-737-unify-type-name-formatting/39872/60


Abstract
========

Add new convenient C APIs to format a type fully qualified name. No longer
format type names differently depending on how types are implemented.

Recommend using the type fully qualified name in error messages and in
``__repr__()`` methods in new C code. Recommend not truncating type
names in new C code.

Add ``%T``, ``%#T``, ``%N`` and ``%#N`` formats to
``PyUnicode_FromFormat()`` to format the fully qualified, respectively,
of an object type and of a type.

Make C code safer by avoiding borrowed reference which can lead to
crashes. The new C API is compatible with the limited C API.


Rationale
=========

Standard library
----------------

In the Python standard library, formatting a type name or the type name
of an object is a common operation to format an error message and to
implement a ``__repr__()`` method. There are different ways to format a
type name which give different outputs.

Example with the ``datetime.timedelta`` type:

* The type short name (``type.__name__``) and the type qualified name
  (``type.__qualname__``) are ``'timedelta'``.
* The type module (``type.__module__``) is ``'datetime'``.
* The type fully qualified name is ``'datetime.timedelta'``.
* The type representation (``repr(type)``) contains the fully qualified
  name: ``<class 'datetime.timedelta'>``.


Python code
^^^^^^^^^^^

In Python, ``type.__name__`` gets the type short name, whereas
``f"{type.__module__}.{type.__qualname__}"`` formats the type "fully
qualified name". Usually, ``type(obj)`` or ``obj.__class__`` are used to
get the type of the object *obj*. Sometimes, the type name is put
between quotes.

Examples:

* ``raise TypeError("str expected, not %s" % type(value).__name__)``
* ``raise TypeError("can't serialize %s" % self.__class__.__name__)``
* ``name = "%s.%s" % (obj.__module__, obj.__qualname__)``

Qualified names were added to types (``type.__qualname__``) in Python
3.3 by :pep:`3155` "Qualified name for classes and functions".

C code
^^^^^^

In C, the most common way to format a type name is to get the
``PyTypeObject.tp_name`` member of the type. Example:

.. code-block:: c

    PyErr_Format(PyExc_TypeError, "globals must be a dict, not %.100s",
                 Py_TYPE(globals)->tp_name);

The type "fully qualified name" is used in a few places:
``PyErr_Display()``, ``type.__repr__()`` implementation, and
``sys.unraisablehook`` implementation.

Using ``Py_TYPE(obj)->tp_name`` is preferred since it is more convenient
than calling ``PyType_GetQualName()`` which requires ``Py_DECREF()``.
Moreover, ``PyType_GetQualName()`` was only added recently, in Python
3.11.

Some functions use ``%R`` (``repr(type)``) to format a type name, the
output contains the type fully qualified name. Example:

.. code-block:: c

    PyErr_Format(PyExc_TypeError,
                 "calling %R should have returned an instance "
                 "of BaseException, not %R",
                 type, Py_TYPE(value));


Using PyTypeObject.tp_name is inconsistent with Python
------------------------------------------------------

The ``PyTypeObject.tp_name`` member is different depending on the type
implementation:

* Static types and heap types in C: *tp_name* is the type fully
  qualified name.
* Python class: *tp_name* is the type short name (``type.__name__``).

So using ``Py_TYPE(obj)->tp_name`` to format an object type name gives
a different output depending if a type is implemented in C or in Python.

It goes against :pep:`399` "Pure Python/C Accelerator Module
Compatibility Requirements" principles which recommends code behaves
the same way if written in Python or in C.

Example:

.. code-block:: pycon

    $ python3.12
    >>> import _datetime; c_obj = _datetime.date(1970, 1, 1)
    >>> import _pydatetime; py_obj = _pydatetime.date(1970, 1, 1)
    >>> my_list = list(range(3))

    >>> my_list[c_obj]  # C type
    TypeError: list indices must be integers or slices, not datetime.date

    >>> my_list[py_obj]  # Python type
    TypeError: list indices must be integers or slices, not date

The error message contains the type fully qualified name
(``datetime.date``) if the type is implemented in C, or the type short
name (``date``) if the type is implemented in Python.


Limited C API
-------------

The ``Py_TYPE(obj)->tp_name`` code cannot be used with the limited C
API, since the ``PyTypeObject`` members are excluded from the limited C
API.

The type name should be read using ``PyType_GetName()``,
``PyType_GetQualName()`` and ``PyType_GetModule()`` functions which are
less convenient to use.


Truncating type names in C
--------------------------

In 1998, when the ``PyErr_Format()`` function was added, the
implementation used a fixed buffer of 500 bytes. The function had the
following comment:

.. code-block:: c

    /* Caller is responsible for limiting the format */

In 2001, the function was modified to allocate a dynamic buffer on the
heap. Too late, the practice of truncating type names, like using the
``%.100s`` format, already became a habit, and developers forgot why
type names are truncated. In Python, type names are not truncated.

Truncating type names in C but not in Python goes against :pep:`399`
"Pure Python/C Accelerator Module Compatibility Requirements" principles
which recommends code behaves the same way if written in Python or in
C.

See the issue: `Replace %.100s by %s in PyErr_Format(): the arbitrary
limit of 500 bytes is outdated
<https://github.com/python/cpython/issues/55042>`__ (2011).


Specification
=============

* Add ``PyType_GetFullyQualifiedName()`` function.
* Add ``PyType_GetModuleName()`` function.
* Add formats to ``PyUnicode_FromFormat()``.
* Recommend using the type fully qualified name in error messages and
  in ``__repr__()`` methods in new C code.
* Recommend not truncating type names in new C code.


Add PyType_GetFullyQualifiedName() function
-------------------------------------------

Add the ``PyType_GetFullyQualifiedName()`` function to get the fully
qualified name of a type: similar to
``f"{type.__module__}.{type.__qualname__}"``, or ``type.__qualname__``
if ``type.__module__`` is not a string or is equal to ``"builtins"`` or
is equal to ``"__main__"``.

API:

.. code-block:: c

    PyObject* PyType_GetFullyQualifiedName(PyTypeObject *type)

On success, return a new reference to the string. On error, raise an
exception and return ``NULL``.


Add PyType_GetModuleName() function
-----------------------------------

Add the ``PyType_GetModuleName()`` function to get the module name of a
type (``type.__module__`` string). API:

.. code-block:: c

    PyObject* PyType_GetModuleName(PyTypeObject *type)

On success, return a new reference to the string. On error, raise an
exception and return ``NULL``.


Add formats to PyUnicode_FromFormat()
-------------------------------------

Add the following formats to ``PyUnicode_FromFormat()``:

* ``%N`` formats the **fully qualified name** of a **type**,
  similar to ``PyType_GetFullyQualifiedName(type)``;
  **N** stands for type **N**\ ame.
* ``%T`` formats the type **fully qualified name** of an object's **type**,
  similar to ``PyType_GetFullyQualifiedName(Py_TYPE(obj))``;
  **T** stands for object **T**\ ype.
* ``%#N`` and ``%#T``: the alternative form uses the **colon** separator
  (``:``), instead of the dot separator (``.``), between the module name
  and the qualified name.

For example, the existing code using *tp_name*:

.. code-block:: c

    PyErr_Format(PyExc_TypeError,
                 "__format__ must return a s


id: pep-0723.rst
content:
PEP: 723
Title: Inline script metadata
Author: Ofek Lev <ofekmeister@gmail.com>
Sponsor: Adam Turner <python@quite.org.uk>
PEP-Delegate: Brett Cannon <brett@python.org>
Discussions-To: https://discuss.python.org/t/31151
Status: Final
Type: Standards Track
Topic: Packaging
Created: 04-Aug-2023
Post-History: `04-Aug-2023 <https://discuss.python.org/t/30979>`__,
              `06-Aug-2023 <https://discuss.python.org/t/31151>`__,
              `23-Aug-2023 <https://discuss.python.org/t/32149>`__,
              `06-Dec-2023 <https://discuss.python.org/t/40418>`__,
Replaces: 722
Resolution: `08-Jan-2024 <https://discuss.python.org/t/40418/82>`__

.. canonical-pypa-spec:: :ref:`packaging:inline-script-metadata`

Abstract
========

This PEP specifies a metadata format that can be embedded in single-file Python
scripts to assist launchers, IDEs and other external tools which may need to
interact with such scripts.


Motivation
==========

Python is routinely used as a scripting language, with Python scripts as a
(better) alternative to shell scripts, batch files, etc. When Python code is
structured as a script, it is usually stored as a single file and does not
expect the availability of any other local code that may be used for imports.
As such, it is possible to share with others over arbitrary text-based means
such as email, a URL to the script, or even a chat window. Code that is
structured like this may live as a single file forever, never becoming a
full-fledged project with its own directory and ``pyproject.toml`` file.

An issue that users encounter with this approach is that there is no standard
mechanism to define metadata for tools whose job it is to execute such scripts.
For example, a tool that runs a script may need to know which dependencies are
required or the supported version(s) of Python.

There is currently no standard tool that addresses this issue, and this PEP
does *not* attempt to define one. However, any tool that *does* address this
issue will need to know what the runtime requirements of scripts are. By
defining a standard format for storing such metadata, existing tools, as well
as any future tools, will be able to obtain that information without requiring
users to include tool-specific metadata in their scripts.


Rationale
=========

This PEP defines a mechanism for embedding metadata *within the script itself*,
and not in an external file.

The metadata format is designed to be similar to the layout of data in the
``pyproject.toml`` file of a Python project directory, to provide a familiar
experience for users who have experience writing Python projects. By using a
similar format, we avoid unnecessary inconsistency between packaging tools,
a common frustration expressed by users in the recent
`packaging survey <https://discuss.python.org/t/22420>`__.

The following are some of the use cases that this PEP wishes to support:

* A user facing CLI that is capable of executing scripts. If we take Hatch as
  an example, the interface would be simply
  ``hatch run /path/to/script.py [args]`` and Hatch will manage the
  environment for that script. Such tools could be used as shebang lines on
  non-Windows systems e.g. ``#!/usr/bin/env hatch run``.
* A script that desires to transition to a directory-type project. A user may
  be rapidly prototyping locally or in a remote REPL environment and then
  decide to transition to a more formal project layout if their idea works
  out. Being able to define dependencies in the script would be very useful to
  have fully reproducible bug reports.
* Users that wish to avoid manual dependency management. For example, package
  managers that have commands to add/remove dependencies or dependency update
  automation in CI that triggers based on new versions or in response to
  CVEs [1]_.


Specification
=============

This PEP defines a metadata comment block format loosely inspired [2]_ by
`reStructuredText Directives`__.

__ https://docutils.sourceforge.io/docs/ref/rst/directives.html

Any Python script may have top-level comment blocks that MUST start with the
line ``# /// TYPE`` where ``TYPE`` determines how to process the content. That
is: a single ``#``, followed by a single space, followed by three forward
slashes, followed by a single space, followed by the type of metadata. Block
MUST end with the line ``# ///``. That is: a single ``#``, followed by a single
space, followed by three forward slashes. The ``TYPE`` MUST only consist of
ASCII letters, numbers and hyphens.

Every line between these two lines (``# /// TYPE`` and ``# ///``) MUST be a
comment starting with ``#``. If there are characters after the ``#`` then the
first character MUST be a space. The embedded content is formed by taking away
the first two characters of each line if the second character is a space,
otherwise just the first character (which means the line consists of only a
single ``#``).

Precedence for an ending line ``# ///`` is given when the next line is not
a valid embedded content line as described above. For example, the following
is a single fully valid block:

.. code:: python

    # /// some-toml
    # embedded-csharp = """
    # /// <summary>
    # /// text
    # ///
    # /// </summary>
    # public class MyClass { }
    # """
    # ///

A starting line MUST NOT be placed between another starting line and its ending
line. In such cases tools MAY produce an error. Unclosed blocks MUST be ignored.

When there are multiple comment blocks of the same ``TYPE`` defined, tools MUST
produce an error.

Tools reading embedded metadata MAY respect the standard Python encoding
declaration. If they choose not to do so, they MUST process the file as UTF-8.

This is the canonical regular expression that MAY be used to parse the
metadata:

.. code:: text

    (?m)^# /// (?P<type>[a-zA-Z0-9-]+)$\s(?P<content>(^#(| .*)$\s)+)^# ///$

In circumstances where there is a discrepancy between the text specification
and the regular expression, the text specification takes precedence.

Tools MUST NOT read from metadata blocks with types that have not been
standardized by this PEP or future ones.

script type
-----------

The first type of metadata block is named ``script`` which contains script
metadata (dependency data and tool configuration).

This document MAY include top-level fields ``dependencies`` and ``requires-python``,
and MAY optionally include a ``[tool]`` table.

The ``[tool]`` table MAY be used by any tool, script runner or otherwise, to configure
behavior. It has the same semantics as the :pep:`tool table <518#tool-table>` in
``pyproject.toml``.

The top-level fields are:

* ``dependencies``: A list of strings that specifies the runtime dependencies
  of the script. Each entry MUST be a valid :pep:`508` dependency.
* ``requires-python``: A string that specifies the Python version(s) with which
  the script is compatible. The value of this field MUST be a valid
  :pep:`version specifier <440#version-specifiers>`.

Script runners MUST error if the specified ``dependencies`` cannot be provided.
Script runners SHOULD error if no version of Python that satisfies the specified
``requires-python`` can be provided.

Example
-------

The following is an example of a script with embedded metadata:

.. code:: python

    # /// script
    # requires-python = ">=3.11"
    # dependencies = [
    #   "requests<3",
    #   "rich",
    # ]
    # ///

    import requests
    from rich.pretty import pprint

    resp = requests.get("https://peps.python.org/api/peps.json")
    data = resp.json()
    pprint([(k, v["title"]) for k, v in data.items()][:10])

Reference Implementation
========================

The following is an example of how to read the metadata on Python 3.11 or
higher.

.. code:: python

   import re
   import tomllib

   REGEX = r'(?m)^# /// (?P<type>[a-zA-Z0-9-]+)$\s(?P<content>(^#(| .*)$\s)+)^# ///$'

   def read(script: str) -> dict | None:
       name = 'script'
       matches = list(
           filter(lambda m: m.gr


id: pep-0643.rst
content:
PEP: 643
Title: Metadata for Package Source Distributions
Author: Paul Moore <p.f.moore@gmail.com>
BDFL-Delegate: Paul Ganssle <paul@ganssle.io>
Discussions-To: https://discuss.python.org/t/pep-643-metadata-for-package-source-distributions/5577
Status: Final
Type: Standards Track
Topic: Packaging
Created: 24-Oct-2020
Post-History: 24-Oct-2020, 01-Nov-2020, 02-Nov-2020, 14-Nov-2020
Resolution: https://discuss.python.org/t/pep-643-metadata-for-package-source-distributions/5577/53


.. canonical-pypa-spec:: :ref:`packaging:core-metadata`


Abstract
========

Python package metadata is stored in the distribution file in a standard
format, defined in the `Core Metadata Specification`_. However, for
source distributions, while the format of the data is defined, there has
traditionally been a lot of inconsistency in what data is recorded in
the source distribution. See `here
<https://discuss.python.org/t/why-isnt-source-distribution-metadata-trustworthy-can-we-make-it-so/2620>`_
for a discussion of this issue.

As a result, metadata consumers are unable to rely on the data available
from source distributions, and need to use the (costly) :pep:`517` build
mechanisms to extract medatata.

This PEP defines a standard that allows build backends to reliably store
package metadata in the source distribution, while still retaining the
necessary flexibility to handle metadata fields that have to be calculated
at build time.


Motivation
==========

There are a number of issues with the way that metadata is currently
stored in source distributions:

* The details of how to store metadata, while standardised, are not
  easy to find.
* The specification requires an old metadata version, and has not been
  updated in line with changes to the core metadata spec.
* There is no way in the spec to distinguish between "this field has been
  omitted because its value will not be known until build time" and "this
  field does not have a value".
* The core metadata specification allows most fields to be optional,
  meaning that the previous issue affects nearly every metadata field.

This PEP proposes an update to the metadata specification to allow
recording of fields which are expected to be "filled in later", and
updates the source distribution specification to clarify that backends
should record sdist metadata using that version of the spec (or later).


Rationale
=========

This PEP allows projects to define source distribution metadata values
as being "dynamic". In this context, saying that a field is "dynamic"
means that the value has not been fixed at the time that the source
distribution was generated. Dynamic values will be supplied by the build
backend at the time when the wheel is generated, and could depend on
details of the build environment.

:pep:`621` has a similar concept, of "dynamic" values that will be
"filled in later", and so we choose to use the same term here by
analogy.


Specification
=============

This PEP defines the relationship between metadata values specified in a
source distribution, and the corresponding values in wheels built from
it. It requires build backends to clearly mark any fields which will
*not* simply be copied unchanged from the sdist to the wheel.

In addition, this PEP makes the `PyPA Specifications`_ document the
canonical location for the specification of the source distribution
format (collecting the information in :pep:`517` and in this PEP).

A new field, ``Dynamic``, will be added to the `Core Metadata Specification`_.
This field will be multiple use, and will be allowed to contain the name
of another core metadata field.

When found in the metadata of a source distribution, the following
rules apply:

1. If a field is *not* marked as ``Dynamic``, then the value of the field
   in any wheel built from the sdist MUST match the value in the sdist.
   If the field is not in the sdist, and not marked as ``Dynamic``, then
   it MUST NOT be present in the wheel.
2. If a field is marked as ``Dynamic``, it may contain any valid value in
   a wheel built from the sdist (including not being present at all).
3. Backends MUST NOT mark a field as ``Dynamic`` if they can determine that
   it was generated from data that will not change at build time.

Backends MAY record the value they calculated for a field they mark as
``Dynamic`` in a source distribution. Consumers, however, MUST NOT treat
this value as canonical, but MAY use it as an hint about what the final
value in a wheel could be.

In any context other than a source distribution, if a field is marked as
``Dynamic``, that indicates that the value was generated at wheel build
time and may not match the value in the sdist (or in other builds of the
project). Backends are not required to record this information, though,
and consumers MUST NOT assume that the lack of a ``Dynamic`` marking has
any significance, except in a source distribution.

The fields ``Name`` and ``Version`` MUST NOT be marked as ``Dynamic``.

As it adds a new metadata field, this PEP updates the core metadata
format to version 2.2.

Source distributions SHOULD use the latest version of the core metadata
specification that was available when they were created.

Build backends are strongly encouraged to only mark fields as
``Dynamic`` when absolutely necessary, and to encourage projects to
avoid backend features that require the use of ``Dynamic``. Projects
should prefer to use environment markers on static values to adapt to
details of the install location.


Backwards Compatibility
=======================

As this proposal increments the core metadata version, it is compatible
with existing source distributions, which will use an older metadata
version. Tools can determine whether a source distribution conforms to
this PEP by checking the metadata version.


Security Implications
=====================

As this specification is purely for the storage of data that is intended
to be publicly available, there are no security implications.


How to Teach This
=================

This is a data storage format for project metadata, and so will not
typically be visible to end users. There is therefore no need to teach
users how to use the format. Developers wanting to reference the
metadata will be able to find the details in the `PyPA Specifications`_.


Rejected Ideas
==============

1. Rather than marking fields as ``Dynamic``, fields should be assumed
   to be dynamic unless explicitly marked as ``Static``.

   This is logically equivalent to the current proposal, but it implies
   that fields being dynamic is the norm. Packaging tools can be much
   more efficient in the presence of metadata that is known to be static,
   so the PEP chooses to make dynamic fields the exception, and require
   backends to "opt in" to making a field dynamic.

   In addition, if dynamic is the default, then in future, as more
   and more metadata becomes static, metadata files will include an
   increasing number of ``Static`` declarations.

2. Rather than having a ``Dynamic`` field, add a special value that
   indicates that a field is "not yet defined".

   Again, this is logically equivalent to the current proposal. It makes
   "being dynamic" an explicit choice, but requires a special value.  As
   some fields can contain arbitrary text, choosing a such a value is
   somewhat awkward (although likely not a problem in practice). There
   does not seem to be enough benefit to this approach to make it worth
   using instead of the proposed mechanism.

3. Special handling of ``Requires-Python``.

   Early drafts of the PEP needed special discussion of ``Requires-Python``,
   because the lack of environment markers for this field meant that it might
   be difficult to require it to be static. The final form of the PEP no longer
   needs this, as the idea of a whitelist of fields allowed to be dynamic was
   dropped.

4. Restrict the use of ``Dynamic`` to a minimal "white list" of
   permitted fields.

   This approach was likely to 


id: pep-0657.rst
content:
PEP: 657
Title: Include Fine Grained Error Locations in Tracebacks
Author: Pablo Galindo <pablogsal@python.org>,
        Batuhan Taskaya <batuhan@python.org>,
        Ammar Askar <ammar@ammaraskar.com>
Discussions-To: https://discuss.python.org/t/pep-657-include-fine-grained-error-locations-in-tracebacks/8629
Status: Final
Type: Standards Track
Created: 08-May-2021
Python-Version: 3.11
Post-History:

Abstract
========

This PEP proposes adding a mapping from each bytecode instruction to the start
and end column offsets of the line that generated them as well as the end line
number. This data will be used to improve tracebacks displayed by the CPython
interpreter in order to improve the debugging experience. The PEP also proposes
adding APIs that allow other tools (such as coverage analysis tools, profilers,
tracers, debuggers) to consume this information from code objects.

Motivation
==========

The primary motivation for this PEP is to improve the feedback presented about
the location of errors to aid with debugging.

Python currently keeps a mapping of bytecode to line numbers from compilation.
The interpreter uses this mapping to point to the source line associated with
an error. While this line-level granularity for instructions is useful, a
single line of Python code can compile into dozens of bytecode operations
making it hard to track which part of the line caused the error.

Consider the following line of Python code::

    x['a']['b']['c']['d'] = 1

If any of the values in the dictionaries are ``None``, the error shown is::

    Traceback (most recent call last):
      File "test.py", line 2, in <module>
        x['a']['b']['c']['d'] = 1
    TypeError: 'NoneType' object is not subscriptable

From the traceback, it is impossible to determine which one of the dictionaries
had the ``None`` element that caused the error. Users often have to attach a
debugger or split up their expression to track down the problem.

However, if the interpreter had a mapping of bytecode to column offsets as well
as line numbers, it could helpfully display::

    Traceback (most recent call last):
      File "test.py", line 2, in <module>
        x['a']['b']['c']['d'] = 1
        ~~~~~~~~~~~^^^^^
    TypeError: 'NoneType' object is not subscriptable

indicating to the user that the object ``x['a']['b']`` must have been ``None``.
This highlighting will occur for every frame in the traceback. For instance, if
a similar error is part of a complex function call chain, the traceback would
display the code associated to the current instruction in every frame::

    Traceback (most recent call last):
      File "test.py", line 14, in <module>
        lel3(x)
        ^^^^^^^
      File "test.py", line 12, in lel3
        return lel2(x) / 23
               ^^^^^^^
      File "test.py", line 9, in lel2
        return 25 + lel(x) + lel(x)
                    ^^^^^^
      File "test.py", line 6, in lel
        return 1 + foo(a,b,c=x['z']['x']['y']['z']['y'], d=e)
                             ~~~~~~~~~~~~~~~~^^^^^
    TypeError: 'NoneType' object is not subscriptable

This problem presents itself in the following situations.

* When passing down multiple objects to function calls while
  accessing the same attribute in them.
  For instance, this error::

    Traceback (most recent call last):
      File "test.py", line 19, in <module>
        foo(a.name, b.name, c.name)
    AttributeError: 'NoneType' object has no attribute 'name'

  With the improvements in this PEP this would show::

    Traceback (most recent call last):
      File "test.py", line 17, in <module>
        foo(a.name, b.name, c.name)
                    ^^^^^^
    AttributeError: 'NoneType' object has no attribute 'name'

* When dealing with lines with complex mathematical expressions,
  especially with libraries such as numpy where arithmetic
  operations can fail based on the arguments. For example: ::

    Traceback (most recent call last):
      File "test.py", line 1, in <module>
        x = (a + b) @ (c + d)
    ValueError: operands could not be broadcast together with shapes (1,2) (2,3)

  There is no clear indication as to which operation failed, was it the addition
  on the left, the right or the matrix multiplication in the middle? With this
  PEP the new error message would look like::

    Traceback (most recent call last):
      File "test.py", line 1, in <module>
        x = (a + b) @ (c + d)
                       ~~^~~
    ValueError: operands could not be broadcast together with shapes (1,2) (2,3)

  Giving a much clearer and easier to debug error message.


Debugging aside, this extra information would also be useful for code
coverage tools, enabling them to measure expression-level coverage instead of
just line-level coverage. For instance, given the following line: ::

    x = foo() if bar() else baz()

coverage, profile or state analysis tools will highlight the full line in both
branches, making it impossible to differentiate what branch was taken. This is
a known problem in pycoverage_.

Similar efforts to this PEP have taken place in other languages such as Java in
the form of JEP358_. ``NullPointerExceptions`` in Java were similarly nebulous when
it came to lines with complicated expressions. A ``NullPointerException`` would
provide very little aid in finding the root cause of an error. The
implementation for JEP358 is fairly complex, requiring walking back through the
bytecode by using a control flow graph analyzer and decompilation techniques to
recover the source code that led to the null pointer. Although the complexity
of this solution is high and requires maintenance for the decompiler every time
Java bytecode is changed, this improvement was deemed to be worth it for the
extra information provided for *just one exception type*.


Rationale
=========

In order to identify the range of source code being executed when exceptions
are raised, this proposal requires adding new data for every bytecode
instruction. This will have an impact on the size of ``pyc`` files on disk and
the size of code objects in memory. The authors of this proposal have chosen
the data types in a way that tries to minimize this impact. The proposed
overhead is storing two ``uint8_t`` (one for the start offset and one for the
end offset) and the end line information for every bytecode instruction (in
the same encoded fashion as the start line is stored currently).

As an illustrative example to gauge the impact of this change, we have
calculated that including the start and end offsets will increase the size of
the standard libraryâs pyc files by 22% (6MB) from 28.4MB to 34.7MB. The
overhead in memory usage will be the same (assuming the *full standard library*
is loaded into the same program). We believe that this is a very acceptable
number since the order of magnitude of the overhead is very small, especially
considering the storage size and memory capabilities of modern computers.
Additionally, in general the memory size of a Python program is not dominated
by code objects. To check this assumption we have executed the test suite of
several popular PyPI projects (including NumPy, pytest, Django and Cython) as
well as several applications (Black, pylint, mypy executed over either mypy or
the standard library) and we found that code objects represent normally 3-6% of
the average memory size of the program.

We understand that the extra cost of this information may not be acceptable for
some users, so we propose an opt-out mechanism which will cause generated code
objects to not have the extra information while also allowing pyc files to not
include the extra information.


Specification
=============

In order to have enough information to correctly resolve the location
within a given line where an error was raised, a map linking bytecode
instructions to column offsets (start and end offset) and end line numbers
is needed. This is similar in fashion to how line numbers are currently linked
to bytecode i


id: pep-0680.rst
content:
PEP: 680
Title: tomllib: Support for Parsing TOML in the Standard Library
Author: Taneli Hukkinen, Shantanu Jain <hauntsaninja at gmail.com>
Sponsor: Petr Viktorin <encukou@gmail.com>
Discussions-To: https://discuss.python.org/t/13040
Status: Final
Type: Standards Track
Created: 01-Jan-2022
Python-Version: 3.11
Post-History: `09-Dec-2021 <https://mail.python.org/archives/list/python-ideas@python.org/thread/IWJ3I32A4TY6CIVQ6ONPEBPWP4TOV2V7/>`__,
              `27-Jan-2022 <https://discuss.python.org/t/pep-680-tomllib-support-for-parsing-toml-in-the-standard-library/13040>`__,
Resolution: https://mail.python.org/archives/list/python-dev@python.org/thread/3AHGWYY562HHO55L4Z2OVYUFZP5W73IS/

.. canonical-doc:: :mod:`python:tomllib`

Abstract
========

This PEP proposes adding the ``tomllib`` module to the standard library for
parsing TOML (Tom's Obvious Minimal Language,
`https://toml.io <https://toml.io/en/>`_).


Motivation
==========

TOML is the format of choice for Python packaging, as evidenced by
:pep:`517`, :pep:`518` and :pep:`621`. This creates a bootstrapping
problem for Python build tools, forcing them to vendor a TOML parsing
package or employ other undesirable workarounds, and causes serious issues
for repackagers and other downstream consumers. Including TOML support in
the standard library would neatly solve all of these issues.

Further, many Python tools are now configurable via TOML, such as
``black``, ``mypy``, ``pytest``, ``tox``, ``pylint`` and ``isort``.
Many that are not, such as ``flake8``, cite the lack of standard library
support as a `main reason why
<https://github.com/PyCQA/flake8/issues/234#issuecomment-812800722>`__.
Given the special place TOML already has in the Python ecosystem, it makes sense
for it to be an included battery.

Finally, TOML as a format is increasingly popular (for the reasons
outlined in :pep:`518`), with various Python TOML libraries having about
2000 reverse dependencies on PyPI (for comparison, ``requests`` has about
28000 reverse dependencies). Hence, this is likely to be a generally useful
addition, even looking beyond the needs of Python packaging and related tools.


Rationale
=========

This PEP proposes basing the standard library support for reading TOML on the
third-party library ``tomli``
(`github.com/hukkin/tomli <https://github.com/hukkin/tomli>`_).

Many projects have recently switched to using ``tomli``, such as ``pip``,
``build``, ``pytest``, ``mypy``, ``black``, ``flit``, ``coverage``,
``setuptools-scm`` and ``cibuildwheel``.

``tomli`` is actively maintained and well-tested. It is about 800 lines
of code with 100% test coverage, and passes all tests in the
`proposed official TOML compliance test suite
<https://github.com/toml-lang/compliance/pull/8>`_, as well as
`the more established BurntSushi/toml-test suite
<https://github.com/BurntSushi/toml-test>`_.


Specification
=============

A new module ``tomllib`` will be added to the Python standard library,
exposing the following public functions:

.. code-block::

   def load(
       fp: SupportsRead[bytes],
       /,
       *,
       parse_float: Callable[[str], Any] = ...,
    ) -> dict[str, Any]: ...

   def loads(
       s: str,
       /,
       *,
       parse_float: Callable[[str], Any] = ...,
   ) -> dict[str, Any]: ...

``tomllib.load`` deserializes a binary file-like object containing a
TOML document to a Python ``dict``.
The ``fp`` argument must have a ``read()`` method with the same API as
``io.RawIOBase.read()``.

``tomllib.loads`` deserializes a ``str`` instance containing a TOML document
to a Python ``dict``.

The ``parse_float`` argument is a callable object that takes as input the
original string representation of a TOML float, and returns a corresponding
Python object (similar to ``parse_float`` in ``json.load``).
For example, the user may pass a function returning a ``decimal.Decimal``,
for use cases where exact precision is important. By default, TOML floats
are parsed as instances of the Python ``float`` type.

The returned object contains only basic Python objects (``str``, ``int``,
``bool``, ``float``, ``datetime.{datetime,date,time}``, ``list``, ``dict`` with
string keys), and the results of ``parse_float``.

``tomllib.TOMLDecodeError`` is raised in the case of invalid TOML.

Note that this PEP does not propose ``tomllib.dump`` or ``tomllib.dumps``
functions; see `Including an API for writing TOML`_ for details.


Maintenance Implications
========================

Stability of TOML
-----------------

The release of TOML 1.0.0 in January 2021 indicates the TOML format should
now be officially considered stable. Empirically, TOML has proven to be a
stable format even prior to the release of TOML 1.0.0. From the
`changelog <https://github.com/toml-lang/toml/blob/master/CHANGELOG.md>`__, we
can see that TOML has had no major changes since April 2020, and has had
two releases in the past five years (2017-2021).

In the event of changes to the TOML specification, we can treat minor
revisions as bug fixes and update the implementation in place. In the event of
major breaking changes, we should preserve support for TOML 1.x.


Maintainability of proposed implementation
------------------------------------------

The proposed implementation (``tomli``) is pure Python, well tested and
weighs in at under 1000 lines of code. It is minimalist, offering a smaller API
surface area than other TOML implementations.

The author of ``tomli`` is willing to help integrate ``tomli`` into the standard
library and help maintain it, `as per this post
<https://github.com/hukkin/tomli/issues/141#issuecomment-998018972>`__.
Furthermore, Python core developer Petr Viktorin has indicated a willingness
to maintain a read API, `as per this post
<https://discuss.python.org/t/adopting-recommending-a-toml-parser/4068/88>`__.

Rewriting the parser in C is not deemed necessary at this time. It is rare for
TOML parsing to be a bottleneck in applications, and users with higher performance
needs can use a third-party library (as is already often the case with JSON,
despite Python offering a standard library C-extension module).


TOML support a slippery slope for other things
----------------------------------------------

As discussed in the `Motivation`_ section, TOML holds a special place in the
Python ecosystem, for reading :pep:`518` ``pyproject.toml`` packaging
and tool configuration files.
This chief reason to include TOML in the standard library does not apply to
other formats, such as YAML or MessagePack.

In addition, the simplicity of TOML distinguishes it from other formats like
YAML, which are highly complicated to construct and parse.

An API for writing TOML may, however, be added in a future PEP.


Backwards Compatibility
=======================

This proposal has no backwards compatibility issues within the standard
library, as it describes a new module.
Any existing third-party module named ``tomllib`` will break, as
``import tomllib`` will import the standard library module.
However, ``tomllib`` is not registered on PyPI, so it is unlikely that any
module with this name is widely used.

Note that we avoid using the more straightforward name ``toml`` to avoid
backwards compatibility implications for users who have pinned versions of the
current ``toml`` PyPI package.
For more details, see the `Alternative names for the module`_ section.


Security Implications
=====================

Errors in the implementation could cause potential security issues.
However, the parser's output is limited to simple data types; inability to load
arbitrary classes avoids security issues common in more "powerful" formats like
pickle and YAML. Also, the implementation will be in pure Python, which reduces
security issues endemic to C, such as buffer overflows.


How to Teach This
=================

The API of ``tomllib`` mimics that of other well-established file format
libraries, such as ``json`` and ``pickle``. The lack of a ``dump`` function will
be 


id: pep-0694.rst
content:
PEP: 694
Title: Upload 2.0 API for Python Package Indexes
Author: Barry Warsaw <barry@python.org>, Donald Stufft <donald@stufft.io>
Discussions-To: https://discuss.python.org/t/pep-694-upload-2-0-api-for-python-package-repositories/16879
Status: Draft
Type: Standards Track
Topic: Packaging
Created: 11-Jun-2022
Post-History: `27-Jun-2022 <https://discuss.python.org/t/pep-694-upload-2-0-api-for-python-package-repositories/16879>`__,
              `06-Jan-2025 <https://discuss.python.org/t/pep-694-pypi-upload-api-2-0/76316>`__
              `14-Apr-2025 <https://discuss.python.org/t/pep-694-pypi-upload-api-2-0/76316/9>`__


Abstract
========

This PEP proposes a standard API for uploading files to a Python package index such as PyPI.  Along
with standardization, the upload API provides additional useful features such as support for:

* an upload session, which can be used to simultaneously publish all wheels in a package release;

* "staging" a release, which can be used to test uploads before publicly publishing them, without the
  need for `test.pypi.org <https://test.pypi.org/>`__;

* artifacts which can be overwritten and replaced, until a session is published;

* asynchronous and "chunked", resumable file uploads, for more efficient use of network bandwidth;

* detailed status on the state of artifact uploads;

* new project creation without requiring the uploading of an artifact.

Once this new upload API is adopted, the existing legacy API can be deprecated, however this PEP
does not propose a deprecation schedule for the legacy API.


Rationale
=========

There is currently no standardized API for uploading files to a Python package index such as
PyPI. Instead, everyone has been forced to reverse engineer the existing `"legacy"
<https://docs.pypi.org/api/upload/>`__ API.

The legacy API, while functional, leaks implementation details of the original PyPI code base,
which has been faithfully replicated in the new code base and alternative implementations.

In addition, there are a number of major issues with the legacy API:

* It is fully synchronous, which forces requests to be held open both for the upload itself, and
  while the index processes the uploaded file to determine success or failure.

* It does not support any mechanism for resuming an upload. With the largest default file size on
  PyPI being around 1GB in size, requiring the entire upload to complete successfully means
  bandwidth is wasted when such uploads experience a network interruption while the request is in
  progress.

* The atomic unit of operation is a single file.  This is problematic when a release logically
  includes an sdist and multiple binary wheels, leading to race conditions where consumers get
  different versions of the package if they are unlucky enough to require a package before their
  platform's wheel has completely uploaded. If the release uploads its sdist first, this may also
  manifest in some consumers seeing only the sdist, triggering a local build from source.

* Status reporting is very limited.  There's no support for reporting multiple errors, warnings,
  deprecations, etc.  Status is limited to the HTTP status code and reason phrase, of which the
  reason phrase has been deprecated since HTTP/2 (:rfc:`RFC 7540 <7540#section-8.1.2.4>`).

* Metadata for a release is submitted alongside the file. However, as this metadata is famously
  unreliable, most installers instead choose to download the entire file and read the metadata from
  there.

* There is no mechanism for allowing an index to do any sort of sanity checks before bandwidth gets
  expended on an upload.  Many cases of invalid metadata or incorrect permissions could be checked
  prior to uploading files.

* There is no support for "staging" a release prior to publishing it to the index.

* Creation of new projects requires the uploading of at least one file, leading to "stub" uploads
  to claim a project namespace.

The new upload API proposed in this PEP solves all of these problems, providing for a much more
flexible, bandwidth friendly approach, with better error reporting, a better release testing
experience, and atomic and simultaneous publishing of all release artifacts.


Legacy API
==========

The following is an overview of the legacy API.  For the detailed description, consult the
`PyPI user guide documentation <https://docs.pypi.org/api/upload/>`__.


Endpoint
--------

The existing upload API lives at a base URL.  For PyPI, that URL is currently
``https://upload.pypi.org/legacy/``.  Clients performing uploads specify the API they want to call
by adding an ``:action`` URL parameter with a value of ``file_upload``. [#fn-action]_

The legacy API also has a ``protocol_version`` parameter, in theory allowing new versions of the API
to be defined.  In practice this has never happened, and the value is always ``1``.

Thus, the effective upload API on PyPI is:
``https://upload.pypi.org/legacy/?:action=file_upload&protocol_version=1``.


Encoding
--------

The data to be submitted is submitted as a ``POST`` request with the content type of
``multipart/form-data``.  This reflects the legacy API's historical nature, which was originally
designed not as an API, but rather as a web form on the initial PyPI implementation, with client code
written to programmatically submit that form.


Content
-------

Roughly speaking, the metadata contained within the package is submitted as parts where the content
disposition is ``form-data``, and the metadata key is the name of the field. The names of these
various pieces of metadata are not documented, and they sometimes, but not always match the names
used in the ``METADATA`` files for package artifacts. The case rarely matches, and the ``form-data``
to ``METADATA`` conversion is inconsistent.

The upload artifact file itself is sent as a ``application/octet-stream`` part with the name of
``content``, and if there is a PGP signature attached, then it will be included as a
``application/octet-stream`` part with the name of ``gpg_signature``.


Authentication
--------------

Upload authentication is also not standardized. On PyPI, authentication is through `API tokens
<https://pypi.org/help/>`__ or `Trusted Publisher (OpenID Connect)
<https://docs.pypi.org/trusted-publishers/>`__.  Other indexes may support different authentication
methods.

.. _spec:

Upload 2.0 API Specification
============================

This PEP draws inspiration from the `Resumable Uploads for HTTP <ietf-draft_>`_ internet draft,
however there are significant differences.  This is largely due to the unique nature of Python
package releases (i.e. metadata, multiple related artifacts, etc.), and the support for an upload
session and release stages.  Where it makes sense to adopt details of the draft, this PEP does so.

This PEP traces the root cause of most of the issues with the existing API to be roughly two things:

- The metadata is submitted alongside the file, rather than being parsed from the
  file itself. [#fn-metadata]_

- It supports only a single request, using only form data, that either succeeds or fails, and all
  actions are atomic within that single request.

To address these issues, this PEP proposes a multi-request workflow, which at a high level involves
these steps:

#. Initiate an upload session, creating a release stage.
#. Upload the file(s) to that stage as part of the upload session.
#. Complete the upload session, publishing or discarding the stage.
#. Optionally check the status of an upload session.


Versioning
----------

This PEP uses the same ``MAJOR.MINOR`` versioning system as used in :pep:`691`, but it is otherwise
independently versioned. The legacy API is considered by this PEP to be version ``1.0``, but this
PEP does not modify the legacy API in any way.

The API proposed in this PEP therefor has the version number ``2.0``.


Root Endpoint
-------------

All URLs described here are relative to the "root endpoint", which may be located an


id: pep-0441.rst
content:
PEP: 441
Title: Improving Python ZIP Application Support
Author: Daniel Holth <dholth@gmail.com>,
        Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://mail.python.org/pipermail/python-dev/2015-February/138277.html
Status: Final
Type: Standards Track
Created: 30-Mar-2013
Python-Version: 3.5
Post-History: 30-Mar-2013, 01-Apr-2013, 16-Feb-2015
Resolution: https://mail.python.org/pipermail/python-dev/2015-February/138578.html

Improving Python ZIP Application Support
========================================

Python has had the ability to execute directories or ZIP-format
archives as scripts since version 2.6 [1]_.  When invoked with a zip
file or directory as its first argument the interpreter adds that
directory to sys.path and executes the ``__main__`` module.  These
archives provide a great way to publish software that needs to be
distributed as a single file script but is complex enough to need to
be written as a collection of modules.

This feature is not as popular as it should be mainly because it was
not promoted as part of Python 2.6 [2]_, so that it is relatively
unknown, but also because the Windows installer does not register a
file extension (other than ``.py``) for this format of file, to associate
with the launcher.

This PEP proposes to fix these problems by re-publicising the feature,
defining the ``.pyz`` and ``.pyzw`` extensions as "Python ZIP Applications"
and "Windowed Python ZIP Applications", and providing some simple
tooling to manage the format.

A New Python ZIP Application Extension
======================================

The terminology "Python Zip Application" will be the formal term used
for a zip-format archive that contains Python code in a form that can
be directly executed by Python (specifically, it must have a
``__main__.py`` file in the root directory of the archive).  The
extension ``.pyz`` will be formally associated with such files.

The Python 3.5 installer will associate ``.pyz`` and ``.pyzw`` "Python
Zip Applications" with the platform launcher so they can be executed.
A ``.pyz`` archive is a console application and a ``.pyzw`` archive is a
windowed application, indicating whether the console should appear
when running the app.

On Unix, it would be ideal if the ``.pyz`` extension and the name
"Python Zip Application" were registered (in the mime types database?).
However, such an association is out of scope for this PEP.

Python Zip applications can be prefixed with a ``#!`` line
pointing to the correct Python interpreter and an optional
explanation::

    #!/usr/bin/env python3
    #  Python application packed with zipapp module
    (binary contents of archive)

On Unix, this allows the OS to run the file with the correct
interpreter, via the standard "shebang" support.  On Windows, the
Python launcher implements shebang support.

However, it is always possible to execute a ``.pyz`` application by
supplying the filename to the Python interpreter directly.

As background, ZIP archives are defined with a footer containing
relative offsets from the end of the file.  They remain valid when
concatenated to the end of any other file.  This feature is completely
standard and is how self-extracting ZIP archives and the bdist_wininst
installer format work.


Minimal Tooling: The zipapp Module
==================================

This PEP also proposes including a module for working with these
archives.  The module will contain functions for working with Python
zip application archives, and a command line interface (via ``python
-m zipapp``) for their creation and manipulation.

More complete tools for managing Python Zip Applications are
encouraged as 3rd party applications on PyPI.  Currently, pyzzer [5]_
and pex [6]_ are two such tools.

Module Interface
----------------

The zipapp module will provide the following functions:

``create_archive(source, target=None, interpreter=None, main=None)``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Create an application archive from *source*.  The source can be any
of the following:

* The name of a directory, in which case a new application archive
  will be created from the content of that directory.
* The name of an existing application archive file, in which case the
  file is copied to the target.  The file name should include the
  ``.pyz`` or ``.pyzw`` extension, if required.
* A file object open for reading in bytes mode.  The content of the
  file should be an application archive, and the file object is
  assumed to be positioned at the start of the archive.

The *target* argument determines where the resulting archive will be
written:

* If it is the name of a file, the archive will be written to that
  file.
* If it is an open file object, the archive will be written to that
  file object, which must be open for writing in bytes mode.
* If the target is omitted (or None), the source must be a directory
  and the target will be a file with the same name as the source, with
  a ``.pyz`` extension added.

The *interpreter* argument specifies the name of the Python
interpreter with which the archive will be executed.  It is written as
a "shebang" line at the start of the archive.  On Unix, this will be
interpreted by the OS, and on Windows it will be handled by the Python
launcher.  Omitting the *interpreter* results in no shebang line being
written.  If an interpreter is specified, and the target is a
filename, the executable bit of the target file will be set.

The *main* argument specifies the name of a callable which will be
used as the main program for the archive.  It can only be specified if
the source is a directory, and the source does not already contain a
``__main__.py`` file.  The *main* argument should take the form
"pkg.module:callable" and the archive will be run by importing
"pkg.module" and executing the given callable with no arguments.  It
is an error to omit *main* if the source is a directory and does not
contain a ``__main__.py`` file, as otherwise the resulting archive
would not be executable.

If a file object is specified for *source* or *target*, it is the
caller's responsibility to close it after calling create_archive.

When copying an existing archive, file objects supplied only need
``read`` and ``readline``, or ``write`` methods.  When creating an
archive from a directory, if the target is a file object it will be
passed to the ``zipfile.ZipFile`` class, and must supply the methods
needed by that class.

``get_interpreter(archive)``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Returns the interpreter specified in the shebang line of the
*archive*.  If there is no shebang, the function returns ``None``.
The *archive* argument can be a filename or a file-like object open
for reading in bytes mode.


Command Line Usage
------------------

The zipapp module can be run with the python ``-m`` flag.  The command
line interface is as follows::

    python -m zipapp directory [options]

        Create an archive from the given directory.  An archive will
        be created from the contents of that directory.  The archive
        will have the same name as the source directory with a .pyz
        extension.

        The following options can be specified:

        -o archive / --output archive

            The destination archive will have the specified name.  The
            given name will be used as written, so should include the
            ".pyz" or ".pyzw" extension.

        -p interpreter / --python interpreter

            The given interpreter will be written to the shebang line
            of the archive.  If this option is not given, the archive
            will have no shebang line.

        -m pkg.mod:fn / --main pkg.mod:fn

            The source directory must not have a __main__.py file. The
            archiver will write a __main__.py file into the target
            which calls fn from the module pkg.mod.

The behaviour of the command line interface matches that of
``zipapp.create_archive()``.

In additio


id: pep-0327.rst
content:
PEP: 327
Title: Decimal Data Type
Author: Facundo Batista <facundo@taniquetil.com.ar>
Status: Final
Type: Standards Track
Created: 17-Oct-2003
Python-Version: 2.4
Post-History: 30-Nov-2003, 02-Jan-2004, 29-Jan-2004


Abstract
========

The idea is to have a Decimal data type, for every use where decimals
are needed but binary floating point is too inexact.

The Decimal data type will support the Python standard functions and
operations, and must comply with the decimal arithmetic ANSI standard
X3.274-1996 [1]_.

Decimal will be floating point (as opposed to fixed point) and will
have bounded precision (the precision is the upper limit on the
number of significant digits in a result).  However, precision is
user-settable, and a notion of significant trailing zeroes is supported
so that fixed-point usage is also possible.

This work is based on code and test functions written by Eric Price,
Aahz and Tim Peters.  Just before Python 2.4a1, the decimal.py
`reference implementation`_ was moved into the standard library; along
with the documentation and the test suite, this was the work of
Raymond Hettinger.  Much of the explanation in this PEP is taken from
Cowlishaw's work [2]_, comp.lang.python and python-dev.


Motivation
==========

Here I'll expose the reasons of why I think a Decimal data type is
needed and why other numeric data types are not enough.

I wanted a Money data type, and after proposing a pre-PEP in
comp.lang.python, the community agreed to have a numeric data type
with the needed arithmetic behaviour, and then build Money over it:
all the considerations about quantity of digits after the decimal
point, rounding, etc., will be handled through Money.  It is not the
purpose of this PEP to have a data type that can be used as Money
without further effort.

One of the biggest advantages of implementing a standard is that
someone already thought out all the creepy cases for you.  And to a
standard GvR redirected me: Mike Cowlishaw's General Decimal
Arithmetic specification [2]_.  This document defines a general
purpose decimal arithmetic.  A correct implementation of this
specification will conform to the decimal arithmetic defined in
ANSI/IEEE standard 854-1987, except for some minor restrictions, and
will also provide unrounded decimal arithmetic and integer arithmetic
as proper subsets.


The problem with binary float
-----------------------------

In decimal math, there are many numbers that can't be represented with
a fixed number of decimal digits, e.g. 1/3 = 0.3333333333.......

In base 2 (the way that standard floating point is calculated), 1/2 =
0.1, 1/4 = 0.01, 1/8 = 0.001, etc.  Decimal 0.2 equals 2/10 equals
1/5, resulting in the binary fractional number
0.001100110011001...  As you can see, the problem is that some decimal
numbers can't be represented exactly in binary, resulting in small
roundoff errors.

So we need a decimal data type that represents exactly decimal
numbers.  Instead of a binary data type, we need a decimal one.


Why floating point?
-------------------

So we go to decimal, but why *floating point*?

Floating point numbers use a fixed quantity of digits (precision) to
represent a number, working with an exponent when the number gets too
big or too small.  For example, with a precision of 5::

       1234 ==>   1234e0
      12345 ==>  12345e0
     123456 ==>  12346e1

(note that in the last line the number got rounded to fit in five digits).

In contrast, we have the example of a ``long`` integer with infinite
precision, meaning that you can have the number as big as you want,
and you'll never lose any information.

In a fixed point number, the position of the decimal point is fixed.
For a fixed point data type, check Tim Peter's FixedPoint at
SourceForge [4]_.  I'll go for floating point because it's easier to
implement the arithmetic behaviour of the standard, and then you can
implement a fixed point data type over Decimal.

But why can't we have a floating point number with infinite precision?
It's not so easy, because of inexact divisions.  E.g.: 1/3 =
0.3333333333333... ad infinitum.  In this case you should store an
infinite amount of 3s, which takes too much memory, ;).

John Roth proposed to eliminate the division operator and force the
user to use an explicit method, just to avoid this kind of trouble.
This generated adverse reactions in comp.lang.python, as everybody
wants to have support for the ``/`` operator in a numeric data type.

With this exposed maybe you're thinking "Hey! Can we just store the 1
and the 3 as numerator and denominator?", which takes us to the next
point.


Why not rational?
-----------------

Rational numbers are stored using two integer numbers, the numerator
and the denominator.  This implies that the arithmetic operations
can't be executed directly (e.g. to add two rational numbers you first
need to calculate the common denominator).

Quoting Alex Martelli:

    The performance implications of the fact that summing two
    rationals (which take O(M) and O(N) space respectively) gives a
    rational which takes O(M+N) memory space is just too troublesome.
    There are excellent Rational implementations in both pure Python
    and as extensions (e.g., gmpy), but they'll always be a "niche
    market" IMHO.  Probably worth PEPping, not worth doing without
    Decimal -- which is the right way to represent sums of money, a
    truly major use case in the real world.

Anyway, if you're interested in this data type, you maybe will want to
take a look at :pep:`239`: Adding a Rational Type to Python.


So, what do we have?
--------------------

The result is a Decimal data type, with bounded precision and floating
point.

Will it be useful?  I can't say it better than Alex Martelli:

    Python (out of the box) doesn't let you have binary floating point
    numbers *with whatever precision you specify*: you're limited to
    what your hardware supplies.  Decimal, be it used as a fixed or
    floating point number, should suffer from no such limitation:
    whatever bounded precision you may specify on number creation
    (your memory permitting) should work just as well.  Most of the
    expense of programming simplicity can be hidden from application
    programs and placed in a suitable decimal arithmetic type.  As per
    http://speleotrove.com/decimal/, *a single data type can be
    used for integer, fixed-point, and floating-point decimal
    arithmetic* -- and for money arithmetic which doesn't drive the
    application programmer crazy.

There are several uses for such a data type.  As I said before, I will
use it as base for Money.  In this case the bounded precision is not
an issue; quoting Tim Peters:

    A precision of 20 would be way more than enough to account for
    total world economic output, down to the penny, since the
    beginning of time.


General Decimal Arithmetic Specification
========================================

Here I'll include information and descriptions that are part of the
specification [2]_ (the structure of the number, the context, etc.).
All the requirements included in this section are not for discussion
(barring typos or other mistakes), as they are in the standard, and
the PEP is just for implementing the standard.

Because of copyright restrictions, I can not copy here explanations
taken from the specification, so I'll try to explain it in my own
words.  I firmly encourage you to read the original specification
document [2]_ for details or if you have any doubt.


The Arithmetic Model
--------------------

The specification is based on a decimal arithmetic model, as defined
by the relevant standards: IEEE 854 [3]_, ANSI X3-274 [1]_, and the
proposed revision [5]_ of IEEE 754 [6]_.

The model has three components:

- Numbers: just the values that the operation uses as input or output.

- Operations: addition, multiplication, etc.

- Context: a set of parameters and rules that the user can select and
  which govern the results of oper


id: pep-0333.rst
content:
PEP: 333
Title: Python Web Server Gateway Interface v1.0
Author: Phillip J. Eby <pje@telecommunity.com>
Discussions-To: web-sig@python.org
Status: Final
Type: Informational
Created: 07-Dec-2003
Post-History: 07-Dec-2003, 08-Aug-2004, 20-Aug-2004, 27-Aug-2004, 27-Sep-2010
Superseded-By: 3333


Preface
=======

Note: For an updated version of this spec that supports Python 3.x and
includes community errata, addenda, and clarifications, please
see :pep:`3333` instead.


Abstract
========

This document specifies a proposed standard interface between web
servers and Python web applications or frameworks, to promote web
application portability across a variety of web servers.


Rationale and Goals
===================

Python currently boasts a wide variety of web application frameworks,
such as Zope, Quixote, Webware, SkunkWeb, PSO, and Twisted Web -- to
name just a few [1]_.  This wide variety of choices can be a problem
for new Python users, because generally speaking, their choice of web
framework will limit their choice of usable web servers, and vice
versa.

By contrast, although Java has just as many web application frameworks
available, Java's "servlet" API makes it possible for applications
written with any Java web application framework to run in any web
server that supports the servlet API.

The availability and widespread use of such an API in web servers for
Python -- whether those servers are written in Python (e.g. Medusa),
embed Python (e.g. mod_python), or invoke Python via a gateway
protocol (e.g. CGI, FastCGI, etc.) -- would separate choice of
framework from choice of web server, freeing users to choose a pairing
that suits them, while freeing framework and server developers to
focus on their preferred area of specialization.

This PEP, therefore, proposes a simple and universal interface between
web servers and web applications or frameworks: the Python Web Server
Gateway Interface (WSGI).

But the mere existence of a WSGI spec does nothing to address the
existing state of servers and frameworks for Python web applications.
Server and framework authors and maintainers must actually implement
WSGI for there to be any effect.

However, since no existing servers or frameworks support WSGI, there
is little immediate reward for an author who implements WSGI support.
Thus, WSGI **must** be easy to implement, so that an author's initial
investment in the interface can be reasonably low.

Thus, simplicity of implementation on *both* the server and framework
sides of the interface is absolutely critical to the utility of the
WSGI interface, and is therefore the principal criterion for any
design decisions.

Note, however, that simplicity of implementation for a framework
author is not the same thing as ease of use for a web application
author.  WSGI presents an absolutely "no frills" interface to the
framework author, because bells and whistles like response objects and
cookie handling would just get in the way of existing frameworks'
handling of these issues.  Again, the goal of WSGI is to facilitate
easy interconnection of existing servers and applications or
frameworks, not to create a new web framework.

Note also that this goal precludes WSGI from requiring anything that
is not already available in deployed versions of Python.  Therefore,
new standard library modules are not proposed or required by this
specification, and nothing in WSGI requires a Python version greater
than 2.2.2.  (It would be a good idea, however, for future versions
of Python to include support for this interface in web servers
provided by the standard library.)

In addition to ease of implementation for existing and future
frameworks and servers, it should also be easy to create request
preprocessors, response postprocessors, and other WSGI-based
"middleware" components that look like an application to their
containing server, while acting as a server for their contained
applications.

If middleware can be both simple and robust, and WSGI is widely
available in servers and frameworks, it allows for the possibility
of an entirely new kind of Python web application framework: one
consisting of loosely-coupled WSGI middleware components.  Indeed,
existing framework authors may even choose to refactor their
frameworks' existing services to be provided in this way, becoming
more like libraries used with WSGI, and less like monolithic
frameworks.  This would then allow application developers to choose
"best-of-breed" components for specific functionality, rather than
having to commit to all the pros and cons of a single framework.

Of course, as of this writing, that day is doubtless quite far off.
In the meantime, it is a sufficient short-term goal for WSGI to
enable the use of any framework with any server.

Finally, it should be mentioned that the current version of WSGI
does not prescribe any particular mechanism for "deploying" an
application for use with a web server or server gateway.  At the
present time, this is necessarily implementation-defined by the
server or gateway.  After a sufficient number of servers and
frameworks have implemented WSGI to provide field experience with
varying deployment requirements, it may make sense to create
another PEP, describing a deployment standard for WSGI servers and
application frameworks.


Specification Overview
======================

The WSGI interface has two sides: the "server" or "gateway" side, and
the "application" or "framework" side.  The server side invokes a
callable object that is provided by the application side.  The
specifics of how that object is provided are up to the server or
gateway.  It is assumed that some servers or gateways will require an
application's deployer to write a short script to create an instance
of the server or gateway, and supply it with the application object.
Other servers and gateways may use configuration files or other
mechanisms to specify where an application object should be
imported from, or otherwise obtained.

In addition to "pure" servers/gateways and applications/frameworks,
it is also possible to create "middleware" components that implement
both sides of this specification.  Such components act as an
application to their containing server, and as a server to a
contained application, and can be used to provide extended APIs,
content transformation, navigation, and other useful functions.

Throughout this specification, we will use the term "a callable" to
mean "a function, method, class, or an instance with a ``__call__``
method".  It is up to the server, gateway, or application implementing
the callable to choose the appropriate implementation technique for
their needs.  Conversely, a server, gateway, or application that is
invoking a callable **must not** have any dependency on what kind of
callable was provided to it.  Callables are only to be called, not
introspected upon.


The Application/Framework Side
------------------------------

The application object is simply a callable object that accepts
two arguments.  The term "object" should not be misconstrued as
requiring an actual object instance: a function, method, class,
or instance with a ``__call__`` method are all acceptable for
use as an application object.  Application objects must be able
to be invoked more than once, as virtually all servers/gateways
(other than CGI) will make such repeated requests.

(Note: although we refer to it as an "application" object, this
should not be construed to mean that application developers will use
WSGI as a web programming API!  It is assumed that application
developers will continue to use existing, high-level framework
services to develop their applications.  WSGI is a tool for
framework and server developers, and is not intended to directly
support application developers.)

Here are two example application objects; one is a function, and the
other is a class::

    def simple_app(environ, start_response):
        """Simplest possible application object"""
        status 


id: pep-0455.rst
content:
PEP: 455
Title: Adding a key-transforming dictionary to collections
Author: Antoine Pitrou <solipsis@pitrou.net>
BDFL-Delegate: Raymond Hettinger
Status: Rejected
Type: Standards Track
Created: 13-Sep-2013
Python-Version: 3.5
Post-History:


Abstract
========

This PEP proposes a new data structure for the ``collections`` module,
called "TransformDict" in this PEP.  This structure is a mutable mapping
which transforms the key using a given function when doing a lookup, but
retains the original key when reading.

Rejection
---------

See the rationale at
https://mail.python.org/pipermail/python-dev/2015-May/140003.html
and for an earlier partial review, see
https://mail.python.org/pipermail/python-dev/2013-October/129937.html .

Rationale
=========

Numerous specialized versions of this pattern exist.  The most common
is a case-insensitive case-preserving dict, i.e. a dict-like container
which matches keys in a case-insensitive fashion but retains the original
casing.  It is a very common need in network programming, as many
protocols feature some arrays of "key / value" properties in their
messages, where the keys are textual strings whose case is specified to
be ignored on receipt but by either specification or custom is to be
preserved or non-trivially canonicalized when retransmitted.

Another common request is an identity dict, where keys are matched
according to their respective id()s instead of normal matching.

Both are instances of a more general pattern, where a given transformation
function is applied to keys when looking them up: that function being
``str.lower`` or ``str.casefold`` in the former example and the built-in
``id`` function in the latter.

(It could be said that the pattern *projects* keys from the user-visible
set onto the internal lookup set.)


Semantics
=========

TransformDict is a ``MutableMapping`` implementation: it faithfully
implements the well-known API of mutable mappings, like ``dict`` itself
and other dict-like classes in the standard library.  Therefore, this PEP
won't rehash the semantics of most TransformDict methods.

The transformation function needn't be bijective, it can be strictly
surjective as in the case-insensitive example (in other words, different
keys can lookup the same value)::

   >>> d = TransformDict(str.casefold)
   >>> d['SomeKey'] = 5
   >>> d['somekey']
   5
   >>> d['SOMEKEY']
   5

TransformDict retains the first key used when creating an entry::

   >>> d = TransformDict(str.casefold)
   >>> d['SomeKey'] = 1
   >>> d['somekey'] = 2
   >>> list(d.items())
   [('SomeKey', 2)]

The original keys needn't be hashable, as long as the transformation
function returns a hashable one::

   >>> d = TransformDict(id)
   >>> l = [None]
   >>> d[l] = 5
   >>> l in d
   True

Constructor
-----------

As shown in the examples above, creating a TransformDict requires passing
the key transformation function as the first argument (much like creating
a ``defaultdict`` requires passing the factory function as first argument).

The constructor also takes other optional arguments which can be used
to initialize the TransformDict with certain key-value pairs.  Those
optional arguments are the same as in the ``dict`` and ``defaultdict``
constructors::

   >>> d = TransformDict(str.casefold, [('Foo', 1)], Bar=2)
   >>> sorted(d.items())
   [('Bar', 2), ('Foo', 1)]

Getting the original key
------------------------

TransformDict also features a lookup method returning the stored key
together with the corresponding value::

    >>> d = TransformDict(str.casefold, {'Foo': 1})
    >>> d.getitem('FOO')
    ('Foo', 1)
    >>> d.getitem('bar')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    KeyError: 'bar'

The method name ``getitem()`` follows the standard ``popitem()`` method
on mutable mappings.

Getting the transformation function
-----------------------------------

TransformDict has a simple read-only property ``transform_func`` which
gives back the transformation function.


Alternative proposals and questions
===================================

Retaining the last original key
-------------------------------

Most python-dev respondents found retaining the first user-supplied key
more intuitive than retaining the last.  Also, it matches the dict
object's own behaviour when using different but equal keys::

   >>> d = {}
   >>> d[1] = 'hello'
   >>> d[1.0] = 'world'
   >>> d
   {1: 'world'}

Furthermore, explicitly retaining the last key in a first-key-retaining
scheme is still possible using the following approach::

   d.pop(key, None)
   d[key] = value

while the converse (retaining the first key in a last-key-retaining
scheme) doesn't look possible without rewriting part of the container's
code.

Using an encoder / decoder pair
-------------------------------

Using a function pair isn't necessary, since the original key is retained
by the container.  Moreover, an encoder / decoder pair would require the
transformation to be bijective, which prevents important use cases
like case-insensitive matching.

Providing a transformation function for values
----------------------------------------------

Dictionary values are not used for lookup, their semantics are totally
irrelevant to the container's operation.  Therefore, there is no point in
having both an "original" and a "transformed" value: the transformed
value wouldn't be used for anything.

Providing a specialized container, not generic
----------------------------------------------

It was asked why we would provide the generic TransformDict construct
rather than a specialized case-insensitive dict variant.  The answer
is that it's nearly as cheap (code-wise and performance-wise) to provide
the generic construct, and it can fill more use cases.

Even case-insensitive dicts can actually elicit different transformation
functions: ``str.lower``, ``str.casefold`` or in some cases ``bytes.lower``
when working with text encoded in an ASCII-compatible encoding.

Other constructor patterns
--------------------------

Two other constructor patterns were proposed by Serhiy Storchaka:

* A type factory scheme::

    d = TransformDict(str.casefold)(Foo=1)

* A subclassing scheme::

    class CaseInsensitiveDict(TransformDict):
        __transform__ = str.casefold

    d = CaseInsensitiveDict(Foo=1)

While both approaches can be defended, they don't follow established
practices in the standard library, and therefore were rejected.


Implementation
==============

A patch for the collections module is tracked on the bug tracker at
http://bugs.python.org/issue18986.


Existing work
=============

Case-insensitive dicts are a popular request:

* http://twistedmatrix.com/documents/current/api/twisted.python.util.InsensitiveDict.html
* https://mail.python.org/pipermail/python-list/2013-May/647243.html
* https://mail.python.org/pipermail/python-list/2005-April/296208.html
* https://mail.python.org/pipermail/python-list/2004-June/241748.html
* http://bugs.python.org/msg197376
* http://stackoverflow.com/a/2082169
* http://stackoverflow.com/a/3296782
* http://code.activestate.com/recipes/66315-case-insensitive-dictionary/
* https://gist.github.com/babakness/3901174
* http://www.wikier.org/blog/key-insensitive-dictionary-in-python
* http://en.sharejs.com/python/14534
* http://www.voidspace.org.uk/python/archive.shtml#caseless

Identity dicts have been requested too:

* https://mail.python.org/pipermail/python-ideas/2010-May/007235.html
* http://www.gossamer-threads.com/lists/python/python/209527

Several modules in the standard library use identity lookups for object
memoization, for example ``pickle``, ``json``, ``copy``, ``cProfile``,
``doctest`` and ``_threading_local``.

Other languages
---------------

C# / .Net
^^^^^^^^^

.Net has a generic ``Dictionary`` class where you can specify a custom
``IEqualityComparer``: http://msdn.microsoft.com/en-us/library/xfhwa508.aspx

Using it is the recommended way to write case-insensitive diction


id: pep-0469.rst
content:
PEP: 469
Title: Migration of dict iteration code to Python 3
Author: Alyssa Coghlan <ncoghlan@gmail.com>
Status: Withdrawn
Type: Standards Track
Created: 18-Apr-2014
Python-Version: 3.5
Post-History: 18-Apr-2014, 21-Apr-2014


Abstract
========

For Python 3, :pep:`3106` changed the design of the ``dict`` builtin and the
mapping API in general to replace the separate list based and iterator based
APIs in Python 2 with a merged, memory efficient set and multiset view
based API. This new style of dict iteration was also added to the Python 2.7
``dict`` type as a new set of iteration methods.

This means that there are now 3 different kinds of dict iteration that may
need to be migrated to Python 3 when an application makes the transition:

* Lists as mutable snapshots: ``d.items()`` -> ``list(d.items())``
* Iterator objects: ``d.iteritems()`` -> ``iter(d.items())``
* Set based dynamic views: ``d.viewitems()`` -> ``d.items()``

There is currently no widely agreed best practice on how to reliably convert
all Python 2 dict iteration code to the common subset of Python 2 and 3,
especially when test coverage of the ported code is limited. This PEP
reviews the various ways the Python 2 iteration APIs may be accessed, and
looks at the available options for migrating that code to Python 3 by way of
the common subset of Python 2.6+ and Python 3.0+.

The PEP also considers the question of whether or not there are any
additions that may be worth making to Python 3.5 that may ease the
transition process for application code that doesn't need to worry about
supporting earlier versions when eventually making the leap to Python 3.


PEP Withdrawal
==============

In writing the second draft of this PEP, I came to the conclusion that
the readability of hybrid Python 2/3 mapping code can actually be best
enhanced by better helper functions rather than by making changes to
Python 3.5+. The main value I now see in this PEP is as a clear record
of the recommended approaches to migrating mapping iteration code from
Python 2 to Python 3, as well as suggesting ways to keep things readable
and maintainable when writing hybrid code that supports both versions.

Notably, I recommend that hybrid code avoid calling mapping iteration
methods directly, and instead rely on builtin functions where possible,
and some additional helper functions for cases that would be a simple
combination of a builtin and a mapping method in pure Python 3 code, but
need to be handled slightly differently to get the exact same semantics in
Python 2.

Static code checkers like pylint could potentially be extended with an
optional warning regarding direct use of the mapping iteration methods in
a hybrid code base.


Mapping iteration models
========================

Python 2.7 provides three different sets of methods to extract the keys,
values and items from a ``dict`` instance, accounting for 9 out of the
18 public methods of the ``dict`` type.

In Python 3, this has been rationalised to just 3 out of 11 public methods
(as the ``has_key`` method has also been removed).


Lists as mutable snapshots
--------------------------

This is the oldest of the three styles of dict iteration, and hence the
one implemented by the ``d.keys()``, ``d.values()`` and ``d.items()``
methods in Python 2.

These methods all return lists that are snapshots of the state of the
mapping at the time the method was called. This has a few consequences:

* the original object can be mutated freely without affecting iteration
  over the snapshot
* the snapshot can be modified independently of the original object
* the snapshot consumes memory proportional to the size of the original
  mapping

The semantic equivalent of these operations in Python 3 are
``list(d.keys())``, ``list(d.values())`` and ``list(d.iteritems())``.


Iterator objects
----------------

In Python 2.2, ``dict`` objects gained support for the then-new iterator
protocol, allowing direct iteration over the keys stored in the dictionary,
thus avoiding the need to build a list just to iterate over the dictionary
contents one entry at a time. ``iter(d)`` provides direct access to the
iterator object for the keys.

Python 2 also provides a ``d.iterkeys()`` method that is essentially
synonymous with ``iter(d)``, along with ``d.itervalues()`` and
``d.iteritems()`` methods.

These iterators provide live views of the underlying object, and hence may
fail if the set of keys in the underlying object is changed during
iteration::

    >>> d = dict(a=1)
    >>> for k in d:
    ...     del d[k]
    ...
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    RuntimeError: dictionary changed size during iteration

As iterators, iteration over these objects is also a one-time operation:
once the iterator is exhausted, you have to go back to the original mapping
in order to iterate again.

In Python 3, direct iteration over mappings works the same way as it does
in Python 2. There are no method based equivalents - the semantic equivalents
of ``d.itervalues()`` and ``d.iteritems()`` in Python 3 are
``iter(d.values())`` and ``iter(d.items())``.

The ``six`` and ``future.utils`` compatibility modules also both provide
``iterkeys()``, ``itervalues()`` and ``iteritems()`` helper functions that
provide efficient iterator semantics in both Python 2 and 3.


Set based dynamic views
-----------------------

The model that is provided in Python 3 as a method based API is that of set
based dynamic views (technically multisets in the case of the ``values()``
view).

In Python 3, the objects returned by ``d.keys()``, ``d.values()`` and
``d. items()`` provide a live view of the current state of
the underlying object, rather than taking a full snapshot of the current
state as they did in Python 2. This change is safe in many circumstances,
but does mean that, as with the direct iteration API, it is necessary to
avoid adding or removing keys during iteration, in order to avoid
encountering the following error::

    >>> d = dict(a=1)
    >>> for k, v in d.items():
    ...     del d[k]
    ...
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    RuntimeError: dictionary changed size during iteration

Unlike the iteration API, these objects are iterables, rather than iterators:
you can iterate over them multiple times, and each time they will iterate
over the entire underlying mapping.

These semantics are also available in Python 2.7 as the ``d.viewkeys()``,
``d.viewvalues()`` and ``d.viewitems()`` methods.

The ``future.utils`` compatibility module also provides
``viewkeys()``, ``viewvalues()`` and ``viewitems()`` helper functions
when running on Python 2.7 or Python 3.x.


Migrating directly to Python 3
==============================

The ``2to3`` migration tool handles direct migrations to Python 3 in
accordance with the semantic equivalents described above:

* ``d.keys()`` -> ``list(d.keys())``
* ``d.values()`` -> ``list(d.values())``
* ``d.items()`` -> ``list(d.items())``
* ``d.iterkeys()`` -> ``iter(d.keys())``
* ``d.itervalues()`` -> ``iter(d.values())``
* ``d.iteritems()`` -> ``iter(d.items())``
* ``d.viewkeys()`` -> ``d.keys()``
* ``d.viewvalues()`` -> ``d.values()``
* ``d.viewitems()`` -> ``d.items()``

Rather than 9 distinct mapping methods for iteration, there are now only the
3 view methods, which combine in straightforward ways with the two relevant
builtin functions to cover all of the behaviours that are available as
``dict`` methods in Python 2.7.

Note that in many cases ``d.keys()`` can be replaced by just ``d``, but the
``2to3`` migration tool doesn't attempt that replacement.

The ``2to3`` migration tool also *does not* provide any automatic assistance
for migrating references to these objects as bound or unbound methods - it
only automates conversions where the API is called immediately.


Migrating to the common subset of Python 2 and 3
================================================

When migrating to the common


id: pep-0482.rst
content:
PEP: 482
Title: Literature Overview for Type Hints
Author: Åukasz Langa <lukasz@python.org>
Discussions-To: python-ideas@python.org
Status: Final
Type: Informational
Topic: Typing
Created: 08-Jan-2015
Post-History:


Abstract
========

This PEP is one of three related to type hinting.  This PEP gives a
literature overview of related work.  The main spec is :pep:`484`.


Existing Approaches for Python
==============================


mypy
----

(This section is a stub, since `mypy`__ is essentially what we're
proposing.)

__ https://mypy-lang.org


Reticulated Python
------------------

`Reticulated Python`__ by Michael Vitousek is an example of
a slightly different approach to gradual typing for Python. It is
described in an actual `academic paper`__ written by
Vitousek with Jeremy Siek and Jim Baker (the latter of Jython fame).

__ https://github.com/mvitousek/reticulated
__ http://wphomes.soic.indiana.edu/jsiek/files/2014/03/retic-python.pdf

PyCharm
-------

PyCharm by JetBrains has been providing a way to specify and check
types for about four years.  The type system suggested by PyCharm__
grew from simple class types to tuple types, generic types,
function types, etc. based on feedback of many users who shared their
experience of using type hints in their code.

__ https://github.com/JetBrains/python-skeletons#types

Others
------

TBD: Add sections on pyflakes__, pylint__, numpy__,
`Argument Clinic`__, pytypedecl__, numba__, obiwan__.

__ https://github.com/pyflakes/pyflakes/
__ https://www.pylint.org
__ https://www.numpy.org
__ https://docs.python.org/3/howto/clinic.html
__ https://github.com/google/pytypedecl
__ https://numba.pydata.org
__ https://pypi.org/project/obiwan

Existing Approaches in Other Languages
======================================

ActionScript
------------

ActionScript__ is a class-based, single inheritance,
object-oriented superset of ECMAScript.  It supports interfaces and
strong runtime-checked static typing.  Compilation supports a âstrict
dialectâ where type mismatches are reported at compile-time.

__ https://livedocs.adobe.com/specs/actionscript/3/

Example code with types::

  package {
    import flash.events.Event;

    public class BounceEvent extends Event {
      public static const BOUNCE:String = "bounce";
      private var _side:String = "none";

      public function get side():String {
        return _side;
      }

      public function BounceEvent(type:String, side:String){
        super(type, true);
        _side = side;
      }

      public override function clone():Event {
        return new BounceEvent(type, _side);
      }
    }
  }

Dart
----

Dart__ is a class-based, single inheritance, object-oriented
language with C-style syntax.  It supports interfaces, abstract classes,
reified generics, and optional typing.

__ https://www.dartlang.org

Types are inferred when possible.  The runtime differentiates between two
modes of execution: *checked mode* aimed for development (catching type
errors at runtime) and *production mode* recommended for speed execution
(ignoring types and asserts).

Example code with types::

  class Point {
      final num x, y;

      Point(this.x, this.y);

      num distanceTo(Point other) {
          var dx = x - other.x;
          var dy = y - other.y;
          return math.sqrt(dx * dx + dy * dy);
      }
  }

Hack
----

Hack__ is a programming language that interoperates seamlessly
with PHP.  It provides opt-in static type checking, type aliasing,
generics, nullable types, and lambdas.

__ https://hacklang.org

Example code with types::

  <?hh
  class MyClass {
    private ?string $x = null;

    public function alpha(): int {
      return 1;
    }

    public function beta(): string {
      return 'hi test';
    }
  }

  function f(MyClass $my_inst): string {
    // Will generate a hh_client error
    return $my_inst->alpha();
  }

TypeScript
----------

TypeScript__ is a typed superset of JavaScript that adds
interfaces, classes, mixins and modules to the language.

__ http://www.typescriptlang.org

Type checks are duck typed.  Multiple valid function signatures are
specified by supplying overloaded function declarations.  Functions and
classes can use generics as type parameterization.  Interfaces can have
optional fields.  Interfaces can specify array and dictionary types.
Classes can have constructors that implicitly add arguments as fields.
Classes can have static fields.  Classes can have private fields.
Classes can have getters/setters for fields (like property).  Types are
inferred.

Example code with types::

  interface Drivable {
      start(): void;
      drive(distance: number): boolean;
      getPosition(): number;
  }

  class Car implements Drivable {
      private _isRunning: boolean;
      private _distanceFromStart: number;

      constructor() {
          this._isRunning = false;
          this._distanceFromStart = 0;
      }

      public start() {
          this._isRunning = true;
      }

      public drive(distance: number): boolean {
          if (this._isRunning) {
              this._distanceFromStart += distance;
              return true;
          }
          return false;
      }

      public getPosition(): number {
          return this._distanceFromStart;
      }
  }


Copyright
=========

This document has been placed in the public domain.


id: pep-0496.rst
content:
PEP: 496
Title: Environment Markers
Author: James Polley <jp@jamezpolley.com>
BDFL-Delegate: Alyssa Coghlan <ncoghlan@gmail.com>
Status: Rejected
Type: Informational
Topic: Packaging
Created: 03-Jul-2015

PEP Status
==========

After this PEP was initially drafted, :pep:`508` was developed and submitted to
fully specify the dependency declaration syntax, including environment markers.
As a result, this PEP ended up being rejected in favour of the more comprehensive
:pep:`508`.

Abstract
========

An **environment marker** describes a condition about the current execution
environment. They are used to indicate when certain dependencies are only
required in particular environments, and to indicate supported platforms
for distributions with additional constraints beyond the availability of a
Python runtime.

Environment markers were first specified in :pep:`345`. :pep:`426`
(which would replace :pep:`345`) proposed extensions to the markers.
When 2.7.10 was released, even these extensions became insufficient due to
their reliance on simple lexical comparisons, and thus this PEP has been born.

Rationale
=========

Many Python packages are written with portability in mind.

For many packages this means they aim to support a wide range of
Python releases. If they depend on libraries such as ``argparse`` -
which started as external libraries, but later got incorporated into
core - specifying a single set of requirements is difficult, as the
set of required packages differs depending on the version of Python in
use.

For other packages, designing for portability means supporting
multiple operating systems. However, the significant differences
between them may mean that particular dependencies are only needed on
particular platforms (relying on ``pywin32`` only on Windows, for
example)"

Environment Markers attempt to provide more flexibility in a list of
requirements by allowing the developer to list requirements that are
specific to a particular environment.

Examples
========

Here are some examples of such markers inside a requirements.txt::

   pywin32 >=1.0 ; sys_platform == 'win32'
   unittest2 >=2.0,<3.0 ; python_version == '2.4' or python_version == '2.5'
   backports.ssl_match_hostname >= 3.4 ; python_version < '2.7.9' or (python_version >= '3.0' and python_version < '3.4')

And here's an example of some conditional metadata included in
setup.py for a distribution that requires PyWin32 both at runtime and
buildtime when using Windows::

   setup(
     install_requires=["pywin32 > 1.0 : sys.platform == 'win32'"],
     setup_requires=["pywin32 > 1.0 : sys.platform == 'win32'"]
     )


Micro-language
==============

The micro-language behind this is as follows. It compares:

* strings with the ``==`` and ``in`` operators (and their opposites)
* version numbers with the ``<``, ``<=``, ``>=``, and ``<`` operators
  in addition to those supported for strings

The usual boolean operators ``and`` and ``or`` can be used to combine
expressions, and parentheses are supported for grouping.

The pseudo-grammar is ::

    MARKER: EXPR [(and|or) EXPR]*
    EXPR: ("(" MARKER ")") | (STREXPR|VEREXPR)
    STREXPR: STRING [STRCMPOP STREXPR]
    STRCMPOP: ==|!=|in|not in
    VEREXPR: VERSION [VERCMPOP VEREXPR]
    VERCMPOP: (==|!=|<|>|<=|>=)


``SUBEXPR`` is either a Python string (such as ``'win32'``) or one of
the ``Strings`` marker variables listed below.

``VEREXPR`` is a :pep:`440` version identifier, or one of the
``Version number`` marker variables listed below. Comparisons between
version numbers are done using :pep:`440` semantics.


Strings
-------

* ``os_name``: ``os.name``
* ``sys_platform``: ``sys.platform``
* ``platform_release``: ``platform.release()``
* ``implementation_name``: ``sys.implementation.name``
* ``platform_machine``: ``platform.machine()``
* ``platform_python_implementation``: ``platform.python_implementation()``


If a particular string value is not available (such as ``sys.implementation.name``
in versions of Python prior to 3.3), the corresponding marker
variable MUST be considered equivalent to the empty string.

If a particular version number value is not available (such as
``sys.implementation.version`` in versions of Python prior to 3.3) the
corresponding marker variable MUST be considered equivalent to ``0``


Version numbers
---------------

* ``python_version``: ``platform.python_version()[:3]``
* ``python_full_version``: see definition below
* ``platform_version``: ``platform.version()``
* ``implementation_version``: see definition below

The ``python_full_version`` and ``implementation_version`` marker variables
are derived from ``sys.version_info`` and ``sys.implementation.version``
respectively, in accordance with the following algorithm::

    def format_full_version(info):
        version = '{0.major}.{0.minor}.{0.micro}'.format(info)
        kind = info.releaselevel
        if kind != 'final':
            version += kind[0] + str(info.serial)
        return version

    python_full_version = format_full_version(sys.version_info)
    implementation_version = format_full_version(sys.implementation.version)

``python_full_version`` will typically correspond to ``sys.version.split()[0]``.


Copyright
=========

This document has been placed in the public domain.


id: pep-0497.rst
content:
PEP: 497
Title: A standard mechanism for backward compatibility
Author: Ed Schofield <ed at pythoncharmers.com>
PEP-Delegate: Brett Cannon <brett@python.org>
Status: Rejected
Type: Process
Created: 04-Aug-2015


Rejection Notice
================
The steering council decided that the ``__past__`` aspect of this proposal
was too complicated for the potential benefit. The other aspect of stronger
requirements for backwards-compatibility should be addressed by :pep:`387`.


Scope
=====

This PEP is complementary to PEPs 5, 236, and 387, and shares similar
goals.

This PEP explains the need for an additional compatibility mechanism
in support of :pep:`5`, "Guidelines for Language Evolution". :pep:`236`,
"Back to the ``__future__``", introduced a mechanism for forward
compatibility in support of :pep:`5` but noted that a new mechanism for
backward compatibility was outside the scope of that PEP. A related
PEP (in progress) introduces such a mechanism for backward
compatibility.

:pep:`5`, "Guidelines for Language Evolution", notes that "This PEP [:pep:`5`]
does not replace or preclude other compatibility strategies such as
dynamic loading of backwards-compatible parsers."


Context
=======

From :pep:`236`: "From time to time, Python makes an incompatible change
to the advertised semantics of core language constructs, or changes
their accidental (implementation-dependent) behavior in some way.
While this is never done capriciously, and is always done with the aim
of improving the language over the long term, over the short term it's
contentious and disrupting. :pep:`5`, Guidelines for Language Evolution,
suggests ways to ease the pain, and this PEP [:pep:`236`] introduces some
machinery in support of that."

Also from :pep:`236`: "The purpose of future_statement is to make life
easier for people who keep current with the latest release in a timely
fashion. We don't hate you if you don't, but your problems are much
harder to solve, and somebody with those problems will need to write a
PEP addressing them. future_statement is aimed at a different
audience."


The current situation
=====================

When an incompatible change to core language syntax or semantics is
being made, Python currently provides the future_statement mechanism
for providing forward compatibility until the release that enforces
the new syntax or semantics, but provides no corresponding standard
mechanism for providing backward compatibility after this release.


Problem
=======

A consequence of this asymmetry is that, with respect to a breaking
change, the older (pre-breaking) version of the Python interpreter is
more capable than the newer (breaking) version; the older interpreter
can use both code designed prior to the change and newer code, whereas
the newer interpreter is only capable of using code that has been
upgraded to support the changed feature.

As an example, consider the changes to the division operator
introduced in :pep:`238` in 2001, soon after :pep:`236` introduced the
future_statement mechanism. :pep:`238` outlines a suite of useful
forward-compatibility mechanisms for "true division" in the Python 2.x
series but omits to include any backward-compatibility mechanisms for
after "true division" was first enforced in Python 3.0. Python versions
since 3.0 do not provide a backward compatibility mechanism such as
``from __past__ import division`` for code that expects the old
"classic division" semantics, whereas Python versions prior to 3.0 do
support both "classic division" code and also forward compatibility
with code expecting "true division". A further consequence of this is
that the "most compatible" interpreter with respect to the variety of
division-related Python code in the wild is Python 2.7, the version
before the breaking change was first enforced.


Backward compatibility as enabler for "downhill upgrades"
=========================================================

In contrast to this situation, newer versions of application software
such as office suites tend to be more capable than earlier versions
with respect to their support for loading different versions of their
data file formats. The pattern is usually that the newer application
versions can transparently load data from either their newer or their
older data formats, and that the newer version defaults to saving data
in the newer format. Newer application software versions tend to be
backward-compatible by default. Forward compatibility is relatively
rare.

This policy puts the user of the newer application software at an
advantage over the user of the older software, which is usually
incapable of loading data in the newer format. Sometimes it is
possible for a user of a newer software application version to export
data in an older version by choosing this option explicitly. In these
cases, the forward-compatibility this enables may or may not be
perfect; some features may be missing or the results may be otherwise
suboptimal. Upgrading is therefore easy, whereas downgrading is
harder.

The emergent behaviour over many users from such a policy of new
attractive features plus backward compatibility features is that a
natural pressure builds up on each individual user to upgrade his or
her own application version, and, the more other users an individual
exchanges data files with, the more acute this pressure becomes.


Proposal - part 1
=================

This PEP makes two specific, related proposals. The first is that:

    :pep:`5` be augmented with a 6th step in the section "Steps for
    Introducing Backwards-Incompatible Features" to indicate that, when an
    incompatible change to core language syntax or semantics is being
    made, Python-dev's policy is to prefer and expect that, wherever
    possible, a mechanism for backward compatibility be considered and
    provided for future Python versions after the breaking change is
    adopted by default, in addition to any mechanisms proposed for forward
    compatibility such as new future_statements. Furthermore, :pep:`387`,
    "Backwards Compatibility Policy" (if accepted) would be
    augmented with the same 6th step.


Example
~~~~~~~

As an example of how this PEP is to be applied, if the latest revision
of the "true division" PEP (238) were proposed today, it would be
considered incomplete. :pep:`238` notes the "severe backwards
compatibility issues" raised by the proposal and describes several
measures for forward compatibility in the Abstract and API Changes
sections. It also mentions some backward compatibility ideas raised on
c.l.py, including "Use ``from __past__ import division`` to use
classic division semantics in a module", but it does not put forward
any backward compatibility plan as part of the proposal.

If this PEP is accepted, it would be expected that a proposal such as
:pep:`238`, because of its large-scale compatibility implications, would
also be accompanied by a backward compatibility plan that enables
users of future Python versions after the breaking change has come
into effect to re-enable the classic division behaviour easily in
their code.


Proposal - part 2
=================

The second proposal is that:

    Python provide a standard backward compatibility mechanism in
    parallel to the ``__future__`` module mechanism for forward
    compatibility.

For reference, this document will refer to this as a "``__past__``"
mechanism hereon, although it need not have all the characteristics
of the ``__future__`` module and ``future_statement`` mechanism.

The specific form and implementation of the ``__past__`` mechanism is
the subject of a separate PEP (in progress).  However, this PEP
recommends that this ``__past__`` mechanism be designed to meet
similar criteria to those outlined in :pep:`296` for ``__future__``.
Specifically:

a. It should enable individual modules to specify obsolete behaviours
to re-enable from older Python versions on a module-by-module basis.

b. It should be flexible enough for both Py


id: pep-0483.rst
content:
PEP: 483
Title: The Theory of Type Hints
Author: Guido van Rossum <guido@python.org>, Ivan Levkivskyi <levkivskyi@gmail.com>
Discussions-To: python-ideas@python.org
Status: Final
Type: Informational
Topic: Typing
Created: 19-Dec-2014
Post-History:


Abstract
========

This PEP lays out the theory referenced by :pep:`484`.


Introduction
============

This document lays out the theory of the new type hinting proposal for
Python 3.5. It's not quite a full proposal or specification because
there are many details that need to be worked out, but it lays out the
theory without which it is hard to discuss more detailed specifications.
We start by recalling basic concepts of type theory; then we explain
gradual typing; then we state some general rules and
define the new special types (such as ``Union``) that can be used
in annotations; and finally we define the approach to generic types
and pragmatic aspects of type hinting.


Notational conventions
----------------------

-  ``t1``, ``t2``, etc. and ``u1``, ``u2``, etc. are types. Sometimes we write
   ``ti`` or ``tj`` to refer to "any of ``t1``, ``t2``, etc."
-  ``T``, ``U`` etc. are type variables (defined with ``TypeVar()``, see below).
-  Objects, classes defined with a class statement, and instances are
   denoted using standard :pep:`8` conventions.
-  the symbol ``==`` applied to types in the context of this PEP means that
   two expressions represent the same type.
-  Note that :pep:`484` makes a distinction between types and classes
   (a type is a concept for the type checker,
   while a class is a runtime concept).  In this PEP we clarify
   this distinction but avoid unnecessary strictness to allow more
   flexibility in the implementation of type checkers.


Background
==========

There are many definitions of the concept of type in the literature.
Here we assume that type is a set of values and a set of functions that
one can apply to these values.

There are several ways to define a particular type:

- By explicitly listing all values.  E.g., ``True`` and ``False``
  form the type ``bool``.
- By specifying functions which can be used with variables of
  a type.  E.g. all objects that have a ``__len__`` method form
  the type ``Sized``.  Both ``[1, 2, 3]`` and ``'abc'`` belong to
  this type, since one can call ``len`` on them::

    len([1, 2, 3])  # OK
    len('abc')      # also OK
    len(42)         # not a member of Sized

- By a simple class definition, for example if one defines a class::

    class UserID(int):
        pass

  then all instances of this class also form a type.
- There are also more complex types.  E.g., one can define the type
  ``FancyList`` as all lists containing only instances of ``int``, ``str``
  or their subclasses.  The value ``[1, 'abc', UserID(42)]`` has this type.

It is important for the user to be able to define types in a form
that can be understood by type checkers.
The goal of this PEP is to propose such a systematic way of defining types
for type annotations of variables and functions using :pep:`3107` syntax.
These annotations can be used to avoid many kind of bugs, for documentation
purposes, or maybe even to increase speed of program execution.
Here we only focus on avoiding bugs by using a static type checker.


Subtype relationships
---------------------

A crucial notion for static type checker is the subtype relationship.
It arises from the question: If ``first_var`` has type ``first_type``, and
``second_var`` has type ``second_type``, is it safe to assign
``first_var = second_var``?

A strong criterion for when it *should* be safe is:

- every value from ``second_type`` is also in the set of values
  of ``first_type``; and
- every function from ``first_type`` is also in the set of functions
  of ``second_type``.

The relation defined thus is called a subtype relation.

By this definition:

-  Every type is a subtype of itself.
-  The set of values becomes smaller in the process of subtyping,
   while the set of functions becomes larger.

An intuitive example: Every ``Dog`` is an ``Animal``, also ``Dog``
has more functions, for example it can bark, therefore ``Dog``
is a subtype of ``Animal``.  Conversely, ``Animal`` is not a subtype of ``Dog``.

A more formal example: Integers are subtype of real numbers.
Indeed, every integer is of course also a real number, and integers
support more operations, such as, e.g., bitwise shifts ``<<`` and ``>>``::

  lucky_number = 3.14    # type: float
  lucky_number = 42      # Safe
  lucky_number * 2       # This works
  lucky_number << 5      # Fails

  unlucky_number = 13    # type: int
  unlucky_number << 5    # This works
  unlucky_number = 2.72  # Unsafe

Let us also consider a tricky example: If ``List[int]`` denotes the type
formed by all lists containing only integer numbers,
then it is *not* a subtype of ``List[float]``, formed by all lists that contain
only real numbers. The first condition of subtyping holds,
but appending a real number only works with ``List[float]`` so that
the second condition fails::

  def append_pi(lst: List[float]) -> None:
      lst += [3.14]

  my_list = [1, 3, 5]  # type: List[int]

  append_pi(my_list)   # Naively, this should be safe...

  my_list[-1] << 5     # ... but this fails

There are two widespread approaches to *declare* subtype information
to type checker.

In nominal subtyping, the type tree is based on the class tree,
i.e., ``UserID`` is considered a subtype of ``int``.
This approach should be used under control of the type checker,
because in Python one can override attributes in an incompatible way::

  class Base:
      answer = '42' # type: str

  class Derived(Base):
      answer = 5 # should be marked as error by type checker

In structural subtyping the subtype relation is deduced from the
declared methods, i.e., ``UserID`` and ``int`` would be considered the same type.
While this may occasionally cause confusion,
structural subtyping is considered more flexible.
We strive to provide support for both approaches, so that
structural information can be used in addition to nominal subtyping.

.. _pep-483-gradual-typing:

Summary of gradual typing
=========================

Gradual typing allows one to annotate only part of a program,
thus leverage desirable aspects of both dynamic and static typing.

We define a new relationship, is-consistent-with, which is similar to
is-subtype-of, except it is not transitive when the new type ``Any`` is
involved. (Neither relationship is symmetric.) Assigning ``a_value``
to ``a_variable`` is OK if the type of ``a_value`` is consistent with
the type of ``a_variable``. (Compare this to "... if the type of ``a_value``
is a subtype of the type of ``a_variable``", which states one of the
fundamentals of OO programming.) The is-consistent-with relationship is
defined by three rules:

-  A type ``t1`` is consistent with a type ``t2`` if ``t1`` is a
   subtype of ``t2``. (But not the other way around.)
-  ``Any`` is consistent with every type. (But ``Any`` is not a subtype
   of every type.)
-  Every type is consistent with ``Any``. (But every type is not a subtype
   of ``Any``.)

That's all! See Jeremy Siek's blog post `What is Gradual
Typing <http://wphomes.soic.indiana.edu/jsiek/what-is-gradual-typing/>`_
for a longer explanation and motivation. ``Any`` can be considered a type
that has all values and all methods. Combined with the definition of
subtyping above, this places ``Any`` partially at the top (it has all values)
and bottom (it has all methods) of the type hierarchy. Contrast this to
``object`` -- it is not consistent with
most types (e.g. you can't use an ``object()`` instance where an
``int`` is expected). IOW both ``Any`` and ``object`` mean
"any type is allowed" when used to annotate an argument, but only ``Any``
can be passed no matter what type is expected (in essence, ``Any``
declares a fallback to dynamic typing and shuts up complaints
from the static checker).

Here's an example showing how these rules w


id: pep-0468.rst
content:
PEP: 468
Title: Preserving the order of \*\*kwargs in a function.
Author: Eric Snow <ericsnowcurrently@gmail.com>
Discussions-To: python-ideas@python.org
Status: Final
Type: Standards Track
Created: 05-Apr-2014
Python-Version: 3.6
Post-History: 05-Apr-2014, 08-Sep-2016
Resolution: https://mail.python.org/pipermail/python-dev/2016-September/146329.html


Abstract
========

The \*\*kwargs syntax in a function definition indicates that the
interpreter should collect all keyword arguments that do not correspond
to other named parameters.  However, Python does not preserved the
order in which those collected keyword arguments were passed to the
function.  In some contexts the order matters.  This PEP dictates that
the collected keyword arguments be exposed in the function body as an
ordered mapping.


Motivation
==========

Python's \*\*kwargs syntax in function definitions provides a powerful
means of dynamically handling keyword arguments.  In some applications
of the syntax (see _`Use Cases`), the semantics applied to the
collected keyword arguments requires that order be preserved.
Unsurprisingly, this is similar to how OrderedDict is related to dict.

Currently to preserved the order you have to do so manually and
separately from the actual function call.  This involves building an
ordered mapping, whether an OrderedDict or an iterable of 2-tuples,
which is then passed as a single argument to the function.
[#arg_unpacking]_

With the capability described in this PEP, that boilerplate would no
longer be required.

For comparison, currently::

   kwargs = OrderedDict()
   kwargs['eggs'] = ...
   ...
   def spam(a, kwargs):
       ...

and with this proposal::

   def spam(a, **kwargs):
       ...

Alyssa (Nick) Coghlan, speaking of some of the uses cases, summed it up well
[#alyssa_obvious]_::

   These *can* all be done today, but *not* by using keyword arguments.
   In my view, the problem to be addressed is that keyword arguments
   *look* like they should work for these cases, because they have a
   definite order in the source code. The only reason they don't work
   is because the interpreter throws that ordering information away.

   It's a textbook case of a language feature becoming an attractive
   nuisance in some circumstances: the simple and obvious solution for
   the above use cases *doesn't actually work* for reasons that aren't
   obviously clear if you don't have a firm grasp of Python's admittedly
   complicated argument handling.

This observation is supported by the appearance of this proposal over
the years and the numerous times that people have been confused by the
constructor for OrderedDict. [#past_threads]_ [#loss_of_order]_
[#compact_dict]_


Use Cases
=========

As Alyssa noted, the current behavior of \*\*kwargs is unintuitive in
cases where one would expect order to matter.  Aside from more specific
cases outlined below, in general "anything else where you want to
control the iteration order *and* set field names and values in a single
call will potentially benefit." [#alyssa_general]_  That matters in the
case of factories (e.g. __init__()) for ordered types.

Serialization
-------------

Obviously OrderedDict would benefit (both __init__() and update()) from
ordered kwargs.  However, the benefit also extends to serialization
APIs [#alyssa_obvious]_::

   In the context of serialisation, one key lesson we have learned is
   that arbitrary ordering is a problem when you want to minimise
   spurious diffs, and sorting isn't a simple solution.

   Tools like doctest don't tolerate spurious diffs at all, but are
   often amenable to a sorting based answer.

   The cases where it would be highly desirable to be able use keyword
   arguments to control the order of display of a collection of key
   value pairs are ones like:

   * printing out key:value pairs in CLI output
   * mapping semantic names to column order in a CSV
   * serialising attributes and elements in particular orders in XML
   * serialising map keys in particular orders in human readable formats
     like JSON and YAML (particularly when they're going to be placed
     under source control)

Debugging
---------

In the words of Raymond Hettinger [#raymond_debug]_::

   It makes it easier to debug if the arguments show-up in the order
   they were created.  AFAICT, no purpose is served by scrambling them.

Other Use Cases
---------------

* Mock objects. [#mock]_
* Controlling object presentation.
* Alternate namedtuple() where defaults can be specified.
* Specifying argument priority by order.


Concerns
========

Performance
-----------

As already noted, the idea of ordered keyword arguments has come up on
a number of occasions.  Each time it has been met with the same
response, namely that preserving keyword arg order would have a
sufficiently adverse effect on function call performance that it's not
worth doing.  However, Guido noted the following [#guido_open]_::

  Making **kwds ordered is still open, but requires careful design and
  implementation to avoid slowing down function calls that don't benefit.

As will be noted below, there are ways to work around this at the
expense of increased complication.  Ultimately the simplest approach is
the one that makes the most sense: pack collected key word arguments
into an OrderedDict.  However, without a C implementation of OrderedDict
there isn't much to discuss.  That changed in Python 3.5.
[#c_ordereddict]_

Note: in Python 3.6 dict is order-preserving.  This virtually eliminates
performance concerns.

Other Python Implementations
----------------------------

Another important issue to consider is that new features must be
cognizant of the multiple Python implementations.  At some point each of
them would be expected to have implemented ordered kwargs.  In this
regard there doesn't seem to be an issue with the idea. [#ironpython]_
An informal survey of the major Python implementations has indicated
that this feature will not be a significant burden.


Specification
=============

Starting in version 3.6 Python will preserve the order of keyword
arguments as passed to a function.  To accomplish this the collected
kwargs will now be an ordered mapping.  Note that this does not necessarily
mean OrderedDict.  dict in CPython 3.6 is now ordered, similar to PyPy.

This will apply only to functions for which the definition uses the
\*\*kwargs syntax for collecting otherwise unspecified keyword
arguments.  Only the order of those keyword arguments will be
preserved.

Relationship to \*\*-unpacking syntax
-------------------------------------

The ** unpacking syntax in function calls has no special connection with
this proposal.  Keyword arguments provided by unpacking will be treated
in exactly the same way as they are now: ones that match defined
parameters are gather there and the remainder will be collected into the
ordered kwargs (just like any other unmatched keyword argument).

Note that unpacking a mapping with undefined order, such as dict, will
preserve its iteration order like normal.  It's just that the order will
remain undefined.  The ordered mapping into which the unpacked key-value
pairs will then be packed will not be able to provide any alternate
ordering.  This should not be surprising.

There have been brief discussions of simply passing these mappings
through to the functions kwargs without unpacking and repacking them,
but that is both outside the scope of this proposal and probably a bad
idea regardless.  (There is a reason those discussions were brief.)

Relationship to inspect.Signature
---------------------------------

Signature objects should need no changes.  The ``kwargs`` parameter of
inspect.BoundArguments (returned by Signature.bind() and
Signature.bind_partial()) will change from a dict to an OrderedDict.

C-API
-----

No changes.

Syntax
------

No syntax is added or changed by this proposal.

Backward-Compatibility
----------------------

The following will change:

* itera


id: pep-0332.rst
content:
PEP: 332
Title: Byte vectors and String/Unicode Unification
Author: Skip Montanaro <skip@pobox.com>
Status: Rejected
Type: Standards Track
Created: 11-Aug-2004
Python-Version: 2.5
Post-History:


Abstract
========

This PEP outlines the introduction of a raw ``bytes`` sequence object
and the unification of the current ``str`` and ``unicode`` objects.


Rejection Notice
================

This PEP is rejected in this form.  The author has expressed lack of
time to continue to shepherd it, and discussion on python-dev has
moved to a slightly different proposal which will (eventually) be
written up as a new PEP.  See the thread starting at
https://mail.python.org/pipermail/python-dev/2006-February/060930.html.


Rationale
=========

Python's current string objects are overloaded.  They serve both to
hold ASCII and non-ASCII character data and to also hold sequences of
raw bytes which have no reasonable interpretation as displayable
character sequences.  This overlap hasn't been a big problem in the
past, but as Python moves closer to requiring source code to be
properly encoded, the use of strings to represent raw byte sequences
will be more problematic.  In addition, as Python's Unicode support
has improved, it's easier to consider strings as ASCII-encoded Unicode
objects.


Proposed Implementation
=======================

The number in parentheses indicates the Python version in which the
feature will be introduced.

- Add a ``bytes`` builtin which is just a synonym for ``str``. (2.5)

- Add a ``b"..."`` string literal which is equivalent to raw string
  literals, with the exception that values which conflict with the
  source encoding of the containing file not generate warnings. (2.5)

- Warn about the use of variables named "bytes". (2.5 or 2.6)

- Introduce a ``bytes`` builtin which refers to a sequence distinct
  from the ``str`` type. (2.6)

- Make ``str`` a synonym for ``unicode``. (3.0)


Bytes Object API
================

TBD.


Issues
======

- Can this be accomplished before Python 3.0?

- Should ``bytes`` objects be mutable or immutable?  (Guido seems to
  like them to be mutable.)


Copyright
=========

This document has been placed in the public domain.


id: pep-0454.rst
content:
PEP: 454
Title: Add a new tracemalloc module to trace Python memory allocations
Author: Victor Stinner <vstinner@python.org>
BDFL-Delegate: Charles-FranÃ§ois Natali <cf.natali@gmail.com>
Status: Final
Type: Standards Track
Created: 03-Sep-2013
Python-Version: 3.4
Resolution: https://mail.python.org/pipermail/python-dev/2013-November/130491.html


Abstract
========

This PEP proposes to add a new ``tracemalloc`` module to trace memory
blocks allocated by Python.


Rationale
=========

Classic generic tools like Valgrind can get the C traceback where a
memory block was allocated. Using such tools to analyze Python memory
allocations does not help because most memory blocks are allocated in
the same C function, in ``PyMem_Malloc()`` for example. Moreover, Python
has an allocator for small objects called "pymalloc" which keeps free
blocks for efficiency. This is not well handled by these tools.

There are debug tools dedicated to the Python language like ``Heapy``
``Pympler`` and ``Meliae`` which lists all alive objects using the
garbage collector module (functions like ``gc.get_objects()``,
``gc.get_referrers()`` and ``gc.get_referents()``), compute their size
(ex: using ``sys.getsizeof()``) and group objects by type. These tools
provide a better estimation of the memory usage of an application.  They
are useful when most memory leaks are instances of the same type and
this type is only instantiated in a few functions. Problems arise when
the object type is very common like ``str`` or ``tuple``, and it is hard
to identify where these objects are instantiated.

Finding reference cycles is also a difficult problem.  There are
different tools to draw a diagram of all references.  These tools
cannot be used on large applications with thousands of objects because
the diagram is too huge to be analyzed manually.


Proposal
========

Using the customized allocation API from :pep:`445`, it becomes easy to
set up a hook on Python memory allocators. A hook can inspect Python
internals to retrieve Python tracebacks. The idea of getting the current
traceback comes from the faulthandler module. The faulthandler dumps
the traceback of all Python threads on a crash, here is the idea is to
get the traceback of the current Python thread when a memory block is
allocated by Python.

This PEP proposes to add a new ``tracemalloc`` module, a debug tool
to trace memory blocks allocated by Python. The module provides the
following information:

* Traceback where an object was allocated
* Statistics on allocated memory blocks per filename and per line
  number: total size, number and average size of allocated memory blocks
* Computed differences between two snapshots to detect memory leaks

The API of the tracemalloc module is similar to the API of the faulthandler
module: ``enable()`` / ``start()``, ``disable()`` / ``stop()`` and
``is_enabled()`` / ``is_tracing()`` functions, an environment variable
(``PYTHONFAULTHANDLER`` and ``PYTHONTRACEMALLOC``), and a ``-X`` command line
option (``-X faulthandler`` and ``-X tracemalloc``). See the `documentation of
the faulthandler module <http://docs.python.org/3/library/faulthandler.html>`_.

The idea of tracing memory allocations is not new. It was first
implemented in the PySizer project in 2005. PySizer was implemented
differently: the traceback was stored in frame objects and some Python
types were linked the trace with the name of object type. PySizer patch
on CPython adds an overhead on performances and memory footprint, even if
the PySizer was not used. tracemalloc attaches a traceback to the
underlying layer, to memory blocks, and has no overhead when the module
is not tracing memory allocations.

The tracemalloc module has been written for CPython. Other
implementations of Python may not be able to provide it.


API
===

To trace most memory blocks allocated by Python, the module should be
started as early as possible by setting the ``PYTHONTRACEMALLOC``
environment variable to ``1``, or by using ``-X tracemalloc`` command
line option. The ``tracemalloc.start()`` function can be called at
runtime to start tracing Python memory allocations.

By default, a trace of an allocated memory block only stores the most
recent frame (1 frame). To store 25 frames at startup: set the
``PYTHONTRACEMALLOC`` environment variable to ``25``, or use the ``-X
tracemalloc=25`` command line option. The ``set_traceback_limit()``
function can be used at runtime to set the limit.


Functions
---------

``clear_traces()`` function:

    Clear traces of memory blocks allocated by Python.

    See also ``stop()``.


``get_object_traceback(obj)`` function:

    Get the traceback where the Python object *obj* was allocated.
    Return a ``Traceback`` instance, or ``None`` if the ``tracemalloc``
    module is not tracing memory allocations or did not trace the
    allocation of the object.

    See also ``gc.get_referrers()`` and ``sys.getsizeof()`` functions.


``get_traceback_limit()`` function:

    Get the maximum number of frames stored in the traceback of a trace.

    The ``tracemalloc`` module must be tracing memory allocations to get
    the limit, otherwise an exception is raised.

    The limit is set by the ``start()`` function.


``get_traced_memory()`` function:

    Get the current size and maximum size of memory blocks traced by the
    ``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``.


``get_tracemalloc_memory()`` function:

    Get the memory usage in bytes of the ``tracemalloc`` module used to
    store traces of memory blocks. Return an ``int``.


``is_tracing()`` function:

    ``True`` if the ``tracemalloc`` module is tracing Python memory
    allocations, ``False`` otherwise.

    See also ``start()`` and ``stop()`` functions.


``start(nframe: int=1)`` function:

    Start tracing Python memory allocations: install hooks on Python
    memory allocators. Collected tracebacks of traces will be limited to
    *nframe* frames. By default, a trace of a memory block only stores
    the most recent frame: the limit is ``1``. *nframe* must be greater
    or equal to ``1``.

    Storing more than ``1`` frame is only useful to compute statistics
    grouped by ``'traceback'`` or to compute cumulative statistics: see
    the ``Snapshot.compare_to()`` and ``Snapshot.statistics()`` methods.

    Storing more frames increases the memory and CPU overhead of the
    ``tracemalloc`` module. Use the ``get_tracemalloc_memory()``
    function to measure how much memory is used by the ``tracemalloc``
    module.

    The ``PYTHONTRACEMALLOC`` environment variable
    (``PYTHONTRACEMALLOC=NFRAME``) and the ``-X`` ``tracemalloc=NFRAME``
    command line option can be used to start tracing at startup.

    See also ``stop()``, ``is_tracing()`` and ``get_traceback_limit()``
    functions.


``stop()`` function:

    Stop tracing Python memory allocations: uninstall hooks on Python
    memory allocators. Clear also traces of memory blocks allocated by
    Python

    Call ``take_snapshot()`` function to take a snapshot of traces
    before clearing them.

    See also ``start()`` and ``is_tracing()`` functions.


``take_snapshot()`` function:

    Take a snapshot of traces of memory blocks allocated by Python.
    Return a new ``Snapshot`` instance.

    The snapshot does not include memory blocks allocated before the
    ``tracemalloc`` module started to trace memory allocations.

    Tracebacks of traces are limited to ``get_traceback_limit()``
    frames. Use the *nframe* parameter of the ``start()`` function to
    store more frames.

    The ``tracemalloc`` module must be tracing memory allocations to
    take a snapshot, see the ``start()`` function.

    See also the ``get_object_traceback()`` function.


Filter
------

``Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False)`` class:

    Filter on traces of memory blocks.

    See the ``fnmatch.fnmatch()`` function for the syntax of
    *filename_p


id: pep-0440.rst
content:
PEP: 440
Title: Version Identification and Dependency Specification
Author: Alyssa Coghlan <ncoghlan@gmail.com>,
        Donald Stufft <donald@stufft.io>
BDFL-Delegate: Alyssa Coghlan <ncoghlan@gmail.com>
Discussions-To: distutils-sig@python.org
Status: Final
Type: Standards Track
Topic: Packaging
Created: 18-Mar-2013
Post-History: 30-Mar-2013, 27-May-2013, 20-Jun-2013,
              21-Dec-2013, 28-Jan-2014, 08-Aug-2014,
              22-Aug-2014
Replaces: 386
Resolution: https://mail.python.org/pipermail/distutils-sig/2014-August/024673.html

.. canonical-pypa-spec:: :ref:`version-specifiers`


Abstract
========

This PEP describes a scheme for identifying versions of Python software
distributions, and declaring dependencies on particular versions.

This document addresses several limitations of the previous attempt at a
standardized approach to versioning, as described in :pep:`345` and :pep:`386`.


Definitions
===========

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this
document are to be interpreted as described in :rfc:`2119`.

"Projects" are software components that are made available for integration.
Projects include Python libraries, frameworks, scripts, plugins,
applications, collections of data or other resources, and various
combinations thereof. Public Python projects are typically registered on
the `Python Package Index <https://pypi.org/>`__.

"Releases" are uniquely identified snapshots of a project.

"Distributions" are the packaged files which are used to publish
and distribute a release.

"Build tools" are automated tools intended to run on development systems,
producing source and binary distribution archives. Build tools may also be
invoked by integration tools in order to build software distributed as
sdists rather than prebuilt binary archives.

"Index servers" are active distribution registries which publish version and
dependency metadata and place constraints on the permitted metadata.

"Publication tools" are automated tools intended to run on development
systems and upload source and binary distribution archives to index servers.

"Installation tools" are integration tools specifically intended to run on
deployment targets, consuming source and binary distribution archives from
an index server or other designated location and deploying them to the target
system.

"Automated tools" is a collective term covering build tools, index servers,
publication tools, integration tools and any other software that produces
or consumes distribution version and dependency metadata.

Version scheme
==============

Distributions are identified by a public version identifier which
supports all defined version comparison operations

The version scheme is used both to describe the distribution version
provided by a particular distribution archive, as well as to place
constraints on the version of dependencies needed in order to build or
run the software.


Public version identifiers
--------------------------

The canonical public version identifiers MUST comply with the following
scheme::

    [N!]N(.N)*[{a|b|rc}N][.postN][.devN]

Public version identifiers MUST NOT include leading or trailing whitespace.

Public version identifiers MUST be unique within a given distribution.

Installation tools SHOULD ignore any public versions which do not comply with
this scheme but MUST also include the normalizations specified below.
Installation tools MAY warn the user when non-compliant or ambiguous versions
are detected.

See also ``Appendix B : Parsing version strings with regular expressions``
which provides a regular expression to check strict conformance with the
canonical format, as well as a more permissive regular expression accepting
inputs that may require subsequent normalization.

Public version identifiers are separated into up to five segments:

* Epoch segment: ``N!``
* Release segment: ``N(.N)*``
* Pre-release segment: ``{a|b|rc}N``
* Post-release segment: ``.postN``
* Development release segment: ``.devN``

Any given release will be a "final release", "pre-release", "post-release" or
"developmental release" as defined in the following sections.

All numeric components MUST be non-negative integers represented as sequences
of ASCII digits.

All numeric components MUST be interpreted and ordered according to their
numeric value, not as text strings.

All numeric components MAY be zero. Except as described below for the
release segment, a numeric component of zero has no special significance
aside from always being the lowest possible value in the version ordering.

.. note::

   Some hard to read version identifiers are permitted by this scheme in
   order to better accommodate the wide range of versioning practices
   across existing public and private Python projects.

   Accordingly, some of the versioning practices which are technically
   permitted by the PEP are strongly discouraged for new projects. Where
   this is the case, the relevant details are noted in the following
   sections.


Local version identifiers
-------------------------

Local version identifiers MUST comply with the following scheme::

    <public version identifier>[+<local version label>]

They consist of a normal public version identifier (as defined in the
previous section), along with an arbitrary "local version label", separated
from the public version identifier by a plus. Local version labels have
no specific semantics assigned, but some syntactic restrictions are imposed.

Local version identifiers are used to denote fully API (and, if applicable,
ABI) compatible patched versions of upstream projects. For example, these
may be created by application developers and system integrators by applying
specific backported bug fixes when upgrading to a new upstream release would
be too disruptive to the application or other integrated system (such as a
Linux distribution).

The inclusion of the local version label makes it possible to differentiate
upstream releases from potentially altered rebuilds by downstream
integrators. The use of a local version identifier does not affect the kind
of a release but, when applied to a source distribution, does indicate that
it may not contain the exact same code as the corresponding upstream release.

To ensure local version identifiers can be readily incorporated as part of
filenames and URLs, and to avoid formatting inconsistencies in hexadecimal
hash representations, local version labels MUST be limited to the following
set of permitted characters:

* ASCII letters (``[a-zA-Z]``)
* ASCII digits (``[0-9]``)
* periods (``.``)

Local version labels MUST start and end with an ASCII letter or digit.

Comparison and ordering of local versions considers each segment of the local
version (divided by a ``.``) separately. If a segment consists entirely of
ASCII digits then that section should be considered an integer for comparison
purposes and if a segment contains any ASCII letters then that segment is
compared lexicographically with case insensitivity. When comparing a numeric
and lexicographic segment, the numeric section always compares as greater than
the lexicographic segment. Additionally a local version with a great number of
segments will always compare as greater than a local version with fewer
segments, as long as the shorter local version's segments match the beginning
of the longer local version's segments exactly.

An "upstream project" is a project that defines its own public versions. A
"downstream project" is one which tracks and redistributes an upstream project,
potentially backporting security and bug fixes from later versions of the
upstream project.

Local version identifiers SHOULD NOT be used when publishing upstream
projects to a public index server, but MAY be used to identify private
builds created directly from the project source. Local
version identifiers SHOULD be used by downstream projects when releasing 


id: pep-0326.rst
content:
PEP: 326
Title: A Case for Top and Bottom Values
Author: Josiah Carlson <jcarlson@uci.edu>,
        Terry Reedy <tjreedy@udel.edu>
Status: Rejected
Type: Standards Track
Created: 20-Dec-2003
Python-Version: 2.4
Post-History: 20-Dec-2003, 03-Jan-2004, 05-Jan-2004, 07-Jan-2004,
              21-Feb-2004

Results
=======

This PEP has been rejected by the BDFL [8]_.  As per the
pseudo-sunset clause [9]_, :pep:`326` is being updated one last time
with the latest suggestions, code modifications, etc., and includes a
link to a module [10]_ that implements the behavior described in the
PEP.  Users who desire the behavior listed in this PEP are encouraged
to use the module for the reasons listed in
`Independent Implementations?`_.


Abstract
========

This PEP proposes two singleton constants that represent a top and
bottom [3]_ value: ``Max`` and ``Min`` (or two similarly suggestive
names [4]_; see `Open Issues`_).

As suggested by their names, ``Max`` and ``Min`` would compare higher
or lower than any other object (respectively).  Such behavior results
in easier to understand code and fewer special cases in which a
temporary minimum or maximum value is required, and an actual minimum
or maximum numeric value is not limited.


Rationale
=========

While ``None`` can be used as an absolute minimum that any value can
attain [1]_, this may be deprecated [4]_ in Python 3.0 and shouldn't
be relied upon.

As a replacement for ``None`` being used as an absolute minimum, as
well as the introduction of an absolute maximum, the introduction of
two singleton constants ``Max`` and ``Min`` address concerns for the
constants to be self-documenting.

What is commonly done to deal with absolute minimum or maximum values,
is to set a value that is larger than the script author ever expects
the input to reach, and hope that it isn't reached.

Guido has brought up [2]_ the fact that there exists two constants
that can be used in the interim for maximum values: sys.maxint and
floating point positive infinity (1e309 will evaluate to positive
infinity).  However, each has their drawbacks.

- On most architectures sys.maxint is arbitrarily small (2**31-1 or
  2**63-1) and can be easily eclipsed by large 'long' integers or
  floating point numbers.

- Comparing long integers larger than the largest floating point
  number representable against any float will result in an exception
  being raised::

        >>> cmp(1.0, 10**309)
        Traceback (most recent call last):
        File "<stdin>", line 1, in ?
        OverflowError: long int too large to convert to float

  Even when large integers are compared against positive infinity::

        >>> cmp(1e309, 10**309)
        Traceback (most recent call last):
        File "<stdin>", line 1, in ?
        OverflowError: long int too large to convert to float

- These same drawbacks exist when numbers are negative.

Introducing ``Max`` and ``Min`` that work as described above does not
take much effort.  A sample Python `reference implementation`_ of both
is included.


Motivation
==========

There are hundreds of algorithms that begin by initializing some set
of values to a logical (or numeric) infinity or negative infinity.
Python lacks either infinity that works consistently or really is the
most extreme value that can be attained.  By adding ``Max`` and
``Min``, Python would have a real maximum and minimum value, and such
algorithms can become clearer due to the reduction of special cases.

``Max`` Examples
---------------------

When testing various kinds of servers, it is sometimes necessary to
only serve a certain number of clients before exiting, which results
in code like the following::

    count = 5

    def counts(stop):
        i = 0
        while i < stop:
            yield i
            i += 1

    for client_number in counts(count):
        handle_one_client()

When using ``Max`` as the value assigned to count, our testing server
becomes a production server with minimal effort.

As another example, in Dijkstra's shortest path algorithm on a graph
with weighted edges (all positive).

1. Set distances to every node in the graph to infinity.
2. Set the distance to the start node to zero.
3. Set visited to be an empty mapping.
4. While shortest distance of a node that has not been visited is less
   than infinity and the destination has not been visited.

   a. Get the node with the shortest distance.
   b. Visit the node.
   c. Update neighbor distances and parent pointers if necessary for
      neighbors that have not been visited.

5. If the destination has been visited, step back through parent
   pointers to find the reverse of the path to be taken.

.. _DijkstraSP_table:

Below is an example of Dijkstra's shortest path algorithm on a graph
with weighted edges using a table (a faster version that uses a heap
is available, but this version is offered due to its similarity to the
description above, the heap version is available via older versions of
this document). ::

    def DijkstraSP_table(graph, S, T):
        table = {}                                                 #3
        for node in graph.iterkeys():
            #(visited, distance, node, parent)
            table[node] = (0, Max, node, None)                     #1
        table[S] = (0, 0, S, None)                                 #2
        cur = min(table.values())                                  #4a
        while (not cur[0]) and cur[1] < Max:                       #4
            (visited, distance, node, parent) = cur
            table[node] = (1, distance, node, parent)              #4b
            for cdist, child in graph[node]:                       #4c
                ndist = distance+cdist                             #|
                if not table[child][0] and ndist < table[child][1]:#|
                    table[child] = (0, ndist, child, node)         #|_
            cur = min(table.values())                              #4a
        if not table[T][0]:
            return None
        cur = T                                                    #5
        path = [T]                                                 #|
        while table[cur][3] is not None:                           #|
            path.append(table[cur][3])                             #|
            cur = path[-1]                                         #|
        path.reverse()                                             #|
        return path                                                #|_

Readers should note that replacing ``Max`` in the above code with an
arbitrarily large number does not guarantee that the shortest path
distance to a node will never exceed that number.  Well, with one
caveat: one could certainly sum up the weights of every edge in the
graph, and set the 'arbitrarily large number' to that total.  However,
doing so does not make the algorithm any easier to understand and has
potential problems with numeric overflows.

.. _DijkstraSP_table_node:

Gustavo Niemeyer [7]_ points out that using a more Pythonic data
structure than tuples, to store information about node distances,
increases readability.  Two equivalent node structures (one using
``None``, the other using ``Max``) and their use in a suitably
modified Dijkstra's shortest path algorithm is given below. ::

    class SuperNode:
        def __init__(self, node, parent, distance, visited):
            self.node = node
            self.parent = parent
            self.distance = distance
            self.visited = visited

    class MaxNode(SuperNode):
        def __init__(self, node, parent=None, distance=Max,
                     visited=False):
            SuperNode.__init__(self, node, parent, distance, visited)
        def __cmp__(self, other):
            return cmp((self.visited, self.distance),
                       (other.visited, other.distance))

    class NoneNode(SuperNode):
        def __init__(self, node, parent=None, distance=None,
                     visited=False):
            Supe


id: pep-0695.rst
content:
PEP: 695
Title: Type Parameter Syntax
Author: Eric Traut <erictr at microsoft.com>
Sponsor: Guido van Rossum <guido@python.org>
Discussions-To: https://mail.python.org/archives/list/typing-sig@python.org/thread/BB2BGYJY2YG5IWESKGTAPUQL3N27ZKVW/
Status: Final
Type: Standards Track
Topic: Typing
Created: 15-Jun-2022
Python-Version: 3.12
Post-History: `20-Jun-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/BB2BGYJY2YG5IWESKGTAPUQL3N27ZKVW/>`__,
              `04-Dec-2022 <https://discuss.python.org/t/pep-695-type-parameter-syntax/21646>`__
Resolution: https://discuss.python.org/t/pep-695-type-parameter-syntax/21646/92

.. canonical-typing-spec:: :ref:`typing:variance-inference`,
                           :ref:`typing:type-aliases`,
                           :ref:`python:type-params`,
                           :ref:`python:type` and
                           :ref:`python:annotation-scopes`.

Abstract
========

This PEP specifies an improved syntax for specifying type parameters within
a generic class, function, or type alias. It also introduces a new statement
for declaring type aliases.


Motivation
==========

:pep:`484` introduced type variables into the language. :pep:`612` built
upon this concept by introducing parameter specifications, and
:pep:`646` added variadic type variables.

While generic types and type parameters have grown in popularity, the
syntax for specifying type parameters still feels "bolted on" to Python.
This is a source of confusion among Python developers.

There is consensus within the Python static typing community that it is time
to provide a formal syntax that is similar to other modern programming
languages that support generic types.

An analysis of 25 popular typed Python libraries revealed that type
variables (in particular, the ``typing.TypeVar`` symbol) were used in
14% of modules.


Points of Confusion
-------------------

While the use of type variables has become widespread, the manner in which
they are specified within code is the source of confusion among many
Python developers. There are a couple of factors that contribute to this
confusion.

The scoping rules for type variables are difficult to understand. Type
variables are typically allocated within the global scope, but their semantic
meaning is valid only when used within the context of a generic class,
function, or type alias. A single runtime instance of a type variable may be
reused in multiple generic contexts, and it has a different semantic meaning
in each of these contexts. This PEP proposes to eliminate this source of
confusion by declaring type parameters at a natural place within a class,
function, or type alias declaration statement.

Generic type aliases are often misused because it is not clear to developers
that a type argument must be supplied when the type alias is used. This leads
to an implied type argument of ``Any``, which is rarely the intent. This PEP
proposes to add new syntax that makes generic type alias declarations
clear.

:pep:`483` and :pep:`484` introduced the concept of "variance" for a type
variable used within a generic class. Type variables can be invariant,
covariant, or contravariant. The concept of variance is an advanced detail
of type theory that is not well understood by most Python developers, yet
they must confront this concept today when defining their first generic
class. This PEP largely eliminates the need for most developers
to understand the concept of variance when defining generic classes.

When more than one type parameter is used with a generic class or type alias,
the rules for type parameter ordering can be confusing. It is normally based on
the order in which they first appear within a class or type alias declaration
statement. However, this can be overridden in a class definition by
including a "Generic" or "Protocol" base class. For example, in the class
declaration ``class ClassA(Mapping[K, V])``, the type parameters are
ordered as ``K`` and then ``V``. However, in the class declaration
``class ClassB(Mapping[K, V], Generic[V, K])``, the type parameters are
ordered as ``V`` and then ``K``. This PEP proposes to make type parameter
ordering explicit in all cases.

The practice of sharing a type variable across multiple generic contexts
creates other problems today. Modern editors provide features like "find
all references" and "rename all references" that operate on symbols at the
semantic level. When a type parameter is shared among multiple generic
classes, functions, and type aliases, all references are semantically
equivalent.

Type variables defined within the global scope also need to be given a name
that starts with an underscore to indicate that the variable is private to
the module. Globally-defined type variables are also often given names to
indicate their variance, leading to cumbersome names like "_T_contra" and
"_KT_co". The current mechanisms for allocating type variables also requires
the developer to supply a redundant name in quotes (e.g. ``T = TypeVar("T")``).
This PEP eliminates the need for the redundant name and cumbersome
variable names.

Defining type parameters today requires importing the ``TypeVar`` and
``Generic`` symbols from the ``typing`` module. Over the past several releases
of Python, efforts have been made to eliminate the need to import ``typing``
symbols for common use cases, and the PEP furthers this goal.


Summary Examples
================

Defining a generic class prior to this PEP looks something like this.

::

    from typing import Generic, TypeVar

    _T_co = TypeVar("_T_co", covariant=True, bound=str)

    class ClassA(Generic[_T_co]):
        def method1(self) -> _T_co:
            ...


With the new syntax, it looks like this.

::

    class ClassA[T: str]:
        def method1(self) -> T:
            ...


Here is an example of a generic function today.

::

    from typing import TypeVar

    _T = TypeVar("_T")

    def func(a: _T, b: _T) -> _T:
        ...

And the new syntax.

::

    def func[T](a: T, b: T) -> T:
        ...


Here is an example of a generic type alias today.

::

    from typing import TypeAlias

    _T = TypeVar("_T")

    ListOrSet: TypeAlias = list[_T] | set[_T]


And with the new syntax.

::

    type ListOrSet[T] = list[T] | set[T]


Specification
=============

Type Parameter Declarations
---------------------------

Here is a new syntax for declaring type parameters for generic
classes, functions, and type aliases. The syntax adds support for
a comma-delimited list of type parameters in square brackets after
the name of the class, function, or type alias.

Simple (non-variadic) type variables are declared with an unadorned name.
Variadic type variables are preceded by ``*`` (see :pep:`646` for details).
Parameter specifications are preceded by ``**`` (see :pep:`612` for details).

::

    # This generic class is parameterized by a TypeVar T, a
    # TypeVarTuple Ts, and a ParamSpec P.
    class ChildClass[T, *Ts, **P]: ...

There is no need to include ``Generic`` as a base class. Its inclusion as
a base class is implied by the presence of type parameters, and it will
automatically be included in the ``__mro__`` and ``__orig_bases__`` attributes
for the class. The explicit use of a ``Generic`` base class will result in a
runtime error.

::

    class ClassA[T](Generic[T]): ...  # Runtime error


A ``Protocol`` base class with type arguments may generate a runtime
error. Type checkers should generate an error in this case because
the use of type arguments is not needed, and the order of type parameters
for the class are no longer dictated by their order in the ``Protocol``
base class.

::

    class ClassA[S, T](Protocol): ... # OK

    class ClassB[S, T](Protocol[S, T]): ... # Recommended type checker error


Type parameter names within a generic class, function, or type alias must be
unique within that same class, function, or type alias. A duplicate name
generates a syntax error at compile time


id: pep-0681.rst
content:
PEP: 681
Title: Data Class Transforms
Author: Erik De Bonte <erikd at microsoft.com>,
        Eric Traut <erictr at microsoft.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra at gmail.com>
Discussions-To: https://mail.python.org/archives/list/typing-sig@python.org/thread/EAALIHA3XEDFDNG2NRXTI3ERFPAD65Z4/
Status: Final
Type: Standards Track
Topic: Typing
Created: 02-Dec-2021
Python-Version: 3.11
Post-History: `24-Apr-2021 <https://mail.python.org/archives/list/typing-sig@python.org/thread/TXL5LEHYX5ZJAZPZ7YHZU7MVFXMVUVWL/>`__,
              `13-Dec-2021 <https://mail.python.org/archives/list/typing-sig@python.org/thread/EAALIHA3XEDFDNG2NRXTI3ERFPAD65Z4/>`__,
              `22-Feb-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/BW6CB6URC4BCN54QSG2STINU2M7V4TQQ/>`__
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/R4A2IYLGFHKFDYJPSDA5NFJ6N7KRPJ6D/

.. canonical-typing-spec:: :ref:`typing:dataclass-transform` and
                           :py:func:`@typing.dataclass_transform <typing.dataclass_transform>`

Abstract
========

:pep:`557` introduced the dataclass to the Python stdlib. Several popular
libraries have behaviors that are similar to dataclasses, but these
behaviors cannot be described using standard type annotations. Such
projects include attrs, pydantic, and object relational mapper (ORM)
packages such as SQLAlchemy and Django.

Most type checkers, linters and language servers have full support for
dataclasses. This proposal aims to generalize this functionality and
provide a way for third-party libraries to indicate that certain
decorator functions, classes, and metaclasses provide behaviors
similar to dataclasses.

These behaviors include:

* Synthesizing an ``__init__`` method based on declared
  data fields.
* Optionally synthesizing ``__eq__``, ``__ne__``, ``__lt__``,
  ``__le__``, ``__gt__`` and ``__ge__`` methods.
* Supporting "frozen" classes, a way to enforce immutability during
  static type checking.
* Supporting "field specifiers", which describe attributes of
  individual fields that a static type checker must be aware of,
  such as whether a default value is provided for the field.

The full behavior of the stdlib dataclass is described in the `Python
documentation <#dataclass-docs_>`_.

This proposal does not affect CPython directly except for the addition
of a ``dataclass_transform`` decorator in ``typing.py``.


Motivation
==========

There is no existing, standard way for libraries with dataclass-like
semantics to declare their behavior to type checkers. To work around
this limitation, Mypy custom plugins have been developed for many of
these libraries, but these plugins don't work with other type
checkers, linters or language servers. They are also costly to
maintain for library authors, and they require that Python developers
know about the existence of these plugins and download and configure
them within their environment.


Rationale
=========

The intent of this proposal is not to support every feature of every
library with dataclass-like semantics, but rather to make it possible
to use the most common features of these libraries in a way that is
compatible with static type checking. If a user values these libraries
and also values static type checking, they may need to avoid using
certain features or make small adjustments to the way they use them.
That's already true for the Mypy custom plugins, which
don't support every feature of every dataclass-like library.

As new features are added to dataclasses in the future, we intend, when
appropriate, to add support for those features on
``dataclass_transform`` as well. Keeping these two feature sets in
sync will make it easier for dataclass users to understand and use
``dataclass_transform`` and will simplify the maintenance of dataclass
support in type checkers.

Additionally, we will consider adding ``dataclass_transform`` support
in the future for features that have been adopted by multiple
third-party libraries but are not supported by dataclasses.


Specification
=============

The ``dataclass_transform`` decorator
-------------------------------------

This specification introduces a new decorator function in
the ``typing`` module named ``dataclass_transform``. This decorator
can be applied to either a function that is itself a decorator,
a class, or a metaclass. The presence of
``dataclass_transform`` tells a static type checker that the decorated
function, class, or metaclass performs runtime "magic" that transforms
a class, endowing it with dataclass-like behaviors.

If ``dataclass_transform`` is applied to a function, using the decorated
function as a decorator is assumed to apply dataclass-like semantics.
If the function has overloads, the ``dataclass_transform`` decorator can
be applied to the implementation of the function or any one, but not more
than one, of the overloads. When applied to an overload, the
``dataclass_transform`` decorator still impacts all usage of the
function.

If ``dataclass_transform`` is applied to a class, dataclass-like
semantics will be assumed for any class that directly or indirectly
derives from the decorated class or uses the decorated class as a
metaclass. Attributes on the decorated class and its base classes
are not considered to be fields.

Examples of each approach are shown in the following sections. Each
example creates a ``CustomerModel`` class with dataclass-like semantics.
The implementation of the decorated objects is omitted for brevity,
but we assume that they modify classes in the following ways:

* They synthesize an ``__init__`` method using data fields declared
  within the class and its parent classes.
* They synthesize ``__eq__`` and ``__ne__`` methods.

Type checkers supporting this PEP will recognize that the
``CustomerModel`` class can be instantiated using the synthesized
``__init__`` method:

.. code-block:: python

  # Using positional arguments
  c1 = CustomerModel(327, "John Smith")

  # Using keyword arguments
  c2 = CustomerModel(id=327, name="John Smith")

  # These calls will generate runtime errors and should be flagged as
  # errors by a static type checker.
  c3 = CustomerModel()
  c4 = CustomerModel(327, first_name="John")
  c5 = CustomerModel(327, "John Smith", 0)

Decorator function example
''''''''''''''''''''''''''

.. code-block:: python

  _T = TypeVar("_T")

  # The ``create_model`` decorator is defined by a library.
  # This could be in a type stub or inline.
  @typing.dataclass_transform()
  def create_model(cls: Type[_T]) -> Type[_T]:
      cls.__init__ = ...
      cls.__eq__ = ...
      cls.__ne__ = ...
      return cls

  # The ``create_model`` decorator can now be used to create new model
  # classes, like this:
  @create_model
  class CustomerModel:
      id: int
      name: str

Class example
'''''''''''''

.. code-block:: python

  # The ``ModelBase`` class is defined by a library. This could be in
  # a type stub or inline.
  @typing.dataclass_transform()
  class ModelBase: ...

  # The ``ModelBase`` class can now be used to create new model
  # subclasses, like this:
  class CustomerModel(ModelBase):
      id: int
      name: str

Metaclass example
'''''''''''''''''

.. code-block:: python

  # The ``ModelMeta`` metaclass and ``ModelBase`` class are defined by
  # a library. This could be in a type stub or inline.
  @typing.dataclass_transform()
  class ModelMeta(type): ...

  class ModelBase(metaclass=ModelMeta): ...

  # The ``ModelBase`` class can now be used to create new model
  # subclasses, like this:
  class CustomerModel(ModelBase):
      id: int
      name: str

Decorator function and class/metaclass parameters
-------------------------------------------------

A decorator function, class, or metaclass that provides dataclass-like
functionality may accept parameters that modify certain behaviors.
This specification defines the following parameters that static type
checkers must honor if they are used by a dataclass 


id: pep-0656.rst
content:
PEP: 656
Title: Platform Tag for Linux Distributions Using Musl
Author: Tzu-ping Chung <uranusjr@gmail.com>
Sponsor: Brett Cannon <brett@python.org>
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/7165
Status: Final
Type: Standards Track
Topic: Packaging
Created: 17-Mar-2021
Post-History: 17-Mar-2021, 18-Apr-2021
Resolution: https://discuss.python.org/t/7165/32


Abstract
========

This PEP proposes a new platform tag series ``musllinux`` for
binary Python package distributions for a Python installation that
depends on musl on a Linux distribution. The tag works similarly to
the "perennial manylinux" platform tags specified in :pep:`600`, but
targeting platforms based on musl instead.


Motivation
==========

With the wide use of containers, distributions such as Alpine Linux
[alpine]_, have been gaining more popularity than ever. Many of them
based on musl [musl]_, a different libc implementation from glibc, and
therefore cannot use the existing ``manylinux`` platform tags. This
means that Python package projects cannot deploy binary distributions
on PyPI for them. Users of such projects demand build constraints from
those projects, putting unnecessary burden on project maintainers.


Rationale
=========

According to the documentation, musl has a stable ABI, and maintains
backwards compatibility [musl-compatibility]_ [compare-libcs]_, so a
binary compiled against an earlier version of musl is guaranteed to
run against a newer musl runtime [musl-compat-ml]_. Therefore, we use
a scheme similar to the glibc-version-based manylinux tags, but
against musl versions instead of glibc.

Logic behind the new platform tag largely follows :pep:`600`
("perennial manylinux"), and requires wheels using this tag make
similar promises. Please refer to :pep:`600` for more details on
rationale and reasoning behind the design.

The ``musllinux`` platform tags only apply to Python interpreters
dynamically linked against the musl libc and executed on the runtime
shared library, on a Linux operating system. Statically linked
interpreters or mixed builds with other libc implementations (such as
glibc) are out of scope and not supported by platform tags defined in
this document. Such interpreters should not claim compatibility with
``musllinux`` platform tags.


Specification
=============

Tags using the new scheme will take the form::

    musllinux_${MUSLMAJOR}_${MUSLMINOR}_${ARCH}

This tag promises the wheel works on any mainstream Linux distribution
that uses musl version ``${MUSLMAJOR}.${MUSLMINOR}``, following the
perennial design. All other system-level dependency requirements rely
on the community's definition to the intentionally vague "mainstream"
description introduced in :pep:`600`. A wheel may make use of newer
system dependencies when all mainstream distributions using the
specified musl version provide the dependency by default; once all
mainstream distributions on the musl version ship a certain dependency
version by default, users relying on older versions are automatically
removed from the coverage of that ``musllinux`` tag.


Reading the musl version
------------------------

The musl version values can be obtained by executing the musl libc
shared library the Python interpreter is currently running on, and
parsing the output::

    import re
    import subprocess

    def get_musl_major_minor(so: str) -> tuple[int, int] | None:
        """Detect musl runtime version.

        Returns a two-tuple ``(major, minor)`` that indicates musl
        library's version, or ``None`` if the given libc .so does not
        output expected information.

        The libc library should output something like this to stderr::

            musl libc (x86_64)
            Version 1.2.2
            Dynamic Program Loader
        """
        proc = subprocess.run([so], stderr=subprocess.PIPE, text=True)
        lines = (line.strip() for line in proc.stderr.splitlines())
        lines = [line for line in lines if line]
        if len(lines) < 2 or lines[0][:4] != "musl":
            return None
        match = re.match(r"Version (\d+)\.(\d+)", lines[1])
        if match:
            return (int(match.group(1)), int(match.group(2)))
        return None

There are currently two possible ways to find the musl library's
location that a Python interpreter is running on, either with the
system ``ldd`` command [ldd]_, or by parsing the ``PT_INTERP``
section's value from the executable's ELF header [elf]_.


Formatting the tag
------------------

Distributions using the tag make similar promises to those described
in :pep:`600`, including:

1. The distribution works on any mainstream Linux distributions with
   musl version ``${MUSLMAJOR}.${MUSLMINOR}`` or later.
2. The distribution's ``${ARCH}`` matches the return value of
   ``sysconfig.get_platform()`` on the host system, replacing period
   (``.``) and hyphen (``-``) characters with underscores (``_``), as
   outlined in :pep:`425` and :pep:`427`.

Example values::

    musllinux_1_1_x86_64   # musl 1.1 running on x86-64.
    musllinux_1_2_aarch64  # musl 1.2 running on ARM 64-bit.

The value can be formatted with the following Python code::

    import sysconfig

    def format_musllinux(musl_version: tuple[int, int]) -> str:
        os_name, sep, arch = sysconfig.get_platform().partition("-")
        assert os_name == "linux" and sep, "Not a Linux"
        arch = arch.replace(".", "_").replace("-", "_")
        return f"musllinux_{musl_version[0]}_{musl_version[1]}_{arch}"

Recommendations to package indexes
----------------------------------

It is recommended for Python package repositories, including PyPI, to
accept platform tags matching the following regular expression::

    musllinux_([0-9]+)_([0-9]+)_([^.-]+)

Python package repositories may impose additional requirements to
reject Wheels with known issues, including but not limited to:

* A ``musllinux_1_1`` wheel containing symbols only available in musl
  1.2 or later.
* Wheel that depends on external libraries not considered generally
  available to the intended audience of the package index.
* A platform tag claiming compatibility to a non-existent musl version
  (like ``musllinux_9000_0``).

Such policies are ultimately up to individual package repositories.
It is not the author's intention to impose restrictions to the
maintainers.


Backwards Compatibility
=======================

There are no backwards compatibility concerns in this PEP.


Rejected Ideas
==============

Create a platform tag based specifically for Alpine Linux
---------------------------------------------------------

Past experience on the ``manylinux`` tag series shows this approach
would be too costly time-wise. The author feels the "works well with
others" rule both is more inclusive and works well enough in practice.


References
==========

.. [alpine] https://alpinelinux.org/

.. [musl] https://musl.libc.org

.. [musl-compatibility] https://wiki.musl-libc.org/compatibility.html

.. [compare-libcs] https://www.etalabs.net/compare_libcs.html

.. [musl-compat-ml] https://mail.python.org/archives/list/distutils-sig@python.org/message/VRXSTNXWHPAVUW253ZCWWMP7WDTBAQDL/

.. [ldd] https://www.unix.com/man-page/posix/1/ldd/

.. [elf] https://refspecs.linuxfoundation.org/elf/elf.pdf


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0642.rst
content:
PEP: 642
Title: Explicit Pattern Syntax for Structural Pattern Matching
Author: Alyssa Coghlan <ncoghlan@gmail.com>
BDFL-Delegate:
Discussions-To: python-dev@python.org
Status: Rejected
Type: Standards Track
Requires: 634
Created: 26-Sep-2020
Python-Version: 3.10
Post-History: 31-Oct-2020, 08-Nov-2020, 03-Jan-2021
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/SQC2FTLFV5A7DV7RCEAR2I2IKJKGK7W3/

Abstract
========

This PEP covers an alternative syntax proposal for :pep:`634`'s structural pattern
matching that requires explicit prefixes on all capture patterns and value
constraints. It also proposes a new dedicated syntax for instance attribute
patterns that aligns more closely with the proposed mapping pattern syntax.

While the result is necessarily more verbose than the proposed syntax in
:pep:`634`, it is still significantly less verbose than the status quo.

As an example, the following match statement would extract "host" and "port"
details from a 2 item sequence, a mapping with "host" and "port" keys, any
object with "host" and "port" attributes, or a "host:port" string, treating
the "port" as optional in the latter three cases::

    port = DEFAULT_PORT
    match expr:
        case [as host, as port]:
            pass
        case {"host" as host, "port" as port}:
            pass
        case {"host" as host}:
            pass
        case object{.host as host, .port as port}:
            pass
        case object{.host as host}:
            pass
        case str{} as addr:
            host, __, optional_port = addr.partition(":")
            if optional_port:
                port = optional_port
        case __ as m:
            raise TypeError(f"Unknown address format: {m!r:.200}")
    port = int(port)


At a high level, this PEP proposes to categorise the different available pattern
types as follows:

* wildcard pattern: ``__``
* group patterns: ``(PTRN)``
* value constraint patterns:
    * equality constraints: ``== EXPR``
    * identity constraints: ``is EXPR``
* structural constraint patterns:
    * sequence constraint patterns: ``[PTRN, as NAME, PTRN as NAME]``
    * mapping constraint patterns: ``{EXPR: PTRN, EXPR as NAME}``
    * instance attribute constraint patterns:
      ``CLS{.NAME, .NAME: PTRN, .NAME == EXPR, .NAME as NAME}``
    * class defined constraint patterns:
      ``CLS(PTRN, PTRN, **{.NAME, .NAME: PTRN, .NAME == EXPR, .NAME as NAME})``
* OR patterns: ``PTRN | PTRN | PTRN``
* AS patterns: ``PTRN as NAME`` (omitting the pattern implies ``__``)

The intent of this approach is to:

* allow an initial form of pattern matching to be developed and released without
  needing to decide up front on the best default options for handling bare names,
  attribute lookups, and literal values
* ensure that pattern matching is defined explicitly at the Abstract Syntax Tree
  level, allowing the specifications of the semantics and the surface syntax for
  pattern matching to be clearly separated
* define a clear and concise "ducktyping" syntax that could potentially be
  adopted in ordinary expressions as a way to more easily retrieve a tuple
  containing multiple attributes from the same object

Relative to :pep:`634`, the proposal also deliberately eliminates any syntax that
"binds to the right" without using the ``as`` keyword (using capture patterns
in :pep:`634`'s mapping patterns and class patterns) or binds to both the left and
the right in the same pattern (using :pep:`634`'s capture patterns with AS patterns)


Relationship with other PEPs
============================

This PEP both depends on and competes with :pep:`634` - the PEP author agrees that
match statements would be a sufficiently valuable addition to the language to
be worth the additional complexity that they add to the learning process, but
disagrees with the idea that "simple name vs literal or attribute lookup"
really offers an adequate syntactic distinction between name binding and value
lookup operations in match patterns (at least for Python).

This PEP agrees with the spirit of :pep:`640` (that the chosen wildcard pattern to
skip a name binding should be supported everywhere, not just in match patterns),
but is now proposing a different spelling for the wildcard syntax (``__`` rather
than ``?``). As such, it competes with :pep:`640` as written, but would complement
a proposal to deprecate the use of ``__`` as an ordinary identifier and instead
turn it into a general purpose wildcard marker that always skips making a new
local variable binding.

While it has not yet been put forward as a PEP, Mark Shannon has a pre-PEP draft
[8]_ expressing several concerns about the runtime semantics of the pattern
matching proposal in :pep:`634`. This PEP is somewhat complementary to that one, as
even though this PEP is mostly about surface syntax changes rather than major
semantic changes, it does propose that the Abstract Syntax Tree definition be
made more explicit to better separate the details of the surface syntax from the
semantics of the code generation step. There is one specific idea in that pre-PEP
draft that this PEP explicitly rejects: the idea that the different kinds of
matching are mutually exclusive. It's entirely possible for the same value to
match different kinds of structural pattern, and which one takes precedence will
intentionally be governed by the order of the cases in the match statement.


Motivation
==========

The original :pep:`622` (which was later split into :pep:`634`, :pep:`635`, and :pep:`636`)
incorporated an unstated but essential assumption in its syntax design: that
neither ordinary expressions *nor* the existing assignment target syntax provide
an adequate foundation for the syntax used in match patterns.

While the PEP didn't explicitly state this assumption, one of the PEP authors
explained it clearly on python-dev [1]_:

    The actual problem that I see is that we have different cultures/intuitions
    fundamentally clashing here.  In particular, so many programmers welcome
    pattern matching as an "extended switch statement" and find it therefore
    strange that names are binding and not expressions for comparison.  Others
    argue that it is at odds with current assignment statements, say, and
    question why dotted names are _/not/_ binding.  What all groups seem to
    have in common, though, is that they refer to _/their/_ understanding and
    interpretation of the new match statement as 'consistent' or 'intuitive'
    --- naturally pointing out where we as PEP authors went wrong with our
    design.

    But here is the catch: at least in the Python world, pattern matching as
    proposed by this PEP is an unprecedented and new way of approaching a common
    problem.  It is not simply an extension of something already there.  Even
    worse: while designing the PEP we found that no matter from which angle you
    approach it, you will run into issues of seeming 'inconsistencies' (which is
    to say that pattern matching cannot be reduced to a 'linear' extension of
    existing features in a meaningful way): there is always something that goes
    fundamentally beyond what is already there in Python.  That's why I argue
    that arguments based on what is 'intuitive' or 'consistent' just do not
    make sense _/in this case/_.

The first iteration of this PEP was then born out of an attempt to show that the
second assertion was not accurate, and that match patterns could be treated
as a variation on assignment targets without leading to inherent contradictions.
(An earlier PR submitted to list this option in the "Rejected Ideas" section
of the original :pep:`622` had previously been declined [2]_).

However, the review process for this PEP strongly suggested that not only did
the contradictions that Tobias mentioned in his email exist, but they were also
concerning enough to cast doubts on the syntax proposal presented in :pep:`634`.
Accordingly, this PEP was changed to go even further


id: pep-0722.rst
content:
PEP: 722
Title: Dependency specification for single-file scripts
Author: Paul Moore <p.f.moore@gmail.com>
PEP-Delegate: Brett Cannon <brett@python.org>
Discussions-To: https://discuss.python.org/t/29905
Status: Rejected
Type: Standards Track
Topic: Packaging
Created: 19-Jul-2023
Post-History: `19-Jul-2023 <https://discuss.python.org/t/29905>`__
Resolution: https://discuss.python.org/t/pep-722-723-decision/36763/


Abstract
========

This PEP specifies a format for including 3rd-party dependencies in a
single-file Python script.


Motivation
==========

Not all Python code is structured as a "project", in the sense of having its own
directory complete with a ``pyproject.toml`` file, and being built into an
installable distribution package. Python is also routinely used as a scripting
language, with Python scripts as a (better) alternative to shell scripts, batch
files, etc. When used to create scripts, Python code is typically stored as a
single file, often in a directory dedicated to such "utility scripts", which
might be in a mix of languages with Python being only one possibility among
many. Such scripts may be shared, often by something as simple as email, or a
link to a URL such as a Github gist. But they are typically *not* "distributed"
or "installed" as part of a normal workflow.

One problem when using Python as a scripting language in this way is how to run
the script in an environment that contains whatever third party dependencies are
required by the script. There is currently no standard tool that addresses this
issue, and this PEP does *not* attempt to define one. However, any tool that
*does* address this issue will need to know what 3rd party dependencies a script
requires. By defining a standard format for storing such data, existing tools,
as well as any future tools, will be able to obtain that information without
requiring users to include tool-specific metadata in their scripts.


Rationale
=========

Because a key requirement is writing single-file scripts, and simple sharing by
giving someone a copy of the script, the PEP defines a mechanism for embedding
dependency data *within the script itself*, and not in an external file.

We define the concept of a *dependency block* that contains information about
what 3rd party packages a script depends on.

In order to identify dependency blocks, the script can simply be read as a text
file. This is deliberate, as Python syntax changes over time, so attempting to
parse the script as Python code would require choosing a specific version of
Python syntax. Also, it is likely that at least some tools will not be written
in Python, and expecting them to implement a Python parser is too much of a
burden.

However, to avoid needing changes to core Python, the format is designed to
appear as comments to the Python parser. It is possible to write code where a
dependency block is *not* interpreted as a comment (for example, by embedding it
in a Python multi-line string), but such uses are discouraged and can easily be
avoided assuming you are not deliberately trying to create a pathological
example.

A `review <language survey_>`_ of how other languages allow scripts to specify
their dependencies shows that a "structured comment" like this is a
commonly-used approach.

Specification
=============

The content of this section will be published in the Python Packaging user
guide, PyPA Specifications section, as a document with the title "Embedding
Metadata in Script Files".

Any Python script may contain a *dependency block*. The dependency block is
identified by reading the script *as a text file* (i.e., the file is not parsed
as Python source code), looking for the first line of the form::

   # Script Dependencies:

The hash character must be at the start of the line with no preceding whitespace.
The text "Script Dependencies" is recognised regardless of case, and the spaces
represent arbitrary whitespace (although at least one space must be present). The
following regular expression recognises the dependency block header line::

    (?i)^#\s+script\s+dependencies:\s*$

Tools reading the dependency block MAY respect the standard Python encoding
declaration. If they choose not to do so, they MUST process the file as UTF-8.

After the header line, all lines in the file up to the first line that doesn't
start with a ``#`` sign are considered *dependency lines* and are treated as
follows:

1. The initial ``#`` sign is stripped.
2. If the line contains the character sequence " # " (SPACE HASH SPACE), then
   those characters and any subsequent characters are discarded. This allows
   dependency blocks to contain inline comments.
3. Whitespace at the start and end of the remaining text is discarded.
4. If the line is now empty, it is ignored.
5. The content of the line MUST now be a valid :pep:`508` dependency specifier.

The requirement for spaces before and after the ``#`` in an inline comment is
necessary to distinguish them from part of a :pep:`508` URL specifier (which
can contain a hash, but without surrounding whitespace).

Consumers MUST validate that at a minimum, all dependencies start with a
``name`` as defined in :pep:`508`, and they MAY validate that all dependencies
conform fully to :pep:`508`. They MUST fail with an error if they find an
invalid specifier.

Example
-------

The following is an example of a script with an embedded dependency block::

    # In order to run, this script needs the following 3rd party libraries
    #
    # Script Dependencies:
    #    requests
    #    rich     # Needed for the output
    #
    #    # Not needed - just to show that fragments in URLs do not
    #    # get treated as comments
    #    pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686

    import requests
    from rich.pretty import pprint

    resp = requests.get("https://peps.python.org/api/peps.json")
    data = resp.json()
    pprint([(k, v["title"]) for k, v in data.items()][:10])


Backwards Compatibility
=======================

As dependency blocks take the form of a structured comment, they can be added
without altering the meaning of existing code.

It is possible that a comment may already exist which matches the form of a
dependency block. While the identifying header text, "Script Dependencies" is
chosen to minimise this risk, it is still possible.

In the rare case where an existing comment would be interpreted incorrectly as a
dependency block, this can be addressed by adding an actual dependency block
(which can be empty if the script has no dependencies) earlier in the code.


Security Implications
=====================

If a script containing a dependency block is run using a tool that automatically
installs dependencies, this could cause arbitrary code to be downloaded and
installed in the user's environment.

The risk here is part of the functionality of the tool being used to run the
script, and as such should already be addressed by the tool itself. The only
additional risk introduced by this PEP is if an untrusted script with a
dependency block is run, when a potentially malicious dependency might be
installed. This risk is addressed by the normal good practice of reviewing code
before running it.


How to Teach This
=================

The format is intended to be close to how a developer might already specify
script dependencies in an explanatory comment. The required structure is
deliberately minimal, so that formatting rules are easy to learn.

Users will need to know how to write Python dependency specifiers. This is
covered by :pep:`508`, but for simple examples (which is expected to be the norm
for inexperienced users) the syntax is either just a package name, or a name and
a version restriction, which is fairly well-understood syntax.

Users will also know how to *run* a script using a tool that interprets
dependency data. This is not covered by this PEP, as it is the responsibility of
such a tool to document how i


id: pep-0736.rst
content:
PEP: 736
Title: Shorthand syntax for keyword arguments at invocation
Author: Joshua Bambrick <jbambrick@google.com>,
        Chris Angelico <rosuav@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-736-keyword-argument-shorthand-final-draft/58504
Status: Rejected
Type: Standards Track
Created: 28-Nov-2023
Python-Version: 3.14
Post-History: `14-Oct-2023 <https://discuss.python.org/t/syntactic-sugar-to-encourage-use-of-named-arguments/36217>`__,
              `17-Jan-2024 <https://discuss.python.org/t/pep-736-shorthand-syntax-for-keyword-arguments-at-invocation/43432>`__,
              `17-Jul-2024 <https://discuss.python.org/t/pep-736-keyword-argument-shorthand-final-draft/58504>`__,
Resolution: `13-Mar-2025 <https://discuss.python.org/t/43432/222>`__

Abstract
========

This PEP proposes to introduce syntactic sugar ``f(x=)`` for the common pattern
where a keyword argument has the same name as that of the variable corresponding
to its value ``f(x=x)``.

Motivation
==========

Keyword argument syntax can become needlessly repetitive and verbose.

Consider the following call:
::

    my_function(
      my_first_variable=my_first_variable,
      my_second_variable=my_second_variable,
      my_third_variable=my_third_variable,
    )

The case of a keyword argument name matching the variable name of its value is
prevalent among Python libraries. This redundancy discourages use of named
arguments and reduces readability by increasing visual noise.

Rationale
=========

There are two ways to invoke a function with arguments: by position and by
keyword. By being explicit, keyword arguments increase readability and
minimise the risk of inadvertent transposition. On the flipside, positional
arguments are often preferred simply to minimise verbosity and visual noise.

We contend that a simple syntactic sugar used to simplify this common pattern
would confer numerous benefits:

Encourages use of named arguments
---------------------------------

By reducing the visual noise that established keyword argument syntax can cause,
this syntax would encourage the use of named arguments, thereby increasing
readability and reducing bugs from argument transposition.

Reduces verbosity
-----------------

By minimising visual noise and in some cases lines of code, we can increase
readability.

Encourages consistent variable names
------------------------------------

A common problem is that semantically identical variables have different names
depending on their contexts. This syntax would encourage authors to use the same
variable name when calling a function as the argument name, which would increase
consistency of variable names used and hence improve readability.

Highlights arguments not following this pattern
-----------------------------------------------

With the current syntax, function calls where many arguments are forwarded from
the local context can make other argument values easy to miss due to the visual
noise. For example::

    add_middleware(
        excluded_urls=excluded_urls,
        server_request=server_request,
        client_request=client_request,
        client_response=client_response,
        span_details=_get_span_details(),
        tracer=tracer,
        meter=meter,
    )

With this syntax, the exceptional arguments become easier to identify::

    add_middleware(
        excluded_urls=,
        server_request=,
        client_request=,
        client_response=,
        span_details=_get_span_details(),
        tracer=,
        meter=,
    )

Applicability to dictionary construction
----------------------------------------

This syntax can be applied to dictionary construction where a similar pattern
frequently occurs (where dictionary keys are identical the names of the
variables assigned as their values), ``{"x": x, "y": y}`` or ``dict(x=x, y=y)``.
With this feature, this can now also be trivially written as ``dict(x=, y=)``.
Whether to further support similar syntax in dictionary literals is an open
question beyond the scope of this PEP.

Specification
=============

We propose to introduce syntactic sugar such that, if the value of a keyword
argument is omitted from a function invocation, the argument's value is inferred
to be the variable matching that name at the invocation scope.

For example, the function invocation:
::

    my_function(my_first_variable=, my_second_variable=, my_third_variable=)

Will be interpreted exactly equivalently to following in existing syntax:
::

    my_function(
      my_first_variable=my_first_variable,
      my_second_variable=my_second_variable,
      my_third_variable=my_third_variable,
    )

If no variable matches that name in the invocation scope, a ``NameError`` is
raised in an identical manner as would be with the established expanded syntax.

This proposal only pertains to function invocations; function definitions are
unaffected by the syntax change. All existing valid syntax is unchanged.

Backwards Compatibility
=======================

Only new syntax is added which was previously syntactically erroneous. No
existing valid syntax is modified. As such, the changes proposed are fully
backwards compatible.

Security Implications
=====================

There are no security implications for this change.

Prior Art
=========

Python already possesses a very similar feature in f-string interpolation where
``f'{x=}'`` is effectively expanded to ``f'x={x}'`` (see
`related GitHub issue <https://github.com/python/cpython/issues/80998>`__).

Several modern languages provide similar features during function invocation,
sometimes referred to as 'punning'. For example:

* In Ruby,  ``f(x:, y:)`` is syntactic sugar for ``f(x: x, y: y)``. See the
  `Ruby 3.1.0 release notes <https://www.ruby-lang.org/en/news/2021/12/25/ruby-3-1-0-released/#:~:text=Other%20Notable%20New%20Features>`__ (search for "keyword arguments").
* In ReasonML, ``f(~x, ~y)`` is syntactic sugar for ``f(~x=x, ~y=y)``. See the
  `ReasonML function documentation <https://reasonml.github.io/docs/en/function#function-application>`__ (search for "punning").
* In SystemVerilog, ``(.mult, .mop1, .data);`` is syntactic sugar for
  ``(.mult(mult), .mop1(mop1),  .data(data));``. See
  `SystemVerilog Implicit Port Connections <http://www.sunburst-design.com/papers/CummingsDesignCon2005_SystemVerilog_ImplicitPorts.pdf>`__.
* In Jakt, ``f(x, y)`` is syntactic sugar for ``f(x: x, y: y)``. See
  `The Jakt programming language <https://github.com/SerenityOS/jakt?tab=readme-ov-file#function-calls>`__.

Beyond function invocation specifically, more languages offer similar features:

* In OCaml, ``let+ x in â¦`` is syntactic sugar for ``let+ x = x in â¦``. See
  `OCaml: Short notation for variable bindings (let-punning) <https://v2.ocaml.org/manual/bindingops.html#ss:letops-punning>`__.
* In JavaScript, ``{ x, y }`` is syntactic sugar for ``{x: x, y: y}``. See
  `JavaScript: Object Initializer <https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Object_initializer>`__.
* In Rust, ``User { x, y }`` is shorthand for ``User {x: x, y: y}``. See
  `Rust: Using the Field Init Shorthand <https://doc.rust-lang.org/book/ch05-01-defining-structs.html#using-the-field-init-shorthand-when-variables-and-fields-have-the-same-name>`__.

Applicability
=============

We analysed popular Python libraries from the last few years using
`this script <https://gist.github.com/joshuabambrick/a850d0e0050129b9252c748fa06c48b2>`__
to compute:

* The number of keyword arguments which were of the form ``f(x=x)`` at
  invocation.
* The percentage of keyword arguments which had the form ``f(x=x)`` at
  invocation.
* The number of lines of code which could be saved by using this syntactic sugar
  to reduce the need for line wraps.

The purpose of this exercise was to compute statistics about the prevalence of
this pattern and should not be interpreted as a recommendation that the proposed
syntactic sugar should be applied universally.

===============


id: pep-0285.rst
content:
PEP: 285
Title: Adding a bool type
Author: Guido van Rossum <guido@python.org>
Status: Final
Type: Standards Track
Created: 08-Mar-2002
Python-Version: 2.3
Post-History: 08-Mar-2002, 30-Mar-2002, 03-Apr-2002


Abstract
========

This PEP proposes the introduction of a new built-in type, bool,
with two constants, ``False`` and ``True``.  The bool type would be a
straightforward subtype (in C) of the int type, and the values
``False`` and ``True`` would behave like 0 and 1 in most respects (for
example, ``False==0`` and ``True==1`` would be true) except ``repr()`` and
``str()``.  All built-in operations that conceptually return a Boolean
result will be changed to return ``False`` or ``True`` instead of 0 or 1;
for example, comparisons, the "not" operator, and predicates like
``isinstance()``.


Review
======

I've collected enough feedback to last me a lifetime, so I declare
the review period officially OVER.  I had Chinese food today; my
fortune cookie said "Strong and bitter words indicate a weak
cause."  It reminded me of some of the posts against this
PEP... :-)

Anyway, here are my BDFL pronouncements.  (Executive summary: I'm
not changing a thing; all variants are rejected.)

1) Should this PEP be accepted?

   => Yes.

   There have been many arguments against the PEP.  Many of them
   were based on misunderstandings.  I've tried to clarify some of
   the most common misunderstandings below in the main text of the
   PEP.  The only issue that weighs at all for me is the tendency
   of newbies to write "if x == True" where "if x" would suffice.
   More about that below too.  I think this is not a sufficient
   reason to reject the PEP.

2) Should ``str(True)`` return "True" or "1"?  "1" might reduce
   backwards compatibility problems, but looks strange.
   (``repr(True)`` would always return "True".)

   => "True".

   Almost all reviewers agree with this.

3) Should the constants be called 'True' and 'False' (similar to
   None) or 'true' and 'false' (as in C++, Java and C99)?

   => True and False.

   Most reviewers agree that consistency within Python is more
   important than consistency with other languages.

4) Should we strive to eliminate non-Boolean operations on bools
   in the future, through suitable warnings, so that for example
   True+1 would eventually (in Python 3000) be illegal?

   => No.

   There's a small but vocal minority that would prefer to see
   "textbook" bools that don't support arithmetic operations at
   all, but most reviewers agree with me that bools should always
   allow arithmetic operations.

5) Should ``operator.truth(x)`` return an int or a bool?

   => bool.

   Tim Peters believes it should return an int, but almost all
   other reviewers agree that it should return a bool.  My
   rationale: ``operator.truth()`` exists to force a Boolean context
   on its argument (it calls the C API ``PyObject_IsTrue())``.
   Whether the outcome is reported as int or bool is secondary; if
   bool exists there's no reason not to use it.  (Under the PEP,
   ``operator.truth()`` now becomes an alias for ``bool()``; that's fine.)

6) Should bool inherit from int?

   => Yes.

   In an ideal world, bool might be better implemented as a
   separate integer type that knows how to perform mixed-mode
   arithmetic.  However, inheriting bool from int eases the
   implementation enormously (in part since all C code that calls
   ``PyInt_Check()`` will continue to work -- this returns true for
   subclasses of int).  Also, I believe this is right in terms of
   substitutability: code that requires an int can be fed a bool
   and it will behave the same as 0 or 1.  Code that requires a
   bool may not work when it is given an int; for example, 3 & 4
   is 0, but both 3 and 4 are true when considered as truth
   values.

7) Should the name 'bool' be changed?

   => No.

   Some reviewers have argued for boolean instead of bool, because
   this would be easier to understand (novices may have heard of
   Boolean algebra but may not make the connection with bool) or
   because they hate abbreviations.  My take: Python uses
   abbreviations judiciously (like 'def', 'int', 'dict') and I
   don't think these are a burden to understanding.  To a newbie,
   it doesn't matter whether it's called a waffle or a bool; it's
   a new word, and they learn quickly what it means.

   One reviewer has argued to make the name 'truth'.  I find this
   an unattractive name, and would actually prefer to reserve this
   term (in documentation) for the more abstract concept of truth
   values that already exists in Python.  For example: "when a
   container is interpreted as a truth value, an empty container
   is considered false and a non-empty one is considered true."

8) Should we strive to require that Boolean operations (like "if",
   "and", "not") have a bool as an argument in the future, so that
   for example "if []:" would become illegal and would have to be
   written as "if bool([]):" ???

   => No!!!

   Some people believe that this is how a language with a textbook
   Boolean type should behave.  Because it was brought up, others
   have worried that I might agree with this position.  Let me
   make my position on this quite clear.  This is not part of the
   PEP's motivation and I don't intend to make this change.  (See
   also the section "Clarification" below.)


Rationale
=========

Most languages eventually grow a Boolean type; even C99 (the new
and improved C standard, not yet widely adopted) has one.

Many programmers apparently feel the need for a Boolean type; most
Python documentation contains a bit of an apology for the absence
of a Boolean type.  I've seen lots of modules that defined
constants "False=0" and "True=1" (or similar) at the top and used
those.  The problem with this is that everybody does it
differently.  For example, should you use "FALSE", "false",
"False", "F" or even "f"?  And should false be the value zero or
None, or perhaps a truth value of a different type that will print
as "true" or "false"?  Adding a standard bool type to the language
resolves those issues.

Some external libraries (like databases and RPC packages) need to
be able to distinguish between Boolean and integral values, and
while it's usually possible to craft a solution, it would be
easier if the language offered a standard Boolean type.  This also
applies to Jython: some Java classes have separately overloaded
methods or constructors for int and boolean arguments.  The bool
type can be used to select the boolean variant.  (The same is
apparently the case for some COM interfaces.)

The standard bool type can also serve as a way to force a value to
be interpreted as a Boolean, which can be used to normalize
Boolean values.  When a Boolean value needs to be normalized to
one of two values, ``bool(x)`` is much clearer than "not not x" and
much more concise than

::

    if x:
        return 1
    else:
        return 0

Here are some arguments derived from teaching Python.  When
showing people comparison operators etc. in the interactive shell,
I think this is a bit ugly::

    >>> a = 13
    >>> b = 12
    >>> a > b
    1
    >>>

If this was::

    >>> a > b
    True
    >>>

it would require a millisecond less thinking each time a 0 or 1
was printed.

There's also the issue (which I've seen baffling even experienced
Pythonistas who had been away from the language for a while) that
if you see::

    >>> cmp(a, b)
    1
    >>> cmp(a, a)
    0
    >>>

you might be tempted to believe that ``cmp()`` also returned a truth
value, whereas in reality it can return three different values
``(-1, 0, 1)``.  If ints were not (normally) used to represent
Booleans results, this would stand out much more clearly as
something completely different.


Specification
=============

The following Python code specifies most of the properties of the
new type::

    class bool(int):

        def __new__(cls, val=0):
            # This constructor always returns an 


id: pep-0291.rst
content:
PEP: 291
Title: Backward Compatibility for the Python 2 Standard Library
Author: Neal Norwitz <nnorwitz@gmail.com>
Status: Superseded
Type: Informational
Created: 06-Jun-2002
Python-Version: 2.3
Post-History:
Superseded-By: 387


Abstract
========

This PEP describes the packages and modules in the Python 2
standard library which should remain backward compatible with
previous versions of Python.  If a package is not listed here,
then it need only remain compatible with the version of Python it
is distributed with.

This PEP has no bearing on the Python 3 standard library.


Rationale
=========

Authors have various reasons why packages and modules should
continue to work with previous versions of Python.  In order to
maintain backward compatibility for these modules while moving the
rest of the standard library forward, it is necessary to know
which modules can be modified and which should use old and
possibly deprecated features.

Generally, authors should attempt to keep changes backward
compatible with the previous released version of Python in order
to make bug fixes easier to backport.

In addition to a package or module being listed in this PEP,
authors must add a comment at the top of each file documenting
the compatibility requirement.

When a major version of Python is released, a Subversion branch is
created for continued maintenance and bug fix releases.  A package
version on a branch may have a different compatibility requirement
than the same package on the trunk (i.e. current bleeding-edge
development).  Where appropriate, these branch compatibilities are
listed below.


Features to Avoid
=================

The following list contains common features to avoid in order
to maintain backward compatibility with each version of Python.
This list is not complete!  It is only meant as a general guide.

Note that the features below were implemented in the version
following the one listed.  For example, features listed next to
1.5.2 were implemented in 2.0.

=======  ======================================================
Version  Features to Avoid
=======  ======================================================
1.5.2    string methods, Unicode, list comprehensions,
         augmented assignment (eg, +=), zip(), import x as y,
         dict.setdefault(), print >> f,
         calling f(\*args, \**kw), plus all features below

2.0      nested scopes, rich comparisons,
         function attributes, plus all features below

2.1      use of object or new-style classes, iterators,
         using generators, nested scopes, or //
         without from __future__ import ... statement,
         isinstance(X, TYP) where TYP is a tuple of types,
         plus all features below

2.2      bool, True, False, basestring, enumerate(),
         {}.pop(), PendingDeprecationWarning,
         Universal Newlines, plus all features below
         plus all features below

2.3      generator expressions, multi-line imports,
         decorators, int/long unification, set/frozenset,
         reversed(), sorted(), "".rsplit(),
         plus all features below

2.4      with statement, conditional expressions,
         combined try/except/finally, relative imports,
         yield expressions or generator.throw/send/close(),
         plus all features below

2.5      with statement without from __future__ import,
         io module, str.format(), except as,
         bytes, b'' literals, property.setter/deleter
=======  ======================================================


Backward Compatible Packages, Modules, and Tools
================================================

==============  ==================  ==============  =====
Package/Module  Maintainer(s)       Python Version  Notes
==============  ==================  ==============  =====
2to3            Benjamin Peterson   2.5
bsddb           - Greg Smith        2.1
                - Barry Warsaw
compiler        Jeremy Hylton       2.1
decimal         Raymond Hettinger   2.3             [2]
distutils       Tarek Ziade         2.3
email           Barry Warsaw        2.1 / 2.3       [1]
pkgutil         Phillip Eby         2.3
platform        Marc-Andre Lemburg  1.5.2
pybench         Marc-Andre Lemburg  1.5.2           [3]
sre             Fredrik Lundh       2.1
subprocess      Peter Astrand       2.2
wsgiref         Phillip J. Eby      2.1
xml (PyXML)     Martin v. Loewis    2.0
xmlrpclib       Fredrik Lundh       2.1
==============  ==================  ==============  =====


====  =============  ==============
Tool  Maintainer(s)  Python Version
====  =============  ==============
None
====  =============  ==============


Notes
-----

1. The email package version 2 was distributed with Python up to
   Python 2.3, and this must remain Python 2.1 compatible.  email
   package version 3 will be distributed with Python 2.4 and will
   need to remain compatible only with Python 2.3.

2. Specification updates will be treated as bugfixes and backported.
   Python 2.3 compatibility will be kept for at least Python 2.4.
   The decision will be revisited for Python 2.5 and not changed
   unless compelling advantages arise.

3. pybench lives under the Tools/ directory. Compatibility with
   older Python versions is needed in order to be able to compare
   performance between Python versions. New features may still
   be used in new tests, which may then be configured to fail
   gracefully on import by the tool in older Python versions.


Copyright
=========

This document has been placed in the public domain.


id: pep-0508.rst
content:
PEP: 508
Title: Dependency specification for Python Software Packages
Author: Robert Collins <rbtcollins@hp.com>
BDFL-Delegate: Donald Stufft <donald@stufft.io>
Discussions-To: distutils-sig@python.org
Status: Final
Type: Standards Track
Topic: Packaging
Created: 11-Nov-2015
Post-History: 05-Nov-2015, 16-Nov-2015
Resolution: https://mail.python.org/pipermail/distutils-sig/2015-November/027868.html

.. canonical-pypa-spec:: :ref:`packaging:dependency-specifiers`


Abstract
========

This PEP specifies the language used to describe dependencies for packages.
It draws a border at the edge of describing a single dependency - the
different sorts of dependencies and when they should be installed is a higher
level problem. The intent is to provide a building block for higher layer
specifications.

The job of a dependency is to enable tools like pip [#pip]_ to find the right
package to install. Sometimes this is very loose - just specifying a name, and
sometimes very specific - referring to a specific file to install. Sometimes
dependencies are only relevant in one platform, or only some versions are
acceptable, so the language permits describing all these cases.

The language defined is a compact line based format which is already in
widespread use in pip requirements files, though we do not specify the command
line option handling that those files permit. There is one caveat - the
URL reference form, specified in :pep:`440` is not actually
implemented in pip, but since :pep:`440` is accepted, we use that format rather
than pip's current native format.

Motivation
==========

Any specification in the Python packaging ecosystem that needs to consume
lists of dependencies needs to build on an approved PEP for such, but
:pep:`426` is mostly aspirational - and there are already existing
implementations of the dependency specification which we can instead adopt.
The existing implementations are battle proven and user friendly, so adopting
them is arguably much better than approving an aspirational, unconsumed, format.

Specification
=============

Examples
--------

All features of the language shown with a name based lookup::

    requests [security,tests] >= 2.8.1, == 2.8.* ; python_version < "2.7"

A minimal URL based lookup::

    pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686

Concepts
--------

A dependency specification always specifies a distribution name. It may
include extras, which expand the dependencies of the named distribution to
enable optional features. The version installed can be controlled using
version limits, or giving the URL to a specific artifact to install. Finally
the dependency can be made conditional using environment markers.

Grammar
-------

We first cover the grammar briefly and then drill into the semantics of each
section later.

A distribution specification is written in ASCII text. We use a parsley
[#parsley]_ grammar to provide a precise grammar. It is expected that the
specification will be embedded into a larger system which offers framing such
as comments, multiple line support via continuations, or other such features.

The full grammar including annotations to build a useful parse tree is
included at the end of the PEP.

Versions may be specified according to the :pep:`440` rules. (Note:
URI is defined in :rfc:`std-66 <3986>`)::

    version_cmp   = wsp* '<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' | '==='
    version       = wsp* ( letterOrDigit | '-' | '_' | '.' | '*' | '+' | '!' )+
    version_one   = version_cmp version wsp*
    version_many  = version_one (wsp* ',' version_one)*
    versionspec   = ( '(' version_many ')' ) | version_many
    urlspec       = '@' wsp* <URI_reference>

Environment markers allow making a specification only take effect in some
environments::

    marker_op     = version_cmp | (wsp* 'in') | (wsp* 'not' wsp+ 'in')
    python_str_c  = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' |
                     '-' | '_' | '*' | '#' | ':' | ';' | ',' | '/' | '?' |
                     '[' | ']' | '!' | '~' | '`' | '@' | '$' | '%' | '^' |
                     '&' | '=' | '+' | '|' | '<' | '>' )
    dquote        = '"'
    squote        = '\\''
    python_str    = (squote (python_str_c | dquote)* squote |
                     dquote (python_str_c | squote)* dquote)
    env_var       = ('python_version' | 'python_full_version' |
                     'os_name' | 'sys_platform' | 'platform_release' |
                     'platform_system' | 'platform_version' |
                     'platform_machine' | 'platform_python_implementation' |
                     'implementation_name' | 'implementation_version' |
                     'extra' # ONLY when defined by a containing layer
                     )
    marker_var    = wsp* (env_var | python_str)
    marker_expr   = marker_var marker_op marker_var
                  | wsp* '(' marker wsp* ')'
    marker_and    = marker_expr wsp* 'and' marker_expr
                  | marker_expr
    marker_or     = marker_and wsp* 'or' marker_and
                      | marker_and
    marker        = marker_or
    quoted_marker = ';' wsp* marker

Optional components of a distribution may be specified using the extras
field::

    identifier_end = letterOrDigit | (('-' | '_' | '.' )* letterOrDigit)
    identifier    = letterOrDigit identifier_end*
    name          = identifier
    extras_list   = identifier (wsp* ',' wsp* identifier)*
    extras        = '[' wsp* extras_list? wsp* ']'

Giving us a rule for name based requirements::

    name_req      = name wsp* extras? wsp* versionspec? wsp* quoted_marker?

And a rule for direct reference specifications::

    url_req       = name wsp* extras? wsp* urlspec wsp+ quoted_marker?

Leading to the unified rule that can specify a dependency.::

    specification = wsp* ( url_req | name_req ) wsp*

Whitespace
----------

Non line-breaking whitespace is mostly optional with no semantic meaning. The
sole exception is detecting the end of a URL requirement.

Names
-----

Python distribution names are currently defined in :pep:`345`. Names
act as the primary identifier for distributions. They are present in all
dependency specifications, and are sufficient to be a specification on their
own. However, PyPI places strict restrictions on names - they must match a
case insensitive regex or they won't be accepted. Accordingly, in this PEP we
limit the acceptable values for identifiers to that regex. A full redefinition
of name may take place in a future metadata PEP. The regex (run with
re.IGNORECASE) is::

    ^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$

Extras
------

An extra is an optional part of a distribution. Distributions can specify as
many extras as they wish, and each extra results in the declaration of
additional dependencies of the distribution **when** the extra is used in a
dependency specification. For instance::

    requests[security]

Extras union in the dependencies they define with the dependencies of the
distribution they are attached to. The example above would result in requests
being installed, and requests own dependencies, and also any dependencies that
are listed in the "security" extra of requests.

If multiple extras are listed, all the dependencies are unioned together.

Versions
--------

See :pep:`440` for more detail on both version numbers and version
comparisons. Version specifications limit the versions of a distribution that
can be used. They only apply to distributions looked up by name, rather than
via a URL. Version comparison are also used in the markers feature. The
optional brackets around a version are present for compatibility with :pep:`345`
but should not be generated, only accepted.

Environment Markers
-------------------

Environment markers allow a dependency specification to provide a rule that
describes when the dependency should be used. For instance, consider a package
that needs argparse. In Python 2.7 argparse is always pre


id: pep-0246.rst
content:
PEP: 246
Title: Object Adaptation
Author: Alex Martelli <aleaxit@gmail.com>,
        Clark C. Evans <cce@clarkevans.com>
Status: Rejected
Type: Standards Track
Created: 21-Mar-2001
Python-Version: 2.5
Post-History: 29-Mar-2001, 10-Jan-2005


Rejection Notice
================

I'm rejecting this PEP.  Something much better is about to happen;
it's too early to say exactly what, but it's not going to resemble
the proposal in this PEP too closely so it's better to start a new
PEP.  GvR.


Abstract
========

This proposal puts forth an extensible cooperative mechanism for
the adaptation of an incoming object to a context which expects an
object supporting a specific protocol (say a specific type, class,
or interface).

This proposal provides a built-in "adapt" function that, for any
object X and any protocol Y, can be used to ask the Python
environment for a version of X compliant with Y.  Behind the
scenes, the mechanism asks object X: "Are you now, or do you know
how to wrap yourself to provide, a supporter of protocol Y?".
And, if this request fails, the function then asks protocol Y:
"Does object X support you, or do you know how to wrap it to
obtain such a supporter?"  This duality is important, because
protocols can be developed after objects are, or vice-versa, and
this PEP lets either case be supported non-invasively with regard
to the pre-existing component[s].

Lastly, if neither the object nor the protocol know about each
other, the mechanism may check a registry of adapter factories,
where callables able to adapt certain objects to certain protocols
can be registered dynamically.  This part of the proposal is
optional: the same effect could be obtained by ensuring that
certain kinds of protocols and/or objects can accept dynamic
registration of adapter factories, for example via suitable custom
metaclasses.  However, this optional part allows adaptation to be
made more flexible and powerful in a way that is not invasive to
either protocols or other objects, thereby gaining for adaptation
much the same kind of advantage that Python standard library's
"copy_reg" module offers for serialization and persistence.

This proposal does not specifically constrain what a protocol
*is*, what "compliance to a protocol" exactly *means*, nor what
precisely a wrapper is supposed to do.  These omissions are
intended to leave this proposal compatible with both existing
categories of protocols, such as the existing system of type and
classes, as well as the many concepts for "interfaces" as such
which have been proposed or implemented for Python, such as the
one in :pep:`245`, the one in Zope3 [2]_, or the ones discussed in
the BDFL's Artima blog in late 2004 and early 2005 [3]_.  However,
some reflections on these subjects, intended to be suggestive and
not normative, are also included.


Motivation
==========

Currently there is no standardized mechanism in Python for
checking if an object supports a particular protocol.  Typically,
existence of certain methods, particularly special methods such as
``__getitem__``, is used as an indicator of support for a particular
protocol.  This technique works well for a few specific protocols
blessed by the BDFL (Benevolent Dictator for Life).  The same can
be said for the alternative technique based on checking
'isinstance' (the built-in class "basestring" exists specifically
to let you use 'isinstance' to check if an object "is a \[built-in\]
string").  Neither approach is easily and generally extensible to
other protocols, defined by applications and third party
frameworks, outside of the standard Python core.

Even more important than checking if an object already supports a
given protocol can be the task of obtaining a suitable adapter
(wrapper or proxy) for the object, if the support is not already
there.  For example, a string does not support the file protocol,
but you can wrap it into a StringIO instance to obtain an object
which does support that protocol and gets its data from the string
it wraps; that way, you can pass the string (suitably wrapped) to
subsystems which require as their arguments objects that are
readable as files.  Unfortunately, there is currently no general,
standardized way to automate this extremely important kind of
"adaptation by wrapping" operations.

Typically, today, when you pass objects to a context expecting a
particular protocol, either the object knows about the context and
provides its own wrapper or the context knows about the object and
wraps it appropriately.  The difficulty with these approaches is
that such adaptations are one-offs, are not centralized in a
single place of the users code, and are not executed with a common
technique, etc.  This lack of standardization increases code
duplication with the same adapter occurring in more than one place
or it encourages classes to be re-written instead of adapted.  In
either case, maintainability suffers.

It would be very nice to have a standard function that can be
called upon to verify an object's compliance with a particular
protocol and provide for a wrapper if one is readily available --
all without having to hunt through each library's documentation
for the incantation appropriate to that particular, specific case.


Requirements
============

When considering an object's compliance with a protocol, there are
several cases to be examined:

a) When the protocol is a type or class, and the object has
   exactly that type or is an instance of exactly that class (not
   a subclass).  In this case, compliance is automatic.

b) When the object knows about the protocol, and either considers
   itself compliant, or knows how to wrap itself suitably.

c) When the protocol knows about the object, and either the object
   already complies or the protocol knows how to suitably wrap the
   object.

d) When the protocol is a type or class, and the object is a
   member of a subclass.  This is distinct from the first case (a)
   above, since inheritance (unfortunately) does not necessarily
   imply substitutability, and thus must be handled carefully.

e) When the context knows about the object and the protocol and
   knows how to adapt the object so that the required protocol is
   satisfied.  This could use an adapter registry or similar
   approaches.

The fourth case above is subtle.  A break of substitutability can
occur when a subclass changes a method's signature, or restricts
the domains accepted for a method's argument ("co-variance" on
arguments types), or extends the co-domain to include return
values which the base class may never produce ("contra-variance"
on return types).  While compliance based on class inheritance
*should* be automatic, this proposal allows an object to signal
that it is not compliant with a base class protocol.

If Python gains some standard "official" mechanism for interfaces,
however, then the "fast-path" case (a) can and should be extended
to the protocol being an interface, and the object an instance of
a type or class claiming compliance with that interface.  For
example, if the "interface" keyword discussed in [3]_ is adopted
into Python, the "fast path" of case (a) could be used, since
instantiable classes implementing an interface would not be
allowed to break substitutability.


Specification
=============

This proposal introduces a new built-in function, ``adapt()``, which
is the basis for supporting these requirements.

The ``adapt()`` function has three parameters:

- ``obj``, the object to be adapted

- ``protocol``, the protocol requested of the object

- ``alternate``, an optional object to return if the object could
  not be adapted

A successful result of the ``adapt()`` function returns either the
object passed ``obj``, if the object is already compliant with the
protocol, or a secondary object ``wrapper``, which provides a view
of the object compliant with the protocol.  The definition of
wrapper is deliberately vague, and a wrapper is allowed to be a
full object with its own sta


id: pep-0520.rst
content:
PEP: 520
Title: Preserving Class Attribute Definition Order
Author: Eric Snow <ericsnowcurrently@gmail.com>
Status: Final
Type: Standards Track
Created: 07-Jun-2016
Python-Version: 3.6
Post-History: 07-Jun-2016, 11-Jun-2016, 20-Jun-2016, 24-Jun-2016
Resolution: https://mail.python.org/pipermail/python-dev/2016-June/145442.html

.. note::
   Since compact dict has landed in 3.6, __definition_order__
   has been removed.  ``cls.__dict__`` now mostly accomplishes the same
   thing instead.

Abstract
========

The class definition syntax is ordered by its very nature. Class
attributes defined there are thus ordered.  Aside from helping with
readability, that ordering is sometimes significant.  If it were
automatically available outside the class definition then the
attribute order could be used without the need for extra boilerplate
(such as metaclasses or manually enumerating the attribute order).
Given that this information already exists, access to the definition
order of attributes is a reasonable expectation.  However, currently
Python does not preserve the attribute order from the class
definition.

This PEP changes that by preserving the order in which attributes
are introduced in the class definition body.  That order will now be
preserved in the ``__definition_order__`` attribute of the class.
This allows introspection of the original definition order, e.g. by
class decorators.

Additionally, this PEP requires that the default class definition
namespace be ordered (e.g. ``OrderedDict``) by default.  The
long-lived class namespace (``__dict__``) will remain a ``dict``.


Motivation
==========

The attribute order from a class definition may be useful to tools
that rely on name order.  However, without the automatic availability
of the definition order, those tools must impose extra requirements on
users.  For example, use of such a tool may require that your class use
a particular metaclass.  Such requirements are often enough to
discourage use of the tool.

Some tools that could make use of this PEP include:

* documentation generators
* testing frameworks
* CLI frameworks
* web frameworks
* config generators
* data serializers
* enum factories (my original motivation)


Background
==========

When a class is defined using a ``class`` statement, the class body
is executed within a namespace.  Currently that namespace defaults to
``dict``.  If the metaclass defines ``__prepare__()`` then the result
of calling it is used for the class definition namespace.

After the execution completes, the definition namespace is
copied into a new ``dict``.  Then the original definition namespace is
discarded.  The new copy is stored away as the class's namespace and
is exposed as ``__dict__`` through a read-only proxy.

The class attribute definition order is represented by the insertion
order of names in the *definition* namespace.  Thus, we can have
access to the definition order by switching the definition namespace
to an ordered mapping, such as ``collections.OrderedDict``.  This is
feasible using a metaclass and ``__prepare__``, as described above.
In fact, exactly this is by far the most common use case for using
``__prepare__``.

At that point, the only missing thing for later access to the
definition order is storing it on the class before the definition
namespace is thrown away.  Again, this may be done using a metaclass.
However, this means that the definition order is preserved only for
classes that use such a metaclass.  There are two practical problems
with that:

First, it requires the use of a metaclass.  Metaclasses introduce an
extra level of complexity to code and in some cases (e.g. conflicts)
are a problem.  So reducing the need for them is worth doing when the
opportunity presents itself.  :pep:`422` and :pep:`487` discuss this at
length.  We have such an opportunity by using an ordered mapping (e.g.
``OrderedDict`` for CPython at least) for the default class definition
namespace, virtually eliminating the need for ``__prepare__()``.

Second, only classes that opt in to using the ``OrderedDict``-based
metaclass will have access to the definition order. This is problematic
for cases where universal access to the definition order is important.


Specification
=============

Part 1:

* all classes have a ``__definition_order__`` attribute
* ``__definition_order__`` is a ``tuple`` of identifiers (or ``None``)
* ``__definition_order__`` is always set:

  1. during execution of the class body, the insertion order of names
     into the class *definition* namespace is stored in a tuple
  2. if ``__definition_order__`` is defined in the class body then it
     must be a ``tuple`` of identifiers or ``None``; any other value
     will result in ``TypeError``
  3. classes that do not have a class definition (e.g. builtins) have
     their ``__definition_order__`` set to ``None``
  4. classes for which ``__prepare__()`` returned something other than
     ``OrderedDict`` (or a subclass) have their ``__definition_order__``
     set to ``None`` (except where #2 applies)

Not changing:

* ``dir()`` will not depend on ``__definition_order__``
* descriptors and custom ``__getattribute__`` methods are unconstrained
  regarding ``__definition_order__``

Part 2:

* the default class *definition* namespace is now an ordered mapping
  (e.g. ``OrderdDict``)
* ``cls.__dict__`` does not change, remaining a read-only proxy around
  ``dict``

Note that Python implementations which have an ordered ``dict`` won't
need to change anything.

The following code demonstrates roughly equivalent semantics for both
parts 1 and 2::

   class Meta(type):
       @classmethod
       def __prepare__(cls, *args, **kwargs):
           return OrderedDict()

   class Spam(metaclass=Meta):
       ham = None
       eggs = 5
       __definition_order__ = tuple(locals())

Why a tuple?
------------

Use of a tuple reflects the fact that we are exposing the order in
which attributes on the class were *defined*.  Since the definition
is already complete by the time ``__definition_order__`` is set, the
content and order of the value won't be changing.  Thus we use a type
that communicates that state of immutability.

Why not a read-only attribute?
------------------------------

There are some valid arguments for making ``__definition_order__``
a read-only attribute (like ``cls.__dict__`` is).  Most notably, a
read-only attribute conveys the nature of the attribute as "complete",
which is exactly correct for ``__definition_order__``.  Since it
represents the state of a particular one-time event (execution of
the class definition body), allowing the value to be replaced would
reduce confidence that the attribute corresponds to the original class
body.  Furthermore, often an immutable-by-default approach helps to
make data easier to reason about.

However, in this case there still isn't a *strong* reason to counter
the well-worn precedent found in Python.  Per Guido::

    I don't see why it needs to be a read-only attribute. There are
    very few of those -- in general we let users play around with
    things unless we have a hard reason to restrict assignment (e.g.
    the interpreter's internal state could be compromised). I don't
    see such a hard reason here.

Also, note that a writeable ``__definition_order__`` allows dynamically
created classes (e.g. by Cython) to still have ``__definition_order__``
properly set.  That could certainly be handled through specific
class-creation tools, such as ``type()`` or the C-API, without the need
to lose the semantics of a read-only attribute.  However, with a
writeable attribute it's a moot point.


Why not "__attribute_order__"?
------------------------------

``__definition_order__`` is centered on the class definition
body.  The use cases for dealing with the class namespace (``__dict__``)
post-definition are a separate matter.  ``__definition_order__`` would
be a significantly misleading name for a feature focused on more than
class definition


id: pep-0534.rst
content:
PEP: 534
Title: Improved Errors for Missing Standard Library Modules
Author: TomÃ¡Å¡ Orsava <tomas.n@orsava.cz>,
        Petr Viktorin <encukou@gmail.com>,
        Alyssa Coghlan <ncoghlan@gmail.com>
Status: Deferred
Type: Standards Track
Created: 05-Sep-2016
Post-History:


Abstract
========

Python is often being built or distributed without its full standard library.
However, there is as of yet no standard, user friendly way of properly
informing the user about the failure to import such missing standard library
modules.

This PEP proposes a mechanism for identifying expected standard library modules
and providing more informative error messages to users when attempts to import
standard library modules fail.


PEP Deferral
============

The PEP authors aren't actively working on this PEP, so if improving these
error messages is an idea that you're interested in pursuing, please get in
touch! (e.g. by posting to the python-dev mailing list).

The key piece of open work is determining how to get the autoconf and Visual
Studio build processes to populate the sysconfig metadata file with the lists
of expected and optional standard library modules.


Motivation
==========

There are several use cases for including only a subset of Python's standard
library.  However, there is so far no user-friendly mechanism for informing
the user *why* a stdlib module is missing and how to remedy the situation
appropriately.


CPython
-------

When one of Python's standard library modules (such as ``_sqlite3``) cannot be
compiled during a CPython build because of missing dependencies (e.g. SQLite
header files), the module is simply skipped.  If you then install this compiled
Python and use it to try to import one of the missing modules, Python will fail
with a ModuleNotFoundError_.

.. _ModuleNotFoundError:
   https://docs.python.org/3.7/library/exceptions.html#ModuleNotFoundError

For example, after deliberately removing ``sqlite-devel`` from the local
system::

   $ ./python -c "import sqlite3"
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
     File "/home/ncoghlan/devel/cpython/Lib/sqlite3/__init__.py", line 23, in <module>
       from sqlite3.dbapi2 import *
     File "/home/ncoghlan/devel/cpython/Lib/sqlite3/dbapi2.py", line 27, in <module>
       from _sqlite3 import *
   ModuleNotFoundError: No module named '_sqlite3'

This can confuse users who may not understand why a cleanly built Python is
missing standard library modules.


Linux and other distributions
-----------------------------

Many Linux and other distributions are already separating out parts of the
standard library to standalone packages.  Among the most commonly excluded
modules are the ``tkinter`` module, since it draws in a dependency on the
graphical environment, ``idlelib``, since it depends on ``tkinter`` (and most
Linux desktop environments provide their own default code editor), and the
``test`` package, as it only serves to test Python internally and is about as
big as the rest of the standard library put together.

The methods of omission of these modules differ.  For example, Debian patches
the file ``Lib/tkinter/__init__.py`` to envelop the line ``import _tkinter`` in
a *try-except* block and upon encountering an ``ImportError`` it simply adds
the following to the error message: ``please install the python3-tk package``
[#debian-patch]_.  Fedora and other distributions simply don't include the
omitted modules, potentially leaving users baffled as to where to find them.

An example from Fedora 29::

    $ python3 -c "import tkinter"
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'tkinter'



Specification
=============

APIs to list expected standard library modules
----------------------------------------------

To allow for easier identification of which module names are *expected* to be
resolved in the standard library, the `sysconfig`_ module will be extended
with two additional functions:

* ``sysconfig.get_stdlib_modules()``, which will provide a list of the names of
  all top level Python standard library modules (including private modules)
* ``sysconfig.get_optional_modules()``, which will list optional public top level
  standard library module names

The results of ``sysconfig.get_optional_modules()`` and the existing
``sys.builtin_module_names`` will both be subsets of the full list provided by
the new ``sysconfig.get_stdlib_modules()`` function.

These added lists will be generated during the Python build process and saved in
the ``_sysconfigdata-*.py`` file along with other `sysconfig`_ values.

Possible reasons for modules being in the "optional" list will be:

* the module relies on an optional build dependency (e.g. ``_sqlite3``,
  ``tkinter``, ``idlelib``)
* the module is private for other reasons and hence may not be present on all
  implementations (e.g. ``_freeze_importlib``, ``_collections_abc``)
* the module is platform specific and hence may not be present in all
  installations (e.g. ``winreg``)
* the ``test`` package may also be freely omitted from Python runtime
  installations, as it is intended for use in testing Python implementations,
  not as a runtime library for Python projects to use (the public API offering
  testing utilities is ``unittest``)

(Note: the ``ensurepip``, ``venv``, and ``distutils`` modules are all considered
mandatory modules in this PEP, even though not all redistributors currently
adhere to that practice)

.. _`sysconfig`: https://docs.python.org/3/library/sysconfig.html


Changes to the default ``sys.excepthook`` implementation
--------------------------------------------------------

The default implementation of the `sys.excepthook`_ function will then be
modified to dispense an appropriate message when it detects a failure to
import a module identified by one of the two new `sysconfig`_ functions as
belonging to the Python standard library.

.. _`sys.excepthook`: https://docs.python.org/3/library/sys.html#sys.excepthook

Revised error message for a module that relies on an optional build dependency
or is otherwise considered optional when Python is installed::

   $ ./python -c "import sqlite3"
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
     File "/home/ncoghlan/devel/cpython/Lib/sqlite3/__init__.py", line 23, in <module>
       from sqlite3.dbapi2 import *
     File "/home/ncoghlan/devel/cpython/Lib/sqlite3/dbapi2.py", line 27, in <module>
       from _sqlite3 import *
   ModuleNotFoundError: Optional standard library module '_sqlite3' was not found

Revised error message for a submodule of an optional top level package when the
entire top level package is missing::

   $ ./python -c "import test.regrtest"
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
   ModuleNotFoundError: Optional standard library module 'test' was not found

Revised error message for a submodule of an optional top level package when the
top level package is present::

   $ ./python -c "import test.regrtest"
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
   ModuleNotFoundError: No submodule named 'test.regrtest' in optional standard library module 'test'

Revised error message for a module that is always expected to be available::

   $ ./python -c "import ensurepip"
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
   ModuleNotFoundError: Standard library module 'ensurepip' was not found

Revised error message for a missing submodule of a standard library package when
the top level package is present::

   $ ./python -c "import encodings.mbcs"
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
   ModuleNotFoundError: No submodule named 'encodings.mbcs' in standard library module 'encodings'

These revised error messages make it clear that the missing modules are expected
to be available f


id: pep-0252.rst
content:
PEP: 252
Title: Making Types Look More Like Classes
Author: Guido van Rossum <guido@python.org>
Status: Final
Type: Standards Track
Created: 19-Apr-2001
Python-Version: 2.2
Post-History:

Abstract
========

This PEP proposes changes to the introspection API for types that
makes them look more like classes, and their instances more like
class instances.  For example, ``type(x)`` will be equivalent to
``x.__class__`` for most built-in types.  When C is ``x.__class__``,
``x.meth(a)`` will generally be equivalent to ``C.meth(x, a)``, and
``C.__dict__`` contains x's methods and other attributes.

This PEP also introduces a new approach to specifying attributes,
using attribute descriptors, or descriptors for short.
Descriptors unify and generalize several different common
mechanisms used for describing attributes: a descriptor can
describe a method, a typed field in the object structure, or a
generalized attribute represented by getter and setter functions.

Based on the generalized descriptor API, this PEP also introduces
a way to declare class methods and static methods.

[Editor's note: the ideas described in this PEP have been incorporated
into Python.  The PEP no longer accurately describes the implementation.]


Introduction
============

One of Python's oldest language warts is the difference between
classes and types.  For example, you can't directly subclass the
dictionary type, and the introspection interface for finding out
what methods and instance variables an object has is different for
types and for classes.

Healing the class/type split is a big effort, because it affects
many aspects of how Python is implemented.  This PEP concerns
itself with making the introspection API for types look the same
as that for classes.  Other PEPs will propose making classes look
more like types, and subclassing from built-in types; these topics
are not on the table for this PEP.


Introspection APIs
==================

Introspection concerns itself with finding out what attributes an
object has.  Python's very general getattr/setattr API makes it
impossible to guarantee that there always is a way to get a list
of all attributes supported by a specific object, but in practice
two conventions have appeared that together work for almost all
objects.  I'll call them the class-based introspection API and the
type-based introspection API; class API and type API for short.

The class-based introspection API is used primarily for class
instances; it is also used by Jim Fulton's ExtensionClasses.  It
assumes that all data attributes of an object x are stored in the
dictionary ``x.__dict__``, and that all methods and class variables
can be found by inspection of x's class, written as ``x.__class__``.
Classes have a ``__dict__`` attribute, which yields a dictionary
containing methods and class variables defined by the class
itself, and a ``__bases__`` attribute, which is a tuple of base
classes that must be inspected recursively.  Some assumptions here
are:

- attributes defined in the instance dict override attributes
  defined by the object's class;

- attributes defined in a derived class override attributes
  defined in a base class;

- attributes in an earlier base class (meaning occurring earlier
  in ``__bases__``) override attributes in a later base class.

(The last two rules together are often summarized as the
left-to-right, depth-first rule for attribute search.  This is the
classic Python attribute lookup rule.  Note that :pep:`253` will
propose to change the attribute lookup order, and if accepted,
this PEP will follow suit.)

The type-based introspection API is supported in one form or
another by most built-in objects.  It uses two special attributes,
``__members__`` and ``__methods__``.  The ``__methods__`` attribute, if
present, is a list of method names supported by the object.  The
``__members__`` attribute, if present, is a list of data attribute
names supported by the object.

The type API is sometimes combined with a ``__dict__`` that works the
same as for instances (for example for function objects in
Python 2.1, ``f.__dict__`` contains f's dynamic attributes, while
``f.__members__`` lists the names of f's statically defined
attributes).

Some caution must be exercised: some objects don't list their
"intrinsic" attributes (like ``__dict__`` and ``__doc__``) in ``__members__``,
while others do; sometimes attribute names occur both in
``__members__`` or ``__methods__`` and as keys in ``__dict__``, in which case
it's anybody's guess whether the value found in ``__dict__`` is used
or not.

The type API has never been carefully specified.  It is part of
Python folklore, and most third party extensions support it
because they follow examples that support it.  Also, any type that
uses ``Py_FindMethod()`` and/or ``PyMember_Get()`` in its tp_getattr
handler supports it, because these two functions special-case the
attribute names ``__methods__`` and ``__members__``, respectively.

Jim Fulton's ExtensionClasses ignore the type API, and instead
emulate the class API, which is more powerful.  In this PEP, I
propose to phase out the type API in favor of supporting the class
API for all types.

One argument in favor of the class API is that it doesn't require
you to create an instance in order to find out which attributes a
type supports; this in turn is useful for documentation
processors.  For example, the socket module exports the SocketType
object, but this currently doesn't tell us what methods are
defined on socket objects.  Using the class API, SocketType would
show exactly what the methods for socket objects are, and we can
even extract their docstrings, without creating a socket.  (Since
this is a C extension module, the source-scanning approach to
docstring extraction isn't feasible in this case.)


Specification of the class-based introspection API
==================================================

Objects may have two kinds of attributes: static and dynamic.  The
names and sometimes other properties of static attributes are
knowable by inspection of the object's type or class, which is
accessible through ``obj.__class__`` or ``type(obj)``.  (I'm using type
and class interchangeably; a clumsy but descriptive term that fits
both is "meta-object".)

(XXX static and dynamic are not great terms to use here, because
"static" attributes may actually behave quite dynamically, and
because they have nothing to do with static class members in C++
or Java.  Barry suggests to use immutable and mutable instead, but
those words already have precise and different meanings in
slightly different contexts, so I think that would still be
confusing.)

Examples of dynamic attributes are instance variables of class
instances, module attributes, etc.  Examples of static attributes
are the methods of built-in objects like lists and dictionaries,
and the attributes of frame and code objects (``f.f_code``,
``c.co_filename``, etc.).  When an object with dynamic attributes
exposes these through its ``__dict__`` attribute, ``__dict__`` is a static
attribute.

The names and values of dynamic properties are typically stored in
a dictionary, and this dictionary is typically accessible as
``obj.__dict__``.  The rest of this specification is more concerned
with discovering the names and properties of static attributes
than with dynamic attributes; the latter are easily discovered by
inspection of ``obj.__dict__``.

In the discussion below, I distinguish two kinds of objects:
regular objects (like lists, ints, functions) and meta-objects.
Types and classes are meta-objects.  Meta-objects are also regular
objects, but we're mostly interested in them because they are
referenced by the ``__class__`` attribute of regular objects (or by
the ``__bases__`` attribute of other meta-objects).

The class introspection API consists of the following elements:

- the ``__class__`` and ``__dict__`` attributes on regular objects;

- the ``__bases__`` and ``__dict__`` attributes on meta-objects;


id: pep-0522.rst
content:
PEP: 522
Title: Allow BlockingIOError in security sensitive APIs
Author: Alyssa Coghlan <ncoghlan@gmail.com>, Nathaniel J. Smith <njs@pobox.com>
Status: Rejected
Type: Standards Track
Requires: 506
Created: 16-Jun-2016
Python-Version: 3.6
Resolution: https://mail.python.org/pipermail/security-sig/2016-August/000101.html


Abstract
========

A number of APIs in the standard library that return random values nominally
suitable for use in security sensitive operations currently have an obscure
operating system dependent failure mode that allows them to return values that
are not, in fact, suitable for such operations.

This is due to some operating system kernels (most notably the Linux kernel)
permitting reads from ``/dev/urandom`` before the system random number
generator is fully initialized, whereas most other operating systems will
implicitly block on such reads until the random number generator is ready.

For the lower level ``os.urandom`` and ``random.SystemRandom`` APIs, this PEP
proposes changing such failures in Python 3.6 from the current silent,
hard to detect, and hard to debug, errors to easily detected and debugged errors
by raising ``BlockingIOError`` with a suitable error message, allowing
developers the opportunity to unambiguously specify their preferred approach
for handling the situation.

For the new high level ``secrets`` API, it proposes to block implicitly if
needed whenever random number is generated by that module, as well as to
expose a new ``secrets.wait_for_system_rng()`` function to allow code otherwise
using the low level APIs to explicitly wait for the system random number
generator to be available.

This change will impact any operating system that offers the ``getrandom()``
system call, regardless of whether the default behaviour of the
``/dev/urandom`` device is to return potentially predictable results when the
system random number generator is not ready (e.g. Linux, NetBSD) or to block
(e.g. FreeBSD, Solaris, Illumos). Operating systems that prevent execution of
userspace code prior to the initialization of the system random number
generator, or do not offer the ``getrandom()`` syscall, will be entirely
unaffected by the proposed change (e.g. Windows, Mac OS X, OpenBSD).

The new exception or the blocking behaviour in the ``secrets`` module would
potentially be encountered in the following situations:

* Python code calling these APIs during Linux system initialization
* Python code running on improperly initialized Linux systems (e.g. embedded
  hardware without adequate sources of entropy to seed the system random number
  generator, or Linux VMs that aren't configured to accept entropy from the
  VM host)


Relationship with other PEPs
============================

This PEP depends on the Accepted :pep:`506`, which adds the ``secrets`` module.

This PEP competes with Victor Stinner's :pep:`524`, which proposes to make
``os.urandom`` itself implicitly block when the system RNG is not ready.


PEP Rejection
=============

For the reference implementation, Guido rejected this PEP in favour of the
unconditional implicit blocking proposal in :pep:`524` (which brings CPython's
behaviour on Linux into line with its behaviour on other operating systems).

This means any further discussion of appropriate default behaviour for
``os.urandom()`` in system Python installations in Linux distributions should
take place on the respective distro mailing lists, rather than on the upstream
CPython mailing lists.


Changes independent of this PEP
===============================

CPython interpreter initialization and ``random`` module initialization have
already been updated to gracefully fall back to alternative seeding options if
the system random number generator is not ready.

This PEP does not compete with the proposal in :pep:`524` to add an
``os.getrandom()`` API to expose the ``getrandom`` syscall on platforms that
offer it. There is sufficient motive for adding that API in the ``os`` module's
role as a thin wrapper around potentially platform dependent operating system
features that it can be added regardless of what happens to the default
behaviour of ``os.urandom()`` on these systems.


Proposal
========

Changing ``os.urandom()`` on platforms with the getrandom() system call
-----------------------------------------------------------------------

This PEP proposes that in Python 3.6+, ``os.urandom()`` be updated to call
the ``getrandom()`` syscall in non-blocking mode if available and raise
``BlockingIOError: system random number generator is not ready; see secrets.token_bytes()``
if the kernel reports that the call would block.

This behaviour will then propagate through to the existing
``random.SystemRandom``, which provides a relatively thin wrapper around
``os.urandom()`` that matches the ``random.Random()`` API.

However, the new ``secrets`` module introduced by :pep:`506` will be updated to
catch the new exception and implicitly wait for the system random number
generator if the exception is ever encountered.

In all cases, as soon as a call to one of these security sensitive APIs
succeeds, all future calls to these APIs in that process will succeed
without blocking (once the operating system random number generator is ready
after system boot, it remains ready).

On Linux and NetBSD, this will replace the previous behaviour of returning
potentially predictable results read from ``/dev/urandom``.

On FreeBSD, Solaris, and Illumos, this will replace the previous behaviour of
implicitly blocking until the system random number generator is ready. However,
it is not clear if these operating systems actually allow userspace code (and
hence Python) to run before the system random number generator is ready.

Note that in all cases, if calling the underlying ``getrandom()`` API reports
``ENOSYS`` rather than returning a successful response or reporting ``EAGAIN``,
CPython will continue to fall back to reading from ``/dev/urandom`` directly.


Adding ``secrets.wait_for_system_rng()``
----------------------------------------

A new exception shouldn't be added without a straightforward recommendation
for how to resolve that error when encountered (however rare encountering
the new error is expected to be in practice). For security sensitive code that
actually does need to use the lower level interfaces to the system random
number generator (rather than the new ``secrets`` module), and does receive
live bug reports indicating this is a real problem for the userbase of that
particular application rather than a theoretical one, this PEP's recommendation
will be to add the following snippet (directly or indirectly) to the
``__main__`` module::

    import secrets
    secrets.wait_for_system_rng()

Or, if compatibility with versions prior to Python 3.6 is needed::

    try:
        import secrets
    except ImportError:
        pass
    else:
        secrets.wait_for_system_rng()

Within the ``secrets`` module itself, this will then be used in
``token_bytes()`` to block implicitly if the new exception is encountered::

    def token_bytes(nbytes=None):
        if nbytes is None:
            nbytes = DEFAULT_ENTROPY
        try:
            result = os.urandom(nbytes)
        except BlockingIOError:
            wait_for_system_rng()
            result = os.urandom(nbytes)
        return result

Other parts of the module will then be updated to use ``token_bytes()`` as
their basic random number generation building block, rather than calling
``os.urandom()`` directly.

Application frameworks covering use cases where access to the system random
number generator is almost certain to be needed (e.g. web frameworks) may
choose to incorporate a call to ``secrets.wait_for_system_rng()`` implicitly
into the commands that start the application such that existing calls to
``os.urandom()`` will be guaranteed to never raise the new exception when using
those frameworks.

For cases where the error is encountered for an application which cannot


id: pep-0244.rst
content:
PEP: 244
Title: The ``directive`` statement
Author: Martin von LÃ¶wis <martin@v.loewis.de>
Status: Rejected
Type: Standards Track
Created: 20-Mar-2001
Python-Version: 2.1
Post-History:


Motivation
==========

From time to time, Python makes an incompatible change to the
advertised semantics of core language constructs, or changes their
accidental (implementation-dependent) behavior in some way.  While
this is never done capriciously, and is always done with the aim
of improving the language over the long term, over the short term
it's contentious and disrupting.

:pep:`5`, Guidelines for Language Evolution suggests ways to ease
the pain, and this PEP introduces some machinery in support of
that.

:pep:`227`, Statically Nested Scopes is the first application, and
will be used as an example here.

When a new, potentially incompatible language feature is added,
some modules and libraries may chose to use it, while others may
not.  This specification introduces a syntax where a module author
can denote whether a certain language feature is used in the
module or not.

In discussion of this PEP, readers commented that there are two
kinds of "settable" language features:

- those that are designed to eventually become the only option, at
  which time specifying use of them is not necessary anymore.  The
  features for which the syntax of the "Back to the ``__future__``"
  :pep:`236`, Back to the ``__future__`` was proposed fall into this
  category.  This PEP supports declaring such features, and
  supports phasing out the "old" meaning of constructs whose
  semantics has changed under the new feature.  However, it
  defines no policy as to what features must be phased out
  eventually.

- those which are designed to stay optional forever, e.g. if they
  change some default setting in the interpreter.  An example for
  such settings might be the request to always emit line-number
  instructions for a certain module; no specific flags of that
  kind are proposed in this specification.

Since a primary goal of this PEP is to support new language
constructs without immediately breaking old libraries, special
care was taken not to break old libraries by introducing the new
syntax.


Syntax
======

A directive_statement is a statement of the form::

   directive_statement: 'directive' ``NAME`` [atom] [';'] NEWLINE

The name in the directive indicates the kind of the directive; it
defines whether the optional atom can be present, and whether
there are further syntactical or semantical restrictions to the
atom.  In addition, depending on the name of the directive,
certain additional syntactical or semantical restrictions may be
placed on the directive (e.g. placement of the directive in the
module may be restricted to the top of the module).

In the directive_statement, ``directive`` is a new
keyword. According to :pep:`5`, this keyword is initially considered as
a keyword only when used in a directive statement, see "Backwards
Compatibility" below.


Semantics
=========

A directive statement instructs the Python interpreter to process
a source file in a different way; the specific details of that
processing depend on the directive name.  The optional atom is
typically interpreted when the source code is processed; details
of that interpretation depend on the directive.


Specific Directives: transitional
=================================

If a syntactical or semantical change is added to Python which is
incompatible, :pep:`5` mandates a transitional evolution of the
language, where the new feature is initially available alongside
with the old one.  Such a transition is possible by means of the
transitional directive.

In a transitional directive, the ``NAME`` is 'transitional'. The atom
MUST be present, and it MUST be a ``NAME``.  The possible values for
that name are defined when the language change is defined.  One
example for such a directive is::

   directive transitional nested_scopes

The transitional directive MUST occur at before any other
statement in a module, except for the documentation string
(i.e. it may appear as the second statement of a module only if
the first statement is a ``STRING+``).


Backwards Compatibility
=======================

Introducing ``directive`` as a new keyword might cause
incompatibilities with existing code.  Following the guideline in
:pep:`5`, in the initial implementation of this specification,
directive is a new keyword only if it was used in a valid
directive_statement (i.e. if it appeared as the first non-string
token in a module).


Unresolved Problems:  directive as the first identifier
=======================================================

Using directive in a module as::

    directive = 1

(i.e. the name directive appears as the first thing in a module)
will treat it as keyword, not as identifier. It would be possible
to classify it as a ``NAME`` with an additional look-ahead token, but
such look-ahead is not available in the Python tokenizer.


Questions and Answers
=====================

**Q:** It looks like this PEP was written to allow definition of source
code character sets.  Is that true?

**A:** No.  Even though the directive facility can be extended to
allow source code encodings, no specific directive is proposed.

**Q:** Then why was this PEP written at all?

**A:** It acts as a counter-proposal to :pep:`236`, which proposes to
overload the import statement with a new meaning.  This PEP
allows to solve the problem in a more general way.

**Q:** But isn't mixing source encodings and language changes like
mixing apples and oranges?

**A:** Perhaps.  To address the difference, the predefined
"transitional" directive has been defined.


Copyright
=========

This document has been placed in the public domain.


id: pep-0250.rst
content:
PEP: 250
Title: Using site-packages on Windows
Author: Paul Moore <p.f.moore@gmail.com>
Status: Final
Type: Standards Track
Created: 30-Mar-2001
Python-Version: 2.2
Post-History: 30-Mar-2001


Abstract
========

The standard Python distribution includes a directory
``Lib/site-packages``, which is used on Unix platforms to hold
locally installed modules and packages.  The ``site.py`` module
distributed with Python includes support for locating other
modules in the site-packages directory.

This PEP proposes that the site-packages directory should be used
on the Windows platform in a similar manner.


Motivation
==========

On Windows platforms, the default setting for ``sys.path`` does not
include a directory suitable for users to install locally
developed modules.  The "expected" location appears to be the
directory containing the Python executable itself.  This is also
the location where distutils (and distutils-generated installers)
installs packages.  Including locally developed code in the same
directory as installed executables is not good practice.

Clearly, users can manipulate ``sys.path``, either in a locally
modified ``site.py``, or in a suitable ``sitecustomize.py``, or even via
``.pth`` files.  However, there should be a standard location for such
files, rather than relying on every individual site having to set
their own policy.

In addition, with distutils becoming more prevalent as a means of
distributing modules, the need for a standard install location for
distributed modules will become more common.  It would be better
to define such a standard now, rather than later when more
distutils-based packages exist which will need rebuilding.

It is relevant to note that prior to Python 2.1, the site-packages
directory was not included in ``sys.path`` for Macintosh platforms.
This has been changed in 2.1, and Macintosh includes ``sys.path`` now,
leaving Windows as the only major platform with no site-specific
modules directory.


Implementation
==============

The implementation of this feature is fairly trivial.  All that
would be required is a change to ``site.py``, to change the section
setting sitedirs.  The Python 2.1 version has::

    if os.sep == '/':
        sitedirs = [makepath(prefix,
                            "lib",
                            "python" + sys.version[:3],
                            "site-packages"),
                    makepath(prefix, "lib", "site-python")]
    elif os.sep == ':':
        sitedirs = [makepath(prefix, "lib", "site-packages")]
    else:
        sitedirs = [prefix]

A suitable change would be to simply replace the last 4 lines with::

    else:
        sitedirs == [prefix, makepath(prefix, "lib", "site-packages")]

Changes would also be required to distutils, to reflect this change
in policy. A patch is available on Sourceforge, patch ID 445744,
which implements this change.  Note that the patch checks the Python
version and only invokes the new behaviour for Python versions from
2.2 onwards. This is to ensure that distutils remains compatible
with earlier versions of Python.

Finally, the executable code which implements the Windows installer
used by the ``bdist_wininst`` command will need changing to use the new
location.  A separate patch is available for this, currently
maintained by Thomas Heller.


Notes
=====

- This change does not preclude packages using the current
  location -- the change only adds a directory to ``sys.path``, it
  does not remove anything.

- Both the current location (``sys.prefix``) and the new directory
  (site-packages) are included in sitedirs, so that ``.pth`` files
  will be recognised in either location.

- This proposal adds a single additional site-packages directory
  to sitedirs. On Unix platforms, two directories are added, one
  for version-independent files (Python code) and one for
  version-dependent code (C extensions). This is necessary on
  Unix, as the sitedirs include a common (across Python versions)
  package location, in ``/usr/local`` by default. As there is no such
  common location available on Windows, there is also no need for
  having two separate package directories.

- If users want to keep DLLs in a single location on Windows, rather
  than keeping them in the package directory, the DLLs subdirectory
  of the Python install directory is already available for that
  purpose. Adding an extra directory solely for DLLs should not be
  necessary.


Open Issues
===========

- Comments from Unix users indicate that there may be issues with
  the current setup on the Unix platform.  Rather than become
  involved in cross-platform issues, this PEP specifically limits
  itself to the Windows platform, leaving changes for other platforms
  to be covered in other PEPs.

- There could be issues with applications which embed Python. To the
  author's knowledge, there should be no problem as a result of this
  change. There have been no comments (supportive or otherwise) from
  users who embed Python.


Copyright
=========

This document has been placed in the public domain.


id: pep-0536.rst
content:
PEP: 536
Title: Final Grammar for Literal String Interpolation
Author: Philipp Angerer <phil.angerer@gmail.com>
Status: Withdrawn
Type: Standards Track
Created: 11-Dec-2016
Python-Version: 3.7
Post-History: `18-Aug-2016 <https://mail.python.org/archives/list/python-ideas@python.org/thread/FOYKXOFWEINPVQSK2XGEHKXSTEVO5WWA/>`__,
              `23-Dec-2016 <https://mail.python.org/archives/list/python-ideas@python.org/thread/YKKEA5NIMMKHZTMRE5UFHST4WQ4NN3XJ/>`__,
              `15-Mar-2019 <https://mail.python.org/archives/list/python-dev@python.org/thread/N43O4KNLZW4U7YZC4NVPCETZIVRDUVU2/>`__
Resolution: https://discuss.python.org/t/pep-536-should-be-marked-as-rejected/35226/4

Abstract
========

:pep:`498` introduced Literal String Interpolation (or âf-stringsâ).
The expression portions of those literals however are subject to
certain restrictions.  This PEP proposes a formal grammar lifting
those restrictions, promoting âf-stringsâ to âf expressionsâ or f-literals.

This PEP expands upon the f-strings introduced by :pep:`498`,
so this text requires familiarity with :pep:`498`.

PEP Withdrawal
==============

This PEP has been withdrawn in favour of :pep:`701`.
:pep:`701` addresses all important points of this PEP.


Terminology
===========

This text will refer to the existing grammar as âf-stringsâ,
and the proposed one as âf-literalsâ.

Furthermore, it will refer to the ``{}``-delimited expressions in
f-literals/f-strings as âexpression portionsâ and the static string content
around them as âstring portionsâ.

Motivation
==========

The current implementation of f-strings in CPython relies on the existing
string parsing machinery and a post processing of its tokens.  This results in
several restrictions to the possible expressions usable within f-strings:

#. It is impossible to use the quote character delimiting the f-string
   within the expression portion::

    >>> f'Magic wand: { bag['wand'] }'
                                 ^
    SyntaxError: invalid syntax

#. A previously considered way around it would lead to escape sequences
   in executed code and is prohibited in f-strings::

    >>> f'Magic wand { bag[\'wand\'] } string'
    SyntaxError: f-string expression portion cannot include a backslash

#. Comments are forbidden even in multi-line f-strings::

    >>> f'''A complex trick: {
    ... bag['bag']  # recursive bags!
    ... }'''
    SyntaxError: f-string expression part cannot include '#'

#. Expression portions need to wrap ``':'`` and ``'!'`` in braces::

    >>> f'Useless use of lambdas: { lambda x: x*2 }'
    SyntaxError: unexpected EOF while parsing

These limitations serve no purpose from a language user perspective and
can be lifted by giving f-literals a regular grammar without exceptions
and implementing it using dedicated parse code.

Rationale
=========

.. https://mail.python.org/pipermail/python-ideas/2016-August/041727.html

The restrictions mentioned in Motivation_ are non-obvious and counter-intuitive
unless the user is familiar with the f-literalsâ implementation details.

As mentioned, a previous version of :pep:`498` allowed escape sequences
anywhere in f-strings, including as ways to encode the braces delimiting
the expression portions and in their code.  They would be expanded before
the code is parsed, which would have had several important ramifications:

#. It would not be clear to human readers which portions are Expressions
and which are strings.  Great material for an âobfuscated/underhanded
Python challengeâ
#. Syntax highlighters are good in parsing nested grammar, but not
in recognizing escape sequences.  ECMAScript 2016 (JavaScript) allows
escape sequences in its identifiers [1]_ and the author knows of no
syntax highlighter able to correctly highlight code making use of this.

As a consequence, the expression portions would be harder to recognize
with and without the aid of syntax highlighting.  With the new grammar,
it is easy to extend syntax highlighters to correctly parse
and display f-literals:

.. raw:: html

   <pre><span style=color:#ff5500>f'Magic wand: </span><span style=color:#3daee9>{</span>bag[<span style=color:#bf0303>'wand'</span>]<span style=color:#3daee9>:^10}</span><span style=color:#ff5500>'</span></pre>

.. This is the output of kate-syntax-highlighter when given that code
   (with some quotes stripped)

Highlighting expression portions with possible escape sequences would
mean to create a modified copy of all rules of the complete expression
grammar, accounting for the possibility of escape sequences in key words,
delimiters, and all other language syntax. One such duplication would
yield one level of escaping depth and have to be repeated for a deeper
escaping in a recursive f-literal. This is the case since no highlighting
engine known to the author supports expanding escape sequences before
applying rules to a certain context. Nesting contexts however is a
standard feature of all highlighting engines.

Familiarity also plays a role: Arbitrary nesting of expressions
without expansion of escape sequences is available in every single
other language employing a string interpolation method that uses
expressions instead of just variable names. [2]_

Specification
=============

:pep:`498` specified f-strings as the following, but places restrictions on it::

    f ' <text> { <expression> <optional !s, !r, or !a> <optional : format specifier> } <text> ... '

All restrictions mentioned in the PEP are lifted from f-literals,
as explained below:

#. Expression portions may now contain strings delimited with the same
   kind of quote that is used to delimit the f-literal.
#. Backslashes may now appear within expressions just like anywhere else
   in Python code.  In case of strings nested within f-literals,
   escape sequences are expanded when the innermost string is evaluated.
#. Comments, using the ``'#'`` character, are possible only in multi-line
   f-literals, since comments are terminated by the end of the line
   (which makes closing a single-line f-literal impossible).
#. Expression portions may contain ``':'`` or ``'!'`` wherever
   syntactically valid.  The first ``':'`` or ``'!'`` that is not part
   of an expression has to be followed a valid coercion or format specifier.

A remaining restriction not explicitly mentioned by :pep:`498` is line breaks
in expression portions.  Since strings delimited by single ``'`` or ``"``
characters are expected to be single line, line breaks remain illegal
in expression portions of single line strings.

.. note:: Is lifting of the restrictions sufficient,
   or should we specify a more complete grammar?

Backwards Compatibility
=======================

f-literals are fully backwards compatible to f-strings,
and expands the syntax considered legal.

Reference Implementation
========================

TBD

References
==========

.. [1] ECMAScript ``IdentifierName`` specification
   ( http://ecma-international.org/ecma-262/6.0/#sec-names-and-keywords )

   Yes, ``const cthulhu = { HÌ¹ÌÌ¦Ì®ÍÌ©ÌÌÍ§ÌÌÌÌ¾EÍ¨ÍÍÌÍ®ÌÍÌ·Ì®Ì£Ì«Ì¤Ì£CÍ¯ÌÍÍÌ¨ÌÍÌ¦ÌÍÌ»OÌÍÍÍÍÌ¬ÌÌ£Ì½Í®ÍÍÌÍ¤ÌÌÍ¢MÌ´Ì¡Ì²Ì­ÍÍÌ¼ÌÌ¯Ì¦ÌÌÍ á¸ÌÌÌÌªÌÍ¥Í¤Í©Ì¾ÍÌÍÍá¹®Ì´Ì·Ì·ÌÌ¼ÍÌ¿Ì¿ÌÌ½ÍHÌÌÌÌÍ\u0042: 42 }`` is valid ECMAScript 2016

.. [2] Wikipedia article on string interpolation
   ( https://en.wikipedia.org/wiki/String_interpolation )

Copyright
=========

This document has been placed in the public domain.


id: pep-0278.rst
content:
PEP: 278
Title: Universal Newline Support
Author: Jack Jansen <jack@cwi.nl>
Status: Final
Type: Standards Track
Created: 14-Jan-2002
Python-Version: 2.3
Post-History:


Abstract
========

This PEP discusses a way in which Python can support I/O on files
which have a newline format that is not the native format on the
platform, so that Python on each platform can read and import
files with CR (Macintosh), LF (Unix) or CR LF (Windows) line
endings.

It is more and more common to come across files that have an end
of line that does not match the standard on the current platform:
files downloaded over the net, remotely mounted filesystems on a
different platform, Mac OS X with its double standard of Mac and
Unix line endings, etc.

Many tools such as editors and compilers already handle this
gracefully, it would be good if Python did so too.


Specification
=============

Universal newline support is enabled by default,
but can be disabled during the configure of Python.

In a Python with universal newline support the feature is
automatically enabled for all import statements and ``execfile()``
calls. There is no special support for ``eval()`` or exec.

In a Python with universal newline support ``open()`` the mode
parameter can also be "U", meaning "open for input as a text file
with universal newline interpretation".  Mode "rU" is also allowed,
for symmetry with "rb". Mode "U" cannot be
combined with other mode flags such as "+". Any line ending in the
input file will be seen as a ``'\n'`` in Python, so little other code has
to change to handle universal newlines.

Conversion of newlines happens in all calls that read data: ``read()``,
``readline()``, ``readlines()``, etc.

There is no special support for output to file with a different
newline convention, and so mode "wU" is also illegal.

A file object that has been opened in universal newline mode gets
a new attribute "newlines" which reflects the newline convention
used in the file.  The value for this attribute is one of None (no
newline read yet), ``"\r"``, ``"\n"``, ``"\r\n"`` or a tuple containing all the
newline types seen.


Rationale
=========

Universal newline support is implemented in C, not in Python.
This is done because we want files with a foreign newline
convention to be import-able, so a Python Lib directory can be
shared over a remote file system connection, or between MacPython
and Unix-Python on Mac OS X.  For this to be feasible the
universal newline convention needs to have a reasonably small
impact on performance, which means a Python implementation is not
an option as it would bog down all imports. And because of files
with multiple newline conventions, which Visual C++ and other
Windows tools will happily produce, doing a quick check for the
newlines used in a file (handing off the import to C code if a
platform-local newline is seen) will not work.  Finally, a C
implementation also allows tracebacks and such (which open the
Python source module) to be handled easily.

There is no output implementation of universal newlines, Python
programs are expected to handle this by themselves or write files
with platform-local convention otherwise.  The reason for this is
that input is the difficult case, outputting different newlines to
a file is already easy enough in Python.

Also, an output implementation would be much more difficult than an
input implementation, surprisingly: a lot of output is done through
``PyXXX_Print()`` methods, and at this point the file object is not
available anymore, only a ``FILE *``. So, an output implementation would
need to somehow go from the ``FILE*`` to the file object, because that
is where the current newline delimiter is stored.

The input implementation has no such problem: there are no cases in
the Python source tree where files are partially read from C,
partially from Python, and such cases are expected to be rare in
extension modules. If such cases exist the only problem is that the
newlines attribute of the file object is not updated during the
``fread()`` or ``fgets()`` calls that are done direct from C.

A partial output implementation, where strings passed to ``fp.write()``
would be converted to use ``fp.newlines`` as their line terminator but
all other output would not is far too surprising, in my view.

Because there is no output support for universal newlines there is
also no support for a mode "rU+": the surprise factor of the
previous paragraph would hold to an even stronger degree.

There is no support for universal newlines in strings passed to
``eval()`` or ``exec``. It is envisioned that such strings always have the
standard ``\n`` line feed, if the strings come from a file that file can
be read with universal newlines.

I think there are no special issues with unicode. utf-16 shouldn't
pose any new problems, as such files need to be opened in binary
mode anyway. Interaction with utf-8 is fine too: values 0x0a and 0x0d
cannot occur as part of a multibyte sequence.

Universal newline files should work fine with iterators and
``xreadlines()`` as these eventually call the normal file
readline/readlines methods.


While universal newlines are automatically enabled for import they
are not for opening, where you have to specifically say ``open(...,
"U")``. This is open to debate, but here are a few reasons for this
design:

- Compatibility.  Programs which already do their own
  interpretation of ``\r\n`` in text files would break. Examples of such
  programs would be editors which warn you when you open a file with
  a different newline convention. If universal newlines was made the
  default such an editor would silently convert your line endings to
  the local convention on save. Programs which open binary files as
  text files on Unix would also break (but it could be argued they
  deserve it :-).

- Interface clarity.  Universal newlines are only supported for
  input files, not for input/output files, as the semantics would
  become muddy.  Would you write Mac newlines if all reads so far
  had encountered Mac newlines?  But what if you then later read a
  Unix newline?

The ``newlines`` attribute is included so that programs that really
care about the newline convention, such as text editors, can
examine what was in a file.  They can then save (a copy of) the
file with the same newline convention (or, in case of a file with
mixed newlines, ask the user what to do, or output in platform
convention).

Feedback is explicitly solicited on one item in the reference
implementation: whether or not the universal newlines routines
should grab the global interpreter lock.  Currently they do not,
but this could be considered living dangerously, as they may
modify fields in a ``FileObject``.  But as these routines are
replacements for ``fgets()`` and ``fread()`` as well it may be difficult
to decide whether or not the lock is held when the routine is
called.  Moreover, the only danger is that if two threads read the
same ``FileObject`` at the same time an extraneous newline may be seen
or the ``newlines`` attribute may inadvertently be set to mixed.  I
would argue that if you read the same ``FileObject`` in two threads
simultaneously you are asking for trouble anyway.

Note that no globally accessible pointers are manipulated in the
``fgets()`` or ``fread()`` replacement routines, just some integer-valued
flags, so the chances of core dumps are zero (he said:-).

Universal newline support can be disabled during configure because it does
have a small performance penalty, and moreover the implementation has
not been tested on all conceivable platforms yet. It might also be silly
on some platforms (WinCE or Palm devices, for instance). If universal
newline support is not enabled then file objects do not have the ``newlines``
attribute, so testing whether the current Python has it can be done with a
simple::

   if hasattr(open, 'newlines'):
      print 'We have universal newline support'

Note that this test uses the ``open()`` function ra


id: pep-0287.rst
content:
PEP: 287
Title: reStructuredText Docstring Format
Author: David Goodger <goodger@python.org>
Discussions-To: doc-sig@python.org
Status: Active
Type: Informational
Created: 25-Mar-2002
Post-History: 02-Apr-2002
Replaces: 216


Abstract
========

When plaintext hasn't been expressive enough for inline documentation,
Python programmers have sought out a format for docstrings.  This PEP
proposes that the `reStructuredText markup`_ be adopted as a standard
markup format for structured plaintext documentation in Python
docstrings, and for PEPs and ancillary documents as well.
reStructuredText is a rich and extensible yet easy-to-read,
what-you-see-is-what-you-get plaintext markup syntax.

Only the low-level syntax of docstrings is addressed here.  This PEP
is not concerned with docstring semantics or processing at all (see
:pep:`256` for a "Road Map to the Docstring PEPs").  Nor is it an attempt
to deprecate pure plaintext docstrings, which are always going to be
legitimate.  The reStructuredText markup is an alternative for those
who want more expressive docstrings.


Benefits
========

Programmers are by nature a lazy breed.  We reuse code with functions,
classes, modules, and subsystems.  Through its docstring syntax,
Python allows us to document our code from within.  The "holy grail"
of the Python Documentation Special Interest Group (Doc-SIG_) has been
a markup syntax and toolset to allow auto-documentation, where the
docstrings of Python systems can be extracted in context and processed
into useful, high-quality documentation for multiple purposes.

Document markup languages have three groups of customers: the authors
who write the documents, the software systems that process the data,
and the readers, who are the final consumers and the most important
group.  Most markups are designed for the authors and software
systems; readers are only meant to see the processed form, either on
paper or via browser software.  ReStructuredText is different: it is
intended to be easily readable in source form, without prior knowledge
of the markup.  ReStructuredText is entirely readable in plaintext
format, and many of the markup forms match common usage (e.g.,
``*emphasis*``), so it reads quite naturally.  Yet it is rich enough
to produce complex documents, and extensible so that there are few
limits.  Of course, to write reStructuredText documents some prior
knowledge is required.

The markup offers functionality and expressivity, while maintaining
easy readability in the source text.  The processed form (HTML etc.)
makes it all accessible to readers: inline live hyperlinks; live links
to and from footnotes; automatic tables of contents (with live
links!); tables; images for diagrams etc.; pleasant, readable styled
text.

The reStructuredText parser is available now, part of the Docutils_
project.  Standalone reStructuredText documents and PEPs can be
converted to HTML; other output format writers are being worked on and
will become available over time.  Work is progressing on a Python
source "Reader" which will implement auto-documentation from
docstrings.  Authors of existing auto-documentation tools are
encouraged to integrate the reStructuredText parser into their
projects, or better yet, to join forces to produce a world-class
toolset for the Python standard library.

Tools will become available in the near future, which will allow
programmers to generate HTML for online help, XML for multiple
purposes, and eventually PDF, DocBook, and LaTeX for printed
documentation, essentially "for free" from the existing docstrings.
The adoption of a standard will, at the very least, benefit docstring
processing tools by preventing further "reinventing the wheel".

Eventually PyDoc, the one existing standard auto-documentation tool,
could have reStructuredText support added.  In the interim it will
have no problem with reStructuredText markup, since it treats all
docstrings as preformatted plaintext.


Goals
=====

These are the generally accepted goals for a docstring format, as
discussed in the Doc-SIG:

1. It must be readable in source form by the casual observer.

2. It must be easy to type with any standard text editor.

3. It must not need to contain information which can be deduced from
   parsing the module.

4. It must contain sufficient information (structure) so it can be
   converted to any reasonable markup format.

5. It must be possible to write a module's entire documentation in
   docstrings, without feeling hampered by the markup language.

reStructuredText meets and exceeds all of these goals, and sets its
own goals as well, even more stringent.  See `Docstring-Significant
Features`_ below.

The goals of this PEP are as follows:

1. To establish reStructuredText as a standard structured plaintext
   format for docstrings (inline documentation of Python modules and
   packages), PEPs, README-type files and other standalone documents.
   "Accepted" status will be sought through Python community consensus
   and eventual BDFL pronouncement.

   Please note that reStructuredText is being proposed as *a*
   standard, not *the only* standard.  Its use will be entirely
   optional.  Those who don't want to use it need not.

2. To solicit and address any related concerns raised by the Python
   community.

3. To encourage community support.  As long as multiple competing
   markups are out there, the development community remains fractured.
   Once a standard exists, people will start to use it, and momentum
   will inevitably gather.

4. To consolidate efforts from related auto-documentation projects.
   It is hoped that interested developers will join forces and work on
   a joint/merged/common implementation.

Once reStructuredText is a Python standard, effort can be focused on
tools instead of arguing for a standard.  Python needs a standard set
of documentation tools.

With regard to PEPs, one or both of the following strategies may be
applied:

a) Keep the existing PEP section structure constructs (one-line
   section headers, indented body text).  Subsections can either be
   forbidden, or supported with reStructuredText-style underlined
   headers in the indented body text.

b) Replace the PEP section structure constructs with the
   reStructuredText syntax.  Section headers will require underlines,
   subsections will be supported out of the box, and body text need
   not be indented (except for block quotes).

Strategy (b) is recommended, and its implementation is complete.

Support for :rfc:`2822` headers has been added to the reStructuredText
parser for PEPs (unambiguous given a specific context: the first
contiguous block of the document).  It may be desired to concretely
specify what over/underline styles are allowed for PEP section
headers, for uniformity.


Rationale
=========

The lack of a standard syntax for docstrings has hampered the
development of standard tools for extracting and converting docstrings
into documentation in standard formats (e.g., HTML, DocBook, TeX).
There have been a number of proposed markup formats and variations,
and many tools tied to these proposals, but without a standard
docstring format they have failed to gain a strong following and/or
floundered half-finished.

Throughout the existence of the Doc-SIG, consensus on a single
standard docstring format has never been reached.  A lightweight,
implicit markup has been sought, for the following reasons (among
others):

1. Docstrings written within Python code are available from within the
   interactive interpreter, and can be "print"ed.  Thus the use of
   plaintext for easy readability.

2. Programmers want to add structure to their docstrings, without
   sacrificing raw docstring readability.  Unadorned plaintext cannot
   be transformed ("up-translated") into useful structured formats.

3. Explicit markup (like XML or TeX) is widely considered unreadable
   by the uninitiated.

4. Implicit markup is aesthetically compatible with the clean and
   minimalist Pyt


id: pep-0293.rst
content:
PEP: 293
Title: Codec Error Handling Callbacks
Author: Walter DÃ¶rwald <walter@livinglogic.de>
Status: Final
Type: Standards Track
Created: 18-Jun-2002
Python-Version: 2.3
Post-History: 19-Jun-2002


Abstract
========

This PEP aims at extending Python's fixed codec error handling
schemes with a more flexible callback based approach.

Python currently uses a fixed error handling for codec error
handlers.  This PEP describes a mechanism which allows Python to
use function callbacks as error handlers.  With these more
flexible error handlers it is possible to add new functionality to
existing codecs by e.g. providing fallback solutions or different
encodings for cases where the standard codec mapping does not
apply.


Specification
=============

Currently the set of codec error handling algorithms is fixed to
either "strict", "replace" or "ignore" and the semantics of these
algorithms is implemented separately for each codec.

The proposed patch will make the set of error handling algorithms
extensible through a codec error handler registry which maps
handler names to handler functions.  This registry consists of the
following two C functions::

    int PyCodec_RegisterError(const char *name, PyObject *error)

    PyObject *PyCodec_LookupError(const char *name)

and their Python counterparts::

    codecs.register_error(name, error)

    codecs.lookup_error(name)

``PyCodec_LookupError`` raises a ``LookupError`` if no callback function
has been registered under this name.

Similar to the encoding name registry there is no way of
unregistering callback functions or iterating through the
available functions.

The callback functions will be used in the following way by the
codecs: when the codec encounters an encoding/decoding error, the
callback function is looked up by name, the information about the
error is stored in an exception object and the callback is called
with this object.  The callback returns information about how to
proceed (or raises an exception).

For encoding, the exception object will look like this::

    class UnicodeEncodeError(UnicodeError):
        def __init__(self, encoding, object, start, end, reason):
            UnicodeError.__init__(self,
                "encoding '%s' can't encode characters " +
                "in positions %d-%d: %s" % (encoding,
                    start, end-1, reason))
            self.encoding = encoding
            self.object = object
            self.start = start
            self.end = end
            self.reason = reason

This type will be implemented in C with the appropriate setter and
getter methods for the attributes, which have the following
meaning:

* ``encoding``: The name of the encoding;
* ``object``: The original unicode object for which ``encode()`` has
  been called;
* ``start``: The position of the first unencodable character;
* ``end``: (The position of the last unencodable character)+1 (or
  the length of object, if all characters from start to the end
  of object are unencodable);
* ``reason``: The reason why ``object[start:end]`` couldn't be encoded.

If object has consecutive unencodable characters, the encoder
should collect those characters for one call to the callback if
those characters can't be encoded for the same reason.  The
encoder is not required to implement this behaviour but may call
the callback for every single character, but it is strongly
suggested that the collecting method is implemented.

The callback must not modify the exception object.  If the
callback does not raise an exception (either the one passed in, or
a different one), it must return a tuple::

    (replacement, newpos)

replacement is a unicode object that the encoder will encode and
emit instead of the unencodable ``object[start:end]`` part, newpos
specifies a new position within object, where (after encoding the
replacement) the encoder will continue encoding.

Negative values for newpos are treated as being relative to
end of object. If newpos is out of bounds the encoder will raise
an ``IndexError``.

If the replacement string itself contains an unencodable character
the encoder raises the exception object (but may set a different
reason string before raising).

Should further encoding errors occur, the encoder is allowed to
reuse the exception object for the next call to the callback.
Furthermore, the encoder is allowed to cache the result of
``codecs.lookup_error``.

If the callback does not know how to handle the exception, it must
raise a ``TypeError``.

Decoding works similar to encoding with the following differences:

* The exception class is named ``UnicodeDecodeError`` and the attribute
  object is the original 8bit string that the decoder is currently
  decoding.

* The decoder will call the callback with those bytes that
  constitute one undecodable sequence, even if there is more than
  one undecodable sequence that is undecodable for the same reason
  directly after the first one.  E.g. for the "unicode-escape"
  encoding, when decoding the illegal string ``\\u00\\u01x``, the
  callback will be called twice (once for ``\\u00`` and once for
  ``\\u01``).  This is done to be able to generate the correct number
  of replacement characters.

* The replacement returned from the callback is a unicode object
  that will be emitted by the decoder as-is without further
  processing instead of the undecodable ``object[start:end]`` part.

There is a third API that uses the old strict/ignore/replace error
handling scheme::

    PyUnicode_TranslateCharmap/unicode.translate

The proposed patch will enhance ``PyUnicode_TranslateCharmap``, so
that it also supports the callback registry.  This has the
additional side effect that ``PyUnicode_TranslateCharmap`` will
support multi-character replacement strings (see SF feature
request #403100 [1]_).

For ``PyUnicode_TranslateCharmap`` the exception class will be named
``UnicodeTranslateError``.  ``PyUnicode_TranslateCharmap`` will collect
all consecutive untranslatable characters (i.e. those that map to
``None``) and call the callback with them.  The replacement returned
from the callback is a unicode object that will be put in the
translated result as-is, without further processing.

All encoders and decoders are allowed to implement the callback
functionality themselves, if they recognize the callback name
(i.e. if it is a system callback like "strict", "replace" and
"ignore").  The proposed patch will add two additional system
callback names: "backslashreplace" and "xmlcharrefreplace", which
can be used for encoding and translating and which will also be
implemented in-place for all encoders and
``PyUnicode_TranslateCharmap``.

The Python equivalent of these five callbacks will look like this::

    def strict(exc):
        raise exc

    def ignore(exc):
        if isinstance(exc, UnicodeError):
            return (u"", exc.end)
        else:
            raise TypeError("can't handle %s" % exc.__name__)

    def replace(exc):
         if isinstance(exc, UnicodeEncodeError):
             return ((exc.end-exc.start)*u"?", exc.end)
         elif isinstance(exc, UnicodeDecodeError):
             return (u"\\ufffd", exc.end)
         elif isinstance(exc, UnicodeTranslateError):
             return ((exc.end-exc.start)*u"\\ufffd", exc.end)
         else:
             raise TypeError("can't handle %s" % exc.__name__)

    def backslashreplace(exc):
         if isinstance(exc,
             (UnicodeEncodeError, UnicodeTranslateError)):
             s = u""
             for c in exc.object[exc.start:exc.end]:
                if ord(c)<=0xff:
                    s += u"\\x%02x" % ord(c)
                elif ord(c)<=0xffff:
                    s += u"\\u%04x" % ord(c)
                else:
                    s += u"\\U%08x" % ord(c)
             return (s, exc.end)
         else:
             raise TypeError("can't handle %s" % exc.__name__)

    def xmlcharrefreplace(exc):
         if isinstance(exc,
             (UnicodeEncodeError, UnicodeTranslateErr


id: pep-0720.rst
content:
PEP: 720
Title: Cross-compiling Python packages
Author: Filipe LaÃ­ns <lains@python.org>
PEP-Delegate:
Status: Draft
Type: Informational
Created: 01-Jul-2023
Python-Version: 3.12


Abstract
========

This PEP attempts to document the status of cross-compilation of downstream
projects.

It should give an overview of the approaches currently used by distributors
(Linux distros, WASM environment providers, etc.) to cross-compile downstream
projects (3rd party extensions, etc.).


Motivation
==========

We write this PEP to express the challenges in cross-compilation and act as a
supporting document in future improvement proposals.


Analysis
========


Introduction
------------

There are a couple different approaches being used to tackle this, with
different levels of interaction required from the user, but they all require a
significant amount of effort. This is due to the lack of standardized
cross-compilation infrastructure on the Python packaging ecosystem, which itself
stems from the complexity of cross-builds, making it a huge undertaking.


Upstream support
----------------

Some major projects like CPython, setuptools, etc. provide some support to help
with cross-compilation, but it's unofficial and at a best-effort basis. For
example, the ``sysconfig`` module allows overwriting the data module name via
the ``_PYTHON_SYSCONFIGDATA_NAME`` environment variable, something that is
required for cross-builds, and setuptools `accepts patches`__ [1]_ to tweak/fix
its logic to be compatible with popular `"environment faking"
<faking the target environment_>`_ workflows [2]_.

The lack of first-party support in upstream projects leads to cross-compilation
being fragile and requiring a significant effort from users, but at the same
time, the lack of standardization makes it harder for upstreams to improve
support as there's no clarity on how this feature should be provided.

.. [1] At the time of writing (Jun 2023), setuptools' compiler interface code,
       the component that most of affects cross-compilation, is developed on the
       `pypa/distutils`__ repository, which gets periodically synced to the
       setuptools repository.

.. [2] We specifically mention *popular* workflows, because this is not
       standardized. Though, many of the most popular implementations
       (crossenv_, conda-forge_'s build system, etc.) work similarly, and this
       is what we are referring to here. For clarity, the implementations we are
       referring to here could be described as *crossenv-style*.

.. __: https://github.com/pypa/distutils/pulls?q=cross
.. __: https://github.com/pypa/distutils

Projects with decent cross-build support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

It seems relevant to point out that there are a few modern Python package
build-backends with, at least, decent cross-compilation support, those being
scikit-build__ and meson-python__. Both these projects integrate external mature
build-systems into Python packaging â CMake__ and Meson__, respectively â so
cross-build support is inherited from them.

.. __: https://github.com/scikit-build/scikit-build
.. __: https://github.com/mesonbuild/meson-python
.. __: https://cmake.org/
.. __: https://mesonbuild.com/


Downstream approaches
---------------------

Cross-compilation approaches fall in a spectrum that goes from, by design,
requiring extensive user interaction to (ideally) almost none. Usually, they'll
be based on one of two main strategies, using a `cross-build environment`_,
or `faking the target environment`_.

.. _approach-cross-environment:

Cross-build environment
~~~~~~~~~~~~~~~~~~~~~~~

This consists of running the Python interpreter normally and utilizing the
cross-build provided by the projects' build-system. However, as we saw above,
upstream support is lacking, so this approach only works for a small-ish set of
projects. When this fails, the usual strategy is to patch the build-system code
to build use the correct toolchain, system details, etc. [3]_.

Since this approach often requires package-specific patching, it requires a lot
of user interaction.

.. admonition:: Examples
   :class: note

   `python-for-android`_, `kivy-ios`_, etc.

.. [3] The scope of the build-system patching varies between users and usually
       depends on the their goal â some (eg. Linux distributions) may patch the
       build-system to support cross-builds, while others might hardcode
       compiler paths and system information in the build-system, to simply make
       the build work.

Faking the target environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Aiming to drop the requirement for user input, a popular approach is trying to
fake the target environment. It generally consists of monkeypatching the Python
interpreter to get it to mimic the interpreter on the target system, which
constitutes of changing many of the ``sys`` module attributes, the ``sysconfig``
data, etc. Using this strategy, build-backends do not need to have any
cross-build support, and should just work without any code changes.

Unfortunately, though, it isn't possible to truly fake the target environment.
There are many reasons for this, one of the main ones being that it breaks code
that actually needs to introspect the running interpreter. As a result,
monkeypatching Python to look like target is very tricky â to achieve the less
amount of breakage, we can only patch certain aspects of the interpreter.
Consequently, build-backends may need some code changes, but these are generally
much smaller than the previous approach. This is an inherent limitation of the
technique, meaning this strategy still requires some user interaction.

Nonetheless, this strategy still works out-of-the-box with significantly more
projects than the approach above, and requires much less effort in these cases.
It is successful in decreasing the amount of user interaction needed, even
though it doesn't succeed in being generic.

.. admonition:: Examples
   :class: note

   `crossenv`_, `conda-forge`_, etc.


Environment introspection
-------------------------

As explained above, most build system code is written with the assumption that
the target system is the same as where the build is occurring, so introspection
is usually used to guide the build.

In this section, we try to document most of the ways this is accomplished. It
should give a decent overview of of environment details that are required by
build systems.

.. list-table::
   :header-rows: 1

   *  - Snippet
      - Description
      - Variance

   *  - .. code-block:: python

            >>> importlib.machinery.EXTENSION_SUFFIXES
            [
               '.cpython-311-x86_64-linux-gnu.so',
               '.abi3.so',
               '.so',
            ]

      - Extension (native module) suffixes supported by this interpreter.
      - This is implementation-defined, but it **usually** differs based on the
        implementation, system architecture, build configuration, Python
        language version, and implementation version â if one exists.

   *  - .. code-block:: python

            >>> importlib.machinery.SOURCE_SUFFIXES
            ['.py']

      - Source (pure-Python) suffixes supported by this interpreter.
      - This is implementation-defined, but it **usually** doesn't differ
        (outside exotic implementations or systems).

   *  - .. code-block:: python

            >>> importlib.machinery.all_suffixes()
            [
               '.py',
               '.pyc',
               '.cpython-311-x86_64-linux-gnu.so',
               '.abi3.so',
               '.so',
            ]

      - All module file suffixes supported by this interpreter. It *should* be the
        union of all ``importlib.machinery.*_SUFFIXES`` attributes.
      - This is implementation-defined, but it **usually** differs based on the
        implementation, system architecture, build configuration, Python
        language version, and implementation version â if one exists. See the


