# embeddings

id: pep-0321.rst
content:
PEP: 321
Title: Date/Time Parsing and Formatting
Author: A.M. Kuchling <amk@amk.ca>
Status: Withdrawn
Type: Standards Track
Created: 16-Sep-2003
Python-Version: 2.4
Post-History:


Abstract
========

Python 2.3 added a number of simple date and time types in the
``datetime`` module.  There's no support for parsing strings in various
formats and returning a corresponding instance of one of the types.
This PEP proposes adding a family of predefined parsing function for
several commonly used date and time formats, and a facility for generic
parsing.

The types provided by the ``datetime`` module all have
``.isoformat()`` and ``.ctime()`` methods that return string
representations of a time, and the ``.strftime()`` method can be used
to construct new formats.  There are a number of additional
commonly-used formats that would be useful to have as part of the
standard library; this PEP also suggests how to add them.


Input Formats
=======================

Useful formats to support include:

* `ISO8601`_
* ARPA/:rfc:`2822`
* `ctime`_
* Formats commonly written by humans such as the American
  "MM/DD/YYYY", the European "YYYY/MM/DD", and variants such as
  "DD-Month-YYYY".
* CVS-style or tar-style dates ("tomorrow", "12 hours ago", etc.)

XXX The Perl `ParseDate.pm`_ module supports many different input formats,
both absolute and relative.  Should we try to support them all?

Options:

1) Add functions to the ``datetime`` module::

        import datetime
        d = datetime.parse_iso8601("2003-09-15T10:34:54")

2) Add class methods to the various types.  There are already various
   class methods such as ``.now()``, so this would be pretty natural.::

        import datetime
        d = datetime.date.parse_iso8601("2003-09-15T10:34:54")

3) Add a separate module (possible names: date, date_parse, parse_date)
   or subpackage (possible names: datetime.parser) containing parsing
   functions::

        import datetime
        d = datetime.parser.parse_iso8601("2003-09-15T10:34:54")


Unresolved questions:

* Naming convention to use.
* What exception to raise on errors?  ValueError, or a specialized exception?
* Should you know what type you're expecting, or should the parsing figure
  it out?  (e.g. ``parse_iso8601("yyyy-mm-dd")`` returns a ``date`` instance,
  but parsing "yyyy-mm-ddThh:mm:ss" returns a ``datetime``.)  Should
  there be an option to signal an error if a time is provided where
  none is expected, or if no time is provided?
* Anything special required for I18N?  For time zones?


Generic Input Parsing
=======================

Is a strptime() implementation that returns ``datetime`` types sufficient?

XXX if yes, describe strptime here.  Can the existing pure-Python
implementation be easily retargeted?


Output Formats
=======================

Not all input formats need to be supported as output formats, because it's
pretty trivial to get the ``strftime()`` argument right for simple things
such as YYYY/MM/DD.   Only complicated formats need to be supported; :rfc:`2822`
is currently the only one I can think of.

Options:

1) Provide predefined format strings, so you could write this::

        import datetime
        d = datetime.datetime(...)
        print d.strftime(d.RFC2822_FORMAT) # or datetime.RFC2822_FORMAT?

2) Provide new methods on all the objects::

        d = datetime.datetime(...)
        print d.rfc822_time()


Relevant functionality in other languages includes the `PHP date`_
function (Python implementation by Simon Willison at
http://simon.incutio.com/archive/2003/10/07/dateInPython)


References
==========

.. _ISO8601: http://www.cl.cam.ac.uk/~mgk25/iso-time.html

.. _ParseDate.pm: http://search.cpan.org/author/MUIR/Time-modules-2003.0211/lib/Time/ParseDate.pm

.. _ctime: http://www.opengroup.org/onlinepubs/007908799/xsh/asctime.html

.. _PHP date: http://www.php.net/date

Other useful links:

http://www.egenix.com/files/python/mxDateTime.html
http://ringmaster.arc.nasa.gov/tools/time_formats.html
http://www.thinkage.ca/english/gcos/expl/b/lib/0tosec.html
https://moin.conectiva.com.br/DateUtil


Copyright
=========

This document has been placed in the public domain.


id: pep-0490.rst
content:
PEP: 490
Title: Chain exceptions at C level
Author: Victor Stinner <vstinner@python.org>
Status: Rejected
Type: Standards Track
Created: 25-Mar-2015
Python-Version: 3.6


Abstract
========

Chain exceptions at C level, as already done at Python level.


Rationale
=========

Python 3 introduced a new killer feature: exceptions are chained by
default, :pep:`3134`.

Example::

    try:
        raise TypeError("err1")
    except TypeError:
        raise ValueError("err2")

Output::

    Traceback (most recent call last):
      File "test.py", line 2, in <module>
        raise TypeError("err1")
    TypeError: err1

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "test.py", line 4, in <module>
        raise ValueError("err2")
    ValueError: err2

Exceptions are chained by default in Python code, but not in
extensions written in C.

A new private ``_PyErr_ChainExceptions()`` function was introduced in
Python 3.4.3 and 3.5 to chain exceptions. Currently, it must be called
explicitly to chain exceptions and its usage is not trivial.

Example of ``_PyErr_ChainExceptions()`` usage from the ``zipimport``
module to chain the previous ``OSError`` to a new ``ZipImportError``
exception::

    PyObject *exc, *val, *tb;
    PyErr_Fetch(&exc, &val, &tb);
    PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);
    _PyErr_ChainExceptions(exc, val, tb);

This PEP proposes to also chain exceptions automatically at C level to
stay consistent and give more information on failures to help
debugging. The previous example becomes simply::

    PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);


Proposal
========

Modify PyErr_*() functions to chain exceptions
----------------------------------------------

Modify C functions raising exceptions of the Python C API to
automatically chain exceptions: modify ``PyErr_SetString()``,
``PyErr_Format()``, ``PyErr_SetNone()``, etc.


Modify functions to not chain exceptions
----------------------------------------

Keeping the previous exception is not always interesting when the new
exception contains information of the previous exception or even more
information, especially when the two exceptions have the same type.

Example of an useless exception chain with ``int(str)``::

    TypeError: a bytes-like object is required, not 'type'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: int() argument must be a string, a bytes-like object or a number, not 'type'

The new ``TypeError`` exception contains more information than the
previous exception. The previous exception should be hidden.

The ``PyErr_Clear()`` function can be called to clear the current
exception before raising a new exception, to not chain the current
exception with a new exception.


Modify functions to chain exceptions
------------------------------------

Some functions save and then restore the current exception. If a new
exception is raised, the exception is currently displayed into
sys.stderr or ignored depending on the function.  Some of these
functions should be modified to chain exceptions instead.

Examples of function ignoring the new exception(s):

* ``ptrace_enter_call()``: ignore exception
* ``subprocess_fork_exec()``: ignore exception raised by enable_gc()
* ``t_bootstrap()`` of the ``_thread`` module: ignore exception raised
  by trying to display the bootstrap function to ``sys.stderr``
* ``PyDict_GetItem()``, ``_PyDict_GetItem_KnownHash()``: ignore
  exception raised by looking for a key in the dictionary
* ``_PyErr_TrySetFromCause()``: ignore exception
* ``PyFrame_LocalsToFast()``: ignore exception raised by
  ``dict_to_map()``
* ``_PyObject_Dump()``: ignore exception. ``_PyObject_Dump()`` is used
  to debug, to inspect a running process, it should not modify the
  Python state.
* ``Py_ReprLeave()``: ignore exception "because there is no way to
  report them"
* ``type_dealloc()``: ignore exception raised by
  ``remove_all_subclasses()``
* ``PyObject_ClearWeakRefs()``: ignore exception?
* ``call_exc_trace()``, ``call_trace_protected()``: ignore exception
* ``remove_importlib_frames()``: ignore exception
* ``do_mktuple()``, helper used by ``Py_BuildValue()`` for example:
  ignore exception?
* ``flush_io()``: ignore exception
* ``sys_write()``, ``sys_format()``: ignore exception
* ``_PyTraceback_Add()``: ignore exception
* ``PyTraceBack_Print()``: ignore exception

Examples of function displaying the new exception to ``sys.stderr``:

* ``atexit_callfuncs()``: display exceptions with
  ``PyErr_Display()`` and return the latest exception, the function
  calls multiple callbacks and only returns the latest exception
* ``sock_dealloc()``: log the ``ResourceWarning`` exception with
  ``PyErr_WriteUnraisable()``
* ``slot_tp_del()``: display exception with
  ``PyErr_WriteUnraisable()``
* ``_PyGen_Finalize()``: display ``gen_close()`` exception with
  ``PyErr_WriteUnraisable()``
* ``slot_tp_finalize()``: display exception raised by the
  ``__del__()`` method with ``PyErr_WriteUnraisable()``
* ``PyErr_GivenExceptionMatches()``: display exception raised by
  ``PyType_IsSubtype()`` with ``PyErr_WriteUnraisable()``


Backward compatibility
======================

A side effect of chaining exceptions is that exceptions store
traceback objects which store frame objects which store local
variables.  Local variables are kept alive by exceptions. A common
issue is a reference cycle between local variables and exceptions: an
exception is stored in a local variable and the frame indirectly
stored in the exception. The cycle only impacts applications storing
exceptions.

The reference cycle can now be fixed with the new
``traceback.TracebackException`` object introduced in Python 3.5. It
stores information required to format a full textual traceback without
storing local variables.

The ``asyncio`` is impacted by the reference cycle issue. This module
is also maintained outside Python standard library to release a
version for Python 3.3.  ``traceback.TracebackException`` will maybe
be backported in a private ``asyncio`` module to fix reference cycle
issues.


Alternatives
============

No change
---------

A new private ``_PyErr_ChainExceptions()`` function is enough to chain
manually exceptions.

Exceptions will only be chained explicitly where it makes sense.


New helpers to chain exceptions
-------------------------------

Functions like ``PyErr_SetString()`` don't chain automatically
exceptions. To make the usage of ``_PyErr_ChainExceptions()`` easier,
new private functions are added:

* ``_PyErr_SetStringChain(exc_type, message)``
* ``_PyErr_FormatChain(exc_type, format, ...)``
* ``_PyErr_SetNoneChain(exc_type)``
* ``_PyErr_SetObjectChain(exc_type, exc_value)``

Helper functions to raise specific exceptions like
``_PyErr_SetKeyError(key)`` or ``PyErr_SetImportError(message, name,
path)`` don't chain exceptions.  The generic
``_PyErr_ChainExceptions(exc_type, exc_value, exc_tb)`` should be used
to chain exceptions with these helper functions.


Appendix
========

PEPs
----

* :pep:`3134` -- Exception Chaining and Embedded Tracebacks
  (Python 3.0):
  new ``__context__`` and ``__cause__`` attributes for exceptions
* :pep:`415` -- Implement context suppression with exception attributes
  (Python 3.3):
  ``raise exc from None``
* :pep:`409` -- Suppressing exception context
  (superseded by the :pep:`415`)


Python C API
------------

The header file ``Include/pyerror.h`` declares functions related to
exceptions.

Functions raising exceptions:

* ``PyErr_SetNone(exc_type)``
* ``PyErr_SetObject(exc_type, exc_value)``
* ``PyErr_SetString(exc_type, message)``
* ``PyErr_Format(exc, format, ...)``

Helpers to raise specific exceptions:

* ``PyErr_BadArgument()``
* ``PyErr_BadInternalCall()``
* ``PyErr_NoMemory()``
* ``PyErr_SetFromErrno(exc)``
* ``PyErr_S


id: pep-0484.rst
content:
PEP: 484
Title: Type Hints
Author: Guido van Rossum <guido@python.org>, Jukka Lehtosalo <jukka.lehtosalo@iki.fi>, Åukasz Langa <lukasz@python.org>
BDFL-Delegate: Mark Shannon
Discussions-To: python-dev@python.org
Status: Final
Type: Standards Track
Topic: Typing
Created: 29-Sep-2014
Python-Version: 3.5
Post-History: 16-Jan-2015, 20-Mar-2015, 17-Apr-2015, 20-May-2015, 22-May-2015
Resolution: https://mail.python.org/pipermail/python-dev/2015-May/140104.html


Abstract
========

:pep:`3107` introduced syntax for function annotations, but the semantics
were deliberately left undefined.  There has now been enough 3rd party
usage for static type analysis that the community would benefit from
a standard vocabulary and baseline tools within the standard library.

This PEP introduces a provisional module to provide these standard
definitions and tools, along with some conventions for situations
where annotations are not available.

Note that this PEP still explicitly does NOT prevent other uses of
annotations, nor does it require (or forbid) any particular processing
of annotations, even when they conform to this specification.  It
simply enables better coordination, as :pep:`333` did for web frameworks.

For example, here is a simple function whose argument and return type
are declared in the annotations::

  def greeting(name: str) -> str:
      return 'Hello ' + name

While these annotations are available at runtime through the usual
``__annotations__`` attribute, *no type checking happens at runtime*.
Instead, the proposal assumes the existence of a separate off-line
type checker which users can run over their source code voluntarily.
Essentially, such a type checker acts as a very powerful linter.
(While it would of course be possible for individual users to employ
a similar checker at run time for Design By Contract enforcement or
JIT optimization, those tools are not yet as mature.)

The proposal is strongly inspired by `mypy <mypy_>`_.  For example, the
type "sequence of integers" can be written as ``Sequence[int]``.  The
square brackets mean that no new syntax needs to be added to the
language.  The example here uses a custom type ``Sequence``, imported
from a pure-Python module ``typing``.  The ``Sequence[int]`` notation
works at runtime by implementing ``__getitem__()`` in the metaclass
(but its significance is primarily to an offline type checker).

The type system supports unions, generic types, and a special type
named ``Any`` which is consistent with (i.e. assignable to and from) all
types.  This latter feature is taken from the idea of gradual typing.
Gradual typing and the full type system are explained in :pep:`483`.

Other approaches from which we have borrowed or to which ours can be
compared and contrasted are described in :pep:`482`.


Rationale and Goals
===================

:pep:`3107` added support for arbitrary annotations on parts of a
function definition.  Although no meaning was assigned to annotations
then, there has always been an `implicit goal to use them for type
hinting <gvr-artima_>`_, which is listed as the first possible use case
in said PEP.

This PEP aims to provide a standard syntax for type annotations,
opening up Python code to easier static analysis and refactoring,
potential runtime type checking, and (perhaps, in some contexts)
code generation utilizing type information.

Of these goals, static analysis is the most important.  This includes
support for off-line type checkers such as mypy, as well as providing
a standard notation that can be used by IDEs for code completion and
refactoring.

Non-goals
---------

While the proposed typing module will contain some building blocks for
runtime type checking -- in particular the ``get_type_hints()``
function -- third party packages would have to be developed to
implement specific runtime type checking functionality, for example
using decorators or metaclasses.  Using type hints for performance
optimizations is left as an exercise for the reader.

It should also be emphasized that **Python will remain a dynamically
typed language, and the authors have no desire to ever make type hints
mandatory, even by convention.**


The meaning of annotations
==========================

Any function without annotations should be treated as having the most
general type possible, or ignored, by any type checker.  Functions
with the ``@no_type_check`` decorator should be treated as having
no annotations.

It is recommended but not required that checked functions have
annotations for all arguments and the return type.  For a checked
function, the default annotation for arguments and for the return type
is ``Any``.  An exception is the first argument of instance and
class methods. If it is not annotated, then it is assumed to have the
type of the containing class for instance methods, and a type object
type corresponding to the containing class object for class methods.
For example, in class ``A`` the first argument of an instance method
has the implicit type ``A``. In a class method, the precise type of
the first argument cannot be represented using the available type
notation.

(Note that the return type of ``__init__`` ought to be annotated with
``-> None``.  The reason for this is subtle.  If ``__init__`` assumed
a return annotation of ``-> None``, would that mean that an
argument-less, un-annotated ``__init__`` method should still be
type-checked?  Rather than leaving this ambiguous or introducing an
exception to the exception, we simply say that ``__init__`` ought to
have a return annotation; the default behavior is thus the same as for
other methods.)

A type checker is expected to check the body of a checked function for
consistency with the given annotations.  The annotations may also be
used to check correctness of calls appearing in other checked functions.

Type checkers are expected to attempt to infer as much information as
necessary.  The minimum requirement is to handle the builtin
decorators ``@property``, ``@staticmethod`` and ``@classmethod``.


Type Definition Syntax
======================

The syntax leverages :pep:`3107`-style annotations with a number of
extensions described in sections below.  In its basic form, type
hinting is used by filling function annotation slots with classes::

  def greeting(name: str) -> str:
      return 'Hello ' + name

This states that the expected type of the ``name`` argument is
``str``.  Analogically, the expected return type is ``str``.

Expressions whose type is a subtype of a specific argument type are
also accepted for that argument.


Acceptable type hints
---------------------

Type hints may be built-in classes (including those defined in
standard library or third-party extension modules), abstract base
classes, types available in the ``types`` module, and user-defined
classes (including those defined in the standard library or
third-party modules).

While annotations are normally the best format for type hints,
there are times when it is more appropriate to represent them
by a special comment, or in a separately distributed stub
file.  (See below for examples.)

Annotations must be valid expressions that evaluate without raising
exceptions at the time the function is defined (but see below for
forward references).

Annotations should be kept simple or static analysis tools may not be
able to interpret the values. For example, dynamically computed types
are unlikely to be understood.  (This is an
intentionally somewhat vague requirement, specific inclusions and
exclusions may be added to future versions of this PEP as warranted by
the discussion.)

In addition to the above, the following special constructs defined
below may be used: ``None``, ``Any``, ``Union``, ``Tuple``,
``Callable``, all ABCs and stand-ins for concrete classes exported
from ``typing`` (e.g. ``Sequence`` and ``Dict``), type variables, and
type aliases.

All newly introduced names used to support features described in
following sections (such as ``Any`` and


id: pep-0485.rst
content:
PEP: 485
Title: A Function for testing approximate equality
Author: Christopher Barker <PythonCHB@gmail.com>
Status: Final
Type: Standards Track
Created: 20-Jan-2015
Python-Version: 3.5
Post-History:
Resolution: https://mail.python.org/pipermail/python-dev/2015-February/138598.html


Abstract
========

This PEP proposes the addition of an isclose() function to the standard
library math module that determines whether one value is approximately equal
or "close" to another value.


Rationale
=========

Floating point values contain limited precision, which results in
their being unable to exactly represent some values, and for errors to
accumulate with repeated computation.  As a result, it is common
advice to only use an equality comparison in very specific situations.
Often an inequality comparison fits the bill, but there are times
(often in testing) where the programmer wants to determine whether a
computed value is "close" to an expected value, without requiring them
to be exactly equal. This is common enough, particularly in testing,
and not always obvious how to do it, that it would be useful addition to
the standard library.


Existing Implementations
------------------------

The standard library includes the ``unittest.TestCase.assertAlmostEqual``
method, but it:

* Is buried in the unittest.TestCase class

* Is an assertion, so you can't use it as a general test at the command
  line, etc. (easily)

* Is an absolute difference test. Often the measure of difference
  requires, particularly for floating point numbers, a relative error,
  i.e. "Are these two values within x% of each-other?", rather than an
  absolute error. Particularly when the magnitude of the values is
  unknown a priori.

The numpy package has the ``allclose()`` and ``isclose()`` functions,
but they are only available with numpy.

The statistics package tests include an implementation, used for its
unit tests.

One can also find discussion and sample implementations on Stack
Overflow and other help sites.

Many other non-python systems provide such a test, including the Boost C++
library and the APL language [4]_.

These existing implementations indicate that this is a common need and
not trivial to write oneself, making it a candidate for the standard
library.


Proposed Implementation
=======================

NOTE: this PEP is the result of extended discussions on the
python-ideas list [1]_.

The new function will go into the math module, and have the following
signature::

  isclose(a, b, rel_tol=1e-9, abs_tol=0.0)

``a`` and ``b``: are the two values to be tested to relative closeness

``rel_tol``: is the relative tolerance -- it is the amount of error
allowed, relative to the larger absolute value of a or b. For example,
to set a tolerance of 5%, pass tol=0.05. The default tolerance is 1e-9,
which assures that the two values are the same within about 9 decimal
digits. ``rel_tol`` must be greater than 0.0

``abs_tol``: is a minimum absolute tolerance level -- useful for
comparisons near zero.

Modulo error checking, etc, the function will return the result of::

  abs(a-b) <= max( rel_tol * max(abs(a), abs(b)), abs_tol )

The name, ``isclose``, is selected for consistency with the existing
``isnan`` and ``isinf``.

Handling of non-finite numbers
------------------------------

The IEEE 754 special values of NaN, inf, and -inf will be handled
according to IEEE rules. Specifically, NaN is not considered close to
any other value, including NaN. inf and -inf are only considered close
to themselves.


Non-float types
---------------

The primary use-case is expected to be floating point numbers.
However, users may want to compare other numeric types similarly. In
theory, it should work for any type that supports ``abs()``,
multiplication, comparisons, and subtraction. However, the implementation
in the math module is written in C, and thus can not (easily) use python's
duck typing. Rather, the values passed into the function will be converted
to the float type before the calculation is performed. Passing in types
(or values) that cannot be converted to floats will raise an appropriate
Exception (TypeError, ValueError, or OverflowError).

The code will be tested to accommodate at least some values of these types:

* ``Decimal``

* ``int``

* ``Fraction``

* ``complex``: For complex, a companion function will be added to the
  ``cmath`` module. In ``cmath.isclose()``, the tolerances are specified
  as floats, and the absolute value of the complex values
  will be used for scaling and comparison. If a complex tolerance is
  passed in, the absolute value will be used as the tolerance.

NOTE: it may make sense to add a ``Decimal.isclose()`` that works properly and
completely with the decimal type, but that is not included as part of this PEP.

Behavior near zero
------------------

Relative comparison is problematic if either value is zero. By
definition, no value is small relative to zero. And computationally,
if either value is zero, the difference is the absolute value of the
other value, and the computed absolute tolerance will be ``rel_tol``
times that value. When ``rel_tol`` is less than one, the difference will
never be less than the tolerance.

However, while mathematically correct, there are many use cases where
a user will need to know if a computed value is "close" to zero. This
calls for an absolute tolerance test. If the user needs to call this
function inside a loop or comprehension, where some, but not all, of
the expected values may be zero, it is important that both a relative
tolerance and absolute tolerance can be tested for with a single
function with a single set of parameters.

There is a similar issue if the two values to be compared straddle zero:
if a is approximately equal to -b, then a and b will never be computed
as "close".

To handle this case, an optional parameter, ``abs_tol`` can be
used to set a minimum tolerance used in the case of very small or zero
computed relative tolerance. That is, the values will be always be
considered close if the difference between them is less than
``abs_tol``

The default absolute tolerance value is set to zero because there is
no value that is appropriate for the general case. It is impossible to
know an appropriate value without knowing the likely values expected
for a given use case. If all the values tested are on order of one,
then a value of about 1e-9 might be appropriate, but that would be far
too large if expected values are on order of 1e-9 or smaller.

Any non-zero default might result in user's tests passing totally
inappropriately. If, on the other hand, a test against zero fails the
first time with defaults, a user will be prompted to select an
appropriate value for the problem at hand in order to get the test to
pass.

NOTE: that the author of this PEP has resolved to go back over many of
his tests that use the numpy ``allclose()`` function, which provides
a default absolute tolerance, and make sure that the default value is
appropriate.

If the user sets the rel_tol parameter to 0.0, then only the
absolute tolerance will effect the result. While not the goal of the
function, it does allow it to be used as a purely absolute tolerance
check as well.


Implementation
--------------

A sample implementation in python is available (as of Jan 22, 2015) on
gitHub:

https://github.com/PythonCHB/close_pep/blob/master/is_close.py

This implementation has a flag that lets the user select which
relative tolerance test to apply -- this PEP does not suggest that
that be retained, but rather that the weak test be selected.

There are also drafts of this PEP and test code, etc. there:

https://github.com/PythonCHB/close_pep


Relative Difference
===================

There are essentially two ways to think about how close two numbers
are to each-other:

Absolute difference: simply ``abs(a-b)``

Relative difference: ``abs(a-b)/scale_factor`` [2]_.

The absolute difference is trivial enough that this proposal focuses
o


id: pep-0491.rst
content:
PEP: 491
Title: The Wheel Binary Package Format 1.9
Author: Daniel Holth <dholth@gmail.com>
Discussions-To: distutils-sig@python.org
Status: Deferred
Type: Standards Track
Topic: Packaging
Created: 16-Apr-2015

Abstract
========

This PEP describes the second version of a built-package format for Python
called "wheel".  Wheel provides a Python-specific, relocatable package format
that allows people to install software more quickly and predictably than
re-building from source each time.

A wheel is a ZIP-format archive with a specially formatted file name and
the ``.whl`` extension.  It contains a single distribution nearly as it
would be installed according to :pep:`376` with a particular installation
scheme.  Simple wheels can be unpacked onto ``sys.path`` and used directly
but wheels are usually installed with a specialized installer.

This version of the wheel specification adds support for installing
distributions into many different directories, and adds a way to find
those files after they have been installed.


PEP Deferral
============

This PEP is not currently being actively pursued, with Python packaging
improvements currently focusing on the package build process rather than
expanding the binary archive format to cover additional use cases.

Some specific elements to be addressed when work on this PEP is resumed in the
future:

- migrating the official wheel format definition to
  https://packaging.python.org/specifications/ (similar to what :pep:`566` did for
  https://packaging.python.org/specifications/core-metadata/)
- updating the PEP itself to focus on the *changes* being made between the
  two versions of the format and the rationale for those changes, rather than
  having to repeat all the information that is unchanged from :pep:`427`
- clarifying that the PEP is deliberately written to allow existing installers
  to be compliant with the specification when using existing install scheme
  definitions, while also allowing the creation of new install scheme
  definitions that take advantage of the richer categorisation scheme for
  the contents of the binary archive


Rationale
=========

Wheel 1.0 is best at installing files into ``site-packages`` and a few
other locations specified by distutils, but users would like to install
files from single distribution into many directories -- perhaps separate
locations for docs, data, and code.  Unfortunately not everyone agrees
on where these install locations should be relative to the root directory.
This version of the format adds many more categories, each of which can be
installed to a different destination based on policy.  Since it might
also be important to locate the installed files at runtime, this version
of the format also adds a way to record the installed paths in a way that
can be read by the installed software.

Details
=======

Installing a wheel 'distribution-1.0-py32-none-any.whl'
-------------------------------------------------------

Wheel installation notionally consists of two phases:

- Unpack.

  a. Parse ``distribution-1.0.dist-info/WHEEL``.
  b. Check that installer is compatible with Wheel-Version.  Warn if
     minor version is greater, abort if major version is greater.
  c. If Root-Is-Purelib == 'true', unpack archive into purelib
     (site-packages).
  d. Else unpack archive into platlib (site-packages).

- Spread.

  a. Unpacked archive includes ``distribution-1.0.dist-info/`` and (if
     there is data) ``distribution-1.0.data/``.
  b. Move each subtree of ``distribution-1.0.data/`` onto its
     destination path. Each subdirectory of ``distribution-1.0.data/``
     is a key into a dict of destination directories, such as
     ``distribution-1.0.data/(purelib|platlib|headers|scripts|data)``.
  c. Update scripts starting with ``#!python`` to point to the correct
     interpreter.  (Note: Python scripts are usually handled by package
     metadata, and not included verbatim in wheel.)
  d. Update ``distribution-1.0.dist.info/RECORD`` with the installed
     paths.
  e. If empty, remove the ``distribution-1.0.data`` directory.
  f. Compile any installed .py to .pyc. (Uninstallers should be smart
     enough to remove .pyc even if it is not mentioned in RECORD.)

In practice, installers will usually extract files directly from the archive
to their destinations without writing a temporary ``distribution-1.0.data/``
directory.

Recommended installer features
''''''''''''''''''''''''''''''

Rewrite ``#!python``.
    In wheel, verbatim scripts are packaged in
    ``{distribution}-{version}.data/scripts/``.  If the first line of
    a file in ``scripts/`` starts with exactly ``b'#!python'``, rewrite to
    point to the correct interpreter.  Unix installers may need to add
    the +x bit to these files if the archive was created on Windows.

    The ``b'#!pythonw'`` convention is allowed. ``b'#!pythonw'`` indicates
    a GUI script instead of a console script.

Generate script wrappers.
    Python scripts are more commonly represented as a ``module:callable``
    string in package metadata, and are not included verbatim in the wheel
    archive's ``scripts`` directory.  This kind of script gives the installer
    an opportunity to generate platform specific wrappers.

Recommended archiver features
'''''''''''''''''''''''''''''

Place ``.dist-info`` at the end of the archive.
    Archivers are encouraged to place the ``.dist-info`` files physically
    at the end of the archive.  This enables some potentially interesting
    ZIP tricks including the ability to amend the metadata without
    rewriting the entire archive.


File Format
-----------

File name convention
''''''''''''''''''''

The wheel filename is ``{distribution}-{version}(-{build
tag})?-{python tag}-{abi tag}-{platform tag}.whl``.

distribution
    Distribution name, e.g. 'django', 'pyramid'.

version
    Distribution version, e.g. 1.0.

build tag
    Optional build number.  Must start with a digit.  A tie breaker
    if two wheels have the same version.  Sort as the empty string
    if unspecified, else sort the initial digits as a number, and the
    remainder lexicographically.

language implementation and version tag
    E.g. 'py27', 'py2', 'py3'.

abi tag
    E.g. 'cp33m', 'abi3', 'none'.

platform tag
    E.g. 'linux_x86_64', 'any'.

For example, ``distribution-1.0-1-py27-none-any.whl`` is the first
build of a package called 'distribution', and is compatible with
Python 2.7 (any Python 2.7 implementation), with no ABI (pure Python),
on any CPU architecture.

The last three components of the filename before the extension are
called "compatibility tags."  The compatibility tags express the
package's basic interpreter requirements and are detailed in :pep:`425`.

Escaping and Unicode
''''''''''''''''''''

Each component of the filename is escaped by replacing runs of
non-alphanumeric characters with an underscore ``_``::

    re.sub("[^\w\d.]+", "_", distribution, re.UNICODE)

The archive filename is Unicode.  The packaging tools may only support
ASCII package names, but Unicode filenames are supported in this
specification.

The filenames *inside* the archive are encoded as UTF-8.  Although some
ZIP clients in common use do not properly display UTF-8 filenames,
the encoding is supported by both the ZIP specification and Python's
``zipfile``.

File contents
'''''''''''''

The contents of a wheel file, where {distribution} is replaced with the
name of the package, e.g. ``beaglevote`` and {version} is replaced with
its version, e.g. ``1.0.0``, consist of:

#. ``/``, the root of the archive, contains all files to be installed in
   ``purelib`` or ``platlib`` as specified in ``WHEEL``.  ``purelib`` and
   ``platlib`` are usually both ``site-packages``.
#. ``{distribution}-{version}.dist-info/`` contains metadata.
#. ``{distribution}-{version}.data/`` contains one subdirectory
   for each non-empty install scheme key not already covered, where
   the subdirectory name is an index into a dictionary of i


id: pep-0446.rst
content:
PEP: 446
Title: Make newly created file descriptors non-inheritable
Author: Victor Stinner <vstinner@python.org>
Status: Final
Type: Standards Track
Created: 05-Aug-2013
Python-Version: 3.4
Replaces: 433


Abstract
========

Leaking file descriptors in child processes causes various annoying
issues and is a known major security vulnerability. Using the
``subprocess`` module with the *close_fds* parameter set to ``True`` is
not possible in all cases.

This PEP proposes to make all file descriptors created by Python
non-inheritable by default to reduce the risk of these issues. This PEP
fixes also a race condition in multi-threaded applications on operating
systems supporting atomic flags to create non-inheritable file
descriptors.

We are aware of the code breakage this is likely to cause, and doing it
anyway for the good of mankind. (Details in the section "Backward
Compatibility" below.)


Rationale
=========

Inheritance of File Descriptors
-------------------------------

Each operating system handles the inheritance of file descriptors
differently. Windows creates non-inheritable handles by default, whereas
UNIX and the POSIX API on Windows create inheritable file descriptors by
default. Python prefers the POSIX API over the native Windows API, to
have a single code base and to use the same type for file descriptors,
and so it creates inheritable file descriptors.

There is one exception: ``os.pipe()`` creates non-inheritable pipes on
Windows, whereas it creates inheritable pipes on UNIX. The reason is an
implementation artifact: ``os.pipe()`` calls ``CreatePipe()`` on Windows
(native API), whereas it calls ``pipe()`` on UNIX (POSIX API). The call
to ``CreatePipe()`` was added in Python in 1994, before the introduction
of ``pipe()`` in the POSIX API in Windows 98. The `issue #4708
<http://bugs.python.org/issue4708>`_ proposes to change ``os.pipe()`` on
Windows to create inheritable pipes.


Inheritance of File Descriptors on Windows
------------------------------------------

On Windows, the native type of file objects is handles (C type
``HANDLE``). These handles have a ``HANDLE_FLAG_INHERIT`` flag which
defines if a handle can be inherited in a child process or not. For the
POSIX API, the C runtime (CRT) also provides file descriptors (C type
``int``). The handle of a file descriptor can be retrieve using the
function ``_get_osfhandle(fd)``. A file descriptor can be created from a
handle using the function ``_open_osfhandle(handle)``.

Using `CreateProcess()
<http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx>`_,
handles are only inherited if their inheritable flag
(``HANDLE_FLAG_INHERIT``) is set **and** the ``bInheritHandles``
parameter of ``CreateProcess()`` is ``TRUE``; all file descriptors
except standard streams (0, 1, 2) are closed in the child process, even
if ``bInheritHandles`` is ``TRUE``. Using the ``spawnv()`` function, all
inheritable handles and all inheritable file descriptors are inherited
in the child process. This function uses the undocumented fields
*cbReserved2* and *lpReserved2* of the `STARTUPINFO
<http://msdn.microsoft.com/en-us/library/windows/desktop/ms686331%28v=vs.85%29.aspx>`_
structure to pass an array of file descriptors.

To replace standard streams (stdin, stdout, stderr) using
``CreateProcess()``, the ``STARTF_USESTDHANDLES`` flag must be set in
the *dwFlags* field of the ``STARTUPINFO`` structure and the
*bInheritHandles* parameter of ``CreateProcess()`` must be set to
``TRUE``. So when at least one standard stream is replaced, all
inheritable handles are inherited by the child process.

The default value of the *close_fds* parameter of ``subprocess`` process
is ``True`` (``bInheritHandles=FALSE``) if *stdin*, *stdout* and
*stderr* parameters are ``None``, ``False`` (``bInheritHandles=TRUE``)
otherwise.

See also:

* `Handle Inheritance
  <http://msdn.microsoft.com/en-us/library/windows/desktop/ms724466%28v=vs.85%29.aspx>`_
* `Stackoverflow: Can TCP SOCKET handles be set not inheritable?
  <http://stackoverflow.com/questions/12058911/can-tcp-socket-handles-be-set-not-inheritable>`_


Only Inherit Some Handles on Windows
------------------------------------

Since Windows Vista, ``CreateProcess()`` supports an extension of the
STARTUPINFO structure: the `STARTUPINFOEX structure
<http://msdn.microsoft.com/en-us/library/ms686329%28v=vs.85%29.aspx>`_.
Using this new structure, it is possible to specify a list of handles to
inherit: ``PROC_THREAD_ATTRIBUTE_HANDLE_LIST``. Read `Programmatically
controlling which handles are inherited by new processes in Win32
<http://blogs.msdn.com/b/oldnewthing/archive/2011/12/16/10248328.aspx>`_
(Raymond Chen, Dec 2011) for more information.

Before Windows Vista, it is possible to make handles inheritable and
call ``CreateProcess()`` with ``bInheritHandles=TRUE``. This option
works if all other handles are non-inheritable. There is a race
condition: if another thread calls ``CreateProcess()`` with
``bInheritHandles=TRUE``, handles will also be inherited in the second
process.

Microsoft suggests to use a lock to avoid the race condition: read
`Q315939: PRB: Child Inherits Unintended Handles During CreateProcess
Call <http://support.microsoft.com/kb/315939/en-us>`_ (last review:
November 2006). The `Python issue #16500 "Add an atfork module"
<http://bugs.python.org/issue16500>`_ proposes to add such lock, it can
be used to make handles non-inheritable without the race condition. Such
lock only protects against a race condition between Python threads; C
threads are not protected.

Another option is to duplicate handles that must be inherited, passing the
values of the duplicated handles to the child process, so the child
process can steal duplicated handles using `DuplicateHandle()
<http://msdn.microsoft.com/en-us/library/windows/apps/ms724251%28v=vs.85%29.aspx>`_
with ``DUPLICATE_CLOSE_SOURCE``. Handle values change between the
parent and the child process because the handles are duplicated (twice);
the parent and/or the child process must be adapted to handle this
change. If the child program cannot be modified, an intermediate program
can be used to steal handles from the parent process before spawning the
final child program. The intermediate program has to pass the handle from the
child process to the parent process. The parent may have to close
duplicated handles if all handles were not stolen, for example if the
intermediate process fails. If the command line is used to pass the
handle values, the command line must be modified when handles are
duplicated, because their values are modified.

This PEP does not include a solution to this problem because there is no
perfect solution working on all Windows versions. This point is deferred
until use cases relying on handle or file descriptor inheritance on
Windows are well known, so we can choose the best solution and carefully
test its implementation.


Inheritance of File Descriptors on UNIX
---------------------------------------

POSIX provides a *close-on-exec* flag on file descriptors to automatically
close a file descriptor when the C function ``execv()`` is
called. File descriptors with the *close-on-exec* flag cleared are
inherited in the child process, file descriptors with the flag set are
closed in the child process.

The flag can be set in two syscalls (one to get current flags, a second
to set new flags) using ``fcntl()``::

    int flags, res;
    flags = fcntl(fd, F_GETFD);
    if (flags == -1) { /* handle the error */ }
    flags |= FD_CLOEXEC;
    /* or "flags &= ~FD_CLOEXEC;" to clear the flag */
    res = fcntl(fd, F_SETFD, flags);
    if (res == -1) { /* handle the error */ }

FreeBSD, Linux, Mac OS X, NetBSD, OpenBSD and QNX also support setting
the flag in a single syscall using ioctl()::

    int res;
    res = ioctl(fd, FIOCLEX, 0);
    if (!res) { /* handle the error */ }

NOTE: The *close-on-exec* flag has no effect on ``fork()``: all file
descriptors are inherited by t


id: pep-0320.rst
content:
PEP: 320
Title: Python 2.4 Release Schedule
Author: Barry Warsaw, Raymond Hettinger, Anthony Baxter
Status: Final
Type: Informational
Topic: Release
Created: 29-Jul-2003
Python-Version: 2.4
Post-History: 01-Dec-2004


Abstract
========

This document describes the development and release schedule for
Python 2.4.  The schedule primarily concerns itself with PEP-sized
items.  Small features may be added up to and including the first
beta release.  Bugs may be fixed until the final release.

There will be at least two alpha releases, two beta releases, and
one release candidate.  The release date was 30th November, 2004.


Release Manager
===============

Anthony Baxter

Martin von Lowis is building the Windows installers, Fred the
doc packages, Sean the RPMs.


Release Schedule
================

- July 9: alpha 1 [completed]

- August 5/6: alpha 2 [completed]

- Sept 3: alpha 3 [completed]

- October 15: beta 1 [completed]

- November 3: beta 2 [completed]

- November 18: release candidate 1 [completed]

- November 30: final [completed]


Completed features for 2.4
==========================

- :pep:`218` Builtin Set Objects.

- :pep:`289` Generator expressions.

- :pep:`292` Simpler String Substitutions to be implemented as a module.

- :pep:`318`: Function/method decorator syntax, using @syntax

- :pep:`322` Reverse Iteration.

- :pep:`327`: A Decimal package for fixed precision arithmetic.

- :pep:`328`: Multi-line Imports

- Encapsulate the decorate-sort-undecorate pattern in a keyword for
  ``list.sort()``.

- Added a builtin called ``sorted()`` which may be used in expressions.

- The ``itertools`` module has two new functions, ``tee()`` and ``groupby()``.

- Add a ``collections`` module with a ``deque()`` object.

- Add two statistical/reduction functions, ``nlargest()`` and ``nsmallest()``
  to the ``heapq`` module.

- Python's windows installer now uses MSI


Deferred until 2.5
==================

- Deprecate and/or remove the modules listed in :pep:`4` (``posixfile``,
  ``gopherlib``, ``pre``, ``others``)

- Remove support for platforms as described in :pep:`11`.

- Finish implementing the Distutils ``bdist_dpkg`` command.  (AMK)

- Add support for reading shadow passwords [1]_

- It would be nice if the built-in SSL socket type could be used
  for non-blocking SSL I/O.  Currently packages such as Twisted
  which implement async servers using SSL have to require third-party
  packages such as pyopenssl.

- AST-based compiler: this branch was not completed in time for
  2.4, but will land on the trunk some time after 2.4 final is
  out, for inclusion in 2.5.

- reST is going to be used a lot in Zope3.  Maybe it could become
  a standard library module?  (Since reST's author thinks it's too
  instable, I'm inclined not to do this.)


Ongoing tasks
=============

The following are ongoing TO-DO items which we should attempt to
work on without hoping for completion by any particular date.

- Documentation: complete the distribution and installation
  manuals.

- Documentation: complete the documentation for new-style
  classes.

- Look over the ``Demos/`` directory and update where required (Andrew
  Kuchling has done a lot of this)

- New tests.

- Fix doc bugs on SF.

- Remove use of deprecated features in the core.

- Document deprecated features appropriately.

- Mark deprecated C APIs with ``Py_DEPRECATED``.

- Deprecate modules which are unmaintained, or perhaps make a new
  category for modules 'Unmaintained'

- In general, lots of cleanup so it is easier to move forward.


Open issues
===========

None at this time.


Carryover features from Python 2.3
==================================

- The import lock could use some redesign. [2]_

- A nicer API to open text files, replacing the ugly (in some
  people's eyes) "U" mode flag.  There's a proposal out there to
  have a new built-in type ``textfile(filename, mode, encoding)``.
  (Shouldn't it have a *bufsize* argument too?)

- New widgets for Tkinter???

  Has anyone gotten the time for this?  *Are* there any new
  widgets in Tk 8.4?  Note that we've got better Tix support
  already (though not on Windows yet).

- :pep:`304` (Controlling Generation of Bytecode Files by Montanaro)
  seems to have lost steam.

- For a class defined inside another class, the ``__name__`` should be
  "outer.inner", and pickling should work.  ([3]_.  I'm no
  longer certain this is easy or even right.)

- Decide on a clearer deprecation policy (especially for modules)
  and act on it.  For a start, see this message from Neal Norwitz [4]_.
  There seems insufficient interest in moving this further in an
  organized fashion, and it's not particularly important.

- Provide alternatives for common uses of the ``types`` module;
  Skip Montanaro has posted a proto-PEP for this idea [5]_.
  There hasn't been any progress on this, AFAICT.

- Use pending deprecation for the ``types`` and ``string`` modules.  This
  requires providing alternatives for the parts that aren't
  covered yet (e.g. ``string.whitespace`` and ``types.TracebackType``).
  It seems we can't get consensus on this.

- :pep:`262` Database of Installed Python Packages (Kuchling)

  This turns out to be useful for Jack Jansen's Python installer,
  so the database is worth implementing.  Code will go in
  sandbox/pep262.

- :pep:`269` Pgen Module for Python (Riehl)

  (Some necessary changes are in; the ``pgen`` module itself needs to
  mature more.)

- :pep:`266` Optimizing Global Variable/Attribute Access (Montanaro)

  :pep:`267` Optimized Access to Module Namespaces (Hylton)

  :pep:`280` Optimizing access to globals (van Rossum)

  These are basically three friendly competing proposals.  Jeremy
  has made a little progress with a new compiler, but it's going
  slowly and the compiler is only the first step.  Maybe we'll be
  able to refactor the compiler in this release.  I'm tempted to
  say we won't hold our breath.

- Lazily tracking tuples? [6]_ [7]_
  Not much enthusiasm I believe.

- :pep:`286` Enhanced Argument Tuples (von Loewis)

  I haven't had the time to review this thoroughly.  It seems a
  deep optimization hack (also makes better correctness guarantees
  though).

- Make '``as``' a keyword.  It has been a pseudo-keyword long enough.
  Too much effort to bother.


References
==========

.. [1] Shadow Password Support Module
       https://bugs.python.org/issue579435

.. [2] PyErr_Warn may cause import deadlock
       https://bugs.python.org/issue683658

.. [3] Nested class __name__
       https://bugs.python.org/issue633930

.. [4] Neal Norwitz, random vs whrandom
       https://mail.python.org/pipermail/python-dev/2002-April/023165.html

.. [5] Skip Montanaro, python/dist/src/Lib types.py,1.26,1.27
       https://mail.python.org/pipermail/python-dev/2002-May/024346.html

.. [6] Daniel Dunbar, Lazily GC tracking tuples
       https://mail.python.org/pipermail/python-dev/2002-May/023926.html

.. [7] GC: untrack simple objects
       https://bugs.python.org/issue558745


Copyright
=========

This document has been placed in the public domain.


id: pep-0334.rst
content:
PEP: 334
Title: Simple Coroutines via SuspendIteration
Author: Clark C. Evans <cce@clarkevans.com>
Status: Withdrawn
Type: Standards Track
Created: 26-Aug-2004
Python-Version: 3.0
Post-History:


Abstract
========

Asynchronous application frameworks such as Twisted [1]_ and Peak
[2]_, are based on a cooperative multitasking via event queues or
deferred execution.  While this approach to application development
does not involve threads and thus avoids a whole class of problems
[3]_, it creates a different sort of programming challenge.  When an
I/O operation would block, a user request must suspend so that other
requests can proceed.  The concept of a coroutine [4]_ promises to
help the application developer grapple with this state management
difficulty.

This PEP proposes a limited approach to coroutines based on an
extension to the :pep:`iterator protocol <234>`.  Currently, an iterator may
raise a StopIteration exception to indicate that it is done producing
values.  This proposal adds another exception to this protocol,
SuspendIteration, which indicates that the given iterator may have
more values to produce, but is unable to do so at this time.


Rationale
=========

There are two current approaches to bringing co-routines to Python.
Christian Tismer's Stackless [6]_ involves a ground-up restructuring
of Python's execution model by hacking the 'C' stack.  While this
approach works, its operation is hard to describe and keep portable. A
related approach is to compile Python code to Parrot [7]_, a
register-based virtual machine, which has coroutines.  Unfortunately,
neither of these solutions is portable with IronPython (CLR) or Jython
(JavaVM).

It is thought that a more limited approach, based on iterators, could
provide a coroutine facility to application programmers and still be
portable across runtimes.

* Iterators keep their state in local variables that are not on the
  "C" stack.  Iterators can be viewed as classes, with state stored in
  member variables that are persistent across calls to its next()
  method.

* While an uncaught exception may terminate a function's execution, an
  uncaught exception need not invalidate an iterator.  The proposed
  exception, SuspendIteration, uses this feature.  In other words,
  just because one call to next() results in an exception does not
  necessarily need to imply that the iterator itself is no longer
  capable of producing values.

There are four places where this new exception impacts:

* The :pep:`255` simple generator mechanism could be extended to safely
  'catch' this SuspendIteration exception, stuff away its current
  state, and pass the exception on to the caller.

* Various iterator filters [9]_ in the standard library, such as
  itertools.izip should be made aware of this exception so that it can
  transparently propagate SuspendIteration.

* Iterators generated from I/O operations, such as a file or socket
  reader, could be modified to have a non-blocking variety.  This
  option would raise a subclass of SuspendIteration if the requested
  operation would block.

* The asyncore library could be updated to provide a basic 'runner'
  that pulls from an iterator; if the SuspendIteration exception is
  caught, then it moves on to the next iterator in its runlist [10]_.
  External frameworks like Twisted would provide alternative
  implementations, perhaps based on FreeBSD's kqueue or Linux's epoll.

While these may seem dramatic changes, it is a very small amount of
work compared with the utility provided by continuations.


Semantics
=========

This section will explain, at a high level, how the introduction of
this new SuspendIteration exception would behave.


Simple Iterators
----------------

The current functionality of iterators is best seen with a simple
example which produces two values 'one' and 'two'. ::

    class States:

        def __iter__(self):
            self._next = self.state_one
            return self

        def next(self):
            return self._next()

        def state_one(self):
            self._next = self.state_two
            return "one"

        def state_two(self):
            self._next = self.state_stop
            return "two"

        def state_stop(self):
            raise StopIteration

    print list(States())

An equivalent iteration could, of course, be created by the
following generator::

    def States():
        yield 'one'
        yield 'two'

    print list(States())


Introducing SuspendIteration
----------------------------

Suppose that between producing 'one' and 'two', the generator above
could block on a socket read.  In this case, we would want to raise
SuspendIteration to signal that the iterator is not done producing,
but is unable to provide a value at the current moment. ::

    from random import randint
    from time import sleep

    class SuspendIteration(Exception):
          pass

    class NonBlockingResource:

        """Randomly unable to produce the second value"""

        def __iter__(self):
            self._next = self.state_one
            return self

        def next(self):
            return self._next()

        def state_one(self):
            self._next = self.state_suspend
            return "one"

        def state_suspend(self):
            rand = randint(1,10)
            if 2 == rand:
                self._next = self.state_two
                return self.state_two()
            raise SuspendIteration()

        def state_two(self):
            self._next = self.state_stop
            return "two"

        def state_stop(self):
            raise StopIteration

    def sleeplist(iterator, timeout = .1):
        """
        Do other things (e.g. sleep) while resource is
        unable to provide the next value
        """
        it = iter(iterator)
        retval = []
        while True:
            try:
                retval.append(it.next())
            except SuspendIteration:
                sleep(timeout)
                continue
            except StopIteration:
                break
        return retval

    print sleeplist(NonBlockingResource())

In a real-world situation, the NonBlockingResource would be a file
iterator, socket handle, or other I/O based producer.  The sleeplist
would instead be an async reactor, such as those found in asyncore or
Twisted.  The non-blocking resource could, of course, be written as a
generator::

    def NonBlockingResource():
        yield "one"
        while True:
            rand = randint(1,10)
            if 2 == rand:
                break
            raise SuspendIteration()
        yield "two"

It is not necessary to add a keyword, 'suspend', since most real
content generators will not be in application code, they will be in
low-level I/O based operations.  Since most programmers need not be
exposed to the SuspendIteration() mechanism, a keyword is not needed.


Application Iterators
---------------------

The previous example is rather contrived, a more 'real-world' example
would be a web page generator which yields HTML content, and pulls
from a database.  Note that this is an example of neither the
'producer' nor the 'consumer', but rather of a filter. ::

    def ListAlbums(cursor):
        cursor.execute("SELECT title, artist FROM album")
        yield '<html><body><table><tr><td>Title</td><td>Artist</td></tr>'
        for (title, artist) in cursor:
            yield '<tr><td>%s</td><td>%s</td></tr>' % (title, artist)
        yield '</table></body></html>'

The problem, of course, is that the database may block for some time
before any rows are returned, and that during execution, rows may be
returned in blocks of 10 or 100 at a time. Ideally, if the database
blocks for the next set of rows, another user connection could be
serviced.  Note the complete absence of SuspendIterator in the above
code.  If done correctly, application developers would be able to
focus on functionality rather than concurrency issues.

The iterator created by the a


id: pep-0452.rst
content:
PEP: 452
Title: API for Cryptographic Hash Functions v2.0
Author: A.M. Kuchling <amk@amk.ca>, Christian Heimes <christian@python.org>
Status: Final
Type: Informational
Created: 15-Aug-2013
Post-History:
Replaces: 247

Abstract
========

There are several different modules available that implement
cryptographic hashing algorithms such as MD5 or SHA.  This
document specifies a standard API for such algorithms, to make it
easier to switch between different implementations.


Specification
=============

All hashing modules should present the same interface.  Additional
methods or variables can be added, but those described in this
document should always be present.

Hash function modules define one function:

``new([string])            (unkeyed hashes)``

``new(key, [string], [digestmod])    (keyed hashes)``
   Create a new hashing object and return it.  The first form is
   for hashes that are unkeyed, such as MD5 or SHA.  For keyed
   hashes such as HMAC, 'key' is a required parameter containing
   a string giving the key to use.  In both cases, the optional
   'string' parameter, if supplied, will be immediately hashed
   into the object's starting state, as if ``obj.update(string)`` was
   called.

   After creating a hashing object, arbitrary bytes can be fed
   into the object using its ``update()`` method, and the hash value
   can be obtained at any time by calling the object's ``digest()``
   method.

   Although the parameter is called 'string', hashing objects operate
   on 8-bit data only. Both 'key' and 'string' must be a bytes-like
   object (bytes, bytearray...). A hashing object may support
   one-dimensional, contiguous buffers as argument, too. Text
   (unicode) is no longer supported in Python 3.x. Python 2.x
   implementations may take ASCII-only unicode as argument, but
   portable code should not rely on the feature.

   Arbitrary additional keyword arguments can be added to this
   function, but if they're not supplied, sensible default values
   should be used.  For example, 'rounds' and 'digest_size'
   keywords could be added for a hash function which supports a
   variable number of rounds and several different output sizes,
   and they should default to values believed to be secure.

Hash function modules define one variable:

``digest_size``
   An integer value; the size of the digest produced by the
   hashing objects created by this module, measured in bytes.
   You could also obtain this value by creating a sample object
   and accessing its 'digest_size' attribute, but it can be
   convenient to have this value available from the module.
   Hashes with a variable output size will set this variable to
   None.

Hashing objects require the following attribute:

``digest_size``
   This attribute is identical to the module-level digest_size
   variable, measuring the size of the digest produced by the
   hashing object, measured in bytes.  If the hash has a variable
   output size, this output size must be chosen when the hashing
   object is created, and this attribute must contain the
   selected size.  Therefore, ``None`` is **not** a legal value for this
   attribute.

``block_size``
   An integer value or ``NotImplemented``; the internal block size
   of the hash algorithm in bytes. The block size is used by the
   HMAC module to pad the secret key to ``digest_size`` or to hash the
   secret key if it is longer than ``digest_size``. If no HMAC
   algorithm is standardized for the hash algorithm, return
   ``NotImplemented`` instead.

``name``
   A text string value; the canonical, lowercase name of the hashing
   algorithm. The name should be a suitable parameter for
   ``hashlib.new``.

Hashing objects require the following methods:

``copy()``
   Return a separate copy of this hashing object.  An update to
   this copy won't affect the original object.

``digest()``
   Return the hash value of this hashing object as a bytes
   containing 8-bit data.  The object is not altered in any way
   by this function; you can continue updating the object after
   calling this function.

``hexdigest()``
   Return the hash value of this hashing object as a string
   containing hexadecimal digits.  Lowercase letters should be used
   for the digits 'a' through 'f'.  Like the ``.digest()`` method, this
   method mustn't alter the object.

``update(string)``
   Hash bytes-like 'string' into the current state of the hashing
   object. ``update()`` can be called any number of times during a
   hashing object's lifetime.

Hashing modules can define additional module-level functions or
object methods and still be compliant with this specification.

Here's an example, using a module named 'MD5'::

    >>> import hashlib
    >>> from Crypto.Hash import MD5
    >>> m = MD5.new()
    >>> isinstance(m, hashlib.CryptoHash)
    True
    >>> m.name
    'md5'
    >>> m.digest_size
    16
    >>> m.block_size
    64
    >>> m.update(b'abc')
    >>> m.digest()
    b'\x90\x01P\x98<\xd2O\xb0\xd6\x96?}(\xe1\x7fr'
    >>> m.hexdigest()
    '900150983cd24fb0d6963f7d28e17f72'
    >>> MD5.new(b'abc').digest()
    b'\x90\x01P\x98<\xd2O\xb0\xd6\x96?}(\xe1\x7fr'


Rationale
=========

The digest size is measured in bytes, not bits, even though hash
algorithm sizes are usually quoted in bits; MD5 is a 128-bit
algorithm and not a 16-byte one, for example.  This is because, in
the sample code I looked at, the length in bytes is often needed
(to seek ahead or behind in a file; to compute the length of an
output string) while the length in bits is rarely used.
Therefore, the burden will fall on the few people actually needing
the size in bits, who will have to multiply digest_size by 8.

It's been suggested that the ``update()`` method would be better named
``append()``.  However, that method is really causing the current
state of the hashing object to be updated, and ``update()`` is already
used by the md5 and sha modules included with Python, so it seems
simplest to leave the name ``update()`` alone.

The order of the constructor's arguments for keyed hashes was a
sticky issue.  It wasn't clear whether the key should come first
or second.  It's a required parameter, and the usual convention is
to place required parameters first, but that also means that the
'string' parameter moves from the first position to the second.
It would be possible to get confused and pass a single argument to
a keyed hash, thinking that you're passing an initial string to an
unkeyed hash, but it doesn't seem worth making the interface
for keyed hashes more obscure to avoid this potential error.


Changes from Version 1.0 to Version 2.0
=======================================

Version 2.0 of API for Cryptographic Hash Functions clarifies some
aspects of the API and brings it up-to-date. It also formalized aspects
that were already de facto standards and provided by most
implementations.

Version 2.0 introduces the following new attributes:

``name``
   The name property was made mandatory by `issue 18532`_.

``block_size``
   The new version also specifies that the return value
   ``NotImplemented`` prevents HMAC support.

Version 2.0 takes the separation of binary and text data in Python
3.0 into account. The 'string' argument to ``new()`` and ``update()`` as
well as the 'key' argument must be bytes-like objects. On Python
2.x a hashing object may also support ASCII-only unicode. The actual
name of argument is not changed as it is part of the public API.
Code may depend on the fact that the argument is called 'string'.


Recommended names for common hashing algorithms
===============================================

+------------+------------+-------------------+
| algorithm  |  variant   |  recommended name |
+============+============+===================+
| MD5        |            |     md5           |
+------------+------------+-------------------+
| RIPEMD-160 |            |     ripemd160     |
+------------+------------+-------------------+
| SHA-1      |            |      sha1


id: pep-0308.rst
content:
PEP: 308
Title: Conditional Expressions
Author: Guido van Rossum, Raymond Hettinger
Status: Final
Type: Standards Track
Created: 07-Feb-2003
Python-Version: 2.5
Post-History: 07-Feb-2003, 11-Feb-2003


Adding a conditional expression
===============================

On 9/29/2005, Guido decided to add conditional expressions in the
form of "X if C else Y". [1]_

The motivating use case was the prevalence of error-prone attempts
to achieve the same effect using "and" and "or". [2]_

Previous community efforts to add a conditional expression were
stymied by a lack of consensus on the best syntax.  That issue was
resolved by simply deferring to a BDFL best judgment call.

The decision was validated by reviewing how the syntax fared when
applied throughout the standard library (this review approximates a
sampling of real-world use cases, across a variety of applications,
written by a number of programmers with diverse backgrounds). [3]_

The following change will be made to the grammar.  (The or_test
symbols is new, the others are modified.)

::

    test: or_test ['if' or_test 'else' test] | lambdef
    or_test: and_test ('or' and_test)*
    ...
    testlist_safe: or_test [(',' or_test)+ [',']]
    ...
    gen_for: 'for' exprlist 'in' or_test [gen_iter]

The new syntax nearly introduced a minor syntactical backwards
incompatibility.  In previous Python versions, the following is
legal::

    [f for f in lambda x: x, lambda x: x**2 if f(1) == 1]

(I.e. a list comprehension where the sequence following 'in' is an
unparenthesized series of lambdas -- or just one lambda, even.)

In Python 3.0, the series of lambdas will have to be
parenthesized, e.g.::

    [f for f in (lambda x: x, lambda x: x**2) if f(1) == 1]

This is because lambda binds less tight than the if-else
expression, but in this context, the lambda could already be
followed by an 'if' keyword that binds less tightly still (for
details, consider the grammar changes shown above).

However, in Python 2.5, a slightly different grammar is used that
is more backwards compatible, but constrains the grammar of a
lambda used in this position by forbidding the lambda's body to
contain an unparenthesized condition expression.  Examples::

    [f for f in (1, lambda x: x if x >= 0 else -1)]    # OK
    [f for f in 1, (lambda x: x if x >= 0 else -1)]    # OK
    [f for f in 1, lambda x: (x if x >= 0 else -1)]    # OK
    [f for f in 1, lambda x: x if x >= 0 else -1]      # INVALID


References
==========

.. [1] Pronouncement
       https://mail.python.org/pipermail/python-dev/2005-September/056846.html

.. [2] Motivating use case:
       https://mail.python.org/pipermail/python-dev/2005-September/056546.html
       https://mail.python.org/pipermail/python-dev/2005-September/056510.html

.. [3] Review in the context of real-world code fragments:
       https://mail.python.org/pipermail/python-dev/2005-September/056803.html


Introduction to earlier draft of the PEP (kept for historical purposes)
=======================================================================

Requests for an if-then-else ("ternary") expression keep coming up
on comp.lang.python.  This PEP contains a concrete proposal of a
fairly Pythonic syntax.  This is the community's one chance: if
this PEP is approved with a clear majority, it will be implemented
in Python 2.4.  If not, the PEP will be augmented with a summary
of the reasons for rejection and the subject better not come up
again.  While the BDFL is co-author of this PEP, he is neither in
favor nor against this proposal; it is up to the community to
decide.  If the community can't decide, the BDFL will reject the
PEP.

After unprecedented community response (very good arguments were
made both pro and con) this PEP has been revised with the help of
Raymond Hettinger.  Without going through a complete revision
history, the main changes are a different proposed syntax, an
overview of proposed alternatives, the state of the current
discussion, and a discussion of short-circuit behavior.

Following the discussion, a vote was held.  While there was an overall
interest in having some form of if-then-else expressions, no one
format was able to draw majority support.  Accordingly, the PEP was
rejected due to the lack of an overwhelming majority for change.
Also, a Python design principle has been to prefer the status quo
whenever there are doubts about which path to take.


Proposal
========

The proposed syntax is as follows::

    (if <condition>: <expression1> else: <expression2>)

Note that the enclosing parentheses are not optional.

The resulting expression is evaluated like this:

- First, <condition> is evaluated.

- If <condition> is true, <expression1> is evaluated and is the
  result of the whole thing.

- If <condition> is false, <expression2> is evaluated and is the
  result of the whole thing.

A natural extension of this syntax is to allow one or more 'elif'
parts::

  (if <cond1>: <expr1> elif <cond2>: <expr2> ... else: <exprN>)

This will be implemented if the proposal is accepted.

The downsides to the proposal are:

* the required parentheses
* confusability with statement syntax
* additional semantic loading of colons

Note that at most one of <expression1> and <expression2> is
evaluated.  This is called a "short-circuit expression"; it is
similar to the way the second operand of 'and' / 'or' is only
evaluated if the first operand is true / false.

A common way to emulate an if-then-else expression is::

    <condition> and <expression1> or <expression2>

However, this doesn't work the same way: it returns <expression2>
when <expression1> is false!  See FAQ 4.16 for alternatives that
work -- however, they are pretty ugly and require much more effort
to understand.


Alternatives
============

Holger Krekel proposed a new, minimally invasive variant::

    <condition> and <expression1> else <expression2>

The concept behind it is that a nearly complete ternary operator
already exists with and/or and this proposal is the least invasive
change that makes it complete.  Many respondants on the
newsgroup found this to be the most pleasing alternative.
However, a couple of respondants were able to post examples
that were mentally difficult to parse.  Later it was pointed
out that this construct works by having the "else" change the
existing meaning of "and".

As a result, there is increasing support for Christian Tismer's
proposed variant of the same idea::

    <condition> then <expression1> else <expression2>

The advantages are simple visual parsing, no required parentheses,
no change in the semantics of existing keywords, not as likely
as the proposal to be confused with statement syntax, and does
not further overload the colon.  The disadvantage is the
implementation costs of introducing a new keyword.  However,
unlike other new keywords, the word "then" seems unlikely to
have been used as a name in existing programs.

---

Many C-derived languages use this syntax::

    <condition> ? <expression1> : <expression2>

Eric Raymond even implemented this.  The BDFL rejected this for
several reasons: the colon already has many uses in Python (even
though it would actually not be ambiguous, because the question
mark requires a matching colon); for people not used to C-derived
language, it is hard to understand.

---

The original version of this PEP proposed the following syntax::

    <expression1> if <condition> else <expression2>

The out-of-order arrangement was found to be too uncomfortable
for many of participants in the discussion; especially when
<expression1> is long, it's easy to miss the conditional while
skimming.

---

Some have suggested adding a new builtin instead of extending the
syntax of the language.  For example::

    cond(<condition>, <expression1>, <expression2>)

This won't work the way a syntax extension will because both
expression1 and expression2 must be evaluated before the function
is called.  There's no way to short-circuit the expression
evalu


id: pep-0687.rst
content:
PEP: 687
Title: Isolating modules in the standard library
Author: Erlend Egeberg Aasland <erlend@python.org>, Petr Viktorin <encukou@gmail.com>
Discussions-To: https://discuss.python.org/t/14824
Status: Accepted
Type: Standards Track
Requires: 489, 573, 630
Created: 04-Apr-2022
Python-Version: 3.12
Post-History: `04-Apr-2022 <https://discuss.python.org/t/14824>`__,
              `11-Apr-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/B3HYQIE4Z5WBJCC3FUZJZHXLM32I4BZA/>`__
Resolution: https://discuss.python.org/t/14824/4

Abstract
========

Extensions in the standard library will be converted to multi-phase
initialization (:pep:`489`) and where possible, all state will be
stored on module objects rather than in process-global variables.


Note on Backdating
==================

Much of this proposal has already been implemented.
We submit this PEP to explain the changes, seek consensus on
whether they are good,  propose the remaining changes,
and set best practices for new modules.



Motivation & Rationale
======================

The informational :pep:`630` describes the background, motivation, rationale,
implications and implementation notes of the proposed changes as they apply
generally to any extension module (not just the standard library).

It is an integral part of this proposal. Read it first.

This PEP discusses specifics of the standard library.


Specification
=============

The body of :pep:`630` will be converted to a HOWTO in the Python
documentation, and that PEP will be retired (marked Final).

All extension modules in the standard library will be converted to multi-phase
initialization introduced in :pep:`489`.

All stdlib extension modules will be *isolated*. That is:

- Types, functions and other objects defined by the module will either be
  immutable, or not shared with other module instances.

- State specific to the module will not be shared with other module instances,
  unless it represents global state.

  For example, ``_csv.field_size_limit`` will get/set a module-specific
  number. On the other hand, functions like ``readline.get_history_item`` or
  ``os.getpid`` will continue to work with state that is process-global
  (external to the module, and possibly shared across other libraries, including
  non-Python ones).

Conversion to heap types
------------------------

Static types that do not need module state access, and have no other reason to
be converted, should stay static.

Types whose methods need access to their module instance will be converted
to heap types following :pep:`630`, with the following considerations:

- All standard library types that used to be static types should remain
  immutable. Heap types must be defined with the ``Py_TPFLAGS_IMMUTABLE_TYPE``
  flag to retain immutability.
  See `bpo-43908 <https://bugs.python.org/issue43908>`__.

  Tests should ensure ``TypeError`` is raised when trying to create a new
  attribute of an immutable type.

- A static type with ``tp_new = NULL`` does not have a public constructor, but
  heap types inherit the constructor from the base class. Make sure types that
  previously were impossible to instantiate retain that feature; use
  ``Py_TPFLAGS_DISALLOW_INSTANTIATION``. Add tests using
  ``test.support.check_disallow_instantiation()``. See
  `bpo-43916 <https://bugs.python.org/issue43916>`__.

- Converted heap types may unintentionally become serializable
  (``pickle``-able). Test that calling ``pickle.dumps`` has the same result
  before and after conversion, and if the test fails, add a ``__reduce__``
  method that raises ``TypeError``. See `PR-21002 <https://github.com/python/cpython/pull/21002/files>`__
  for an example.

These issues will be added to the Devguide to help any future conversions.

If another kind of issue is found, the module in question should be unchanged
until a solution is found and added to the Devguide, and already
converted modules are checked and fixed.


Process
-------

The following process should be added to the Devguide, and remain until
all modules are converted.
Any new findings should be documented there or in the general HOWTO.

Part 1: Preparation
...................

1. Open a discussion, either on the bug tracker or on Discourse. Involve the
   module maintainer and/or code owner. Explain the reason and rationale for
   the changes.
2. Identify global state performance bottlenecks.
   Create a proof-of-concept implementation, and measure the performance impact.
   ``pyperf`` is a good tool for benchmarking.
3. Create an implementation plan. For small modules with few types, a single PR
   may do the job. For larger modules with lots of types, and possibly also
   external library callbacks, multiple PR's will be needed.


Part 2: Implementation
......................

Note: this is a suggested implementation plan for a complex module, based on
lessons learned with other modules. Feel free to simplify it for
smaller modules.

1. Add Argument Clinic where possible; it enables you to easily use the
   defining class to fetch module state from type methods.
2. Prepare for module state; establish a module state ``struct``, add an instance
   as a static global variable, and create helper stubs for fetching the module
   state.
3. Add relevant global variables to the module state ``struct``, and modify code
   that accesses the global state to use the module state helpers instead. This
   step may be broken into several PR's.
4. Where necessary, convert static types to heap types.
5. Convert the global module state struct to true module state.
6. Implement multi-phase initialisation.

Steps 4 through 6 should preferably land in a single alpha development phase.


Backwards Compatibility
=======================

Extension modules in the standard library will now be loadable more than once.
For example, deleting such a module from ``sys.modules`` and re-importing it
will result in a fresh module instance, isolated from any previously loaded
instances.

This may affect code that expected the previous behavior: globals of
extension modules were shallowly copied from the first loaded module.


Security Implications
=====================

None known.


How to Teach This
=================

A large part of this proposal is a HOWTO aimed at experienced users,
which will be moved to the documentation.

Beginners should not be affected.


Reference Implementation
========================

Most of the changes are now in the main branch, as commits for these issues:

- `bpo-40077, Convert static types to heap types: use PyType_FromSpec() <https://bugs.python.org/issue40077>`_
- `bpo-46417, Clear static types in Py_Finalize() for embedded Python <https://bugs.python.org/issue46417>`_
- `bpo-1635741, Py_Finalize() doesn't clear all Python objects at exit <https://bugs.python.org/issue1635741>`_

As an example, changes and fix-ups done in the ``_csv`` module are:

- `GH-23224, Remove static state from the _csv module <https://github.com/python/cpython/pull/23224>`_
- `GH-26008, Allow subclassing of csv.Error <https://github.com/python/cpython/pull/26008>`_
- `GH-26074, Add GC support to _csv heap types <https://github.com/python/cpython/pull/26074>`_
- `GH-26351, Make heap types converted during 3.10 alpha immutable <https://github.com/python/cpython/pull/26351>`_


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0693.rst
content:
PEP: 693
Title: Python 3.12 Release Schedule
Author: Thomas Wouters <thomas@python.org>
Status: Active
Type: Informational
Topic: Release
Created: 24-May-2022
Python-Version: 3.12


Abstract
========

This document describes the development and release schedule for
Python 3.12.

Release Manager and Crew
========================

- 3.12 Release Manager: Thomas Wouters
- Windows installers: Steve Dower
- Mac installers: Ned Deily
- Documentation: Julien Palard

Release Schedule
================

3.12.0 schedule
---------------

Note: the dates below use a 17-month development period that results
in a 12-month release cadence between feature versions, as defined by
:pep:`602`.

Actual:

- 3.12 development begins: Sunday, 2022-05-08
- 3.12.0 alpha 1: Monday, 2022-10-24
- 3.12.0 alpha 2: Monday, 2022-11-14
- 3.12.0 alpha 3: Tuesday, 2022-12-06
- 3.12.0 alpha 4: Tuesday, 2023-01-10
- 3.12.0 alpha 5: Tuesday, 2023-02-07
- 3.12.0 alpha 6: Tuesday, 2023-03-07
- 3.12.0 alpha 7: Tuesday, 2023-04-04
- 3.12.0 beta 1: Monday, 2023-05-22
  (No new features beyond this point.)
- 3.12.0 beta 2: Tuesday, 2023-06-06
- 3.12.0 beta 3: Monday, 2023-06-19
- 3.12.0 beta 4: Tuesday, 2023-07-11
- 3.12.0 candidate 1: Sunday, 2023-08-06
- 3.12.0 candidate 2: Wednesday, 2023-09-06
- 3.12.0 candidate 3: Tuesday, 2023-09-19
- 3.12.0 final:  Monday, 2023-10-02

Bugfix releases
---------------

Actual:

- 3.12.1: Thursday, 2023-12-07
- 3.12.2: Tuesday, 2024-02-06
- 3.12.3: Tuesday, 2024-04-09
- 3.12.4: Thursday, 2024-06-06
- 3.12.5: Tuesday, 2024-08-06
- 3.12.6: Friday, 2024-09-06
- 3.12.7: Tuesday, 2024-10-01
- 3.12.8: Tuesday, 2024-12-03
- 3.12.9: Tuesday, 2025-02-04
- 3.12.10: Tuesday, 2025-04-08 (final regular bugfix release with binary installers)

Source-only security fix releases
---------------------------------

Provided irregularly on an as-needed basis until October 2028.

3.12 Lifespan
-------------

3.12 received bugfix updates approximately every 2 months for
approximately 18 months.  Some time after the release of 3.13.0 final,
the tenth and final 3.12 bugfix update was released.  After that,
it is expected that security updates (source only) will be released
until 5 years after the release of 3.12.0 final, so until approximately
October 2028.


Features for 3.12
=================

New features can be found in `Whatâs New In Python 3.12
<https://docs.python.org/3.12/whatsnew/3.12.html>`__.


Copyright
=========

This document is placed in the public domain or under the CC0-1.0-Universal
license, whichever is more permissive.


id: pep-0644.rst
content:
PEP: 644
Title: Require OpenSSL 1.1.1 or newer
Author: Christian Heimes <christian@python.org>
Discussions-To: https://discuss.python.org/t/pep-644-require-openssl-1-1-or-newer/5584
Status: Final
Type: Standards Track
Created: 27-Oct-2020
Python-Version: 3.10
Post-History: 27-Oct-2020, 03-Mar-2021, 17-Mar-2021, 17-Apr-2021
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/INLCO2EZVQW7R7J2OL6HWVLVU3TQRAZV/


Abstract
========

This PEP proposes for CPythonâs standard library to support only OpenSSL
1.1.1 LTS or newer. Support for OpenSSL versions past end-of-lifetime,
incompatible forks, and other TLS libraries are dropped.


Motivation
==========

Python makes use of OpenSSL in ``hashlib``, ``hmac``, and ``ssl`` modules. OpenSSL
provides fast implementations of cryptographic primitives and a full TLS
stack including handling of X.509 certificates. The ``ssl`` module is used by
standard library modules like ``urllib`` and 3rd party modules like ``urllib3``
to implement secure variants of internet protocols. ``pip`` uses the ``ssl``
module to securely download packages from PyPI. Any bug in the ``ssl`` module's
bindings to OpenSSL can lead to a severe security issue.

Over time OpenSSL's public API has evolved and changed. Version 1.0.2
introduced new APIs to verify and match hostnames. OpenSSL 1.1.0 made
internal structs opaque and introduced new APIs that replace direct access of
struct members. Version 3.0.0 will deprecate more APIs due to internal
reorganization that moves cryptographic algorithms out of the core and into
providers. Forks like LibreSSL and BoringSSL have diverged in different
directions.

Currently Python versions 3.6 to 3.9 are compatible with OpenSSL 1.0.2,
1.1.0, and 1.1.1. For the most part Python also works with LibreSSL >= 2.7.1
with some missing features and broken tests.

Due to limited resources and time it becomes increasingly hard to support
multiple versions and forks as well as test and verify correctness. Besides
multiple incompatible APIs there are build time flags,
distribution-specific patches, and local crypto-policy settings that add to
plethora of combinations. On the other hand, the Python core team has only
a couple of domain experts who are familiar with TLS and OpenSSL internals
and even fewer who are active maintainers.

Requiring OpenSSL 1.1.1 would allow us to give the vast majority of users a
better experience, reduce our maintenance overhead and thus free resources
to implement new features. Users would be able to rely on the presence of
new features and consistent behavior, ultimately resulting in a more robust
experience.


Impact
======

OpenSSL 1.1.1 is the default variant and version of OpenSSL on almost all
supported platforms and distributions. Itâs also the only version that still
receives security support from upstream [9]_.

No macOS and Windows user will be affected by the deprecation. The python.org
installer and alternative distributions like Conda ship with most recent
OpenSSL version.

As of October 2020 and according to DistroWatch [1]_ most current BSD and
Linux distributions ship with OpenSSL 1.1.1 as well. Some older releases of
long-term support (LTS) and enterprise distributions have older versions of
OpenSSL or LibreSSL. By the time Python 3.10 will be generally available,
several of these distributions will have reached end of lifetime, end of
general support, or moved from LibreSSL to OpenSSL.

Other software has dropped support for OpenSSL 1.0.2 as well. For example,
PyCA cryptography 3.2 (2020-10-25) removed compatibility with OpenSSL 1.0.2.


OpenSSL 1.0.2 LTS
-----------------

released: 2015-02
end of lifetime: 2019-12

OpenSSL 1.0.2 added hostname verification, ALPN support, and elliptic curves.

- CentOS 7 (EOL 2024-06)
- Debian 8 Jessie (EOL 2020-07)
- Linux Mint 18.3 (EOL 2021-04)
- RHEL 7 (full support ends 2019-08, maintenance 2 support ends 2024-06)
- SUSE Enterprise Linux 12-SP5 (general supports ends 2024-10)
- Ubuntu 16.04 LTS / Xenial (general support ends 2021-04)


OpenSSL 1.1.0
-------------

released: 2016-08
end of lifetime: 2019-09

OpenSSL 1.1.0 removed or disabled insecure ciphers by default and added
support for ChaCha20-Poly1305, BLAKE2 (basic features), X25519 and CT. The
majority of structs were made opaque and new APIs were introduced. OpenSSL
1.1.0 is not API compatible with 1.0.2.

- Debian 9 Stretch (security support ended 2020-07, LTS until 2022-06)
- Ubuntu 18.04 LTS / Bionic (general support ends 2023-04)


OpenSSL 1.1.1 LTS
-----------------

released: 2018-08
end of lifetime: 2023-09 (planned)

OpenSSL 1.1.1 added TLS 1.3, SHA-3, X448 and Ed448.

- Alpine (switched back to OpenSSL in 2018 [4]_)
- Arch Linux current
- CentOS 8.0+
- Debian 10 Buster
- Debian 11 Bullseye (ETA 2021-06)
- Fedora 29+
- FreeBSD 11.3+
- Gentoo Linux stable (dropped LibreSSL as alternative in January 2021 [10]_)
- HardenedBSD (switched back to OpenSSL in 2018 [3]_)
- Linux Mint 19.3+
- macOS (python.org installer)
- NetBSD 8.2+
- openSUSE 15.2+
- RHEL 8.0+
- Slackware current
- SUSE Enterprise Linux 15-SP2
- Ubuntu 18.10+
- Ubuntu 20.04 LTS / Focal
- VoidLinux (switched back to OpenSSL in March 2021 [5]_)
- Windows (python.org installer, Conda)

Major CI providers provide images with OpenSSL 1.1.1.

- AppVeyor (with image ``Ubuntu2004``)
- CircleCI (with recent ``cimg/base:stable`` or ``cimg/base:stable-20.04``)
- GitHub Actions (with ``runs-on: ubuntu-20.04``)
- Giblab CI (with Debian Stretch, Ubuntu Focal, CentOS 8, RHEL 8, or Fedora
  runner)
- Packit
- TravisCI (with ``dist: focal``)
- Zuul


OpenSSL 3.0.0
-------------

released: n/a (planned for mid/late 2021)

OpenSSL 3.0.0 is currently under development. Major changes include
relicensing to Apache License 2.0 and a new API for cryptographic algorithms
providers. Most changes are internal refactorings and donât affect public
APIs. [8]_


LibreSSL
--------

created: 2014-04 (forked from OpenSSL 1.0.1g)

- DragonFly BSD
- Hyperbola GNU/Linux-libre
- OpenBSD
- OpenELEC (discontinued)
- TrueOS (discontinued)

Some distributions like FreeBSD and OPNsense also feature LibreSSL
instead of OpenSSL as non-standard TLS libraries. Gentoo discontinued
LibreSSL as an alternative to OpenSSL in January 2021 [10]_ due to
compatibility issues and little testing.

OpenBSD ports has a port ``security/openssl/1.1`` which is documented as
"[...] is present to provide support for applications which cannot be made
compatible with LibReSSL" [7]_. The package could be used by OpenBSD to
provide a working ssl module.


BoringSSL
---------

created: 2014-06

BoringSSL is Googleâs fork of OpenSSL. Itâs not intended for general use and
therefore not supported by Python. There are no guarantees of API or ABI
stability.  Vendored copies of BoringSSL are used in Chrome/Chromium browser,
Android, and on Apple platforms [6]_.


Benefits
========

TLS 1.3
-------

OpenSSL 1.1.1 introduced support for the new TLS 1.3 version. The latest
version of the TLS protocol has a faster handshake and is more secure than
the previous versions.

Thread and fork safety
----------------------

Starting with release 1.1.0c, OpenSSL is fully fork and thread safe.
Bindings no longer need any workarounds or additional callbacks to support
multithreading.

SHA-3
-----

Since 1.1.0, OpenSSL ships with SHA-3 and SHAKE implementations.
Python's builtin SHA-3 support is based on the reference implementation. The
internal ``_sha3`` code is fairly large and the resulting shared library close
to 0.5 MB. Python could drop the builtin implementation and rely on OpenSSL's
``libcrypto`` instead.

So far LibreSSL upstream development has refused to add SHA-3 support. [2]_


Compatibility
=============

OpenSSL downstream patches and options
--------------------------------------

OpenSSL features more than 70 configure and build time options in the form
of  ``OPENSSL_NO_*`` macros. Around 60 options affect the presence of features
like 


id: pep-0650.rst
content:
PEP: 650
Title: Specifying Installer Requirements for Python Projects
Author: Vikram Jayanthi <vikramjayanthi@google.com>,
        Dustin Ingram <di@python.org>,
        Brett Cannon <brett@python.org>
Discussions-To: https://discuss.python.org/t/pep-650-specifying-installer-requirements-for-python-projects/6657
Status: Withdrawn
Type: Standards Track
Topic: Packaging
Created: 16-Jul-2020
Post-History: 14-Jan-2021


Abstract
========

Python package installers are not completely interoperable with each
other. While pip is the most widely used installer and a de facto
standard, other installers such as Poetry_ or Pipenv_ are popular as
well due to offering unique features which are optimal for certain
workflows and not directly in line with how pip operates.

While the abundance of installer options is good for end-users with
specific needs, the lack of interoperability between them makes it
hard to support all potential installers. Specifically, the lack of a
standard requirements file for declaring dependencies means that each
tool must be explicitly used in order to install dependencies
specified with their respective format. Otherwise tools must emit a
requirements file which leads to potential information loss for the
installer as well as an added export step as part of a developer's
workflow.

By providing a standardized API that can be used to invoke a
compatible installer, we can solve this problem without needing to
resolve individual concerns, unique requirements, and
incompatibilities between different installers and their lock files.

Installers that implement the specification can be invoked in a
uniform way, allowing users to use their installer of choice as if
they were invoking it directly.

Terminology
===========

Installer interface
    The interface by which an *installer backend* and a
    *universal installer* interact.

Universal installer
    An installer that can invoke an *installer backend* by calling the
    optional invocation methods of the *installer interface*. This can
    also be thought of as the installer frontend, Ã  la the build_
    project for :pep:`517`.

Installer backend
    An installer that implements the *installer interface*, allowing
    it to be invoked by a *universal installer*. An
    *installer backend* may also be a *universal installer* as well,
    but it is not required. In comparison to :pep:`517`, this would
    be Flit_. *Installer backends* may be wrapper packages around
    a backing installer, e.g. Poetry could choose to not support this
    API, but a package could act as a wrapper to invoke Poetry as
    appropriate to use Poetry to perform an installation.

Dependency group
    A set of dependencies that are related and required to be
    installed simultaneously for some purpose. For example, a
    "test" dependency group could include the dependencies required to
    run the test suite. How dependency groups are specified is up to
    the *installer backend*.


Motivation
==========

This specification allows anyone to invoke and interact with
*installer backends* that implement the specified interface, allowing
for a universally supported layer on top of existing tool-specific
installation processes.

This in turn would enable the use of all installers that implement the
specified interface to be used in environments that support a single
*universal installer*, as long as that installer implements this
specification as well.

Below, we identify various use-cases applicable to stakeholders in the
Python community and anyone who interacts with Python package
installers. For developers or companies, this PEP would allow for
increased functionality and flexibility with Python package
installers.

Providers
---------

Providers are the parties (organization, person, community, etc.) that
supply a service or software tool which interacts with Python
packaging and consequently Python package installers. Two different
types of providers are considered:

Platform/Infrastructure Providers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Platform providers (cloud environments, application hosting, etc.) and
infrastructure service providers need to support package installers
for their users to install Python dependencies. Most only support pip,
however there is user demand for other Python installers. Most
providers do not want to maintain support for more than one installer
because of the complexity it adds to their software or service and the
resources it takes to do so.

Via this specification, we can enable a provider-supported
*universal installer* to invoke the user-desired *installer backend*
without the providerâs platform needing to have specific knowledge of
said backend. What this means is if Poetry implemented the installer
backend API proposed by this PEP (or some other package wrapped Poetry
to provide the API), then platform providers would support Poetry
implicitly.

IDE Providers
^^^^^^^^^^^^^

Integrated development environments may interact with Python package
installation and management. Most only support pip as a Python package
installer, and users are required to find work arounds to install
their dependencies using other package installers. Similar to the
situation with PaaS & IaaS providers, IDE providers do not want to
maintain support for N different Python installers. Instead,
implementers of the installer interface (*installer backends*) could
be invoked by the IDE by it acting as a *universal installer*.

Developers
----------

Developers are teams, people, or communities that code and use Python
package installers and Python packages. Three different types of
developers are considered:

Developers using PaaS & IaaS providers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Most PaaS and IaaS providers only support one Python package
installer: pip_. (Some exceptions include Heroku's Python buildpack_,
which supports pip and Pipenv_). This dictates the installers that
developers can use while working with these providers, which might not
be optimal for their application or workflow.

Installers adopting this PEP to become *installer backends* would allow
users to use third party platforms/infrastructure without having to
worry about which Python package installer they are required to use as
long as the provider uses a *universal installer*.

Developers using IDEs
^^^^^^^^^^^^^^^^^^^^^

Most IDEs only support pip or a few Python package installers.
Consequently, developers must use workarounds or hacky methods to
install their dependencies if they use an unsupported package
installer.

If the IDE uses/provides a *universal installer* it would allow for
any *installer backend* that the developer wanted to be used to
install dependencies, freeing them of any extra work to install their
dependencies in order to integrate into the IDE's workflow more
closely.

Developers working with other developers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Developers want to be able to use the installer of their choice while
working with other developers, but currently have to synchronize their
installer choice for compatibility of dependency installation. If all
preferred installers instead implemented the specified interface, it
would allow for cross use of installers, allowing developers to choose
an installer regardless of their collaboratorâs preference.

Upgraders & Package Infrastructure Providers
--------------------------------------------

Package upgraders and package infrastructure in CI/CD such as
Dependabot_, PyUP_, etc. currently support a few installers. They work
by parsing and editing the installer-specific dependency files
directly (such as ``requirements.txt`` or ``poetry.lock``) with
relevant package information such as upgrades, downgrades, or new
hashes. Similar to Platform and IDE providers, most of these providers
do not want to support N different Python package installers as that
would require supporting N different file types.

Currently, these services/bots have to implement


id: pep-0678.rst
content:
PEP: 678
Title: Enriching Exceptions with Notes
Author: Zac Hatfield-Dodds <zac@zhd.dev>
Sponsor: Irit Katriel
Discussions-To: https://discuss.python.org/t/pep-678-enriching-exceptions-with-notes/13374
Status: Final
Type: Standards Track
Requires: 654
Created: 20-Dec-2021
Python-Version: 3.11
Post-History: `27-Jan-2022 <https://discuss.python.org/t/pep-678-enriching-exceptions-with-notes/13374>`__
Resolution: https://discuss.python.org/t/pep-678-enriching-exceptions-with-notes/13374/100

.. canonical-doc:: :meth:`python:BaseException.add_note` and :attr:`python:BaseException.__notes__`

   See :ref:`python:tut-exception-notes` for a user-focused tutorial.

Abstract
========
Exception objects are typically initialized with a message that describes the
error which has occurred.  Because further information may be available when
the exception is caught and re-raised, or included in an ``ExceptionGroup``,
this PEP proposes to add ``BaseException.add_note(note)``, a
``.__notes__`` attribute holding a list of notes so added, and to
update the builtin traceback formatting code to include notes in the formatted
traceback following the exception string.

This is particularly useful in relation to :pep:`654` ``ExceptionGroup``\ s,
which make previous workarounds ineffective or confusing.  Use cases have been
identified in the standard library, Hypothesis and ``cattrs`` packages, and
common code patterns with retries.


Motivation
==========
When an exception is created in order to be raised, it is usually initialized
with information that describes the error that has occurred. There are cases
where it is useful to add information after the exception was caught. For
example,

- testing libraries may wish to show the values involved in a failing
  assertion, or the steps to reproduce a failure (e.g. ``pytest`` and
  ``hypothesis``; example below).
- code which retries an operation on error may wish to associate an iteration,
  timestamp, or other explanation with each of several errors - especially if
  re-raising them in an ``ExceptionGroup``.
- programming environments for novices can provide more detailed descriptions
  of various errors, and tips for resolving them.

Existing approaches must pass this additional information around while keeping
it in sync with the state of raised, and potentially caught or chained,
exceptions. This is already error-prone, and made more difficult by :pep:`654`
``ExceptionGroup``\ s, so the time is right for a built-in solution.  We
therefore propose to add:

- a new method ``BaseException.add_note(note: str)``,
- ``BaseException.__notes__``, a list of note strings added using
  ``.add_note()``, and
- support in the builtin traceback formatting code such that notes are
  displayed in the formatted traceback following the exception string.


Example usage
-------------
::

   >>> try:
   ...     raise TypeError('bad type')
   ... except Exception as e:
   ...     e.add_note('Add some information')
   ...     raise
   ...
   Traceback (most recent call last):
     File "<stdin>", line 2, in <module>
   TypeError: bad type
   Add some information
   >>>

When collecting exceptions into an exception group, we may want to add context
information for the individual errors. In the following example with
`Hypothesis' proposed support for ExceptionGroup
<https://github.com/HypothesisWorks/hypothesis/pull/3191>`__, each exception
includes a note of the minimal failing example::

    from hypothesis import given, strategies as st, target

    @given(st.integers())
    def test(x):
        assert x < 0
        assert x > 0


    + Exception Group Traceback (most recent call last):
    |   File "test.py", line 4, in test
    |     def test(x):
    |
    |   File "hypothesis/core.py", line 1202, in wrapped_test
    |     raise the_error_hypothesis_found
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | ExceptionGroup: Hypothesis found 2 distinct failures.
    +-+---------------- 1 ----------------
        | Traceback (most recent call last):
        |   File "test.py", line 6, in test
        |     assert x > 0
        |     ^^^^^^^^^^^^
        | AssertionError: assert -1 > 0
        |
        | Falsifying example: test(
        |     x=-1,
        | )
        +---------------- 2 ----------------
        | Traceback (most recent call last):
        |   File "test.py", line 5, in test
        |     assert x < 0
        |     ^^^^^^^^^^^^
        | AssertionError: assert 0 < 0
        |
        | Falsifying example: test(
        |     x=0,
        | )
        +------------------------------------


Non-goals
---------
Tracking multiple notes as a list, rather than by concatenating strings when
notes are added, is intended to maintain the distinction between the
individual notes. This might be required in specialized use cases, such
as translation of the notes by packages like ``friendly-traceback``.

However, ``__notes__`` is *not* intended to carry structured data.  If your
note is for use by a program rather than display to a human, `we recommend
<https://discuss.python.org/t/accepting-pep-654-exception-groups-and-except/10813/26>`__
instead (or additionally) choosing a convention for an attribute, e.g.
``err._parse_errors = ...`` on the error or ``ExceptionGroup``.

As a rule of thumb, we suggest that you should prefer `exception chaining
<https://docs.python.org/3/tutorial/errors.html#exception-chaining>`__ when the
error is going to be re-raised or handled as an individual error, and prefer
``.add_note()`` when you want to avoid changing the exception type or
are collecting multiple exception objects to handle together. [1]_


Specification
=============

``BaseException`` gains a new method ``.add_note(note: str)``. If ``note`` is
a string, ``.add_note(note)`` appends it to the ``__notes__`` list, creating
the attribute if it does not already exist.  If ``note`` is not a string,
``.add_note()`` raises ``TypeError``.

Libraries may clear existing notes by modifying or deleting the ``__notes__``
list, if it has been created, including clearing all notes with
``del err.__notes__``.  This allows full control over the attached notes,
without overly complicating the API or adding multiple names to
``BaseException.__dict__``.

When an exception is displayed by the interpreter's builtin traceback-rendering code,
its notes (if there are any) appear immediately after the exception message, in the order
in which they were added, with each note starting on a new line.

If ``__notes__`` has been created, ``BaseExceptionGroup.subgroup`` and
``BaseExceptionGroup.split`` create a new list for each new instance, containing
the same contents as the original exception group's ``__notes__``.

We *do not* specify the expected behaviour when users have assigned a non-list
value to ``__notes__``, or a list which contains non-string elements.
Implementations might choose to emit warnings, discard or ignore bad values,
convert them to strings, raise an exception, or do something else entirely.


Backwards Compatibility
=======================

System-defined or "dunder" names (following the pattern ``__*__``) are part of
the language specification, with `unassigned names reserved for future use and
subject to breakage without warning
<https://docs.python.org/3/reference/lexical_analysis.html#reserved-classes-of-identifiers>`__.
We are also unaware of any code which *would* be broken by adding ``__notes__``.

We were also unable to find any code which would be broken by the addition of
``BaseException.add_note()``: while searching Google and `GitHub finds several
definitions <https://grep.app/search?q=.add_note%28&filter[lang][0]=Python>`__
of an ``.add_note()`` method, none of them are on a subclass of
``BaseException``.


How to Teach This
=================

The ``add_note()`` method and ``__notes__`` attribute will be documented as
part of the language standard, and explained as part of `the "Errors and
Exceptions" tutorial <https://github.com/python/cpyth


id: pep-0730.rst
content:
PEP: 730
Title: Adding iOS as a supported platform
Author: Russell Keith-Magee <russell@keith-magee.com>
Sponsor: Ned Deily <nad@python.org>
Discussions-To: https://discuss.python.org/t/pep730-adding-ios-as-a-supported-platform/35854
Status: Final
Type: Standards Track
Created: 09-Oct-2023
Python-Version: 3.13
Resolution: https://discuss.python.org/t/pep-730-adding-ios-as-a-supported-platform/35854/66

.. canonical-doc:: :ref:`python:using-ios`


Abstract
========

This PEP proposes adding iOS as a supported platform in CPython. The initial
goal is to achieve Tier 3 support for Python 3.13. This PEP describes the
technical aspects of the changes that are required to support iOS. It also
describes the project management concerns related to adoption of iOS as a Tier 3
platform.

Motivation
==========

Over the last 15 years, mobile platforms have become increasingly important
parts of the computing landscape. iOS is one of two operating systems that
control the vast majority of these devices. However, there is no official
support for iOS in CPython.

The `BeeWare Project <https://beeware.org>`__ and `Kivy <https://kivy.org>`__
have both supported iOS for almost 10 years. This support has been able to
generate applications that have been accepted for publication in the iOS App
Store. This demonstrates the technical feasibility of iOS support.

It is important for the future of Python as a language that it is able to be
used on any hardware or OS that has widespread adoption. If Python cannot be
used a on a platform that has widespread use, adoption of the language will be
impacted as potential users will adopt other languages that *do* provide support
for these platforms.

Rationale
=========

Development landscape
---------------------

iOS provides a single API, but 2 distinct ABIs - ``iphoneos`` (physical
devices), and ``iphonesimulator``. Each of these ABIs can be provided on
multiple CPU architectures. At time of writing, Apple officially supports
``arm64`` on the device ABI, and ``arm64`` and ``x86_64`` are supported on the
simulator ABI.

As with macOS, iOS supports the creation of "fat" binaries that contains
multiple CPU architectures. However, fat binaries *cannot* span ABIs. That is,
it is possible to have a fat *simulator* binary, and a fat *device* binary, but
it is not possible to create a single fat "iOS" binary that covers both
simulator and device needs. To support distribution of a single development
artefact, Apple uses an "XCframework" structure - a wrapper around multiple ABIs
that implement a common API.

iOS runs on a Darwin kernel, similar to macOS. However, there is a need to
differentiate between macOS and iOS at an implementation level, as there are
significant platform differences between iOS and macOS.

iOS code is compiled for compatibility against a minimum iOS version.

Apple frequently refers to "iPadOS" in their marketing material. However, from a
development perspective, there is no discernable difference between iPadOS and
iOS. A binary that has been compiled for the ``iphoneos`` or ``iphonesimulator``
ABIs can be deployed on iPad.

Other Apple platforms, such as tvOS, watchOS, and visionOS, use different ABIs,
and are not covered by this PEP.

POSIX compliance
----------------

iOS is broadly a POSIX platform. However, similar to WASI/Emscripten, there are
POSIX APIs that exist on iOS, but cannot be used; and POSIX APIs that don't
exist at all.

Most notable of these is the fact that iOS does not provide any form of
multiprocess support. ``fork`` and ``spawn`` both *exist* in the iOS API;
however, if they are invoked, the invoking iOS process stops, and the new
process doesn't start.

Unlike WASI/Emscripten, threading *is* supported on iOS.

There are also significant limits to socket handling. Due to process sandboxing,
there is no availability of interprocess communication via socket. However,
sockets for network communication *are* available.

Dynamic libraries
-----------------

The iOS `App Store guidelines
<https://developer.apple.com/app-store/review/guidelines>`__ allow apps to be
written in languages other than Objective C or Swift. However, they have very
strict guidelines about the structure of apps that are submitted for
distribution.

iOS apps can use dynamically loaded libraries; however, there are very strict
requirements on how dynamically loaded content is packaged for use on iOS:

* Dynamic binary content must be compiled as dynamic libraries, not shared
  objects or binary bundles.

* They must be packaged in the app bundle as Frameworks.

* Each Framework can only contain a single dynamic library.

* The Framework *must* be contained in the iOS App's ``Frameworks`` folder.

* A Framework may not contain any non-library content.

This imposes some constraints on the operation of CPython. It is not possible
store binary modules in the ``lib-dynload`` and/or ``site-packages`` folders;
they must be stored in the app's Frameworks folder, with each module wrapped in
a Framework. This also means that the common assumption that a Python module can
construct the location of a binary module by using the ``__file__`` attribute of
the Python module no longer holds.

As with macOS, compiling a binary module that is accessible from a
statically-linked build of Python requires the use of the ``--undefined
dynamic_lookup`` option to avoid linking ``libpython3.x`` into every binary
module. However, on iOS, this compiler flag raises a deprecation warning when it
is used. A warning from this flag has been observed on macOS as well - however,
responses from Apple staff suggest that they `do not intend to break the CPython
ecosystem by removing this option
<https://github.com/python/cpython/issues/97524#issuecomment-1458855301>`__. As
Python does not currently have a notable presence on iOS, it is difficult to
judge whether iOS usage of this flag would fall under the same umbrella.

Console and interactive usage
-----------------------------

Distribution of a traditional CPython REPL or interactive "python.exe" should
not be considered a goal of this work.

Mobile devices (including iOS) do not provide a TTY-style console. They do not
provide ``stdin``, ``stdout`` or ``stderr``. iOS provides a system log, and it
is possible to install a redirection so that all ``stdout`` and ``stderr``
content is redirected to the system log; but there is no analog for ``stdin``.

In addition, iOS places restrictions on downloading additional code at runtime
(as this behavior would be functionally indistinguishable from trying to work
around App Store review). As a result, a traditional "create a virtual
environment and pip install" development experience will not be viable on iOS.

It is *possible* to build an native iOS application that provides a REPL
interface. This would be closer to an IDLE-style user experience; however,
Tkinter cannot be used on iOS, so any app would require a ground-up rewrite. The
iOS app store already contains several examples of apps in this category (e.g.,
`Pythonista <http://www.omz-software.com/pythonista/>`__ and `Pyto
<https://pyto.readthedocs.io/>`__). The focus of this work would be to provide
an embedded distribution that IDE-style native interfaces could utilize, not a
user-facing "app" interface to iOS on Python.

Specification
=============

Platform identification
-----------------------

``sys``
'''''''

``sys.platform`` will identify as ``"ios"`` on both simulator and physical
devices.

``sys.implementation._multiarch`` will describe the ABI and CPU architecture:

* ``"arm64-iphoneos"`` for ARM64 devices
* ``"arm64-iphonesimulator"`` for ARM64 simulators
* ``"x86_64-iphonesimulator"`` for x86_64 simulators

``platform``
''''''''''''

``platform`` will be modified to support returning iOS-specific details. Most of
the values returned by the ``platform`` module will match those returned by
``os.uname()``, with the exception of:

* ``platform.system()`` - ``"iOS"`` or ``iPadOS`` (depending on


id: pep-0724.rst
content:
PEP: 724
Title: Stricter Type Guards
Author: Rich Chiodo <rchiodo at microsoft.com>,
        Eric Traut <erictr at microsoft.com>,
        Erik De Bonte <erikd at microsoft.com>,
Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-724-stricter-type-guards/34124
Status: Withdrawn
Type: Standards Track
Topic: Typing
Created: 28-Jul-2023
Python-Version: 3.13
Post-History: `30-Dec-2021 <https://mail.python.org/archives/list/typing-sig@python.org/thread/EMUD2D424OI53DCWQ4H5L6SJD2IXBHUL/>`__,
              `19-Sep-2023 <https://discuss.python.org/t/pep-724-stricter-type-guards/34124>`__,

Status
======

This PEP is withdrawn. The Typing Council was unable to reach consensus on
this proposal, and the authors decided to withdraw it.


Abstract
========

:pep:`647` introduced the concept of a user-defined type guard function which
returns ``True`` if the type of the expression passed to its first parameter
matches its return ``TypeGuard`` type. For example, a function that has a
return type of ``TypeGuard[str]`` is assumed to return ``True`` if and only if
the type of the expression passed to its first input parameter is a ``str``.
This allows type checkers to narrow types when a user-defined type guard
function returns ``True``.

This PEP refines the ``TypeGuard`` mechanism introduced in :pep:`647`. It
allows type checkers to narrow types when a user-defined type guard function
returns ``False``. It also allows type checkers to apply additional (more
precise) type narrowing under certain circumstances when the type guard
function returns ``True``.


Motivation
==========

User-defined type guard functions enable a type checker to narrow the type of
an expression when it is passed as an argument to the type guard function. The
``TypeGuard`` mechanism introduced in :pep:`647` is flexible, but this
flexibility imposes some limitations that developers have found inconvenient
for some uses.

Limitation 1: Type checkers are not allowed to narrow a type in the case where
the type guard function returns ``False``. This means the type is not narrowed
in the negative ("else") clause.

Limitation 2: Type checkers must use the ``TypeGuard`` return type if the type
guard function returns ``True`` regardless of whether additional narrowing can
be applied based on knowledge of the pre-narrowed type.

The following code sample demonstrates both of these limitations.

.. code-block:: python

    def is_iterable(val: object) -> TypeGuard[Iterable[Any]]:
        return isinstance(val, Iterable)

    def func(val: int | list[int]):
        if is_iterable(val):
            # The type is narrowed to 'Iterable[Any]' as dictated by
            # the TypeGuard return type
            reveal_type(val)  # Iterable[Any]
        else:
            # The type is not narrowed in the "False" case
            reveal_type(val)  # int | list[int]

        # If "isinstance" is used in place of the user-defined type guard
        # function, the results differ because type checkers apply additional
        # logic for "isinstance"

        if isinstance(val, Iterable):
            # Type is narrowed to "list[int]" because this is
            # a narrower (more precise) type than "Iterable[Any]"
            reveal_type(val)  # list[int]
        else:
            # Type is narrowed to "int" because the logic eliminates
            # "list[int]" from the original union
            reveal_type(val)  # int


:pep:`647` imposed these limitations so it could support use cases where the
return ``TypeGuard`` type was not a subtype of the input type. Refer to
:pep:`647` for examples.

Rationale
=========

There are a number of issues where a stricter ``TypeGuard`` would have
been a solution:

* `Python typing issue - TypeGuard doesn't intersect like isinstance <https://github.com/python/typing/issues/1351>`__
* `Pyright issue - TypeGuard not eliminating possibility on branch <https://github.com/microsoft/pyright/issues/3450>`__
* `Pyright issue - Type narrowing for Literal doesn't work <https://github.com/microsoft/pyright/issues/3466>`__
* `Mypy issue - TypeGuard is incompatible with exhaustive check <https://github.com/python/mypy/issues/15305>`__
* `Mypy issue - Incorrect type narrowing for inspect.isawaitable <https://github.com/python/mypy/issues/15520>`__
* `Typeshed issue - asyncio.iscoroutinefunction is not a TypeGuard <https://github.com/python/typeshed/issues/8009>`__


Specification
=============

The use of a user-defined type guard function involves five types:

* I = ``TypeGuard`` input type
* R = ``TypeGuard`` return type
* A = Type of argument passed to type guard function (pre-narrowed)
* NP = Narrowed type (positive)
* NN = Narrowed type (negative)

.. code-block:: python

    def guard(x: I) -> TypeGuard[R]: ...

    def func1(val: A):
        if guard(val):
            reveal_type(val)  # NP
        else:
            reveal_type(val)  # NN


This PEP proposes some modifications to :pep:`647` to address the limitations
discussed above. These limitations are safe to eliminate only when a specific
condition is met. In particular, when the output type ``R`` of a user-defined
type guard function is consistent [#isconsistent]_ with the type of its first
input parameter (``I``), type checkers should apply stricter type guard
semantics.

  .. code-block:: python

     # Stricter type guard semantics are used in this case because
     # "Kangaroo | Koala" is consistent with "Animal"
     def is_marsupial(val: Animal) -> TypeGuard[Kangaroo | Koala]:
         return isinstance(val, Kangaroo | Koala)

     # Stricter type guard semantics are not used in this case because
     # "list[T]"" is not consistent with "list[T | None]"
     def has_no_nones(val: list[T | None]) -> TypeGuard[list[T]]:
         return None not in val

When stricter type guard semantics are applied, the application of a
user-defined type guard function changes in two ways.

* Type narrowing is applied in the negative ("else") case.

.. code-block:: python

    def is_str(val: str | int) -> TypeGuard[str]:
        return isinstance(val, str)

    def func(val: str | int):
        if not is_str(val):
            reveal_type(val)  # int

* Additional type narrowing is applied in the positive "if" case if applicable.

.. code-block:: python

    def is_cardinal_direction(val: str) -> TypeGuard[Literal["N", "S", "E", "W"]]:
        return val in ("N", "S", "E", "W")

    def func(direction: Literal["NW", "E"]):
        if is_cardinal_direction(direction):
            reveal_type(direction)  # "Literal[E]"
        else:
            reveal_type(direction)  # "Literal[NW]"


The type-theoretic rules for type narrowing are specified in the following
table.

============ ======================= ===================
\            Non-strict type guard   Strict type guard
============ ======================= ===================
Applies when R not consistent with I R consistent with I
NP is ..     :math:`R`               :math:`A \land R`
NN is ..     :math:`A`               :math:`A \land \neg{R}`
============ ======================= ===================

In practice, the theoretic types for strict type guards cannot be expressed
precisely in the Python type system. Type checkers should fall back on
practical approximations of these types. As a rule of thumb, a type checker
should use the same type narrowing logic -- and get results that are consistent
with -- its handling of "isinstance". This guidance allows for changes and
improvements if the type system is extended in the future.


Additional Examples
===================

``Any`` is consistent [#isconsistent]_ with any other type, which means
stricter semantics can be applied.

.. code-block:: python

     # Stricter type guard semantics are used in this case because
     # "str" is consistent with "Any"
    def is_str(x: Any) -> TypeGuard[str]:
        return isinstance(x, str)

    def test(x: float | str):
        if is_str(x):
     


id: pep-0042.rst
content:
PEP: 42
Title: Feature Requests
Author: Jeremy Hylton <jeremy@alum.mit.edu>
Status: Withdrawn
Type: Process
Created: 12-Sep-2000
Post-History:

.. withdrawn::

   It is `obsolete`_.
   All new feature requests should  either go to the `Python bug tracker`_
   for very simple requests or the `Ideas Discourse category`_ for
   everything else.  The rest of this document is retained for historical
   purposes only.


Introduction
============

This PEP contains a list of feature requests that may be considered
for future versions of Python.  Large feature requests should not be
included here, but should be described in separate PEPs; however a
large feature request that doesn't have its own PEP can be listed here
until its own PEP is created.  See :pep:`0` for details.

This PEP was created to allow us to close bug reports that are really
feature requests.  Marked as Open, they distract from the list of real
bugs (which should ideally be less than a page).  Marked as Closed,
they tend to be forgotten.  The procedure now is:  if a bug report is
really a feature request, add the feature request to this PEP; mark
the bug as "feature request", "later", and "closed"; and add a comment
to the bug saying that this is the case (mentioning the PEP
explicitly).  It is also acceptable to move large feature requests
directly from the bugs database to a separate PEP.

This PEP should really be separated into four different categories
(categories due to Laura Creighton):

1. BDFL rejects as a bad idea.  Don't come back with it.

2. BDFL will put in if somebody writes the code.  (Or at any rate,
   BDFL will say 'change this and I will put it in' if you show up
   with code.)

   possibly divided into:

      a) BDFL would really like to see some code!

      b) BDFL is never going to be enthusiastic about this, but
         will work it in when it's easy.

3. If you show up with code, BDFL will make a pronouncement.  It might
   be ICK.

4. This is too vague.  This is rejected, but only on the grounds of
   vagueness.  If you like this enhancement, make a new PEP.


Core Language / Builtins
========================

* The parser should handle more deeply nested parse trees.

  The following will fail -- ``eval("["*50`` + ``"]"*50)`` -- because
  the parser has a hard-coded limit on stack size.  This limit should
  be raised or removed.  Removal would be hard because the current
  compiler can overflow the C stack if the nesting is too deep.

  https://bugs.python.org/issue215555

* Non-accidental IEEE-754 support (Infs, NaNs, settable traps, etc).
  Big project.

* Windows:  Trying to create (or even access) files with certain
  magic names can hang or crash Windows systems.  This is really a
  bug in the OSes, but some apps try to shield users from it.  When
  it happens, the symptoms are very confusing.

  Hang using files named prn.txt, etc https://bugs.python.org/issue481171

* eval and free variables: It might be useful if there was a way to
  pass bindings for free variables to eval when a code object with
  free variables is passed. https://bugs.python.org/issue443866

Standard Library
================

* The urllib module should support proxies which require
  authentication.  See SourceForge bug #210619 for information:

  https://bugs.python.org/issue210619

* os.rename() should be modified to handle EXDEV errors on platforms
  that don't allow rename() to operate across filesystem boundaries
  by copying the file over and removing the original. Linux is one
  system that requires this treatment.

  https://bugs.python.org/issue212317

* signal handling doesn't always work as expected.  E.g. if
  sys.stdin.readline() is interrupted by a (returning) signal
  handler, it returns "".  It would be better to make it raise an
  exception (corresponding to EINTR) or to restart.  But these
  changes would have to applied to all places that can do blocking
  interruptible I/O.  So it's a big project.

  https://bugs.python.org/issue210599

* Extend Windows utime to accept directory paths.

  https://bugs.python.org/issue214245

* Extend copy.py to module & function types.

  https://bugs.python.org/issue214553

* Better checking for bad input to ``marshal.load*().``

  https://bugs.python.org/issue214754

* rfc822.py should be more lenient than the spec in the types of
  address fields it parses.  Specifically, an invalid address of the
  form "From: Amazon.com <delivers-news2@amazon.com>" should be
  parsed correctly.

  https://bugs.python.org/issue210678

* cgi.py's FieldStorage class should be more conservative with memory
  in the face of large binary file uploads.

  https://bugs.python.org/issue210674

  There are two issues here: first, because
  read_lines_to_outerboundary() uses readline() it is possible that a
  large amount of data will be read into memory for a binary file
  upload.  This should probably look at the Content-Type header of the
  section and do a chunked read if it's a binary type.

  The second issue was related to the self.lines attribute, which was
  removed in revision 1.56 of cgi.py (see also):

  https://bugs.python.org/issue219806

* urllib should support proxy definitions that contain just the host
  and port

  https://bugs.python.org/issue210849

* urlparse should be updated to comply with :rfc:`2396`, which defines
  optional parameters for each segment of the path.

  https://bugs.python.org/issue210834

* The exceptions raised by pickle and cPickle are currently
  different; these should be unified (probably the exceptions should
  be defined in a helper module that's imported by both). [No bug
  report; I just thought of this.]

* More standard library routines should support Unicode.  For
  example, urllib.quote() could convert Unicode strings to UTF-8 and
  then do the usual %HH conversion.  But this is not the only one!

  https://bugs.python.org/issue216716

* There should be a way to say that you don't mind if ``str()`` or
  ``__str__()`` return a Unicode string object.  Or a different function
  -- ``ustr()`` has been proposed.  Or something...

  http://sf.net/patch/?func=detailpatch&patch_id=101527&group_id=5470

* Killing a thread from another thread.  Or maybe sending a signal.
  Or maybe raising an asynchronous exception.

  https://bugs.python.org/issue221115

* The debugger (pdb) should understand packages.

  https://bugs.python.org/issue210631

* Jim Fulton suggested the following:

  ::

    I wonder if it would be a good idea to have a new kind of
    temporary file that stored data in memory unless:

    - The data exceeds some size, or

    - Somebody asks for a fileno.

    Then the cgi module (and other apps) could use this thing in a
    uniform way.

  https://bugs.python.org/issue415692

* Jim Fulton pointed out that binascii's b2a_base64() function has
  situations where it makes sense not to append a newline, or to
  append something else than a newline.

  Proposal:

  - add an optional argument giving the delimiter string to be
    appended, defaulting to "\\n"

  - possibly special-case None as the delimiter string to avoid adding
    the pad bytes too???

  https://bugs.python.org/issue415694

* pydoc should be integrated with the HTML docs, or at least be able
  to link to them.

  https://bugs.python.org/issue405554

* Distutils should deduce dependencies for .c and .h files.

  https://bugs.python.org/issue472881

* asynchat is buggy in the face of multithreading.

  https://bugs.python.org/issue595217

* It would be nice if the higher level modules (httplib, smtplib,
  nntplib, etc.) had options for setting socket timeouts.

  https://bugs.python.org/issue723287

* The curses library is missing two important calls: newterm() and
  delscreen()

  https://bugs.python.org/issue665572, http://bugs.debian.org/175590

* It would be nice if the built-in SSL socket type could be used for
  non-blocking SSL I/O.  Currently packages such as Twisted which
  implement async servers using SSL have to r


id: pep-0718.rst
content:
PEP: 718
Title: Subscriptable functions
Author: James Hilton-Balfe <gobot1234yt@gmail.com>
Sponsor: Guido van Rossum <guido@python.org>
Discussions-To: https://discuss.python.org/t/28457/
Status: Draft
Type: Standards Track
Topic: Typing
Created: 23-Jun-2023
Python-Version: 3.15
Post-History: `24-Jun-2023 <https://discuss.python.org/t/28457/>`__

Abstract
--------

This PEP proposes making function objects subscriptable for typing purposes. Doing so
gives developers explicit control over the types produced by the type checker where
bi-directional inference (which allows for the types of parameters of anonymous
functions to be inferred) and other methods than specialisation are insufficient. It
also brings functions in line with regular classes in their ability to be
subscriptable.

Motivation
----------

Unknown Types
^^^^^^^^^^^^^

Currently, it is not possible to infer the type parameters to generic functions in
certain situations:

.. code-block:: python

   def make_list[T](*args: T) -> list[T]: ...
   reveal_type(make_list())  # type checker cannot infer a meaningful type for T

Making instances of ``FunctionType`` subscriptable would allow for this constructor to
be typed:

.. code-block:: python

   reveal_type(make_list[int]())  # type is list[int]

Currently you have to use an assignment to provide a precise type:

.. code-block:: python

   x: list[int] = make_list()
   reveal_type(x)  # type is list[int]

but this code is unnecessarily verbose taking up multiple lines for a simple function
call.

Similarly, ``T`` in this example cannot currently be meaningfully inferred, so ``x`` is
untyped without an extra assignment:

.. code-block:: python

   def factory[T](func: Callable[[T], Any]) -> Foo[T]: ...

   reveal_type(factory(lambda x: "Hello World" * x))

If function objects were subscriptable, however, a more specific type could be given:

.. code-block:: python

   reveal_type(factory[int](lambda x: "Hello World" * x))  # type is Foo[int]

Undecidable Inference
^^^^^^^^^^^^^^^^^^^^^

There are even cases where subclass relations make type inference impossible. However,
if you can specialise the function type checkers can infer a meaningful type.

.. code-block:: python

   def foo[T](x: Sequence[T] | T) -> list[T]: ...

   reveal_type(foo[bytes](b"hello"))

Currently, type checkers do not consistently synthesise a type here.

Unsolvable Type Parameters
^^^^^^^^^^^^^^^^^^^^^^^^^^

Currently, with unspecialised literals, it is not possible to determine a type for
situations similar to:

.. code-block:: python

   def foo[T](x: list[T]) -> T: ...
   reveal_type(foo([]))  # type checker cannot infer T (yet again)

.. code-block:: python

   reveal_type(foo[int]([]))  # type is int

It is also useful to be able to specify in cases in which a certain type must be passed
to a function beforehand:

.. code-block:: python

   words = ["hello", "world"]
   foo[int](words)  # Invalid: list[str] is incompatible with list[int]

Allowing subscription makes functions and methods consistent with generic classes where
they weren't already. Whilst all of the proposed changes can be implemented using
callable generic classes, syntactic sugar would be highly welcome.

Due to this, specialising the function and using it as a new factory is fine

.. code-block:: python

   make_int_list = make_list[int]
   reveal_type(make_int_list())  # type is list[int]

Monomorphisation and Reification
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This proposal also opens the door to
`monomorphisation <https://en.wikipedia.org/wiki/Monomorphization>`_ and
`reified types <https://en.wikipedia.org/wiki/Reification_(computer_science)>`_.

This would allow for a functionality which anecdotally has been requested many times.

*Please note this feature is not being proposed by the PEP, but may be implemented in
the future.*

The syntax for such a feature may look something like:

.. code-block:: python

   def foo[T]():
      return T.__value__

   assert foo[int]() is int

Rationale
---------

Function objects in this PEP is used to refer to ``FunctionType``\ , ``MethodType``\ ,
``BuiltinFunctionType``\ , ``BuiltinMethodType`` and ``MethodWrapperType``\ .

For ``MethodType`` you should be able to write:

.. code-block:: python

   class Foo:
       def make_list[T](self, *args: T) -> list[T]: ...

   Foo().make_list[int]()

and have it work similarly to a ``FunctionType``.

For ``BuiltinFunctionType``, so builtin generic functions (e.g. ``max`` and ``min``)
work like ones defined in Python. Built-in functions should behave as much like
functions implemented in Python as possible.

``BuiltinMethodType`` is the same type as ``BuiltinFunctionType``.

``MethodWrapperType`` (e.g. the type of ``object().__str__``) is useful for
generic magic methods.

Specification
-------------

Function objects should implement ``__getitem__`` to allow for subscription at runtime
and return an instance of ``types.GenericAlias`` with ``__origin__`` set as the
callable and ``__args__`` as the types passed.

Type checkers should support subscripting functions and understand that the parameters
passed to the function subscription should follow the same rules as a generic callable
class.

Setting ``__orig_class__``
^^^^^^^^^^^^^^^^^^^^^^^^^^

Currently, ``__orig_class__`` is an attribute set in ``GenericAlias.__call__`` to the
instance of the ``GenericAlias`` that created the called class e.g.

.. code-block:: python

   class Foo[T]: ...

   assert Foo[int]().__orig_class__ == Foo[int]

Currently, ``__orig_class__`` is unconditionally set; however, to avoid potential
erasure on any created instances, this attribute should not be set if ``__origin__`` is
an instance of any function object.

The following code snippet would fail at runtime without this change as
``__orig_class__`` would be ``bar[str]`` and not ``Foo[int]``.

.. code-block:: python

   def bar[U]():
       return Foo[int]()

   assert bar[str]().__orig_class__  == Foo[int]

Interactions with ``@typing.overload``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Overloaded functions should work much the same as already, since they have no effect on
the runtime type. The only change is that more situations will be decidable and the
behaviour/overload can be specified by the developer rather than leaving it to ordering
of overloads/unions.

Backwards Compatibility
-----------------------
Currently these classes are not subclassable and so there are no backwards
compatibility concerns with regards to classes already implementing
``__getitem__``.

Reference Implementation
------------------------

The runtime changes proposed can be found here
https://github.com/Gobot1234/cpython/tree/function-subscript

Acknowledgements
----------------

Thank you to Alex Waygood and Jelle Zijlstra for their feedback on this PEP and Guido
for some motivating examples.

Copyright
---------

This document is placed in the public domain or under the CC0-1.0-Universal license,
whichever is more permissive.


id: pep-0297.rst
content:
PEP: 297
Title: Support for System Upgrades
Author: Marc-AndrÃ© Lemburg <mal@lemburg.com>
Status: Rejected
Type: Standards Track
Created: 19-Jul-2001
Python-Version: 2.6
Post-History:


Rejection Notice
================

This PEP is rejected for failure to generate significant interest.


Abstract
========

This PEP proposes strategies to allow the Python standard library
to be upgraded in parts without having to reinstall the complete
distribution or having to wait for a new patch level release.


Problem
=======

Python currently does not allow overriding modules or packages in
the standard library per default. Even though this is possible by
defining a ``PYTHONPATH`` environment variable (the paths defined in
this variable are prepended to the Python standard library path),
there is no standard way of achieving this without changing the
configuration.

Since Python's standard library is starting to host packages which
are also available separately, e.g. the distutils, email and PyXML
packages, which can also be installed independently of the Python
distribution, it is desirable to have an option to upgrade these
packages without having to wait for a new patch level release of
the Python interpreter to bring along the changes.

On some occasions, it may also be desirable to update modules of
the standard library without going through the whole Python release
cycle, e.g. in order to provide hot-fixes for security problems.


Proposed Solutions
==================

This PEP proposes two different but not necessarily conflicting
solutions:

1. Adding a new standard search path to ``sys.path``:
   ``$stdlibpath/system-packages`` just before the ``$stdlibpath``
   entry. This complements the already existing entry for site
   add-ons ``$stdlibpath/site-packages`` which is appended to the
   ``sys.path`` at interpreter startup time.

   To make use of this new standard location, distutils will need
   to grow support for installing certain packages in
   ``$stdlibpath/system-packages`` rather than the standard location
   for third-party packages ``$stdlibpath/site-packages``.

2. Tweaking distutils to install directly into ``$stdlibpath`` for the
   system upgrades rather than into ``$stdlibpath/site-packages``.

The first solution has a few advantages over the second:

* upgrades can be easily identified (just look in
  ``$stdlibpath/system-packages``)

* upgrades can be de-installed without affecting the rest
  of the interpreter installation

* modules can be virtually removed from packages; this is
  due to the way Python imports packages: once it finds the
  top-level package directory it stay in this directory for
  all subsequent package submodule imports

* the approach has an overall much cleaner design than the
  hackish install on top of an existing installation approach

The only advantages of the second approach are that the Python
interpreter does not have to changed and that it works with
older Python versions.

Both solutions require changes to distutils. These changes can
also be implemented by package authors, but it would be better to
define a standard way of switching on the proposed behaviour.


Scope
=====

Solution 1: Python 2.6 and up

Solution 2: all Python versions supported by distutils


Credits
=======

None


References
==========

None


Copyright
=========

This document has been placed in the public domain.


id: pep-0283.rst
content:
PEP: 283
Title: Python 2.3 Release Schedule
Author: Guido van Rossum
Status: Final
Type: Informational
Topic: Release
Created: 27-Feb-2002
Python-Version: 2.3
Post-History: 27-Feb-2002


Abstract
========

This document describes the development and release schedule for
Python 2.3.  The schedule primarily concerns itself with PEP-sized
items.  Small features may be added up to and including the first
beta release.  Bugs may be fixed until the final release.

There will be at least two alpha releases, two beta releases, and
one release candidate.  Alpha and beta releases will be spaced at
least 4 weeks apart (except if an emergency release must be made
to correct a blunder in the previous release; then the blunder
release does not count).  Release candidates will be spaced at
least one week apart (excepting again blunder corrections).

===========  ===========
alpha 1      31 Dec 2002
alpha 2      19 Feb 2003
beta 1       25 Apr 2003
beta 2       29 Jun 2003
candidate 1  18 Jul 2003
candidate 2  24 Jul 2003
final        29 Jul 2003
===========  ===========

Release Manager
===============

Barry Warsaw, Jeremy Hylton, Tim Peters


Completed features for 2.3
==========================

This list is not complete.  See ``Doc/whatsnew/whatsnew23.tex`` in CVS
for more, and of course ``Misc/NEWS`` for the full list.

- Tk 8.4 update.

- The ``bool`` type and its constants, ``True`` and ``False`` (:pep:`285`).

- ``PyMalloc`` was greatly enhanced and is enabled by default.

- Universal newline support (:pep:`278`).

- :pep:`263` Defining Python Source Code Encodings, Lemburg

  Implemented (at least phase 1, which is all that's planned for
  2.3).

- Extended slice notation for all built-in sequences.  The patch
  by Michael Hudson is now all checked in.

- Speed up list iterations by filling ``tp_iter`` and other tweaks.
  See https://bugs.python.org/issue560736; also done for ``xrange`` and
  tuples.

- Timeout sockets.  https://bugs.python.org/issue555085

- Stage B0 of the ``int``/``long`` integration (:pep:`237`).  This means
  issuing a ``FutureWarning`` about situations where ``hex`` or ``oct``
  conversions or left shifts returns a different value for an ``int``
  than for a ``long`` with the same value.  The semantics do *not*
  change in Python 2.3; that will happen in Python 2.4.

- Nuke ``SET_LINENO`` from all code objects (providing a different way
  to set debugger breakpoints).  This can boost ``pystone`` by >5%.
  https://bugs.python.org/issue587993, now checked in.  (Unfortunately
  the ``pystone`` boost didn't happen.  What happened?)

- Write a ``pymemcompat.h`` that people can bundle with their
  extensions and then use the 2.3 memory interface with all
  Pythons in the range 1.5.2 to 2.3.  (Michael Hudson checked in
  ``Misc/pymemcompat.h``.)

- Add a new concept, "pending deprecation", with associated
  warning ``PendingDeprecationWarning``.  This warning is normally
  suppressed, but can be enabled by a suitable ``-W`` option.  Only a
  few things use this at this time.

- Warn when an extension type's ``tp_compare`` returns anything except
  -1, 0 or 1.  https://bugs.python.org/issue472523

- Warn for assignment to ``None`` (in various forms).

- :pep:`218` Adding a Built-In Set Object Type, Wilson

  Alex Martelli contributed a new version of Greg Wilson's
  prototype, and I've reworked that quite a bit.  It's in the
  standard library now as the module ``sets``, although some details
  may still change until the first beta release.  (There are no
  plans to make this a built-in type, for now.)

- :pep:`293` Codec error handling callbacks, DÃ¶rwald

  Fully implemented.  Error handling in ``unicode.encode`` or
  ``str.decode`` can now be customized.

- :pep:`282` A Logging System, Mick

  Vinay Sajip's implementation has been packagized and imported.
  (Documentation and unit tests still pending.)
  https://bugs.python.org/issue578494

- A modified MRO (Method Resolution Order) algorithm.  Consensus
  is that we should adopt C3.  Samuele Pedroni has contributed a
  draft implementation in C, see https://bugs.python.org/issue619475
  This has now been checked in.

- A new command line option parser.  Greg Ward's Optik package
  (http://optik.sf.net) has been adopted, converted to a single
  module named ``optparse``.  See also
  http://www.python.org/sigs/getopt-sig/

- A standard ``datetime`` type.  This started as a wiki:
  http://www.zope.org/Members/fdrake/DateTimeWiki/FrontPage.  A
  prototype was coded in ``nondist/sandbox/datetime/``.  Tim Peters
  has finished the C implementation and checked it in.

- :pep:`273` Import Modules from Zip Archives, Ahlstrom

  Implemented as a part of the :pep:`302` implementation work.

- :pep:`302` New Import Hooks, JvR

  Implemented (though the 2.3a1 release contained some bugs that
  have been fixed post-release).

- A new pickling protocol. See :pep:`307`.

- :pep:`305` (CSV File API, by Skip Montanaro et al.) is in; this is
  the csv module.

- Raymond Hettinger's ``itertools`` module is in.

- :pep:`311` (Simplified GIL Acquisition for Extensions, by Mark
  Hammond) has been included in beta 1.

- Two new ``PyArg_Parse*()`` format codes, 'k' returns an unsigned C
  long int that receives the lower ``LONG_BIT`` bits of the Python
  argument, truncating without range checking. 'K' returns an
  unsigned C long long int that receives the lower ``LONG_LONG_BIT``
  bits, truncating without range checking.  (SF 595026; Thomas
  Heller did this work.)

- A new version of IDLE was imported from the IDLEfork project
  (http://idlefork.sf.net).  The code now lives in the ``idlelib``
  package in the standard library and the ``idle`` script is installed
  by ``setup.py``.


Planned features for 2.3
========================

Too late for anything more to get done here.


Ongoing tasks
=============

The following are ongoing TO-DO items which we should attempt to
work on without hoping for completion by any particular date.

- Documentation: complete the distribution and installation
  manuals.

- Documentation: complete the documentation for new-style
  classes.

- Look over the ``Demos/`` directory and update where required (Andrew
  Kuchling has done a lot of this)

- New tests.

- Fix doc bugs on SF.

- Remove use of deprecated features in the core.

- Document deprecated features appropriately.

- Mark deprecated C APIs with ``Py_DEPRECATED``.

- Deprecate modules which are unmaintained, or perhaps make a new
  category for modules 'Unmaintained'

- In general, lots of cleanup so it is easier to move forward.


Open issues
===========

There are some issues that may need more work and/or thought
before the final release (and preferably before the first beta
release):  No issues remaining.


Features that did not make it into Python 2.3
=============================================

- The import lock could use some redesign.  (SF 683658.)

- Set API issues; is the sets module perfect?

  I expect it's good enough to stop polishing it until we've had
  more widespread user experience.

- A nicer API to open text files, replacing the ugly (in some
  people's eyes) "U" mode flag.  There's a proposal out there to
  have a new built-in type ``textfile(filename, mode, encoding)``.
  (Shouldn't it have a *bufsize* argument too?)

  Ditto.

- New widgets for Tkinter???

  Has anyone gotten the time for this?  *Are* there any new
  widgets in Tk 8.4?  Note that we've got better Tix support
  already (though not on Windows yet).

- Fredrik Lundh's basetime proposal:

  http://effbot.org/ideas/time-type.htm

  I believe this is dead now.

- :pep:`304` (Controlling Generation of Bytecode Files by Montanaro)
  seems to have lost steam.

- For a class defined inside another class, the ``__name__`` should be
  ``"outer.inner"``, and pickling should work.  (SF 633930.  I'm no
  longer certain this is easy or even right.)

- reST is going to be used a lot in Zope3.  Maybe it could become
  a standard library mo


id: pep-0532.rst
content:
PEP: 532
Title: A circuit breaking protocol and binary operators
Author: Alyssa Coghlan <ncoghlan@gmail.com>,
        Mark E. Haase <mehaase@gmail.com>
Status: Deferred
Type: Standards Track
Created: 30-Oct-2016
Python-Version: 3.8
Post-History: 05-Nov-2016

PEP Deferral
============

Further consideration of this PEP has been deferred until Python 3.8 at the
earliest.

Abstract
========

Inspired by :pep:`335`, :pep:`505`, :pep:`531`, and the related discussions, this PEP
proposes the definition of a new circuit breaking protocol (using the
method names ``__then__`` and ``__else__``) that provides a common underlying
semantic foundation for:

* conditional expressions: ``LHS if COND else RHS``
* logical conjunction: ``LHS and RHS``
* logical disjunction: ``LHS or RHS``
* the None-aware operators proposed in :pep:`505`
* the rich comparison chaining model proposed in :pep:`535`

Taking advantage of the new protocol, it further proposes that the definition
of conditional expressions be revised to also permit the use of ``if`` and
``else`` respectively as right-associative and left-associative general
purpose short-circuiting operators:

* Right-associative short-circuiting: ``LHS if RHS``
* Left-associative short-circuiting: ``LHS else RHS``

In order to make logical inversion (``not EXPR``) consistent with the above
changes, it also proposes the introduction of a new logical inversion protocol
(using the method name ``__not__``).

To force short-circuiting of a circuit breaker without having to evaluate
the expression creating it twice, a new ``operator.short_circuit(obj)``
helper function will be added to the operator module.

Finally, a new standard ``types.CircuitBreaker`` type is proposed to decouple
an object's truth value (as used to determine control flow) from the value
it returns from short-circuited circuit breaking expressions, with the
following factory functions added to the operator module to represent
particularly common switching idioms:

* switching on ``bool(obj)``: ``operator.true(obj)``
* switching on ``not bool(obj)``: ``operator.false(obj)``
* switching on ``obj is value``: ``operator.is_sentinel(obj, value)``
* switching on ``obj is not value``: ``operator.is_not_sentinel(obj, value)``


Relationship with other PEPs
============================

This PEP builds on an extended history of work in other proposals. Some of
the key proposals are discussed below.


PEP 531: Existence checking protocol
------------------------------------

This PEP is a direct successor to :pep:`531`, replacing the existence checking
protocol and the new ``?then`` and ``?else`` syntactic operators defined there
with the new circuit breaking protocol and adjustments to conditional
expressions and the ``not`` operator.


PEP 505: None-aware operators
-----------------------------

This PEP complements the None-aware operator proposals in :pep:`505`, by offering
an underlying protocol-driven semantic framework that explains their
short-circuiting behaviour as highly optimised syntactic sugar for particular
uses of conditional expressions.

Given the changes proposed by this PEP:

* ``LHS ?? RHS`` would roughly be ``is_not_sentinel(LHS, None) else RHS``
* ``EXPR?.attr`` would roughly be ``EXPR.attr if is_not_sentinel(EXPR, None)``
* ``EXPR?[key]`` would roughly be ``EXPR[key] if is_not_sentinel(EXPR, None)``

In all three cases, the dedicated syntactic form would be optimised to avoid
actually creating the circuit breaker instance and instead implement the
underlying control flow directly. In the latter two cases, the syntactic form
would also avoid evaluating ``EXPR`` twice.

This means that while the None-aware operators would remain highly specialised
and specific to None, other sentinel values would still be usable through the
more general protocol-driven proposal in this PEP.


PEP 335: Overloadable Boolean operators
---------------------------------------

:pep:`335` proposed the ability to overload the short-circuiting ``and`` and
``or`` operators directly, with the ability to overload the semantics of
comparison chaining being one of the consequences of that change. The
proposal in an earlier version of this PEP to instead handle the element-wise
comparison use case by changing the semantic definition of comparison chaining
is drawn directly from Guido's rejection of :pep:`335` [1]_.

However, initial feedback on this PEP indicated that the number of different
proposals that it covered made it difficult to read, so that part of the
proposal has been separated out as :pep:`535`.


PEP 535: Rich comparison chaining
---------------------------------

As noted above, :pep:`535` is a proposal to build on the circuit breaking protocol
defined in this PEP in order to expand the rich comparison support introduced
in :pep:`207` to also handle comparison chaining operations like
``LEFT_BOUND < VALUE < RIGHT_BOUND``.


Specification
=============

The circuit breaking protocol (``if-else``)
-------------------------------------------

Conditional expressions (``LHS if COND else RHS``) are currently interpreted
as an expression level equivalent to::

    if COND:
        _expr_result = LHS
    else:
        _expr_result = RHS

This PEP proposes changing that expansion to allow the checked condition to
implement a new "circuit breaking" protocol that allows it to see, and
potentially alter, the result of either or both branches of the expression::

    _cb = COND
    _type_cb = type(cb)
    if _cb:
        _expr_result = LHS
        if hasattr(_type_cb, "__then__"):
            _expr_result = _type_cb.__then__(_cb, _expr_result)
    else:
        _expr_result = RHS
        if hasattr(_type_cb, "__else__"):
            _expr_result = _type_cb.__else__(_cb, _expr_result)

As shown, interpreter implementations would be required to access only the
protocol method needed for the branch of the conditional expression that is
actually executed. Consistent with other protocol methods, the special methods
would be looked up via the circuit breaker's type, rather than directly on the
instance.


Circuit breaking operators (binary ``if`` and binary ``else``)
--------------------------------------------------------------

The proposed name of the protocol doesn't come from the proposed changes to
the semantics of conditional expressions. Rather, it comes from the proposed
addition of ``if`` and ``else`` as general purpose protocol driven
short-circuiting operators to complement the existing ``True`` and ``False``
based short-circuiting operators (``or`` and ``and``, respectively) as well
as the ``None`` based short-circuiting operator proposed in :pep:`505` (``??``).

Together, these two operators would be known as the circuit breaking operators.

In order to support this usage, the definition of conditional expressions in
the language grammar would be updated to make both the ``if`` clause and
the ``else`` clause optional::

    test: else_test ['if' or_test ['else' test]] | lambdef
    else_test: or_test ['else' test]

Note that we would need to avoid the apparent simplification to
``else_test ('if' else_test)*`` in order to make it easier for compiler
implementations to correctly preserve the semantics of normal conditional
expressions.

The definition of the ``test_nocond`` node in the grammar (which deliberately
excludes conditional expressions) would remain unchanged, so the circuit
breaking operators would require parentheses when used in the ``if``
clause of comprehensions and generator expressions just as conditional
expressions themselves do.

This grammar definition means precedence/associativity in the otherwise
ambiguous case of ``expr1 if cond else expr2 else expr3`` resolves as
``(expr1 if cond else expr2) else epxr3``. However, a guideline will also be
added to :pep:`8` to say "don't do that", as such a construct will be inherently
confusing for readers, regardless of how the interpreter executes it.

The right-associative circuit breaking operator (``LHS


id: pep-0254.rst
content:
PEP: 254
Title: Making Classes Look More Like Types
Author: Guido van Rossum <guido@python.org>
Status: Rejected
Type: Standards Track
Created: 18-Jun-2001
Python-Version: 2.2
Post-History:


Abstract
========

This PEP has not been written yet.  Watch this space!


Status
======

This PEP was a stub entry and eventually abandoned without having
been filled-out.  Substantially most of the intended functionality
was implemented in Py2.2 with new-style types and classes.


Copyright
=========

This document has been placed in the public domain.


id: pep-0240.rst
content:
PEP: 240
Title: Adding a Rational Literal to Python
Author: Christopher A. Craig <python-pep@ccraig.org>, Moshe Zadka <moshez@zadka.site.co.il>
Status: Rejected
Type: Standards Track
Created: 11-Mar-2001
Python-Version: 2.2
Post-History: 16-Mar-2001


Abstract
========

A :pep:`different PEP <239>` suggests adding a builtin rational type to
Python.  This PEP suggests changing the ddd.ddd float literal to a
rational in Python, and modifying non-integer division to return
it.


BDFL Pronouncement
==================

This PEP is rejected.  The needs outlined in the rationale section
have been addressed to some extent by the acceptance of :pep:`327`
for decimal arithmetic.  Guido also noted, "Rational arithmetic
was the default 'exact' arithmetic in ABC and it did not work out as
expected".  See the python-dev discussion on 17 June 2005 [1]_.


Rationale
=========

Rational numbers are useful for exact and unsurprising arithmetic.
They give the correct results people have been taught in various
math classes.  Making the "obvious" non-integer type one with more
predictable semantics will surprise new programmers less than
using floating point numbers. As quite a few posts on c.l.py and
on tutor@python.org have shown, people often get bit by strange
semantics of floating point numbers: for example, ``round(0.98, 2)``
still gives 0.97999999999999998.


Proposal
========

Literals conforming to the regular expression ``'\d*.\d*'`` will be
rational numbers.


Backwards Compatibility
=======================

The only backwards compatible issue is the type of literals
mentioned above.  The following migration is suggested:

1. The next Python after approval will allow
   ``from __future__ import rational_literals``
   to cause all such literals to be treated as rational numbers.

2. Python 3.0 will have a warning, turned on by default, about
   such literals in the absence of a ``__future__`` statement.  The
   warning message will contain information about the ``__future__``
   statement, and indicate that to get floating point literals,
   they should be suffixed with "e0".

3. Python 3.1 will have the warning turned off by default.  This
   warning will stay in place for 24 months, at which time the
   literals will be rationals and the warning will be removed.


Common Objections
=================

Rationals are slow and memory intensive!
(Relax, I'm not taking floats away, I'm just adding two more characters.
``1e0`` will still be a float)

Rationals must present themselves as a decimal float or they will be
horrible for users expecting decimals (i.e. ``str(.5)`` should return ``'.5'`` and
not ``'1/2'``).  This means that many rationals must be truncated at some
point, which gives us a new loss of precision.



References
==========

.. [1] Raymond Hettinger, Propose rejection of PEPs 239 and 240 -- a builtin
       rational type and rational literals
       https://mail.python.org/pipermail/python-dev/2005-June/054281.html


Copyright
=========

This document has been placed in the public domain.


id: pep-0526.rst
content:
PEP: 526
Title: Syntax for Variable Annotations
Author: Ryan Gonzalez <rymg19@gmail.com>, Philip House <phouse512@gmail.com>, Ivan Levkivskyi <levkivskyi@gmail.com>, Lisa Roach <lisaroach14@gmail.com>, Guido van Rossum <guido@python.org>
Status: Final
Type: Standards Track
Topic: Typing
Created: 09-Aug-2016
Python-Version: 3.6
Post-History: 30-Aug-2016, 02-Sep-2016
Resolution: https://mail.python.org/pipermail/python-dev/2016-September/146282.html

.. canonical-typing-spec:: :ref:`python:annassign`,
                           :ref:`typing:classvar` and
                           :py:data:`typing.ClassVar`

Status
======

This PEP has been provisionally accepted by the BDFL.
See the acceptance message for more color:
https://mail.python.org/pipermail/python-dev/2016-September/146282.html


Notice for Reviewers
====================

This PEP was drafted in a separate repo:
https://github.com/phouse512/peps/tree/pep-0526.

There was preliminary discussion on python-ideas and at
https://github.com/python/typing/issues/258.

Before you bring up an objection in a public forum please at least
read the summary of `rejected ideas <PEP 526 rejected_>`_ listed at the end of this PEP.


Abstract
========

:pep:`484` introduced type hints, a.k.a. type annotations.  While its
main focus was function annotations, it also introduced the notion of
type comments to annotate variables::

  # 'primes' is a list of integers
  primes = []  # type: List[int]

  # 'captain' is a string (Note: initial value is a problem)
  captain = ...  # type: str

  class Starship:
      # 'stats' is a class variable
      stats = {}  # type: Dict[str, int]

This PEP aims at adding syntax to Python for annotating the types of variables
(including class variables and instance variables),
instead of expressing them through comments::

  primes: List[int] = []

  captain: str  # Note: no initial value!

  class Starship:
      stats: ClassVar[Dict[str, int]] = {}

:pep:`484` explicitly states that type comments are intended to help with
type inference in complex cases, and this PEP does not change this
intention.  However, since in practice type comments have also been
adopted for class variables and instance variables, this PEP also
discusses the use of type annotations for those variables.


Rationale
=========

Although type comments work well enough, the fact that they're
expressed through comments has some downsides:

- Text editors often highlight comments differently from type annotations.

- There's no way to annotate the type of an undefined variable; one needs to
  initialize it to ``None`` (e.g. ``a = None # type: int``).

- Variables annotated in a conditional branch are difficult to read::

    if some_value:
        my_var = function() # type: Logger
    else:
        my_var = another_function() # Why isn't there a type here?

- Since type comments aren't actually part of the language, if a Python script
  wants to parse them, it requires a custom parser instead of just using
  ``ast``.

- Type comments are used a lot in typeshed. Migrating typeshed to use
  the variable annotation syntax instead of type comments would improve
  readability of stubs.

- In situations where normal comments and type comments are used together, it is
  difficult to distinguish them::

    path = None  # type: Optional[str]  # Path to module source

- It's impossible to retrieve the annotations at runtime outside of
  attempting to find the module's source code and parse it at runtime,
  which is inelegant, to say the least.

The majority of these issues can be alleviated by making the syntax
a core part of the language. Moreover, having a dedicated annotation syntax
for class and instance variables (in addition to method annotations) will
pave the way to static duck-typing as a complement to nominal typing defined
by :pep:`484`.

Non-goals
*********

While the proposal is accompanied by an extension of the ``typing.get_type_hints``
standard library function for runtime retrieval of annotations, variable
annotations are not designed for runtime type checking. Third party packages
will have to be developed to implement such functionality.

It should also be emphasized that **Python will remain a dynamically typed
language, and the authors have no desire to ever make type hints mandatory,
even by convention.** Type annotations should not be confused with variable
declarations in statically typed languages. The goal of annotation syntax is
to provide an easy way to specify structured type metadata
for third party tools.

This PEP does not require type checkers to change their type checking
rules. It merely provides a more readable syntax to replace type
comments.


Specification
=============

Type annotation can be added to an assignment statement or to a single
expression indicating the desired type of the annotation target to a third
party type checker::

  my_var: int
  my_var = 5  # Passes type check.
  other_var: int  = 'a'  # Flagged as error by type checker,
                         # but OK at runtime.

This syntax does not introduce any new semantics beyond :pep:`484`, so that
the following three statements are equivalent::

  var = value # type: annotation
  var: annotation; var = value
  var: annotation = value

Below we specify the syntax of type annotations
in different contexts and their runtime effects.

We also suggest how type checkers might interpret annotations, but
compliance to these suggestions is not mandatory.  (This is in line
with the attitude towards compliance in :pep:`484`.)

Global and local variable annotations
*************************************

The types of locals and globals can be annotated as follows::

  some_number: int           # variable without initial value
  some_list: List[int] = []  # variable with initial value

Being able to omit the initial value allows for easier typing of variables
assigned in conditional branches::

  sane_world: bool
  if 2+2 == 4:
      sane_world = True
  else:
      sane_world = False

Note that, although the syntax does allow tuple packing, it does *not* allow
one to annotate the types of variables when tuple unpacking is used::

  # Tuple packing with variable annotation syntax
  t: Tuple[int, ...] = (1, 2, 3)
  # or
  t: Tuple[int, ...] = 1, 2, 3  # This only works in Python 3.8+

  # Tuple unpacking with variable annotation syntax
  header: str
  kind: int
  body: Optional[List[str]]
  header, kind, body = message

Omitting the initial value leaves the variable uninitialized::

  a: int
  print(a)  # raises NameError

However, annotating a local variable will cause the interpreter to always make
it a local::

  def f():
      a: int
      print(a)  # raises UnboundLocalError
      # Commenting out the a: int makes it a NameError.

as if the code were::

  def f():
      if False: a = 0
      print(a)  # raises UnboundLocalError

Duplicate type annotations will be ignored. However, static type
checkers may issue a warning for annotations of the same variable
by a different type::

  a: int
  a: str  # Static type checker may or may not warn about this.

.. _classvar:

Class and instance variable annotations
***************************************

Type annotations can also be used to annotate class and instance variables
in class bodies and methods. In particular, the value-less notation ``a: int``
allows one to annotate instance variables that should be initialized
in ``__init__`` or ``__new__``. The proposed syntax is as follows::

  class BasicStarship:
      captain: str = 'Picard'               # instance variable with default
      damage: int                           # instance variable without default
      stats: ClassVar[Dict[str, int]] = {}  # class variable

Here ``ClassVar`` is a special class defined by the typing module that
indicates to the static type checker that this variable should not be
set on instances.

Note that a ``ClassVar`` parameter cannot include any type variables, regardless
of the level 


id: pep-0268.rst
content:
PEP: 268
Title: Extended HTTP functionality and WebDAV
Author: Greg Stein <gstein@lyra.org>
Status: Rejected
Type: Standards Track
Created: 20-Aug-2001
Python-Version: 2.x
Post-History: 21-Aug-2001


Rejection Notice
================

This PEP has been rejected. It has failed to generate sufficient
community support in the six years since its proposal.


Abstract
========

This PEP discusses new modules and extended functionality for Python's
HTTP support. Notably, the addition of authenticated requests, proxy
support, authenticated proxy usage, and WebDAV_ capabilities.


Rationale
=========

Python has been quite popular as a result of its "batteries included"
positioning. One of the most heavily used protocols, HTTP (see
:rfc:`2616`), has been included with Python for years (``httplib``). However,
this support has not kept up with the full needs and requirements of
many HTTP-based applications and systems. In addition, new protocols
based on HTTP, such as WebDAV and XML-RPC, are becoming useful and are
seeing increasing usage. Supplying this functionality meets Python's
"batteries included" role and also keeps Python at the leading edge of
new technologies.

While authentication and proxy support are two very notable features
missing from Python's core HTTP processing, they are minimally handled
as part of Python's URL handling (``urllib`` and
``urllib2``). However, applications that need fine-grained or
sophisticated HTTP handling cannot make use of the features while they
reside in urllib. Refactoring these features into a location where
they can be directly associated with an HTTP connection will improve
their utility for both urllib and for sophisticated applications.

The motivation for this PEP was from several people requesting these
features directly, and from a number of feature requests on
SourceForge. Since the exact form of the modules to be provided and
the classes/architecture used could be subject to debate, this PEP was
created to provide a focal point for those discussions.


Specification
=============

Two modules will be added to the standard library: ``httpx`` (HTTP
extended functionality), and ``davlib`` (WebDAV library).

[ suggestions for module names are welcome; ``davlib`` has some
precedence, but something like ``webdav`` might be desirable ]


HTTP Authentication
-------------------

The ``httpx`` module will provide a mixin for performing HTTP
authentication (for both proxy and origin server authentication). This
mixin (``httpx.HandleAuthentication``) can be combined with the
``HTTPConnection`` and the ``HTTPSConnection`` classes (the mixin may
possibly work with the HTTP and HTTPS compatibility classes, but that
is not a requirement).

The mixin will delegate the authentication process to one or more
"authenticator" objects, allowing multiple connections to share
authenticators. The use of a separate object allows for a long term
connection to an authentication system (e.g. LDAP). An authenticator
for the Basic and Digest mechanisms (see :rfc:`2617`) will be
provided. User-supplied authenticator subclasses can be registered and
used by the connections.

A "credentials" object (``httpx.Credentials``) is also associated with
the mixin, and stores the credentials (e.g. username and password)
needed by the authenticators. Subclasses of Credentials can be created
to hold additional information (e.g. NT domain).

The mixin overrides the ``getresponse()`` method to detect ``401
(Unauthorized)`` and ``407 (Proxy Authentication Required)``
responses. When this is found, the response object, the connection,
and the credentials are passed to the authenticator corresponding with
the authentication scheme specified in the response (multiple
authenticators are tried in decreasing order of security if multiple
schemes are in the response). Each authenticator can examine the
response headers and decide whether and how to resend the request with
the correct authentication headers. If no authenticator can
successfully handle the authentication, then an exception is raised.

Resending a request, with the appropriate credentials, is one of the
more difficult portions of the authentication system. The difficulty
arises in recording what was sent originally: the request line, the
headers, and the body. By overriding putrequest, putheader, and
endheaders, we can capture all but the body. Once the endheaders
method is called, then we capture all calls to send() (until the next
putrequest method call) to hold the body content. The mixin will have
a configurable limit for the amount of data to hold in this fashion
(e.g. only hold up to 100k of body content). Assuming that the entire
body has been stored, then we can resend the request with the
appropriate authentication information.

If the body is too large to be stored, then the ``getresponse()``
simply returns the response object, indicating the 401 or 407
error. Since the authentication information has been computed and
cached (into the Credentials object; see below), the caller can simply
regenerate the request. The mixin will attach the appropriate
credentials.

A "protection space" (see :rfc:`2617`, section 1.2) is defined as a tuple
of the host, port, and authentication realm. When a request is
initially sent to an HTTP server, we do not know the authentication
realm (the realm is only returned when authentication fails). However,
we do have the path from the URL, and that can be useful in
determining the credentials to send to the server.  The Basic
authentication scheme is typically set up hierarchically: the
credentials for ``/path`` can be tried for ``/path/subpath``. The
Digest authentication scheme has explicit support for the hierarchical
setup. The ``httpx.Credentials`` object will store credentials for
multiple protection spaces, and can be looked up in two different
ways:

1. looked up using ``(host, port, path)`` -- this lookup scheme is
   used when generating a request for a path where we don't know the
   authentication realm.

2. looked up using ``(host, port, realm)`` -- this mechanism is used
   during the authentication process when the server has specified that
   the Request-URI resides within a specific realm.

The ``HandleAuthentication`` mixin will override ``putrequest()`` to
automatically insert credentials, if available. The URL from the
putrequest is used to determine the appropriate authentication
information to use.

It is also important to note that two sets of credentials are used,
and stored by the mixin. One set for any proxy that may be used, and
one used for the target origin server. Since proxies do not have
paths, the protection spaces in the proxy credentials will always use
"/" for storing and looking up via a path.


Proxy Handling
--------------

The ``httpx`` module will provide a mixin for using a proxy to perform
HTTP(S) operations. This mixin (``httpx.UseProxy``) can be combined
with the ``HTTPConnection`` and the ``HTTPSConnection`` classes (the
mixin may possibly work with the HTTP and HTTPS compatibility classes,
but that is not a requirement).

The mixin will record the ``(host, port)`` of the proxy to use. XXX
will be overridden to use this host/port combination for connections
and to rewrite request URLs into the absoluteURIs referring to the
origin server (these URIs are passed to the proxy server).

Proxy authentication is handled by the ``httpx.HandleAuthentication``
class since a user may directly use ``HTTP(S)Connection`` to speak
with proxies.


WebDAV Features
---------------

The ``davlib`` module will provide a mixin for sending WebDAV requests
to a WebDAV-enabled server. This mixin (``davlib.DAVClient``) can be
combined with the ``HTTPConnection`` and the ``HTTPSConnection``
classes (the mixin may possibly work with the HTTP and HTTPS
compatibility classes, but that is not a requirement).

The mixin provides methods to perform the various HTTP methods defined
by HTTP in :rfc:`2616`, and by WebDAV in :rfc:`2518`.

A custom r


id: pep-0518.rst
content:
PEP: 518
Title: Specifying Minimum Build System Requirements for Python Projects
Author: Brett Cannon <brett@python.org>,
        Nathaniel J. Smith <njs@pobox.com>,
        Donald Stufft <donald@stufft.io>
BDFL-Delegate: Alyssa Coghlan
Discussions-To: distutils-sig@python.org
Status: Final
Type: Standards Track
Topic: Packaging
Created: 10-May-2016
Post-History: 10-May-2016,
              11-May-2016,
              13-May-2016
Resolution: https://mail.python.org/pipermail/distutils-sig/2016-May/028969.html


Abstract
========

This PEP specifies how Python software packages should specify what
build dependencies they have in order to execute their chosen build
system. As part of this specification, a new configuration file is
introduced for software packages to use to specify their build
dependencies (with the expectation that the same configuration file
will be used for future configuration details).


Rationale
=========

When Python first developed its tooling for building distributions of
software for projects, distutils [#distutils]_ was the chosen
solution. As time went on, setuptools [#setuptools]_ gained popularity
to add some features on top of distutils. Both used the concept of a
``setup.py`` file that project maintainers executed to build
distributions of their software (as well as users to install said
distribution).

Using an executable file to specify build requirements under distutils
isn't an issue as distutils is part of Python's standard library.
Having the build tool as part of Python means that a ``setup.py`` has
no external dependency that a project maintainer needs to worry about
to build a distribution of their project. There was no need to specify
any dependency information as the only dependency is Python.

But when a project chooses to use setuptools, the use of an executable
file like ``setup.py`` becomes an issue. You can't execute a
``setup.py`` file without knowing its dependencies, but currently
there is no standard way to know what those dependencies are in an
automated fashion without executing the ``setup.py`` file where that
information is stored. It's a catch-22 of a file not being runnable
without knowing its own contents which can't be known programmatically
unless you run the file.

Setuptools tried to solve this with a ``setup_requires`` argument to
its ``setup()`` function [#setup_args]_. This solution has a number
of issues, such as:

* No tooling (besides setuptools itself) can access this information
  without executing the ``setup.py``, but ``setup.py`` can't be
  executed without having these items installed.
* While setuptools itself will install anything listed in this, they
  won't be installed until *during* the execution of the ``setup()``
  function, which means that the only way to actually use anything
  added here is through increasingly complex machinations that delay
  the import and usage of these modules until later on in the
  execution of the ``setup()`` function.
* This cannot include ``setuptools`` itself nor can it include a
  replacement to ``setuptools``, which means that projects such as
  ``numpy.distutils`` are largely incapable of utilizing it and
  projects cannot take advantage of newer setuptools features until
  their users naturally upgrade the version of setuptools to a newer
  one.
* The items listed in ``setup_requires`` get implicitly installed
  whenever you execute the ``setup.py`` but one of the common ways
  that the ``setup.py`` is executed is via another tool, such as
  ``pip``, who is already managing dependencies. This means that
  a command like ``pip install spam`` might end up having both
  pip and setuptools downloading and installing packages and end
  users needing to configure *both* tools (and for ``setuptools``
  without being in control of the invocation) to change settings
  like which repository it installs from. It also means that users
  need to be aware of the discovery rules for both tools, as one
  may support different package formats or determine the latest
  version differently.

This has culminated in a situation where use of ``setup_requires``
is rare, where projects tend to either simply copy and paste snippets
between ``setup.py`` files or they eschew it all together in favor
of simply documenting elsewhere what they expect the user to have
manually installed prior to attempting to build or install their
project.

All of this has led pip [#pip]_ to simply assume that setuptools is
necessary when executing a ``setup.py`` file. The problem with this,
though, is it doesn't scale if another project began to gain traction
in the community as setuptools has. It also prevents other projects
from gaining traction due to the friction required to use it with a
project when pip can't infer the fact that something other than
setuptools is required.

This PEP attempts to rectify the situation by specifying a way to list
the minimal dependencies of the build system of a project in a
declarative fashion in a specific file. This allows a project to list
what build dependencies it has to go from e.g. source checkout to
wheel, while not falling into the catch-22 trap that a ``setup.py``
has where tooling can't infer what a project needs to build itself.
Implementing this PEP will allow projects to specify what build system
they depend on upfront so that tools like pip can make sure that they
are installed in order to run the build system to build the project.

To provide more context and motivation for this PEP, think of the
(rough) steps required to produce a built artifact for a project:

1. The source checkout of the project.
2. Installation of the build system.
3. Execute the build system.

This PEP covers step #2. :pep:`517` covers step #3, including how to have
the build system dynamically specify more dependencies that the build
system requires to perform its job. The purpose of this PEP though, is
to specify the minimal set of requirements for the build system to
simply begin execution.


Specification
=============

File Format
-----------

The build system dependencies will be stored in a file named
``pyproject.toml`` that is written in the TOML format [#toml]_.

This format was chosen as it is human-usable (unlike JSON [#json]_),
it is flexible enough (unlike configparser [#configparser]_), stems
from a standard (also unlike configparser [#configparser]_), and it
is not overly complex (unlike YAML [#yaml]_). The TOML format is
already in use by the Rust community as part of their
Cargo package manager [#cargo]_ and in private email stated they have
been quite happy with their choice of TOML. A more thorough
discussion as to why various alternatives were not chosen can be read
in the `Other file formats`_ section. The authors do realize, though,
that choice of configuration file format is ultimately subjective and
a choice had to be made and the authors prefer TOML for this situation.

Below we list the tables that tools are expected to recognize/respect.
Tables not specified in this PEP are reserved for future use by other
PEPs.

build-system table
------------------

The ``[build-system]`` table is used to store build-related data.
Initially only one key of the table will be valid and is mandatory
for the table: ``requires``. This key must have a value of a list
of strings representing :pep:`508` dependencies required to execute the
build system (currently that means what dependencies are required to
execute a ``setup.py`` file).

For the vast majority of Python projects that rely upon setuptools,
the ``pyproject.toml`` file will be::

  [build-system]
  # Minimum requirements for the build system to execute.
  requires = ["setuptools", "wheel"]  # PEP 508 specifications.

Because the use of setuptools and wheel are so expansive in the
community at the moment, build tools are expected to use the example
configuration file above as their default semantics when a
``pyproject.toml`` file is not present.

Tools should not require the existence of 


id: pep-0256.rst
content:
PEP: 256
Title: Docstring Processing System Framework
Author: David Goodger <goodger@python.org>
Discussions-To: doc-sig@python.org
Status: Rejected
Type: Standards Track
Created: 01-Jun-2001
Post-History: 13-Jun-2001


Rejection Notice
================

This proposal seems to have run out of steam.


Abstract
========

Python lends itself to inline documentation.  With its built-in
docstring syntax, a limited form of `Literate Programming`_ is easy to
do in Python.  However, there are no satisfactory standard tools for
extracting and processing Python docstrings.  The lack of a standard
toolset is a significant gap in Python's infrastructure; this PEP aims
to fill the gap.

The issues surrounding docstring processing have been contentious and
difficult to resolve.  This PEP proposes a generic Docstring
Processing System (DPS) framework, which separates out the components
(program and conceptual), enabling the resolution of individual issues
either through consensus (one solution) or through divergence (many).
It promotes standard interfaces which will allow a variety of plug-in
components (input context readers, markup parsers, and output format
writers) to be used.

The concepts of a DPS framework are presented independently of
implementation details.


Road Map to the Docstring PEPs
==============================

There are many aspects to docstring processing.  The "Docstring PEPs"
have broken up the issues in order to deal with each of them in
isolation, or as close as possible.  The individual aspects and
associated PEPs are as follows:

* Docstring syntax.  :pep:`287`, "reStructuredText Docstring Format",
  proposes a syntax for Python docstrings, PEPs, and
  other uses.

* Docstring semantics consist of at least two aspects:

  - Conventions: the high-level structure of docstrings.  Dealt with
    in :pep:`257`, "Docstring Conventions".

  - Methodology: rules for the informational content of docstrings.
    Not addressed.

* Processing mechanisms.  This PEP (:pep:`256`) outlines the high-level
  issues and specification of an abstract docstring processing system
  (DPS).  :pep:`258`, "Docutils Design Specification", is an
  overview of the design and implementation of one DPS under
  development.

* Output styles: developers want the documentation generated from
  their source code to look good, and there are many different ideas
  about what that means.  :pep:`258` touches on "Stylist Transforms".
  This aspect of docstring processing has yet to be fully explored.

By separating out the issues, we can form consensus more easily
(smaller fights ;-), and accept divergence more readily.


Rationale
=========

There are standard inline documentation systems for some other
languages.  For example, Perl has POD_ ("Plain Old Documentation") and
Java has Javadoc_, but neither of these mesh with the Pythonic way.
POD syntax is very explicit, but takes after Perl in terms of
readability.  Javadoc is HTML-centric; except for "``@field``" tags,
raw HTML is used for markup.  There are also general tools such as
Autoduck_ and Web_ (Tangle & Weave), useful for multiple languages.

There have been many attempts to write auto-documentation systems
for Python (not an exhaustive list):

- Marc-Andre Lemburg's doc.py_

- Daniel Larsson's pythondoc_ & gendoc_

- Doug Hellmann's HappyDoc_

- Laurence Tratt's Crystal (no longer available on the web)

- Ka-Ping Yee's pydoc_ (pydoc.py is now part of the Python standard
  library; see below)

- Tony Ibbs' docutils_ (Tony has donated this name to the `Docutils
  project`_)

- Edward Loper's STminus_ formalization and related efforts

These systems, each with different goals, have had varying degrees of
success.  A problem with many of the above systems was over-ambition
combined with inflexibility.  They provided a self-contained set of
components: a docstring extraction system, a markup parser, an
internal processing system and one or more output format writers with
a fixed style.  Inevitably, one or more aspects of each system had
serious shortcomings, and they were not easily extended or modified,
preventing them from being adopted as standard tools.

It has become clear (to this author, at least) that the "all or
nothing" approach cannot succeed, since no monolithic self-contained
system could possibly be agreed upon by all interested parties.  A
modular component approach designed for extension, where components
may be multiply implemented, may be the only chance for success.
Standard inter-component APIs will make the DPS components
comprehensible without requiring detailed knowledge of the whole,
lowering the barrier for contributions, and ultimately resulting in a
rich and varied system.

Each of the components of a docstring processing system should be
developed independently.  A "best of breed" system should be chosen,
either merged from existing systems, and/or developed anew.  This
system should be included in Python's standard library.


PyDoc & Other Existing Systems
------------------------------

PyDoc became part of the Python standard library as of release 2.1.
It extracts and displays docstrings from within the Python interactive
interpreter, from the shell command line, and from a GUI window into a
web browser (HTML).  Although a very useful tool, PyDoc has several
deficiencies, including:

- In the case of the GUI/HTML, except for some heuristic hyperlinking
  of identifier names, no formatting of the docstrings is done.  They
  are presented within ``<p><small><tt>`` tags to avoid unwanted line
  wrapping.  Unfortunately, the result is not attractive.

- PyDoc extracts docstrings and structural information (class
  identifiers, method signatures, etc.) from imported module objects.
  There are security issues involved with importing untrusted code.
  Also, information from the source is lost when importing, such as
  comments, "additional docstrings" (string literals in non-docstring
  contexts; see :pep:`258`), and the order of definitions.

The functionality proposed in this PEP could be added to or used by
PyDoc when serving HTML pages.  The proposed docstring processing
system's functionality is much more than PyDoc needs in its current
form.  Either an independent tool will be developed (which PyDoc may
or may not use), or PyDoc could be expanded to encompass this
functionality and *become* the docstring processing system (or one
such system).  That decision is beyond the scope of this PEP.

Similarly for other existing docstring processing systems, their
authors may or may not choose compatibility with this framework.
However, if this framework is accepted and adopted as the Python
standard, compatibility will become an important consideration in
these systems' future.


Specification
=============

The docstring processing system framework is broken up as follows:

1. Docstring conventions.  Documents issues such as:

   - What should be documented where.

   - First line is a one-line synopsis.

   :pep:`257` documents some of these issues.

2. Docstring processing system design specification.  Documents
   issues such as:

   - High-level spec: what a DPS does.

   - Command-line interface for executable script.

   - System Python API.

   - Docstring extraction rules.

   - Readers, which encapsulate the input context.

   - Parsers.

   - Document tree: the intermediate internal data structure.  The
     output of the Parser and Reader, and the input to the Writer all
     share the same data structure.

   - Transforms, which modify the document tree.

   - Writers for output formats.

   - Distributors, which handle output management (one file, many
     files, or objects in memory).

   These issues are applicable to any docstring processing system
   implementation.  :pep:`258` documents these issues.

3. Docstring processing system implementation.

4. Input markup specifications: docstring syntax.  :pep:`287`
   proposes a standard syntax.

5. Input parser implementations.

6.


id: pep-0530.rst
content:
PEP: 530
Title: Asynchronous Comprehensions
Author: Yury Selivanov <yury@edgedb.com>
Discussions-To: python-dev@python.org
Status: Final
Type: Standards Track
Created: 03-Sep-2016
Python-Version: 3.6
Post-History: 03-Sep-2016


Abstract
========

:pep:`492` and :pep:`525` introduce support for native coroutines and
asynchronous generators using ``async`` / ``await`` syntax.  This PEP
proposes to add asynchronous versions of list, set, dict comprehensions
and generator expressions.


Rationale and Goals
===================

Python has extensive support for synchronous comprehensions, allowing
to produce lists, dicts, and sets with a simple and concise syntax.  We
propose implementing similar syntactic constructions for the
asynchronous code.

To illustrate the readability improvement, consider the following
example::

    result = []
    async for i in aiter():
        if i % 2:
            result.append(i)

With the proposed asynchronous comprehensions syntax, the above code
becomes as short as::

    result = [i async for i in aiter() if i % 2]

The PEP also makes it possible to use the ``await`` expressions in
all kinds of comprehensions::

    result = [await fun() for fun in funcs]


Specification
=============

Asynchronous Comprehensions
---------------------------

We propose to allow using ``async for`` inside list, set and dict
comprehensions.  Pending :pep:`525` approval, we can also allow creation
of asynchronous generator expressions.

Examples:

* set comprehension: ``{i async for i in agen()}``;

* list comprehension: ``[i async for i in agen()]``;

* dict comprehension: ``{i: i ** 2 async for i in agen()}``;

* generator expression: ``(i ** 2 async for i in agen())``.

It is allowed to use ``async for`` along with  ``if`` and ``for``
clauses in asynchronous comprehensions and generator expressions::

    dataset = {data for line in aiter()
                    async for data in line
                    if check(data)}

Asynchronous comprehensions are only allowed inside an ``async def``
function.

In principle, asynchronous generator expressions are allowed in
any context.  However, in Python 3.6, due to ``async`` and ``await``
soft-keyword status, asynchronous generator expressions are only
allowed in an ``async def`` function.  Once ``async`` and ``await``
become reserved keywords in Python 3.7, this restriction will be
removed.


``await`` in Comprehensions
---------------------------

We propose to allow the use of ``await`` expressions in both
asynchronous and synchronous comprehensions::

    result = [await fun() for fun in funcs]
    result = {await fun() for fun in funcs}
    result = {fun: await fun() for fun in funcs}

    result = [await fun() for fun in funcs if await smth]
    result = {await fun() for fun in funcs if await smth}
    result = {fun: await fun() for fun in funcs if await smth}

    result = [await fun() async for fun in funcs]
    result = {await fun() async for fun in funcs}
    result = {fun: await fun() async for fun in funcs}

    result = [await fun() async for fun in funcs if await smth]
    result = {await fun() async for fun in funcs if await smth}
    result = {fun: await fun() async for fun in funcs if await smth}

This is only valid in ``async def`` function body.


Grammar Updates
---------------

The proposal requires one change on the grammar level: adding the
optional "async" keyword to ``comp_for``::

    comp_for: [ASYNC] 'for' exprlist 'in' or_test [comp_iter]

The ``comprehension`` AST node will have the new ``is_async`` argument.


Backwards Compatibility
-----------------------

The proposal is fully backwards compatible.


Acceptance
==========

:pep:`530` was accepted by Guido, September 6, 2016 [1]_.


Implementation
==============

The implementation is tracked in issue 28008 [3]_.  The reference
implementation git repository is available at [2]_.


References
==========

.. [1] https://mail.python.org/pipermail/python-ideas/2016-September/042141.html

.. [2] https://github.com/1st1/cpython/tree/asyncomp

.. [3] http://bugs.python.org/issue28008


Acknowledgments
===============

I thank Guido van Rossum, Victor Stinner and Elvis Pranskevichus
for their feedback, code reviews, and discussions around this
PEP.

Copyright
=========

This document has been placed in the public domain.


id: pep-0524.rst
content:
PEP: 524
Title: Make os.urandom() blocking on Linux
Author: Victor Stinner <vstinner@python.org>
Status: Final
Type: Standards Track
Created: 20-Jun-2016
Python-Version: 3.6


Abstract
========

Modify ``os.urandom()`` to block on Linux 3.17 and newer until the OS
urandom is initialized to increase the security.

Add also a new ``os.getrandom()`` function (for Linux and Solaris) to be
able to choose how to handle when ``os.urandom()`` is going to block on
Linux.


The bug
=======

Original bug
------------

Python 3.5.0 was enhanced to use the new ``getrandom()`` syscall
introduced in Linux 3.17 and Solaris 11.3. The problem is that users
started to complain that Python 3.5 blocks at startup on Linux in
virtual machines and embedded devices: see issues `#25420
<http://bugs.python.org/issue25420>`_ and `#26839
<http://bugs.python.org/issue26839>`_.

On Linux, ``getrandom(0)`` blocks until the kernel initialized urandom
with 128 bits of entropy. The issue #25420 describes a Linux build
platform blocking at ``import random``. The issue #26839 describes a
short Python script used to compute a MD5 hash, systemd-cron, script
called very early in the init process. The system initialization blocks
on this script which blocks on ``getrandom(0)`` to initialize Python.

The Python initialization requires random bytes to implement a
counter-measure against the hash denial-of-service (hash DoS), see:

* `Issue #13703: Hash collision security issue
  <http://bugs.python.org/issue13703>`_
* :pep:`PEP 456: Secure and interchangeable hash algorithm
  <456>`

Importing the ``random`` module creates an instance of
``random.Random``: ``random._inst``. On Python 3.5, random.Random
constructor reads 2500 bytes from ``os.urandom()`` to seed a Mersenne
Twister RNG (random number generator).

Other platforms may be affected by this bug, but in practice, only Linux
systems use Python scripts to initialize the system.


Status in Python 3.5.2
----------------------

Python 3.5.2 behaves like Python 2.7 and Python 3.4. If the system
urandom is not initialized, the startup does not block, but
``os.urandom()`` can return low-quality entropy (even it is not easily
guessable).


Use Cases
=========

The following use cases are used to help to choose the right compromise
between security and practicability.


Use Case 1: init script
-----------------------

Use a Python 3 script to initialize the system, like systemd-cron. If
the script blocks, the system initialize is stuck too. The issue #26839
is a good example of this use case.

Use case 1.1: No secret needed
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If the init script doesn't have to generate any secure secret, this use
case is already handled correctly in Python 3.5.2: Python startup
doesn't block on system urandom anymore.

Use case 1.2: Secure secret required
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If the init script has to generate a secure secret, there is no safe
solution.

Falling back to weak entropy is not acceptable, it would
reduce the security of the program.

Python cannot produce itself secure entropy, it can only wait until
system urandom is initialized. But in this use case, the whole system
initialization is blocked by this script, so the system fails to boot.

The real answer is that the system initialization must not be blocked by
such script. It is ok to start the script very early at system
initialization, but the script may blocked a few seconds until it is
able to generate the secret.

Reminder: in some cases, the initialization of the system urandom never
occurs and so programs waiting for system urandom blocks forever.


Use Case 2: Web server
----------------------

Run a Python 3 web server serving web pages using HTTP and HTTPS
protocols. The server is started as soon as possible.

The first target of the hash DoS attack was web server: it's important
that the hash secret cannot be easily guessed by an attacker.

If serving a web page needs a secret to create a cookie, create an
encryption key, ..., the secret must be created with good entropy:
again, it must be hard to guess the secret.

A web server requires security. If a choice must be made between
security and running the server with weak entropy, security is more
important. If there is no good entropy: the server must block or fail
with an error.

The question is if it makes sense to start a web server on a host before
system urandom is initialized.

The issues #25420 and #26839 are restricted to the Python startup, not
to generate a secret before the system urandom is initialized.


Fix system urandom
==================

Load entropy from disk at boot
-------------------------------

Collecting entropy can take up to several minutes. To accelerate the
system initialization, operating systems store entropy on disk at
shutdown, and then reload entropy from disk at the boot.

If a system collects enough entropy at least once, the system urandom
will be initialized quickly, as soon as the entropy is reloaded from
disk.


Virtual machines
----------------

Virtual machines don't have a direct access to the hardware and so have
less sources of entropy than bare metal. A solution is to add a
`virtio-rng device
<https://fedoraproject.org/wiki/Features/Virtio_RNG>`_ to pass entropy
from the host to the virtual machine.


Embedded devices
----------------

A solution for embedded devices is to plug an hardware RNG.

For example, Raspberry Pi have an hardware RNG but it's not used by
default. See: `Hardware RNG on Raspberry Pi
<http://fios.sector16.net/hardware-rng-on-raspberry-pi/>`_.



Denial-of-service when reading random
=====================================

Don't use /dev/random but /dev/urandom
--------------------------------------

The ``/dev/random`` device should only used for very specific use cases.
Reading from ``/dev/random`` on Linux is likely to block. Users don't
like when an application blocks longer than 5 seconds to generate a
secret. It is only expected for specific cases like generating
explicitly an encryption key.

When the system has no available entropy, choosing between blocking
until entropy is available or falling back on lower quality entropy is a
matter of compromise between security and practicability. The choice
depends on the use case.

On Linux, ``/dev/urandom`` is secure, it should be used instead of
``/dev/random``. See `Myths about /dev/urandom
<http://www.2uo.de/myths-about-urandom/>`_ by Thomas HÃ¼hn: "Fact:
/dev/urandom is the preferred source of cryptographic randomness on
UNIX-like systems"


getrandom(size, 0) can block forever on Linux
---------------------------------------------

The origin of the Python issue #26839 is the `Debian bug
report #822431
<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=822431>`_: in fact,
``getrandom(size, 0)`` blocks forever on the virtual machine. The system
succeeded to boot because systemd killed the blocked process after 90
seconds.

Solutions like `Load entropy from disk at boot`_ reduces the risk of
this bug.


Rationale
=========

On Linux, reading the ``/dev/urandom`` can return "weak" entropy before
urandom is fully initialized, before the kernel collected 128 bits of
entropy. Linux 3.17 adds a new ``getrandom()`` syscall which allows to
block until urandom is initialized.

On Python 3.5.2, os.urandom() uses the
``getrandom(size, GRND_NONBLOCK)``, but falls back on reading the
non-blocking ``/dev/urandom`` if ``getrandom(size, GRND_NONBLOCK)``
fails with ``EAGAIN``.

Security experts promotes ``os.urandom()`` to generate cryptographic
keys because it is implemented with a `Cryptographically secure
pseudo-random number generator (CSPRNG)
<https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator>`_.
By the way, ``os.urandom()`` is preferred over ``ssl.RAND_bytes()`` for
different reasons.

This PEP proposes to modify os.urandom() to use ``getrandom()`` in
blocking mode to not return weak entropy, but also ensure that Python
will not block at startup


id: pep-0242.rst
content:
PEP: 242
Title: Numeric Kinds
Author: Paul F. Dubois <paul@pfdubois.com>
Status: Withdrawn
Type: Standards Track
Created: 17-Mar-2001
Python-Version: 2.2
Post-History: 17-Apr-2001

.. withdrawn::

   The kinds module will not be added to the standard library.

   There was no opposition to the proposal but only mild interest in
   using it, not enough to justify adding the module to the standard
   library.  Instead, it will be made available as a separate
   distribution item at the Numerical Python site.  At the next
   release of Numerical Python, it will no longer be a part of the
   Numeric distribution.

Abstract
========

This proposal gives the user optional control over the precision
and range of numeric computations so that a computation can be
written once and run anywhere with at least the desired precision
and range.  It is backward compatible with existing code.  The
meaning of decimal literals is clarified.


Rationale
=========

Currently it is impossible in every language except Fortran 90 to
write a program in a portable way that uses floating point and
gets roughly the same answer regardless of platform -- or refuses
to compile if that is not possible.  Python currently has only one
floating point type, equal to a C double in the C implementation.

No type exists corresponding to single or quad floats.  It would
complicate the language to try to introduce such types directly
and their subsequent use would not be portable.  This proposal is
similar to the Fortran 90 "kind" solution, adapted to the Python
environment.  With this facility an entire calculation can be
switched from one level of precision to another by changing a
single line.  If the desired precision does not exist on a
particular machine, the program will fail rather than get the
wrong answer.  Since coding in this style would involve an early
call to the routine that will fail, this is the next best thing to
not compiling.


Supported Kinds of Ints and Floats
==================================

Complex numbers are treated separately below, since Python can be
built without them.

Each Python compiler may define as many "kinds" of integer and
floating point numbers as it likes, except that it must support at
least two kinds of integer corresponding to the existing int and
long, and must support at least one kind of floating point number,
equivalent to the present float.

The range and precision of these required kinds are processor
dependent, as at present, except for the "long integer" kind,
which can hold an arbitrary integer.

The built-in functions ``int()``, ``long()``, and ``float()`` convert inputs
to these default kinds as they do at present.  (Note that a
Unicode string is actually a different "kind" of string and that a
sufficiently knowledgeable person might be able to expand this PEP
to cover that case.)

Within each type (integer, floating) the compiler supports a
linearly-ordered set of kinds, with the ordering determined by the
ability to hold numbers of an increased range and/or precision.


Kind Objects
============

Two new standard functions are defined in a module named "kinds".
They return callable objects called kind objects.  Each int or
floating kind object f has the signature ``result = f(x)``, and each
complex kind object has the signature ``result = f(x, y=0.)``.

``int_kind(n)``
   For an integer argument ``n >= 1``, return a callable object whose
   result is an integer kind that will hold an integer number in
   the open interval (``-10**n``, ``10**n``).  The kind object accepts
   arguments that are integers including longs.  If ``n == 0``,
   returns the kind object corresponding to the Python literal 0.

``float_kind(nd, n)``
   For ``nd >= 0`` and ``n >= 1``, return a callable object whose result
   is a floating point kind that will hold a floating-point
   number with at least nd digits of precision and a base-10
   exponent in the closed interval ``[-n, n]``.  The kind object
   accepts arguments that are integer or float.

   If nd and n are both zero, returns the kind object
   corresponding to the Python literal 0.0.

The compiler will return a kind object corresponding to the least
of its available set of kinds for that type that has the desired
properties.  If no kind with the desired qualities exists in a
given implementation an ``OverflowError`` exception is thrown.  A kind
function converts its argument to the target kind, but if the
result does not fit in the target kind's range, an ``OverflowError``
exception is thrown.

Besides their callable behavior, kind objects have attributes
giving the traits of the kind in question.

1. ``name`` is the name of the kind.  The standard kinds are called
   int, long, double.

2. ``typecode`` is a single-letter string that would be appropriate
   for use with ``Numeric`` or module ``array`` to form an array of this
   kind.  The standard types' typecodes are 'i', 'O', 'd'
   respectively.

3. Integer kinds have these additional attributes: ``MAX``, equal to
   the maximum permissible integer of this kind, or ``None`` for the
   long kind. ``MIN``, equal to the most negative permissible integer
   of this kind, or ``None`` for the long kind.

4. Float kinds have these additional attributes whose properties
   are equal to the corresponding value for the corresponding C
   type in the standard header file "float.h".  ``MAX``, ``MIN``, ``DIG``,
   ``MANT_DIG``, ``EPSILON``, ``MAX_EXP``, ``MAX_10_EXP``, ``MIN_EXP``,
   ``MIN_10_EXP``, ``RADIX``, ``ROUNDS``
   (== ``FLT_RADIX``, ``FLT_ROUNDS`` in float.h).  These
   values are of type integer except for ``MAX``, ``MIN``, and ``EPSILON``,
   which are of the Python floating type to which the kind
   corresponds.


Attributes of Module kinds
==========================

``int_kinds`` is a list of the available integer kinds, sorted from lowest
to highest kind.  By definition, ``int_kinds[-1]`` is the long kind.

``float_kinds`` is a list of the available floating point kinds, sorted
from lowest to highest kind.

``default_int_kind`` is the kind object corresponding to the Python
literal 0

``default_long_kind`` is the kind object corresponding to the Python
literal 0L

``default_float_kind`` is the kind object corresponding to the Python
literal 0.0


Complex Numbers
===============

If supported, complex numbers have real and imaginary parts that
are floating-point numbers with the same kind.  A Python compiler
must support a complex analog of each floating point kind it
supports, if it supports complex numbers at all.

If complex numbers are supported, the following are available in
module kinds:

``complex_kind(nd, n)``
   Return a callable object whose result is a complex kind that
   will hold a complex number each of whose components (.real,
   .imag) is of kind ``float_kind(nd, n)``.  The kind object will
   accept one argument that is of any integer, real, or complex
   kind, or two arguments, each integer or real.

``complex_kinds`` is a list of the available complex kinds, sorted
from lowest to highest kind.

``default_complex_kind`` is the kind object corresponding to the
Python literal 0.0j.  The name of this kind
is doublecomplex, and its typecode is 'D'.

Complex kind objects have these addition attributes:

``floatkind`` is the kind object of the corresponding float type.


Examples
========

In module myprecision.py::

    import kinds
    tinyint = kinds.int_kind(1)
    single = kinds.float_kind(6, 90)
    double = kinds.float_kind(15, 300)
    csingle = kinds.complex_kind(6, 90)

In the rest of my code::

    from myprecision import tinyint, single, double, csingle
    n = tinyint(3)
    x = double(1.e20)
    z = 1.2
    # builtin float gets you the default float kind, properties unknown
    w = x * float(x)
    # but in the following case we know w has kind "double".
    w = x * double(z)

    u = csingle(x + z * 1.0j)
    u2 = csingle(x+z, 1.0)

Note how that entire code can then be changed to a higher
precision by chang


id: pep-0295.rst
content:
PEP: 295
Title: Interpretation of multiline string constants
Author: Stepan Koltsov <yozh@mx1.ru>
Status: Rejected
Type: Standards Track
Created: 22-Jul-2002
Python-Version: 3.0
Post-History:


Abstract
========

This PEP describes an interpretation of multiline string constants
for Python.  It suggests stripping spaces after newlines and
stripping a newline if it is first character after an opening
quotation.


Rationale
=========

This PEP proposes an interpretation of multiline string constants
in Python.  Currently, the value of string constant is all the
text between quotations, maybe with escape sequences substituted,
e.g.::

    def f():
        """
        la-la-la
        limona, banana
        """

    def g():
        return "This is \
        string"

    print repr(f.__doc__)
    print repr(g())

prints::

    '\n\tla-la-la\n\tlimona, banana\n\t'
    'This is \tstring'

This PEP suggest two things:

- ignore the first character after opening quotation, if it is
  newline

- ignore in string constants all spaces and tabs up to
  first non-whitespace character, but no more than current
  indentation.

After applying this, previous program will print::

    'la-la-la\nlimona, banana\n'
    'This is string'

To get this result, previous programs could be rewritten for
current Python as (note, this gives the same result with new
strings meaning)::

    def f():
        """\
    la-la-la
    limona, banana
    """

    def g():
      "This is \
    string"

Or stripping can be done with library routines at runtime (as
pydoc does), but this decreases program readability.


Implementation
==============

I'll say nothing about CPython, Jython or Python.NET.

In original Python, there is no info about the current indentation
(in spaces) at compile time, so space and tab stripping should be
done at parse time.  Currently no flags can be passed to the
parser in program text (like ``from __future__ import xxx``).  I
suggest enabling or disabling of this feature at Python compile
time depending of CPP flag ``Py_PARSE_MULTILINE_STRINGS``.


Alternatives
============

New interpretation of string constants can be implemented with flags
'i' and 'o' to string constants, like::

   i"""
   SELECT * FROM car
   WHERE model = 'i525'
   """ is in new style,

   o"""SELECT * FROM employee
   WHERE birth < 1982
   """ is in old style, and

   """
   SELECT employee.name, car.name, car.price FROM employee, car
   WHERE employee.salary * 36 > car.price
   """ is in new style after Python-x.y.z and in old style otherwise.

Also this feature can be disabled if string is raw, i.e. if flag 'r'
specified.


Copyright
=========

This document has been placed in the Public Domain.


id: pep-0281.rst
content:
PEP: 281
Title: Loop Counter Iteration with range and xrange
Author: Magnus Lie Hetland <magnus@hetland.org>
Status: Rejected
Type: Standards Track
Created: 11-Feb-2002
Python-Version: 2.3
Post-History:


Abstract
========

This PEP describes yet another way of exposing the loop counter in
for-loops. It basically proposes that the functionality of the
function ``indices()`` from :pep:`212` be included in the existing
functions ``range()`` and ``xrange()``.


Pronouncement
=============

In commenting on :pep:`279`'s ``enumerate()`` function, this PEP's author
offered, "I'm quite happy to have it make :pep:`281` obsolete."
Subsequently, :pep:`279` was accepted into Python 2.3.

On 17 June 2005, the BDFL concurred with it being obsolete and
hereby rejected the PEP.  For the record, he found some of the
examples to somewhat jarring in appearance::

   >>> range(range(5), range(10), range(2))
   [5, 7, 9]


Motivation
==========

It is often desirable to loop over the indices of a sequence.  PEP
212 describes several ways of doing this, including adding a
built-in function called indices, conceptually defined as::

   def indices(sequence):
       return range(len(sequence))

On the assumption that adding functionality to an existing built-in
function may be less intrusive than adding a new built-in function,
this PEP proposes adding this functionality to the existing
functions ``range()`` and ``xrange()``.


Specification
=============

It is proposed that all three arguments to the built-in functions
``range()`` and ``xrange()`` are allowed to be objects with a length
(i.e. objects implementing the ``__len__`` method).  If an argument
cannot be interpreted as an integer (i.e. it has no ``__int__``
method), its length will be used instead.

Examples::

   >>> range(range(10))
   [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
   >>> range(range(5), range(10))
   [5, 6, 7, 8, 9]
   >>> range(range(5), range(10), range(2))
   [5, 7, 9]
   >>> list(xrange(range(10)))
   [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
   >>> list(xrange(xrange(10)))
   [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

   # Number the lines of a file:
   lines = file.readlines()
   for num in range(lines):
       print num, lines[num]


Alternatives
============

A natural alternative to the above specification is allowing
``xrange()`` to access its arguments in a lazy manner.  Thus, instead
of using their length explicitly, ``xrange`` can return one index for
each element of the stop argument until the end is reached.  A
similar lazy treatment makes little sense for the start and step
arguments since their length must be calculated before iteration
can begin.  (Actually, the length of the step argument isn't needed
until the second element is returned.)

A pseudo-implementation (using only the stop argument, and assuming
that it is iterable) is::

   def xrange(stop):
       i = 0
       for x in stop:
           yield i
           i += 1

Testing whether to use ``int()`` or lazy iteration could be done by
checking for an ``__iter__`` attribute.  (This example assumes the
presence of generators, but could easily have been implemented as a
plain iterator object.)

It may be questionable whether this feature is truly useful, since
one would not be able to access the elements of the iterable object
inside the for loop through indexing.

Example::

   # Printing the numbers of the lines of a file:
   for num in range(file):
       print num # The line itself is not accessible

A more controversial alternative (to deal with this) would be to
let ``range()`` behave like the function ``irange()`` of :pep:`212` when
supplied with a sequence.

Example::

   >>> range(5)
   [0, 1, 2, 3, 4]
   >>> range('abcde')
   [(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd'), (4, 'e')]


Backwards Compatibility
=======================

The proposal could cause backwards incompatibilities if arguments
are used which implement both ``__int__`` and ``__len__`` (or ``__iter__`` in
the case of lazy iteration with ``xrange``).  The author does not
believe that this is a significant problem.


Copyright
=========

This document has been placed in the public domain.


id: pep-0732.rst
content:
PEP: 732
Title: The Python Documentation Editorial Board
Author: Joanna Jablonski
Sponsor: Mariatta Wijaya
Discussions-To: https://discuss.python.org/t/pep-732-the-python-documentation-editorial-board/36710
Status: Active
Type: Process
Topic: Governance
Created: 14-Oct-2023
Post-History: `20-Oct-2023 <https://discuss.python.org/t/pep-732-the-python-documentation-editorial-board/36710>`__
Resolution: https://discuss.python.org/t/pep-732-the-python-documentation-editorial-board/36710/9


Abstract
========

This PEP:

* Establishes the Python Documentation Editorial Board
* Proposes how the editorial board will work

Motivation
==========

The Steering Council approved the creation of a
`Documentation Working Group <https://github.com/python/docs-community/blame/main/docs/workgroup/workgroup_charter.rst>`_
in March 2021 to set direction for the docs. This group
is now called the Editorial Board to differentiate it
from the Documentation Working Group that was created
since then to focus on more tactical work.

The purpose of the Python documentation is to serve the
present and future end users of Python. As such, the core
development community and the greater Python documentation
contributors work together to achieve this:

.. image:: pep-0732-concentric.drawio.svg
   :align: center
   :alt: Three concentric circles. At the centre: Documentation Editorial
         Board, trusted group. Around this: Documentation Working Group,
         volunteers who contribute to the docs. Finally, the outer circle
         is the world, includes readers of the documentation.
   :class: invert-in-dark-mode
   :width: 75%


Specification
=============

Mandate
-------

The editorial board will:

* Ensure processes are in place to maintain and improve the quality of Python's documentation
* Foster Python documentation as a community resource to serve the current and future users
* Act in alignment with the `Python Software Foundation mission
  <https://www.python.org/psf/mission/>`_, which is to advance the Python
  programming language, and to support and facilitate the growth of a diverse
  and international community of Python programmers
* Ensure that contributing to documentation is accessible, inclusive, and sustainable
* Establish appropriate decision-making processes for documentation content
* Seek to achieve consensus among contributors prior to making decisions
* Be the final arbiter for documentation content decisions

Responsibilities
----------------

The board has authority to make decisions about Pythonâs
documentation, as scoped below. For example, it can:

* Set big-picture strategy for Pythonâs documentation
* Set the intended structure for documentation
* Make style and editorial decisions for both writing and design
* Handle documentation governance (for example, delegation of decision-making
  to subject-matter experts, resolution of disagreements, decisions.)

Scope
-----

The Editorial board oversees the content and strategy for the following:

.. list-table::
   :widths: 50 50
   :header-rows: 1

   * - In scope
     - Not in scope
   * - CPython documentation (docs.python.org)
     - Code comments in CPython codebase
   * - CPython devguide (devguide.python.org)
     - CPython docstrings
   * - Translations of CPython docs
     - PEPs (peps.python.org)
   * -
     - PyPA documentation
   * -
     - www.python.org
   * -
     - The Python Wiki (wiki.python.org)


Composition
~~~~~~~~~~~

The Python Documentation Editorial Board is composed of five members.

Editorial Board Members
~~~~~~~~~~~~~~~~~~~~~~~

The initial Editorial Board members are:

* Mariatta Wijaya
* Ned Batchelder
* Joanna Jablonski
* Guido van Rossum
* Carol Willing

Editorial Board Member Qualifications
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Editorial board members should have:

* A good grasp of the philosophy of the Python project
* A background in Python education and developer-facing documentation
* A solid track record of being constructive and helpful
* A history of making significant contributions to Python
* A willingness to dedicate time to improving Python's docs

Members of the Editorial Board should have experience in education,
communication, technical writing, Pythonâs documentation, accessibility,
translation, or community management.

Term
~~~~

Editorial Board members serve for an indefinite term, though it is
generally expected that there will be changes in Editorial Board
composition each year. Editorial Board members will confirm annually
whether they wish to continue as a board member. Members may resign
at any time.

If a board member drops out of touch and cannot be contacted for a
month or longer, then the rest of the board may vote to replace them.

Changes to the Editorial Board's Size
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Annually after each major Python release, the Editorial Board will
review whether the board's size should change. This provides
flexibility if the needs of the documentation community change
over time. A simple majority is needed to make a decision to
increase the board's size where quorum is 80% of the current board.

As the sponsoring organization of the Documentation Editorial
Board, the Steering Council may change the number of members of
the Board at any time, including appointing new members or
dismissing existing members.

Vacancies
~~~~~~~~~

If a vacancy exists on the board for any reason, the Documentation
Editorial Board will publicly announce a call for prospective
board members. Prospective board members would submit a brief
document stating qualifications and their motivation to serve.
The sitting members of the Editorial Board will select new board
members by a simple majority where quorum is 80% of the current board.

Amendments
==========

This PEP serves as a charter for the Docs Editorial Board. Changes
to its operation can be made either through a new PEP or through
a change to this PEP. In either case, the change would be decided
upon by the Steering Council after discussion in the community.


PEP Acceptance
==============

:pep:`732` was `accepted by the Python Steering Council on December 11, 2023
<https://discuss.python.org/t/pep-732-the-python-documentation-editorial-board/36710/8>`_.

The Steering Council commented that, while they don't disagree with the
scoping set out in the PEP, it would probably make
sense for the Editorial Board to consider expanding the scope to include
docstrings in the standard library, once the Board is sufficiently established
and the higher priorities have been taken care of.


Contact
=======

To ask the Editorial Board for a decision,
community members may open an issue in the
`python/editorial-board <https://github.com/python/editorial-board/>`__
repository.


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0726.rst
content:
PEP: 726
Title: Module ``__setattr__`` and ``__delattr__``
Author: Sergey B Kirpichev <skirpichev@gmail.com>
Sponsor: Adam Turner <python@quite.org.uk>
Discussions-To: https://discuss.python.org/t/32640/
Status: Rejected
Type: Standards Track
Created: 24-Aug-2023
Python-Version: 3.13
Post-History: `06-Apr-2023 <https://discuss.python.org/t/25506/>`__,
              `31-Aug-2023 <https://discuss.python.org/t/32640/>`__,
Resolution: https://discuss.python.org/t/32640/32


Abstract
========

This PEP proposes supporting user-defined ``__setattr__``
and ``__delattr__`` methods on modules to extend customization
of module attribute access beyond :pep:`562`.

Motivation
==========

There are several potential uses of a module ``__setattr__``:

1. To prevent setting an attribute at all (i.e. make it read-only)
2. To validate the value to be assigned
3. To intercept setting an attribute and update some other state

Proper support for read-only attributes would also require adding the
``__delattr__`` function to prevent their deletion.

It would be convenient to directly support such customization, by recognizing
``__setattr__`` and ``__delattr__`` methods defined in a module that would act
like normal :py:meth:`python:object.__setattr__` and
:py:meth:`python:object.__delattr__` methods, except that they will be defined
on module *instances*.  Together with existing ``__getattr__`` and ``__dir__``
methods this will streamline all variants of customizing module attribute access.

For example

.. code:: python

   # mplib.py

   CONSTANT = 3.14
   prec = 53
   dps = 15

   def dps_to_prec(n):
       """Return the number of bits required to represent n decimals accurately."""
       return max(1, int(round((int(n)+1)*3.3219280948873626)))

   def prec_to_dps(n):
       """Return the number of accurate decimals that can be represented with n bits."""
       return max(1, int(round(int(n)/3.3219280948873626)-1))

   def validate(n):
       n = int(n)
       if n <= 0:
           raise ValueError('Positive integer expected')
       return n

   def __setattr__(name, value):
       if name == 'CONSTANT':
           raise AttributeError('Read-only attribute!')
       if name == 'dps':
           value = validate(value)
           globals()['dps'] = value
           globals()['prec'] = dps_to_prec(value)
           return
       if name == 'prec':
           value = validate(value)
           globals()['prec'] = value
           globals()['dps'] = prec_to_dps(value)
           return
       globals()[name] = value

   def __delattr__(name):
       if name in ('CONSTANT', 'dps', 'prec'):
           raise AttributeError('Read-only attribute!')
       del globals()[name]

.. code:: pycon

  >>> import mplib
  >>> mplib.foo = 'spam'
  >>> mplib.CONSTANT = 42
  Traceback (most recent call last):
    ...
  AttributeError: Read-only attribute!
  >>> del mplib.foo
  >>> del mplib.CONSTANT
  Traceback (most recent call last):
    ...
  AttributeError: Read-only attribute!
  >>> mplib.prec
  53
  >>> mplib.dps
  15
  >>> mplib.dps = 5
  >>> mplib.prec
  20
  >>> mplib.dps = 0
  Traceback (most recent call last):
    ...
  ValueError: Positive integer expected


Existing Options
================

The current workaround is assigning the ``__class__`` of a module object to a
custom subclass of :py:class:`python:types.ModuleType` (see [1]_).

For example, to prevent modification or deletion of an attribute we could use:

.. code:: python

   # mod.py

   import sys
   from types import ModuleType

   CONSTANT = 3.14

   class ImmutableModule(ModuleType):
       def __setattr__(name, value):
           raise AttributeError('Read-only attribute!')

       def __delattr__(name):
           raise AttributeError('Read-only attribute!')

   sys.modules[__name__].__class__ = ImmutableModule

But this variant is slower (~2x) than the proposed solution.  More
importantly, it also brings a noticeable speed regression (~2-3x) for
attribute *access*.

Specification
=============

The ``__setattr__`` function at the module level should accept two
arguments, the name of an attribute and the value to be assigned,
and return :py:obj:`None` or raise an :exc:`AttributeError`.

.. code:: python

   def __setattr__(name: str, value: typing.Any, /) -> None: ...

The ``__delattr__`` function should accept one argument,
the name of an attribute, and return :py:obj:`None` or raise an
:py:exc:`AttributeError`:

.. code:: python

   def __delattr__(name: str, /) -> None: ...

The ``__setattr__`` and ``__delattr__`` functions are looked up in the
module ``__dict__``.  If present, the appropriate function is called to
customize setting the attribute or its deletion, else the normal
mechanism (storing/deleting the value in the module dictionary) will work.

Defining module ``__setattr__`` or ``__delattr__`` only affects lookups made
using the attribute access syntax --- directly accessing the module globals
(whether by ``globals()`` within the module, or via a reference to the module's
globals dictionary) is unaffected.  For example:

.. code:: pycon

   >>> import mod
   >>> mod.__dict__['foo'] = 'spam'  # bypasses __setattr__, defined in mod.py

or

.. code:: python

   # mod.py

   def __setattr__(name, value):
      ...

   foo = 'spam'  # bypasses __setattr__
   globals()['bar'] = 'spam'  # here too

   def f():
       global x
       x = 123

   f()  # and here

To use a module global and trigger ``__setattr__`` (or ``__delattr__``),
one can access it via ``sys.modules[__name__]`` within the module's code:

.. code:: python

   # mod.py

   sys.modules[__name__].foo = 'spam'  # bypasses __setattr__

   def __setattr__(name, value):
       ...

   sys.modules[__name__].bar = 'spam'  # triggers __setattr__

This limitation is intentional (just as for the :pep:`562`), because the
interpreter highly optimizes access to module globals and disabling all that
and going through special methods written in Python would slow down the code
unacceptably.


How to Teach This
=================

The "Customizing module attribute access" [1]_ section of the documentation
will be expanded to include new functions.


Reference Implementation
========================

The reference implementation for this PEP can be found in `CPython PR #108261
<https://github.com/python/cpython/pull/108261>`__.


Backwards compatibility
=======================

This PEP may break code that uses module level (global) names
``__setattr__`` and ``__delattr__``, but the language reference
explicitly reserves *all* undocumented dunder names, and allows
"breakage without warning" [2]_.

The performance implications of this PEP are small, since additional
dictionary lookup is much cheaper than storing/deleting the value in
the dictionary.  Also it is hard to imagine a module that expects the
user to set (and/or delete) attributes enough times to be a
performance concern.  On another hand, proposed mechanism allows to
override setting/deleting of attributes without affecting speed of
attribute access, which is much more likely scenario to get a
performance penalty.


Discussion
==========

As pointed out by Victor Stinner, the proposed API could be useful already in
the stdlib, for example to ensure that :py:obj:`sys.modules` type is always a
:py:class:`dict`:

.. code:: pycon

   >>> import sys
   >>> sys.modules = 123
   >>> import asyncio
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "<frozen importlib._bootstrap>", line 1260, in _find_and_load
   AttributeError: 'int' object has no attribute 'get'

or to prevent deletion of critical :py:mod:`sys` attributes, which makes the
code more complicated.  For example, code using :py:obj:`sys.stderr` has to
check if the attribute exists and if it's not :py:obj:`None`.  Currently, it's
possible to remove any :py:mod:`sys` attribute, including functions:

.. code:: pycon

   >>> import sys
   >>> del sys.excepthook
   >>> 1+  # notice the next line
   sys.exce


id: pep-0646.rst
content:
PEP: 646
Title: Variadic Generics
Author: Mark Mendoza <mendoza.mark.a@gmail.com>,
        Matthew Rahtz <mrahtz@google.com>,
        Pradeep Kumar Srinivasan <gohanpra@gmail.com>,
        Vincent Siles <vsiles@fb.com>
Sponsor: Guido van Rossum <guido@python.org>
Status: Final
Type: Standards Track
Topic: Typing
Created: 16-Sep-2020
Python-Version: 3.11
Post-History: 07-Oct-2020, 23-Dec-2020, 29-Dec-2020
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/OR5RKV7GAVSGLVH3JAGQ6OXFAXIP5XDX/

.. canonical-doc:: :ref:`typing:typevartuple` and :py:class:`typing.TypeVarTuple`

Abstract
========

:pep:`484` introduced ``TypeVar``, enabling creation of generics parameterised
with a single type. In this PEP, we introduce ``TypeVarTuple``, enabling parameterisation
with an *arbitrary* number of types - that is, a *variadic* type variable,
enabling *variadic* generics. This enables a wide variety of use cases.
In particular, it allows the type of array-like structures
in numerical computing libraries such as NumPy and TensorFlow to be
parameterised with the array *shape*, enabling static type checkers
to catch shape-related bugs in code that uses these libraries.

Acceptance
==========

This PEP was accepted for Python 3.11, with the caveat that details around
multiple unpackings in a type expression aren't specified precisely.
This gives individual type checkers some leeway, but can be tightened
in future PEPs.

Motivation
==========

Variadic generics have long been a requested feature, for a myriad of
use cases [#typing193]_. One particular use case - a use case with potentially
large impact, and the main case this PEP targets - concerns typing in
numerical libraries.

In the context of numerical computation with libraries such as NumPy and TensorFlow,
the *shape* of variables is often just as important as the variable *type*.
For example, consider the following function which converts a batch [#batch]_
of videos to grayscale:

::

    def to_gray(videos: Array): ...

From the signature alone, it is not obvious what shape of array [#array]_
we should pass for the ``videos`` argument. Possibilities include, for
example,

  batch Ã time Ã height Ã width Ã channels

and

  time Ã batch Ã channels Ã height Ã width. [#timebatch]_

This is important for three reasons:

* **Documentation**. Without the required shape being clear in the signature,
  the user must hunt in the docstring or the code in question to determine
  what the input/output shape requirements are.
* **Catching shape bugs before runtime**. Ideally, use of incorrect shapes
  should be an error we can catch ahead of time using static analysis.
  (This is particularly important for machine learning code, where iteration
  times can be slow.)
* **Preventing subtle shape bugs**. In the worst case, use of the wrong shape
  will result in the program appearing to run fine, but with a subtle bug
  that can take days to track down. (See `this exercise`_ in a popular machine learning
  tutorial for a particularly pernicious example.)

Ideally, we should have some way of making shape requirements explicit in
type signatures. Multiple proposals [#numeric-stack]_ [#typing-ideas]_
[#syntax-proposal]_ have suggested the use of the standard generics syntax for
this purpose. We would write:

::

    def to_gray(videos: Array[Time, Batch, Height, Width, Channels]): ...

However, note that arrays can be of arbitrary rank - ``Array`` as used above is
generic in an arbitrary number of axes. One way around this would be to use a different
``Array`` class for each rank...

::

    Axis1 = TypeVar('Axis1')
    Axis2 = TypeVar('Axis2')

    class Array1(Generic[Axis1]): ...

    class Array2(Generic[Axis1, Axis2]): ...

...but this would be cumbersome, both for users (who would have to sprinkle 1s and 2s
and so on throughout their code) and for the authors of array libraries (who would have to duplicate implementations throughout multiple classes).

Variadic generics are necessary for an ``Array`` that is generic in an arbitrary
number of axes to be cleanly defined as a single class.

Summary Examples
================

Cutting right to the chase, this PEP allows an ``Array`` class that is generic
in its shape (and datatype) to be defined using a newly-introduced
arbitrary-length type variable, ``TypeVarTuple``, as follows:

::

    from typing import TypeVar, TypeVarTuple

    DType = TypeVar('DType')
    Shape = TypeVarTuple('Shape')

    class Array(Generic[DType, *Shape]):

        def __abs__(self) -> Array[DType, *Shape]: ...

        def __add__(self, other: Array[DType, *Shape]) -> Array[DType, *Shape]: ...

Such an ``Array`` can be used to support a number of different kinds of
shape annotations. For example, we can add labels describing the
semantic meaning of each axis:

::

    from typing import NewType

    Height = NewType('Height', int)
    Width = NewType('Width', int)

    x: Array[float, Height, Width] = Array()

We could also add annotations describing the actual size of each axis:

::

    from typing import Literal as L


    x: Array[float, L[480], L[640]] = Array()

For consistency, we use semantic axis annotations as the basis of the examples
in this PEP, but this PEP is agnostic about which of these two (or possibly other)
ways of using ``Array`` is preferable; that decision is left to library authors.

(Note also that for the rest of this PEP, for conciseness of example, we use
a simpler version of ``Array`` which is generic only in the shape - *not* the
data type.)

Specification
=============

In order to support the above use cases, we introduce
``TypeVarTuple``. This serves as a placeholder not for a single type
but for a *tuple* of types.

In addition, we introduce a new use for the star operator: to 'unpack'
``TypeVarTuple`` instances and tuple types such as ``Tuple[int,
str]``. Unpacking a ``TypeVarTuple`` or tuple type is the typing
equivalent of unpacking a variable or a tuple of values.

Type Variable Tuples
--------------------

In the same way that a normal type variable is a stand-in for a single
type such as ``int``, a type variable *tuple* is a stand-in for a *tuple* type such as
``Tuple[int, str]``.

Type variable tuples are created with:

::

    from typing import TypeVarTuple

    Ts = TypeVarTuple('Ts')

Using Type Variable Tuples in Generic Classes
'''''''''''''''''''''''''''''''''''''''''''''

Type variable tuples behave like a number of individual type variables packed in a
``Tuple``. To understand this, consider the following example:

::

  Shape = TypeVarTuple('Shape')

  class Array(Generic[*Shape]): ...

  Height = NewType('Height', int)
  Width = NewType('Width', int)
  x: Array[Height, Width] = Array()

The ``Shape`` type variable tuple here behaves like ``Tuple[T1, T2]``,
where ``T1`` and ``T2`` are type variables. To use these type variables
as type parameters of ``Array``, we must *unpack* the type variable tuple using
the star operator: ``*Shape``. The signature of ``Array`` then behaves
as if we had simply written ``class Array(Generic[T1, T2]): ...``.

In contrast to ``Generic[T1, T2]``, however, ``Generic[*Shape]`` allows
us to parameterise the class with an *arbitrary* number of type parameters.
That is, in addition to being able to define rank-2 arrays such as
``Array[Height, Width]``, we could also define rank-3 arrays, rank-4 arrays,
and so on:

::

  Time = NewType('Time', int)
  Batch = NewType('Batch', int)
  y: Array[Batch, Height, Width] = Array()
  z: Array[Time, Batch, Height, Width] = Array()

Using Type Variable Tuples in Functions
'''''''''''''''''''''''''''''''''''''''

Type variable tuples can be used anywhere a normal ``TypeVar`` can.
This includes class definitions, as shown above, as well as function
signatures and variable annotations:

::

    class Array(Generic[*Shape]):

        def __init__(self, shape: Tuple[*Shape]):
            self._shape: Tuple[*Shape] = shape

        def get_shape(self) -> 


id: pep-0652.rst
content:
PEP: 652
Title: Maintaining the Stable ABI
Author: Petr Viktorin <encukou@gmail.com>
Discussions-To: https://discuss.python.org/t/pre-pep-maintaining-the-stable-abi/6986/
Status: Final
Type: Standards Track
Created: 09-Feb-2021
Python-Version: 3.10
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/IN4XMFLQJ6D6V67EXU27GV3QWSEHHNNH/


.. canonical-doc:: :ref:`python:stable` (user docs) and
                   :ref:`devguide:c-api` (development docs)

Abstract
========

CPython's Limited C-API and Stable ABI, introduced in :pep:`384`,
will be formalized in a single definitive file, tested, and documented.


Motivation
==========

:pep:`384` defined a Limited API and Stable ABI, which allow extenders and
embedders of CPython to compile extension modules that are binary-compatible
with any subsequent version of 3.x.
In theory, this brings several advantages:

* A module can be built only once per platform and support multiple versions
  of Python, reducing time, power and maintainer attention needed for builds
  (in exchange for potentially worse performance).

* Binary wheels using the Stable ABI work with new versions of CPython
  throughout the pre-release period, and can be tested in environments where
  building from source is not practical.

* As a welcome side effect of the Limited API's hiding of implementation
  details, this API is becoming a viable target for alternate Python
  implementations that would be incompatible with the full C API.

However, in hindsight, :pep:`384` and its implementation has several issues:

* It is ill-defined. According to :pep:`384`, functions are *opt-out*:
  all functions not specially marked are part of the Stable ABI.
  In practice, for Windows there's a list that's *opt-in*.
  For users there is a ``#define`` that should make only the Stable ABI
  available, but there is no process that ensures it is kept up-to date.
  Neither is there a process for updating the documentation.
* Until recently, the Stable ABI was not tested at all. It tends to break.
  For example, changing a function to a macro can break the Stable ABI as the
  function symbol is removed.
* There is no way to deprecate parts of the Limited API.
* It is incomplete. Some operations are not available in the Stable ABI,
  with little reason except "we forgot".
  (This last point is one the PEP will not help with, however.)

This PEP defines the Limited API more clearly and introducess process
designed to make the Stable ABI and Limited API more useful and robust.


Rationale
=========

This PEP contains a lot of clarifications and definitions, but just one big
technical change: the Stable ABI will be explicitly listed in
a human-maintained âmanifestâ file.

There have been efforts to collect such lists automatically, e.g. by scanning
the symbols exported from Python.
Such automation might seem easier to maintain than a handcrafted file,
but has major issues: for example, the set exported symbols has
platform-specific variations.
Also, the cost of updating an explicit manifest is small compared
to the overall work that should go into changing API that will need to
be supported forever (or until Python 3 reaches end of life, if that
comes sooner).

This PEP proposes automatically generating things *from* the manifest:
initially documentation and DLL contents, with later possibilities
for also automating tests.


Stable ABI vs. Limited API
==========================

:pep:`384` and this document deal with the *Limited API* and the *Stable ABI*,
two related but distinct concepts. In short:

* The *Stable ABI* is a promise that certain extensions compiled with
  CPython 3.x will be binary compatible with all subsequent versions
  of CPython 3.x.
* The *Limited API* is a subset of CPython's C API that produces such extensions.

This section clarifies these terms and defines some of their semantics
(either pre-existing or newly proposed here).

The word âExtensionsâ is used as a shorthand for all code that uses the
Python API, e.g. extension modules or software that embeds Python.


Stable ABI
----------

The CPython *Stable ABI* is a promise that extensions built against
a specific Stable ABI version will be compatible with any newer
interpreter of the same major version.

The Stable ABI does not define a complete binary interface:
important details like the layout of structures in memory or function
calling conventions are determined by the platform and the compiler and
its settings.
The Stable ABI promise only applies if these lower-details are also stable.

For example, an extension built with the CPython 3.10 Stable ABI will be usable
with CPython 3.11, 3.12, etc.
It will not necessarily be compatible with CPython 4.0, nor with CPython 3.10
on a different platform.

The Stable ABI is not generally forward-compatible: an extension built and
tested with CPython 3.10 will not generally be compatible with CPython 3.9.

.. note::
   For example, starting in Python 3.10, the ``Py_tp_doc`` slot may be set to
   ``NULL``, while in older versions, a ``NULL`` value will likely crash the
   interpreter.

The Stable ABI trades performance for its stability.
For example, extensions built for a specific CPython version will automatically
use faster macros instead of functions in the Stable ABI.

Future Python versions may deprecate some members of the Stable ABI.
Deprecated members will still work, but may suffer from issues like reduced
performance or, in the most extreme cases, memory/resource leaks.

Limited API
-----------

The Stable ABI promise holds for extensions compiled from code that restricts
itself to the *Limited API* (application programming interface).
The Limited API is a subset of CPython's C API.

Extensions that target the Limited API should define the preprocessor macro
``Py_LIMITED_API`` to either ``3`` or the current ``PYTHON_API_VERSION``.
This will enable Stable ABI versions of several functions and limit definitions
to the Limited API.
(However, note that the macro is not perfect: due to technical issues or
oversights, some non-limited API might be exposed even with it defined.)

The Limited API is not guaranteed to be *stable*.
In the future, parts of the Limited API may be deprecated.
They may even be removed, as long as the *Stable ABI* is kept
stable and Python's general backwards compatibility policy, :pep:`387`,
is followed.

.. note::

   For example, a function declaration might be removed from public header
   files but kept in the library.
   This is currently a possibility for the future; this PEP does not to propose
   a concrete process for deprecations and removals.

The goal for the Limited API is to cover everything needed to interact
with the interpreter.
The main reason to not include a public API in the Limited subset
should be that it needs implementation details that change between CPython
versions (like struct memory layouts) â usually for performance reasons.

The Limited API is not limited to CPython. Other implementations are
encouraged to implement it and help drive its design.


Specification
=============

To make the Stable ABI more useful and robust, the following changes
are proposed.


Stable ABI Manifest
-------------------

All members of the Stable ABI â functions, typedefs, structs, data, macros,
and constants â will be explicitly listed in a single "manifest" file,
``Misc/stable_abi.txt``.

For structs, any fields that users of the Stable ABI are allowed to access
will be listed explicitly.

The manifest will also serve as the definitive list of the Limited API.
Members that are not part of the Limited API, but are part of the Stable ABI
(e.g. ``PyObject.ob_type``, which is accessible by the ``Py_TYPE`` macro),
will be annotated as such.

For items that are only available on some systems, the manifest will record the
feature macro that determines their presence (such as ``MS_WINDOWS`` or
``HAVE_FORK``).
To make the implementation (and usage from non-C lang


id: pep-0685.rst
content:
PEP: 685
Title: Comparison of extra names for optional distribution dependencies
Author: Brett Cannon <brett@python.org>
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/14141
Status: Accepted
Type: Standards Track
Topic: Packaging
Created: 08-Mar-2022
Post-History: `08-Mar-2022 <https://discuss.python.org/t/14141>`__
Resolution: https://discuss.python.org/t/pep-685-comparison-of-extra-names-for-optional-distribution-dependencies/14141/55


.. canonical-pypa-spec::


Abstract
========

This PEP specifies how to normalize `distribution extra <Provides-Extra_>`_
names when performing comparisons.
This prevents tools from either failing to find an extra name, or
accidentally matching against an unexpected name.


Motivation
==========

The `Provides-Extra`_ core metadata specification states that an extra's
name "must be a valid Python identifier".
:pep:`508` specifies that the value of an ``extra`` marker may contain a
letter, digit, or any one of ``.``, ``-``, or ``_`` after the initial character.
There is no other `PyPA specification
<https://packaging.python.org/en/latest/specifications/>`_
which outlines how extra names should be written or normalized for comparison.
Due to the amount of packaging-related code in existence,
it is important to evaluate current practices by the community and
standardize on one that doesn't break most existing code, while being
something tool authors can agree to following.

The issue of there being no consistent standard was brought forward by an
`initial discussion <https://discuss.python.org/t/7614>`__
noting that the extra ``adhoc-ssl`` was not considered equal to the name
``adhoc_ssl`` by pip 22.


Rationale
=========

:pep:`503` specifies how to normalize distribution names::

    re.sub(r"[-_.]+", "-", name).lower()

This collapses any run of the characters ``-``, ``_`` and ``.``
down to a single ``-``.
For example, ``---`` ``.`` and ``__`` all get converted to just ``-``.
This does **not** produce a valid Python identifier, per
the core metadata 2.2 specification for extra names.

`Setuptools 60 performs normalization <https://github.com/pypa/setuptools/blob/b2f7b8f92725c63b164d5776f85e67cc560def4e/pkg_resources/__init__.py#L1324-L1330>`__
via::

    re.sub(r'[^A-Za-z0-9-.]+', '_', name).lower()

The use of an underscore/``_`` differs from PEP 503's use of a hyphen/``-``,
and it also normalizes characters outside of those allowed by :pep:`508`.
Runs of ``.`` and ``-``, unlike PEP 503, do **not** get normalized to one ``_``,
e.g. ``..`` stays the same. To note, this is inconsistent with this function's
docstring, which *does* specify that all non-alphanumeric characters
(which would include ``-`` and ``.``) are normalized and collapsed.

For pip 22, its
"extra normalisation behaviour is quite convoluted and erratic" [pip-erratic]_
and so its use is not considered.

.. [pip-erratic] Tzu-ping Chung on Python Discourse <https://discuss.python.org/t/7614/10


Specification
=============

When comparing extra names, tools MUST normalize the names being compared
using the semantics outlined in :pep:`PEP 503 <0503#normalized-names>`
for names::

    re.sub(r"[-_.]+", "-", name).lower()

The `core metadata`_ specification will be updated such that the allowed
names for `Provides-Extra`_ matches what :pep:`508` specifies for names.
This will bring extra naming in line with that of the Name_ field.
Because this changes what is considered valid, it will lead to a core
metadata version increase to ``2.3``.

For tools writing `core metadata`_,
they MUST write out extra names in their normalized form.
This applies to the `Provides-Extra`_ field and the
:pep:`extra marker <0508#extras>` when used in the `Requires-Dist`_ field.

Tools generating metadata MUST raise an error if a user specified
two or more extra names which would normalize to the same name.
Tools generating metadata MUST raise an error if an invalid extra
name is provided as appropriate for the specified core metadata version.
If a project's metadata specifies an older core metadata version and
the name would be invalid with newer core metadata versions,
tools reading that metadata SHOULD warn the user.
Tools SHOULD warn users when an invalid extra name is read and SHOULD
ignore the name to avoid ambiguity.
Tools MAY raise an error instead of a warning when reading an
invalid name, if they so desire.


Backwards Compatibility
=======================

Moving to :pep:`503` normalization and :pep:`508` name acceptance
allows for all preexisting, valid names to continue to be valid.

Based on research looking at a collection of wheels on PyPI [pypi-results]_,
the risk of extra name clashes is limited to 73 instances when considering
all extras names on PyPI, valid or not (not just those within a single package)
while *only* looking at valid names leads to only 3 clashes:

* ``dev-test``: ``dev_test``, ``dev-test``, ``dev.test``
* ``dev-lint``: ``dev-lint``, ``dev.lint``, ``dev_lint``
* ``apache-beam``: ``apache-beam``, ``apache.beam``

By requiring tools writing core metadata to only record the normalized name,
the issue of preexisting, invalid extra names should diminish over time.

.. [pypi-results] Paul Moore on Python Discourse https://discuss.python.org/t/14141/17


Security Implications
=====================

It is possible that for a distribution that has conflicting extra names, a
tool ends up installing dependencies that somehow weaken the security
of the system.
This is only hypothetical and if it were to occur,
it would probably be more of a security concern for the distributions
specifying such extras names rather than the distribution that pulled
them in together.


How to Teach This
=================

This should be transparent to users on a day-to-day basis.
It will be up to tools to educate/stop users when they select extra
names which conflict.


Reference Implementation
========================

No reference implementation is provided aside from the code above,
but the expectation is the `packaging project`_ will provide a
function in its ``packaging.utils`` module that will implement extra name
normalization.
It will also implement extra name comparisons appropriately.
Finally, if the project ever gains the ability to write out metadata,
it will also implement this PEP.


Transition Plan
===============

There is a risk that a build tool will produce core metadata
conforming to version 2.3 and thus this PEP but which is consumed by a
tool that is unaware of this PEP (if that tool chooses to attempt to
read a core metadata version it does not directly support).
In such a case there is a chance that a user may specify an extra
using an non-normalized name which worked previously but which fails
now.

As such, consumers of this PEP should be prioritized more than
producers so that users can be notified that they are specifying extra
names which are not normalized (and thus may break in the future).


Rejected Ideas
==============

Using setuptools 60's normalization
-----------------------------------

Initially, this PEP proposed using setuptools ``safe_extra()`` for normalization
to try to minimize backwards-compatibility issues.
However, after checking various wheels on PyPI,
it became clear that standardizing **all** naming on :pep:`508` and
:pep:`503` semantics was easier and better long-term,
while causing minimal backwards compatibility issues.


Open Issues
===========

N/A


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


.. _core metadata: https://packaging.python.org/en/latest/specifications/core-metadata/
.. _Name: https://packaging.python.org/en/latest/specifications/core-metadata/#name
.. _packaging project: https://packaging.pypa.io
.. _Provides-Extra: https://packaging.python.org/en/latest/specifications/core-metadata/#provides-extra-multiple-use
.. _Requires-Dist: https://packaging.python.org/en/late


id: pep-0691.rst
content:
PEP: 691
Title: JSON-based Simple API for Python Package Indexes
Author: Donald Stufft <donald@stufft.io>,
        Pradyun Gedam <pradyunsg@gmail.com>,
        Cooper Lees <me@cooperlees.com>,
        Dustin Ingram <di@python.org>
PEP-Delegate: Brett Cannon <brett@python.org>
Discussions-To: https://discuss.python.org/t/pep-691-json-based-simple-api-for-python-package-indexes/15553
Status: Accepted
Type: Standards Track
Topic: Packaging
Created: 04-May-2022
Post-History: `05-May-2022 <https://discuss.python.org/t/pep-691-json-based-simple-api-for-python-package-indexes/15553>`__
Resolution: https://discuss.python.org/t/pep-691-json-based-simple-api-for-python-package-indexes/15553/70


Abstract
========

The "Simple Repository API" that was defined in :pep:`503` (and was in use much
longer than that) has served us reasonably well for a very long time. However,
the reliance on using HTML as the data exchange mechanism has several
shortcomings.

There are two major issues with an HTML-based API:

- While HTML5 is a standard, it's an incredibly complex standard and ensuring
  completely correct parsing of it involves complex logic that does not
  currently exist within the Python standard library (nor the standard library
  of many other languages).

  This means that to actually accept everything that is technically valid, tools
  have to pull in large dependencies or they have to rely on the standard library's
  ``html.parser`` library, which is lighter weight but potentially doesn't
  fully support HTML5.

- HTML5 is primarily designed as a markup language to present documents for human
  consumption. Our use of it is driven largely for historical and accidental
  reasons, and it's unlikely anyone would design an API that relied on it if
  they were starting from scratch.

  The primary issue with using a markup format designed for human consumption
  is that there's not a great way to actually encode data within HTML. We've
  gotten around this by limiting the data we put in this API and being creative
  with how we can cram data into the API (for instance, hashes are embedded as
  URL fragments, adding the ``data-yanked`` attribute in :pep:`592`).

:pep:`503` was largely an attempt to standardize what was already in use, so it
did not propose any large changes to the API.

In the intervening years, we've regularly talked about an "API V2" that would
re-envision the entire API of PyPI. However, due to limited time constraints,
that effort has not gained much, if any, traction beyond people thinking that
it would be nice to do.

This PEP attempts to take a different route. It doesn't fundamentally change
the overall API structure, but instead specifies a new serialization of the
existing data contained in existing :pep:`503` responses in a format that is
easier for software to parse rather than using a human centric document format.


Goals
=====

- **Enable zero configuration discovery.** Clients of the simple API **MUST** be
  able to gracefully determine whether a target repository supports this PEP
  without relying on any form of out of band communication (configuration, prior
  knowledge, etc). Individual clients **MAY** choose to require configuration
  to enable the use of this API, however.
- **Enable clients to drop support for "legacy" HTML parsing.** While it is expected
  that most clients will keep supporting HTML-only repositories for a while, if not
  forever, it should be possible for a client to choose to support only the new
  API formats and no longer invoke an HTML parser.
- **Enable repositories to drop support for "legacy" HTML formats.** Similar to
  clients, it is expected that most repositories will continue to support HTML
  responses for a long time, or forever. It should be possible for a repository to
  choose to only support the new formats.
- **Maintain full support for existing HTML-only clients.** We **MUST** not break
  existing clients that are accessing the API as a strictly :pep:`503` API. The only
  exception to this, is if the repository itself has chosen to no longer support
  the HTML format.
- **Minimal additional HTTP requests.** Using this API **MUST** not drastically
  increase the amount of HTTP requests an installer must do in order to function.
  Ideally it will require 0 additional requests, but if needed it may require one
  or two additional requests (total, not per dependency).
- **Minimal additional unique responses.** Due to the nature of how large
  repositories like PyPI cache responses, this PEP should not introduce a
  significantly or combinatorially large number of additional unique responses
  that the repository may produce.
- **Supports TUF.** This PEP **MUST** be able to function within the bounds of
  what TUF can support (:pep:`458`), and must be able to be secured using it.
- **Require only the standard library, or small external dependencies for clients.**
  Parsing an API response should ideally require nothing but the standard
  library, however it would be acceptable to require a small, pure Python
  dependency.


Specification
=============

To enable response parsing with only the standard library, this PEP specifies that
all responses (besides the files themselves, and the HTML responses from
:pep:`503`) should be serialized using `JSON <https://www.json.org/>`_.

To enable zero configuration discovery and to minimize the amount of additional HTTP
requests, this PEP extends :pep:`503` such that all of the API endpoints (other than the
files themselves) will utilize HTTP content negotiation to allow client and server to
select the correct serialization format to serve, i.e. either HTML or JSON.


Versioning
----------

Versioning will adhere to :pep:`629` format (``Major.Minor``), which has defined the
existing HTML responses to be ``1.0``. Since this PEP does not introduce new features
into the API, rather it describes a different serialization format for the existing
features, this PEP does not change the existing ``1.0`` version, and instead just
describes how to serialize that into JSON.

Similar to :pep:`629`, the major version number **MUST** be incremented if any
changes to the new format would result in no longer being able to expect existing
clients to meaningfully understand the format.

Likewise, the minor version **MUST** be incremented if features are
added or removed from the format, but existing clients would be expected to continue
to meaningfully understand the format.

Changes that would not result in existing clients being unable to meaningfully
understand the format and which do not represent features being added or removed
may occur without changing the version number.

This is intentionally vague, as this PEP believes it is best left up to future PEPs
that make any changes to the API to investigate and decide whether or not that
change should increment the major or minor version.

Future versions of the API may add things that can only be represented in a subset
of the available serializations of that version. All serializations version numbers,
within a major version, **SHOULD** be kept in sync, but the specifics of how a
feature serializes into each format may differ, including whether or not that feature
is present at all.

It is the intent of this PEP that the API should be thought of as URL endpoints that
return data, whose interpretation is defined by the version of that data, and then
serialized into the target serialization format.


.. _json-serialization:

JSON Serialization
------------------

The URL structure from :pep:`503` still applies, as this PEP only adds an additional
serialization format for the already existing API.

The following constraints apply to all JSON serialized responses described in this
PEP:

* All JSON responses will *always* be a JSON object rather than an array or other
  type.

* While JSON doesn't natively support an URL type, any value that represents an
  URL in this API may be either absolute or relative as long 


id: pep-0478.rst
content:
PEP: 478
Title: Python 3.5 Release Schedule
Author: Larry Hastings <larry@hastings.org>
Status: Final
Type: Informational
Topic: Release
Created: 22-Sep-2014
Python-Version: 3.5


Abstract
========

This document describes the development and release schedule for
Python 3.5.  The schedule primarily concerns itself with PEP-sized
items.

.. Small features may be added up to the first beta
   release.  Bugs may be fixed until the final release,
   which is planned for September 2015.


Release Manager and Crew
========================

- 3.5 Release Manager: Larry Hastings
- Windows installers: Steve Dower
- Mac installers: Ned Deily
- Documentation: Georg Brandl


Release Schedule
================

Python 3.5 has now reached its end-of-life and has been retired.
No more releases will be made.

These are all the historical releases of Python 3.5,
including their release dates.

- 3.5.0 alpha 1: February 8, 2015
- 3.5.0 alpha 2: March 9, 2015
- 3.5.0 alpha 3: March 29, 2015
- 3.5.0 alpha 4: April 19, 2015
- 3.5.0 beta 1: May 24, 2015
  (Beta 1 is also "feature freeze"--no new features beyond this point.)
- 3.5.0 beta 2: May 31, 2015
- 3.5.0 beta 3: July 5, 2015
- 3.5.0 beta 4: July 26, 2015
- 3.5.0 release candidate 1: August 10, 2015
- 3.5.0 release candidate 2: August 25, 2015
- 3.5.0 release candidate 3: September 7, 2015
- 3.5.0 final: September 13, 2015
- 3.5.1 release candidate 1: November 22, 2015
- 3.5.1 final: December 6, 2015
- 3.5.2 release candidate 1: Sunday, June 12, 2016
- 3.5.2 final: Sunday, June 26, 2016
- 3.5.3 candidate 1: January 2, 2017
- 3.5.3 final: January 17, 2017
- 3.5.4 candidate 1: July 25, 2017
- 3.5.4 final: August 8, 2017
- 3.5.5 candidate 1: January 23, 2018
- 3.5.5 final: February 4, 2018
- 3.5.6 candidate 1: July 19, 2018
- 3.5.6 final: August 2, 2018
- 3.5.7 candidate 1: March 4, 2019
- 3.5.7 final: March 18, 2019
- 3.5.8 candidate 1: September 9, 2019
- 3.5.8 candidate 2: October 12, 2019
- 3.5.8 final: October 29, 2019
- 3.5.9 final: November 1, 2019
- 3.5.10 rc1: August 21, 2020
- 3.5.10 final: September 5, 2020




Features for 3.5
================

* :pep:`441`, improved Python zip application support
* :pep:`448`, additional unpacking generalizations
* :pep:`461`, "%-formatting" for bytes and bytearray objects
* :pep:`465`, a new operator ("@") for matrix multiplication
* :pep:`471`, os.scandir(), a fast new directory traversal function
* :pep:`475`, adding support for automatic retries of interrupted system calls
* :pep:`479`, change StopIteration handling inside generators
* :pep:`484`, the typing module, a new standard for type annotations
* :pep:`485`, math.isclose(), a function for testing approximate equality
* :pep:`486`, making the Windows Python launcher aware of virtual environments
* :pep:`488`, eliminating .pyo files
* :pep:`489`, a new and improved mechanism for loading extension modules
* :pep:`492`, coroutines with async and await syntax


Copyright
=========

This document has been placed in the public domain.


id: pep-0322.rst
content:
PEP: 322
Title: Reverse Iteration
Author: Raymond Hettinger <python@rcn.com>
Status: Final
Type: Standards Track
Created: 24-Sep-2003
Python-Version: 2.4
Post-History: 24-Sep-2003


Abstract
========

This proposal is to add a builtin function to support reverse
iteration over sequences.


Motivation
==========

For indexable objects, current approaches for reverse iteration are
error prone, unnatural, and not especially readable::

    for i in xrange(n-1, -1, -1):
        print seqn[i]

One other current approach involves reversing a list before iterating
over it.  That technique wastes computer cycles, memory, and lines of
code::

    rseqn = list(seqn)
    rseqn.reverse()
    for value in rseqn:
        print value

Extended slicing is a third approach that minimizes the code overhead
but does nothing for memory efficiency, beauty, or clarity.

Reverse iteration is much less common than forward iteration, but it
does arise regularly in practice.  See `Real World Use Cases`_ below.


Proposal
========

Add a builtin function called *reversed()* that makes a reverse
iterator over sequence objects that support __getitem__() and
__len__().

The above examples then simplify to::

    for i in reversed(xrange(n)):
        print seqn[i]

::

    for elem in reversed(seqn):
        print elem

The core idea is that the clearest, least error-prone way of specifying
reverse iteration is to specify it in a forward direction and then say
*reversed*.

The implementation could be as simple as::

    def reversed(x):
        if hasattr(x, 'keys'):
            raise ValueError("mappings do not support reverse iteration")
        i = len(x)
        while i > 0:
            i -= 1
            yield x[i]

No language syntax changes are needed.  The proposal is fully backwards
compatible.

A C implementation and unit tests are at:  https://bugs.python.org/issue834422

BDFL Pronouncement
==================

This PEP has been conditionally accepted for Py2.4.  The condition means
that if the function is found to be useless, it can be removed before
Py2.4b1.


Alternative Method Names
========================

* *reviter*  -- Jeremy Fincher's suggestion matches use of iter()
* *ireverse* -- uses the itertools naming convention
* *inreverse* -- no one seems to like this one except me

The name *reverse* is not a candidate because it duplicates the name
of the list.reverse() which mutates the underlying list.


Discussion
==========

The case against adoption of the PEP is a desire to keep the number of
builtin functions small.  This needs to weighed against the simplicity
and convenience of having it as builtin instead of being tucked away in
some other namespace.


Real World Use Cases
====================

Here are some instances of reverse iteration taken from the standard
library and comments on why reverse iteration was necessary:

* atexit.exit_handlers() uses::

    while _exithandlers:
        func, targs, kargs = _exithandlers.pop()
            . . .

  In this application popping is required, so the new function would
  not help.

* heapq.heapify() uses ``for i in xrange(n//2 - 1, -1, -1)`` because
  higher-level orderings are more easily formed from pairs of
  lower-level orderings.  A forward version of this algorithm is
  possible; however, that would complicate the rest of the heap code
  which iterates over the underlying list in the opposite direction.
  The replacement code ``for i in reversed(xrange(n//2))`` makes
  clear the range covered and how many iterations it takes.

* mhlib.test() uses::

    testfolders.reverse();
    for t in testfolders:
        do('mh.deletefolder(%s)' % `t`)

  The need for reverse iteration arises because the tail of the
  underlying list is altered during iteration.

* platform._dist_try_harder() uses
  ``for n in range(len(verfiles)-1,-1,-1)`` because the loop deletes
  selected elements from *verfiles* but needs to leave the rest of
  the list intact for further iteration.

* random.shuffle() uses ``for i in xrange(len(x)-1, 0, -1)`` because
  the algorithm is most easily understood as randomly selecting
  elements from an ever diminishing pool.  In fact, the algorithm can
  be run in a forward direction but is less intuitive and rarely
  presented that way in literature.  The replacement code
  ``for i in reversed(xrange(1, len(x)))`` is much easier
  to verify visually.

* rfc822.Message.__delitem__() uses::

    list.reverse()
    for i in list:
        del self.headers[i]

  The need for reverse iteration arises because the tail of the
  underlying list is altered during iteration.


Rejected Alternatives
=====================

Several variants were submitted that attempted to apply *reversed()*
to all iterables by running the iterable to completion, saving the
results, and then returning a reverse iterator over the results.
While satisfying some notions of full generality, running the input
to the end is contrary to the purpose of using iterators
in the first place.  Also, a small disaster ensues if the underlying
iterator is infinite.

Putting the function in another module or attaching it to a type object
is not being considered.  Like its cousins, *zip()* and *enumerate()*,
the function needs to be directly accessible in daily programming.  Each
solves a basic looping problem:  lock-step iteration, loop counting, and
reverse iteration.  Requiring some form of dotted access would interfere
with their simplicity, daily utility, and accessibility.  They are core
looping constructs, independent of any one application domain.


Copyright
=========

This document has been placed in the public domain.


id: pep-0444.rst
content:
PEP: 444
Title: Python Web3 Interface
Author: Chris McDonough <chrism@plope.com>,
        Armin Ronacher <armin.ronacher@active-4.com>
Discussions-To: web-sig@python.org
Status: Deferred
Type: Informational
Created: 19-Jul-2010


Abstract
========

This document specifies a proposed second-generation standard
interface between web servers and Python web applications or
frameworks.

PEP Deferral
============

Further exploration of the concepts covered in this PEP has been deferred
for lack of a current champion interested in promoting the goals of the PEP
and collecting and incorporating feedback, and with sufficient available
time to do so effectively.

Note that since this PEP was first created, :pep:`3333` was created as a more
incremental update that permitted use of WSGI on Python 3.2+. However, an
alternative specification that furthers the Python 3 goals of a cleaner
separation of binary and text data may still be valuable.

Rationale and Goals
===================

This protocol and specification is influenced heavily by the Web
Services Gateway Interface (WSGI) 1.0 standard described in :pep:`333`.
The high-level rationale for having any standard that allows
Python-based web servers and applications to interoperate is outlined
in :pep:`333`.  This document essentially uses :pep:`333` as a template, and
changes its wording in various places for the purpose of forming a
different standard.

Python currently boasts a wide variety of web application frameworks
which use the WSGI 1.0 protocol.  However, due to changes in the
language, the WSGI 1.0 protocol is not compatible with Python 3.  This
specification describes a standardized WSGI-like protocol that lets
Python 2.6, 2.7 and 3.1+ applications communicate with web servers.
Web3 is clearly a WSGI derivative; it only uses a different name than
"WSGI" in order to indicate that it is not in any way backwards
compatible.

Applications and servers which are written to this specification are
meant to work properly under Python 2.6.X, Python 2.7.X and Python
3.1+.  Neither an application nor a server that implements the Web3
specification can be easily written which will work under Python 2
versions earlier than 2.6 nor Python 3 versions earlier than 3.1.

.. note::

   Whatever Python 3 version fixed http://bugs.python.org/issue4006 so
   ``os.environ['foo']`` returns surrogates (ala :pep:`383`) when the
   value of 'foo' cannot be decoded using the current locale instead
   of failing with a KeyError is the *true* minimum Python 3 version.
   In particular, however, Python 3.0 is not supported.

.. note::

   Python 2.6 is the first Python version that supported an alias for
   ``bytes`` and the ``b"foo"`` literal syntax.  This is why it is the
   minimum version supported by Web3.

Explicability and documentability are the main technical drivers for
the decisions made within the standard.


Differences from WSGI
=====================

- All protocol-specific environment names are prefixed with ``web3.``
  rather than ``wsgi.``, eg. ``web3.input`` rather than
  ``wsgi.input``.

- All values present as environment dictionary *values* are explicitly
  *bytes* instances instead of native strings.  (Environment *keys*
  however are native strings, always ``str`` regardless of
  platform).

- All values returned by an application must be bytes instances,
  including status code, header names and values, and the body.

- Wherever WSGI 1.0 referred to an ``app_iter``, this specification
  refers to a ``body``.

- No ``start_response()`` callback (and therefore no ``write()``
  callable nor ``exc_info`` data).

- The ``readline()`` function of ``web3.input`` must support a size
  hint parameter.

- The ``read()`` function of ``web3.input`` must be length delimited.
  A call without a size argument must not read more than the content
  length header specifies.  In case a content length header is absent
  the stream must not return anything on read.  It must never request
  more data than specified from the client.

- No requirement for middleware to yield an empty string if it needs
  more information from an application to produce output (e.g. no
  "Middleware Handling of Block Boundaries").

- Filelike objects passed to a "file_wrapper" must have an
  ``__iter__`` which returns bytes (never text).

- ``wsgi.file_wrapper`` is not supported.

- ``QUERY_STRING``, ``SCRIPT_NAME``, ``PATH_INFO`` values required to
  be placed in environ by server (each as the empty bytes instance if
  no associated value is received in the HTTP request).

- ``web3.path_info`` and ``web3.script_name`` should be put into the
  Web3 environment, if possible, by the origin Web3 server.  When
  available, each is the original, plain 7-bit ASCII, URL-encoded
  variant of its CGI equivalent derived directly from the request URI
  (with %2F segment markers and other meta-characters intact).  If the
  server cannot provide one (or both) of these values, it must omit
  the value(s) it cannot provide from the environment.

- This requirement was removed: "middleware components **must not**
  block iteration waiting for multiple values from an application
  iterable.  If the middleware needs to accumulate more data from the
  application before it can produce any output, it **must** yield an
  empty string."

- ``SERVER_PORT`` must be a bytes instance (not an integer).

- The server must not inject an additional ``Content-Length`` header
  by guessing the length from the response iterable.  This must be set
  by the application itself in all situations.

- If the origin server advertises that it has the ``web3.async``
  capability, a Web3 application callable used by the server is
  permitted to return a callable that accepts no arguments.  When it
  does so, this callable is to be called periodically by the origin
  server until it returns a non-``None`` response, which must be a
  normal Web3 response tuple.

  .. XXX (chrism) Needs a section of its own for explanation.


Specification Overview
======================

The Web3 interface has two sides: the "server" or "gateway" side, and
the "application" or "framework" side.  The server side invokes a
callable object that is provided by the application side.  The
specifics of how that object is provided are up to the server or
gateway.  It is assumed that some servers or gateways will require an
application's deployer to write a short script to create an instance
of the server or gateway, and supply it with the application object.
Other servers and gateways may use configuration files or other
mechanisms to specify where an application object should be imported
from, or otherwise obtained.

In addition to "pure" servers/gateways and applications/frameworks, it
is also possible to create "middleware" components that implement both
sides of this specification.  Such components act as an application to
their containing server, and as a server to a contained application,
and can be used to provide extended APIs, content transformation,
navigation, and other useful functions.

Throughout this specification, we will use the term "application
callable" to mean "a function, a method, or an instance with a
``__call__`` method".  It is up to the server, gateway, or application
implementing the application callable to choose the appropriate
implementation technique for their needs.  Conversely, a server,
gateway, or application that is invoking a callable **must not** have
any dependency on what kind of callable was provided to it.
Application callables are only to be called, not introspected upon.


The Application/Framework Side
------------------------------

The application object is simply a callable object that accepts one
argument.  The term "object" should not be misconstrued as requiring
an actual object instance: a function, method, or instance with a
``__call__`` method are all acceptable for use as an application
object.  Application objects must be able to be invoked more than
once,


id: pep-0450.rst
content:
PEP: 450
Title: Adding A Statistics Module To The Standard Library
Author: Steven D'Aprano <steve@pearwood.info>
Status: Final
Type: Standards Track
Created: 01-Aug-2013
Python-Version: 3.4
Post-History: 13-Sep-2013


Abstract
========

This PEP proposes the addition of a module for common statistics functions such
as mean, median, variance and standard deviation to the Python standard
library. See also http://bugs.python.org/issue18606


Rationale
=========

The proposed statistics module is motivated by the "batteries included"
philosophy towards the Python standard library.  Raymond Hettinger and other
senior developers have requested a quality statistics library that falls
somewhere in between high-end statistics libraries and ad hoc code. [1]_
Statistical functions such as mean, standard deviation and others are obvious
and useful batteries, familiar to any Secondary School student. Even cheap
scientific calculators typically include multiple statistical functions such
as:

- mean
- population and sample variance
- population and sample standard deviation
- linear regression
- correlation coefficient

Graphing calculators aimed at Secondary School students typically include all
of the above, plus some or all of:

- median
- mode
- functions for calculating the probability of random variables from the
  normal, t, chi-squared, and F distributions
- inference on the mean

and others [2]_.  Likewise spreadsheet applications such as Microsoft Excel,
LibreOffice and Gnumeric include rich collections of statistical
functions [3]_.

In contrast, Python currently has no standard way to calculate even the
simplest and most obvious statistical functions such as mean.  For those who
need statistical functions in Python, there are two obvious solutions:

- install numpy and/or scipy [4]_;

- or use a Do It Yourself solution.

Numpy is perhaps the most full-featured solution, but it has a few
disadvantages:

- It may be overkill for many purposes.  The documentation for numpy even warns

     "It can be hard to know what functions are available in numpy.  This is
     not a complete list, but it does cover most of them."[5]_

  and then goes on to list over 270 functions, only a small number of which are
  related to statistics.

- Numpy is aimed at those doing heavy numerical work, and may be intimidating
  to those who don't have a background in computational mathematics and
  computer science.  For example, ``numpy.mean`` takes four arguments::

      mean(a, axis=None, dtype=None, out=None)

  although fortunately for the beginner or casual numpy user, three are
  optional and ``numpy.mean`` does the right thing in simple cases::

      >>>  numpy.mean([1, 2, 3, 4])
      2.5

- For many people, installing numpy may be difficult or impossible.  For
  example, people in corporate environments may have to go through a difficult,
  time-consuming process before being permitted to install third-party
  software.  For the casual Python user, having to learn about installing
  third-party packages in order to average a list of numbers is unfortunate.

This leads to option number 2, DIY statistics functions.  At first glance, this
appears to be an attractive option, due to the apparent simplicity of common
statistical functions.  For example::

    def mean(data):
        return sum(data)/len(data)

    def variance(data):
        # Use the Computational Formula for Variance.
        n = len(data)
        ss = sum(x**2 for x in data) - (sum(data)**2)/n
        return ss/(n-1)

    def standard_deviation(data):
        return math.sqrt(variance(data))

The above appears to be correct with a casual test::

    >>> data = [1, 2, 4, 5, 8]
    >>> variance(data)
    7.5

But adding a constant to every data point should not change the variance::

    >>> data = [x+1e12 for x in data]
    >>> variance(data)
    0.0

And variance should *never* be negative::

    >>> variance(data*100)
    -1239429440.1282566

By contrast, the proposed reference implementation gets the exactly correct
answer 7.5 for the first two examples, and a reasonably close answer for the
third: 6.012. numpy does no better [6]_.

Even simple statistical calculations contain traps for the unwary, starting
with the Computational Formula itself.  Despite the name, it is numerically
unstable and can be extremely inaccurate, as can be seen above.  It is
completely unsuitable for computation by computer [7]_.  This problem plagues
users of many programming language, not just Python [8]_, as coders reinvent
the same numerically inaccurate code over and over again [9]_, or advise others
to do so [10]_.

It isn't just the variance and standard deviation. Even the mean is not quite
as straightforward as it might appear.  The above implementation seems too
simple to have problems, but it does:

- The built-in ``sum`` can lose accuracy when dealing with floats of wildly
  differing magnitude.  Consequently, the above naive ``mean`` fails this
  "torture test"::

      assert mean([1e30, 1, 3, -1e30]) == 1

  returning 0 instead of 1, a purely computational error of 100%.

- Using ``math.fsum`` inside ``mean`` will make it more accurate with float
  data, but it also has the side-effect of converting any arguments to float
  even when unnecessary.  E.g. we should expect the mean of a list of Fractions
  to be a Fraction, not a float.

While the above mean implementation does not fail quite as catastrophically as
the naive variance does, a standard library function can do much better than
the DIY versions.

The example above involves an especially bad set of data, but even for more
realistic data sets accuracy is important.  The first step in interpreting
variation in data (including dealing with ill-conditioned data) is often to
standardize it to a series with variance 1 (and often mean 0).  This
standardization requires accurate computation of the mean and variance of the
raw series.  Naive computation of mean and variance can lose precision very
quickly.  Because precision bounds accuracy, it is important to use the most
precise algorithms for computing mean and variance that are practical, or the
results of standardization are themselves useless.


Comparison To Other Languages/Packages
======================================

The proposed statistics library is not intended to be a competitor to such
third-party libraries as numpy/scipy, or of proprietary full-featured
statistics packages aimed at professional statisticians such as Minitab, SAS
and Matlab.  It is aimed at the level of graphing and scientific calculators.

Most programming languages have little or no built-in support for statistics
functions.  Some exceptions:

R
-

R (and its proprietary cousin, S) is a programming language designed for
statistics work. It is extremely popular with statisticians and is extremely
feature-rich [11]_.

C#
--

The C# LINQ package includes extension methods to calculate the average of
enumerables [12]_.

Ruby
----

Ruby does not ship with a standard statistics module, despite some apparent
demand [13]_.  Statsample appears to be a feature-rich third-party library,
aiming to compete with R [14]_.

PHP
---

PHP has an extremely feature-rich (although mostly undocumented) set of
advanced statistical functions [15]_.

Delphi
------

Delphi includes standard statistical functions including Mean, Sum,
Variance, TotalVariance, MomentSkewKurtosis in its Math library [16]_.

GNU Scientific Library
----------------------

The GNU Scientific Library includes standard statistical functions,
percentiles, median and others [17]_.  One innovation I have borrowed from the
GSL is to allow the caller to optionally specify the pre-calculated mean of
the sample (or an a priori known population mean) when calculating the variance
and standard deviation [18]_.


Design Decisions Of The Module
==============================

My intention is to start small and grow the library as needed, rather than try
to include everything from the 


id: pep-0336.rst
content:
PEP: 336
Title: Make None Callable
Author: Andrew McClelland <eternalsquire@comcast.net>
Status: Rejected
Type: Standards Track
Created: 28-Oct-2004
Post-History:


Abstract
========

``None`` should be a callable object that when called with any
arguments has no side effect and returns ``None``.


BDFL Pronouncement
==================

This PEP is rejected.  It is considered a feature that ``None`` raises
an error when called.  The proposal falls short in tests for
obviousness, clarity, explicitness, and necessity.  The provided Switch
example is nice but easily handled by a simple lambda definition.
See python-dev discussion on 17 June 2005 [1]_.


Motivation
==========

To allow a programming style for selectable actions that is more
in accordance with the minimalistic functional programming goals
of the Python language.


Rationale
=========

Allow the use of ``None`` in method tables as a universal no effect
rather than either (1) checking a method table entry against ``None``
before calling, or (2) writing a local no effect method with
arguments similar to other functions in the table.

The semantics would be effectively::

    class None:

        def __call__(self, *args):
            pass


How To Use
==========

Before, checking function table entry against ``None``::

    class Select:

        def a(self, input):
            print 'a'

        def b(self, input):
            print 'b'

        def c(self, input):
            print 'c'

        def __call__(self, input):
         function = { 1 : self.a,
               2 : self.b,
               3 : self.c
            }.get(input, None)
         if function:  return function(input)

Before, using a local no effect method::

    class Select:

        def a(self, input):
            print 'a'

        def b(self, input):
            print 'b'

        def c(self, input):
            print 'c'

        def nop(self, input):
         pass

        def __call__(self, input):
            return { 1 : self.a,
            2 : self.b,
            3 : self.c
            }.get(input, self.nop)(input)

After::

    class Select:

        def a(self, input):
            print 'a'

        def b(self, input):
            print 'b'

        def c(self, input):
            print 'c'

        def __call__(self, input):
         return { 1 : self.a,
            2 : self.b,
            3 : self.c
            }.get(input, None)(input)


References
==========

.. [1] Raymond Hettinger, Propose to reject PEP 336 -- Make None Callable
       https://mail.python.org/pipermail/python-dev/2005-June/054280.html


Copyright
=========

This document has been placed in the public domain.


id: pep-0487.rst
content:
PEP: 487
Title: Simpler customisation of class creation
Author: Martin Teichmann <lkb.teichmann@gmail.com>,
Status: Final
Type: Standards Track
Created: 27-Feb-2015
Python-Version: 3.6
Post-History: 27-Feb-2015, 05-Feb-2016, 24-Jun-2016, 02-Jul-2016, 13-Jul-2016
Replaces: 422
Resolution: https://mail.python.org/pipermail/python-dev/2016-July/145629.html


Abstract
========

Currently, customising class creation requires the use of a custom metaclass.
This custom metaclass then persists for the entire lifecycle of the class,
creating the potential for spurious metaclass conflicts.

This PEP proposes to instead support a wide range of customisation
scenarios through a new ``__init_subclass__`` hook in the class body,
and a hook to initialize attributes.

The new mechanism should be easier to understand and use than
implementing a custom metaclass, and thus should provide a gentler
introduction to the full power of Python's metaclass machinery.


Background
==========

Metaclasses are a powerful tool to customize class creation. They have,
however, the problem that there is no automatic way to combine metaclasses.
If one wants to use two metaclasses for a class, a new metaclass combining
those two needs to be created, typically manually.

This need often occurs as a surprise to a user: inheriting from two base
classes coming from two different libraries suddenly raises the necessity
to manually create a combined metaclass, where typically one is not
interested in those details about the libraries at all. This becomes
even worse if one library starts to make use of a metaclass which it
has not done before. While the library itself continues to work perfectly,
suddenly every code combining those classes with classes from another library
fails.

Proposal
========

While there are many possible ways to use a metaclass, the vast majority
of use cases falls into just three categories: some initialization code
running after class creation, the initialization of descriptors and
keeping the order in which class attributes were defined.

The first two categories can easily be achieved by having simple hooks
into the class creation:

1. An ``__init_subclass__`` hook that initializes
   all subclasses of a given class.
2. upon class creation, a ``__set_name__`` hook is called on all the
   attribute (descriptors) defined in the class, and

The third category is the topic of another PEP, :pep:`520`.

As an example, the first use case looks as follows::

   >>> class QuestBase:
   ...    # this is implicitly a @classmethod (see below for motivation)
   ...    def __init_subclass__(cls, swallow, **kwargs):
   ...        cls.swallow = swallow
   ...        super().__init_subclass__(**kwargs)

   >>> class Quest(QuestBase, swallow="african"):
   ...    pass

   >>> Quest.swallow
   'african'

The base class ``object`` contains an empty ``__init_subclass__``
method which serves as an endpoint for cooperative multiple inheritance.
Note that this method has no keyword arguments, meaning that all
methods which are more specialized have to process all keyword
arguments.

This general proposal is not a new idea (it was first suggested for
inclusion in the language definition `more than 10 years ago`_, and a
similar mechanism has long been supported by `Zope's ExtensionClass`_),
but the situation has changed sufficiently in recent years that
the idea is worth reconsidering for inclusion.

The second part of the proposal adds an ``__set_name__``
initializer for class attributes, especially if they are descriptors.
Descriptors are defined in the body of a
class, but they do not know anything about that class, they do not
even know the name they are accessed with. They do get to know their
owner once ``__get__`` is called, but still they do not know their
name. This is unfortunate, for example they cannot put their
associated value into their object's ``__dict__`` under their name,
since they do not know that name.  This problem has been solved many
times, and is one of the most important reasons to have a metaclass in
a library. While it would be easy to implement such a mechanism using
the first part of the proposal, it makes sense to have one solution
for this problem for everyone.

To give an example of its usage, imagine a descriptor representing weak
referenced values::

    import weakref

    class WeakAttribute:
        def __get__(self, instance, owner):
            return instance.__dict__[self.name]()

        def __set__(self, instance, value):
            instance.__dict__[self.name] = weakref.ref(value)

        # this is the new initializer:
        def __set_name__(self, owner, name):
            self.name = name

Such a ``WeakAttribute`` may, for example, be used in a tree structure
where one wants to avoid cyclic references via the parent::

    class TreeNode:
        parent = WeakAttribute()

        def __init__(self, parent):
            self.parent = parent

Note that the ``parent`` attribute is used like a normal attribute,
yet the tree contains no cyclic references and can thus be easily
garbage collected when out of use. The ``parent`` attribute magically
becomes ``None`` once the parent ceases existing.

While this example looks very trivial, it should be noted that until
now such an attribute cannot be defined without the use of a metaclass.
And given that such a metaclass can make life very hard, this kind of
attribute does not exist yet.

Initializing descriptors could simply be done in the
``__init_subclass__`` hook. But this would mean that descriptors can
only be used in classes that have the proper hook, the generic version
like in the example would not work generally. One could also call
``__set_name__`` from within the base implementation of
``object.__init_subclass__``. But given that it is a common mistake
to forget to call ``super()``, it would happen too often that suddenly
descriptors are not initialized.


Key Benefits
============


Easier inheritance of definition time behaviour
-----------------------------------------------

Understanding Python's metaclasses requires a deep understanding of
the type system and the class construction process. This is legitimately
seen as challenging, due to the need to keep multiple moving parts (the code,
the metaclass hint, the actual metaclass, the class object, instances of the
class object) clearly distinct in your mind. Even when you know the rules,
it's still easy to make a mistake if you're not being extremely careful.

Understanding the proposed implicit class initialization hook only requires
ordinary method inheritance, which isn't quite as daunting a task. The new
hook provides a more gradual path towards understanding all of the phases
involved in the class definition process.


Reduced chance of metaclass conflicts
-------------------------------------

One of the big issues that makes library authors reluctant to use metaclasses
(even when they would be appropriate) is the risk of metaclass conflicts.
These occur whenever two unrelated metaclasses are used by the desired
parents of a class definition. This risk also makes it very difficult to
*add* a metaclass to a class that has previously been published without one.

By contrast, adding an ``__init_subclass__`` method to an existing type poses
a similar level of risk to adding an ``__init__`` method: technically, there
is a risk of breaking poorly implemented subclasses, but when that occurs,
it is recognised as a bug in the subclass rather than the library author
breaching backwards compatibility guarantees.


New Ways of Using Classes
=========================

Subclass registration
---------------------

Especially when writing a plugin system, one likes to register new
subclasses of a plugin baseclass. This can be done as follows::

   class PluginBase:
       subclasses = []

       def __init_subclass__(cls, **kwargs):
           super().__init_subclass__(**kwargs)
           cls.subclasses.append(cls)

In this example, ``Plu


id: pep-0493.rst
content:
PEP: 493
Title: HTTPS verification migration tools for Python 2.7
Author: Alyssa Coghlan <ncoghlan@gmail.com>,
        Robert Kuska <rkuska@redhat.com>,
        Marc-AndrÃ© Lemburg <mal@lemburg.com>
BDFL-Delegate: Barry Warsaw
Status: Final
Type: Standards Track
Created: 10-May-2015
Python-Version: 2.7.12
Post-History: 06-Jul-2015, 11-Nov-2015, 24-Nov-2015, 24-Feb-2016
Resolution: https://mail.python.org/pipermail/python-dev/2016-March/143450.html


Abstract
========

:pep:`476` updated Python's default handling of HTTPS certificates in client
modules to align with certificate handling in web browsers, by validating
that the certificates received belonged to the server the client was attempting
to contact. The Python 2.7 long term maintenance series was judged to be in
scope for this change, with the new behaviour introduced in the Python 2.7.9
maintenance release.

This has created a non-trivial barrier to adoption for affected Python 2.7
maintenance releases, so this PEP proposes additional Python 2.7 specific
features that allow system administrators and other users to more easily
decouple the decision to verify server certificates in HTTPS client modules
from the decision to update to newer Python 2.7 maintenance releases.


Rationale
=========

:pep:`476` changed Python's default behaviour to align with expectations
established by web browsers in regards to the semantics of HTTPS URLs:
starting with Python 2.7.9 and 3.4.3, HTTPS clients in the standard library
validate server certificates by default.

However, it is also the case that this change *does* cause problems for
infrastructure administrators operating private intranets that rely on
self-signed certificates, or otherwise encounter problems with the new default
certificate verification settings.

To manage these kinds of situations, web browsers provide users with "click
through" warnings that allow the user to add the server's certificate to the
browser's certificate store. Network client tools like ``curl`` and ``wget``
offer options to switch off certificate checking entirely (by way of
``curl --insecure`` and ``wget --no-check-certificate``, respectively).

At a different layer of the technology stack, Linux security modules like
``SELinux`` and ``AppArmor``, while enabled by default by distribution vendors,
offer relatively straightforward mechanisms for turning them off.

At the moment, no such convenient mechanisms exist to disable Python's
default certificate checking for a whole process.

:pep:`476` did attempt to address this question, by covering how to revert to the
old settings process wide by monkeypatching the ``ssl`` module to restore the
old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed
to allow system administrators to disable the feature by default in their
Standard Operating Environment definition has been determined to be
insufficient in at least some cases. The specific case that led to the
initial creation of this PEP is the one where a Linux distributor aims to
provide their users with a
`smoother migration path <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__
than the standard one provided by consuming upstream CPython 2.7 releases
directly, but other potential challenges have also been pointed out with
updating embedded Python runtimes and other user level installations of Python.

Rather than allowing a plethora of mutually incompatible migration techniques
to bloom, this PEP proposes an additional feature to be added to Python 2.7.12
to make it easier to revert a process to the past behaviour of skipping
certificate validation in HTTPS client modules. It also provides additional
recommendations to redistributors backporting these features to versions of
Python prior to Python 2.7.9.

Alternatives
------------

In the absence of clear upstream guidance and recommendations, commercial
redistributors will still make their own design decisions in the interests of
their customers. The main approaches available are:

* Continuing to rebase on new Python 2.7.x releases, while providing no
  additional assistance beyond the mechanisms defined in :pep:`476` in migrating
  from unchecked to checked hostnames in standard library HTTPS clients
* Gating availability of the changes in default handling of HTTPS connections
  on upgrading from Python 2 to Python 3
* For Linux distribution vendors, gating availability of the changes in default
  handling of HTTPS connections on upgrading to a new operating system version
* Implementing one or both of the backport suggestions described in this PEP,
  regardless of the formal status of the PEP


Scope Limitations
=================

These changes are being proposed purely as tools for helping to manage the
transition to the new default certificate handling behaviour in the context
of Python 2.7. They are not being proposed as new features for Python 3, as
it is expected that the vast majority of client applications affected by this
problem without the ability to update the application itself will be Python 2
applications.

It would likely be desirable for a future version of Python 3 to allow the
default certificate handling for secure protocols to be configurable on a
per-protocol basis, but that question is beyond the scope of this PEP.


Requirements for capability detection
=====================================

As the proposals in this PEP aim to facilitate backports to earlier Python
versions, the Python version number cannot be used as a reliable means for
detecting them. Instead, they are designed to allow the presence
or absence of the feature to be determined using the following technique::

    python -c "import ssl; ssl.<_relevant_attribute>"

This will fail with ``AttributeError`` (and hence a non-zero return code) if
the relevant capability is not available.

The feature detection attributes defined by this PEP are:

* ``ssl._https_verify_certificates``: runtime configuration API
* ``ssl._https_verify_envvar``: environment based configuration
* ``ssl._cert_verification_config``: file based configuration (:pep:`476` opt-in)

The marker attributes are prefixed with an underscore to indicate the
implementation dependent and security sensitive nature of these capabilities.


Feature: Configuration API
==========================

This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x
releases. It consists of a new ``ssl._https_verify_certificates()`` to specify
the default handling of HTTPS certificates in standard library client libraries.

It is not proposed to forward port this change to Python 3, so Python 3
applications that need to support skipping certificate verification will still
need to define their own suitable security context.

Feature detection
-----------------

The marker attribute on the ``ssl`` module related to this feature is the
``ssl._https_verify_certificates`` function itself.

Specification
-------------

The ``ssl._https_verify_certificates`` function will work as follows::

    def _https_verify_certificates(enable=True):
        """Verify server HTTPS certificates by default?"""
        global _create_default_https_context
        if enable:
            _create_default_https_context = create_default_context
        else:
            _create_default_https_context = _create_unverified_context

If called without arguments, or with ``enable`` set to a true value, then
standard library client modules will subsequently verify HTTPS certificates by default, otherwise they will skip verification.

If called with ``enable`` set to a false value, then standard library client
modules will subsequently skip verifying HTTPS certificates by default.

Security Considerations
-----------------------

The inclusion of this feature will allow security sensitive applications to
include the following forward-compatible snippet in their code::

    if hasattr(ssl, "_https_verify_certificates"):
        ssl._https_verify_certificates()

Some develope


id: pep-0492.rst
content:
PEP: 492
Title: Coroutines with async and await syntax
Author: Yury Selivanov <yury@edgedb.com>
Discussions-To: python-dev@python.org
Status: Final
Type: Standards Track
Created: 09-Apr-2015
Python-Version: 3.5
Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015, 05-May-2015


Abstract
========

The growth of Internet and general connectivity has triggered the
proportionate need for responsive and scalable code.  This proposal
aims to answer that need by making writing explicitly asynchronous,
concurrent Python code easier and more Pythonic.

It is proposed to make *coroutines* a proper standalone concept in
Python, and introduce new supporting syntax.  The ultimate goal
is to help establish a common, easily approachable, mental
model of asynchronous programming in Python and make it as close to
synchronous programming as possible.

This PEP assumes that the asynchronous tasks are scheduled and
coordinated by an Event Loop similar to that of stdlib module
``asyncio.events.AbstractEventLoop``.  While the PEP is not tied to any
specific Event Loop implementation, it is relevant only to the kind of
coroutine that uses ``yield`` as a signal to the scheduler, indicating
that the coroutine will be waiting until an event (such as IO) is
completed.

We believe that the changes proposed here will help keep Python
relevant and competitive in a quickly growing area of asynchronous
programming, as many other languages have adopted, or are planning to
adopt, similar features: [2]_, [5]_, [6]_, [7]_, [8]_, [10]_.


API Design and Implementation Revisions
=======================================

1. Feedback on the initial beta release of Python 3.5 resulted in a
   redesign of the object model supporting this PEP to more clearly
   separate native coroutines from generators - rather than being a
   new kind of generator, native coroutines are now their own
   completely distinct type (implemented in [17]_).

   This change was implemented based primarily due to problems
   encountered attempting to integrate support for native coroutines
   into the Tornado web server (reported in [18]_).

2. In CPython 3.5.2, the ``__aiter__`` protocol was updated.

   Before 3.5.2, ``__aiter__`` was expected to return an *awaitable*
   resolving to an *asynchronous iterator*.  Starting with 3.5.2,
   ``__aiter__`` should return asynchronous iterators directly.

   If the old protocol is used in 3.5.2, Python will raise a
   ``PendingDeprecationWarning``.

   In CPython 3.6, the old ``__aiter__`` protocol will still be
   supported with a ``DeprecationWarning`` being raised.

   In CPython 3.7, the old ``__aiter__`` protocol will no longer be
   supported: a ``RuntimeError`` will be raised if ``__aiter__``
   returns anything but an asynchronous iterator.

   See [19]_ and [20]_ for more details.


Rationale and Goals
===================

Current Python supports implementing coroutines via generators (PEP
342), further enhanced by the ``yield from`` syntax introduced in PEP
380. This approach has a number of shortcomings:

* It is easy to confuse coroutines with regular generators, since they
  share the same syntax; this is especially true for new developers.

* Whether or not a function is a coroutine is determined by a presence
  of ``yield``  or ``yield from`` statements in its *body*, which can
  lead to unobvious errors when such statements appear in or disappear
  from function body during refactoring.

* Support for asynchronous calls is limited to expressions where
  ``yield`` is allowed syntactically, limiting the usefulness of
  syntactic features, such as ``with`` and ``for`` statements.

This proposal makes coroutines a native Python language feature, and
clearly separates them from generators.  This removes
generator/coroutine ambiguity, and makes it possible to reliably define
coroutines without reliance on a specific library.  This also enables
linters and IDEs to improve static code analysis and refactoring.

Native coroutines and the associated new syntax features make it
possible to define context manager and iteration protocols in
asynchronous terms. As shown later in this proposal, the new ``async
with`` statement lets Python programs perform asynchronous calls when
entering and exiting a runtime context, and the new ``async for``
statement makes it possible to perform asynchronous calls in iterators.


Specification
=============

This proposal introduces new syntax and semantics to enhance coroutine
support in Python.

This specification presumes knowledge of the implementation of
coroutines in Python (:pep:`342` and :pep:`380`).  Motivation for the syntax
changes proposed here comes from the asyncio framework (:pep:`3156`) and
the "Cofunctions" proposal (:pep:`3152`, now rejected in favor of this
specification).

From this point in this document we use the word *native coroutine* to
refer to functions declared using the new syntax.  *generator-based
coroutine* is used where necessary to refer to coroutines that are
based on generator syntax.  *coroutine* is used in contexts where both
definitions are applicable.


New Coroutine Declaration Syntax
--------------------------------

The following new syntax is used to declare a *native coroutine*::

    async def read_data(db):
        pass

Key properties of *coroutines*:

* ``async def`` functions are always coroutines, even if they do not
  contain ``await`` expressions.

* It is a ``SyntaxError`` to have ``yield`` or ``yield from``
  expressions in an ``async`` function.

* Internally, two new code object flags were introduced:

  - ``CO_COROUTINE`` is used to mark *native coroutines*
    (defined with new syntax).

  - ``CO_ITERABLE_COROUTINE`` is used to make *generator-based
    coroutines* compatible with *native coroutines* (set by
    `types.coroutine()`_ function).

* Regular generators, when called, return a *generator object*;
  similarly, coroutines return a *coroutine* object.

* ``StopIteration`` exceptions are not propagated out of coroutines,
  and are replaced with a ``RuntimeError``.  For regular generators
  such behavior requires a future import (see :pep:`479`).

* When a *native coroutine* is garbage collected, a ``RuntimeWarning``
  is raised if it was never awaited on (see also
  `Debugging Features`_).

* See also `Coroutine objects`_ section.


types.coroutine()
-----------------

A new function ``coroutine(fn)`` is added to the ``types`` module.  It
allows interoperability between existing *generator-based coroutines*
in asyncio and *native coroutines* introduced by this PEP::

    @types.coroutine
    def process_data(db):
        data = yield from read_data(db)
        ...

The function applies ``CO_ITERABLE_COROUTINE`` flag to
generator-function's code object, making it return a *coroutine* object.

If ``fn`` is not a *generator function*, it is wrapped.  If it returns
a *generator*, it will be wrapped in an *awaitable* proxy object
(see below the definition of awaitable objects).

Note, that the ``CO_COROUTINE`` flag is not applied by
``types.coroutine()`` to make it possible to separate *native
coroutines* defined with new syntax, from *generator-based coroutines*.


Await Expression
----------------

The following new ``await`` expression is used to obtain a result of
coroutine execution::

    async def read_data(db):
        data = await db.fetch('SELECT ...')
        ...

``await``, similarly to ``yield from``, suspends execution of
``read_data`` coroutine until ``db.fetch`` *awaitable* completes and
returns the result data.

It uses the ``yield from`` implementation with an extra step of
validating its argument.  ``await`` only accepts an *awaitable*, which
can be one of:

* A *native coroutine* object returned from a *native coroutine
  function*.

* A *generator-based coroutine* object returned from a function
  decorated with ``types.coroutine()``.

* An object with an ``__await__`` method returning an iterator.

  Any ``yield from`` chain of calls ends 


id: pep-0486.rst
content:
PEP: 486
Title: Make the Python Launcher aware of virtual environments
Author: Paul Moore <p.f.moore@gmail.com>
Status: Final
Type: Standards Track
Created: 12-Feb-2015
Python-Version: 3.5
Post-History: 12-Feb-2015
Resolution: https://mail.python.org/pipermail/python-dev/2015-February/138579.html


Abstract
========

The Windows installers for Python include a launcher that locates the
correct Python interpreter to run (see :pep:`397`).  However, the
launcher is not aware of virtual environments (virtualenv [1]_ or PEP
405 based), and so cannot be used to run commands from the active
virtualenv.

This PEP proposes making the launcher "virtualenv aware".  This means
that when run without specifying an explicit Python interpreter to
use, the launcher will use the currently active virtualenv, if any,
before falling back to the configured default Python.


Rationale
=========

Windows users with multiple copies of Python installed need a means of
selecting which one to use.  The Python launcher provides this
facility by means of a ``py`` command that can be used to run either a
configured "default" Python or a specific interpreter, by means of
command line arguments.  So typical usage would be::

    # Run the Python interactive interpreter
    py

    # Execute an installed module
    py -m pip install pytest
    py -m pytest

When using virtual environments, the ``py`` launcher is unaware that a
virtualenv is active, and will continue to use the system Python.  So
different command invocations are needed to run the same commands in a
virtualenv::

    # Run the Python interactive interpreter
    python

    # Execute an installed module (these could use python -m,
    # which is longer to type but is a little more similar to the
    # launcher approach)
    pip install pytest
    py.test

Having to use different commands is error-prone, and in many cases
the error is difficult to spot immediately. The PEP proposes making
the ``py`` command usable with virtual environments, so that the first
form of command can be used in all cases.


Implementation
==============

Both ``virtualenv`` and the core ``venv`` module set an environment
variable ``VIRTUAL_ENV`` when activating a virtualenv.  This PEP
proposes that the launcher checks for the ``VIRTUAL_ENV`` environment
variable whenever it would run the "default" Python interpreter for
the system (i.e., when no specific version flags such as ``py -2.7``
are used) and if present, run the Python interpreter for the
virtualenv rather than the default system Python.

The "default" Python interpreter referred to above is (as per :pep:`397`)
either the latest version of Python installed on the system, or
a version configured via the ``py.ini`` configuration file.  When the
user specifies an explicit Python version on the command line, this
will always be used (as at present).


Impact on Script Launching
==========================


As well as interactive use, the launcher is used as the Windows file
association for Python scripts.  In that case, a "shebang" (``#!``)
line at the start of the script is used to identify the interpreter to
run.  A fully-qualified path can be used, or a version-specific Python
(``python3`` or ``python2``, or even ``python3.5``), or the generic
``python``, which means to use the default interpreter.

The launcher also looks for the specific shebang line
``#!/usr/bin/env python``. On Unix, the ``env`` program searches for a
command on ``$PATH`` and runs the command so located. Similarly, with
this shebang line, the launcher will look for a copy of ``python.exe``
on the user's current ``%PATH%`` and will run that copy.

As activating a virtualenv means that it is added to ``PATH``, no
special handling is needed to run scripts with the active virtualenv -
they just need to use the ``#!/usr/bin/env python`` shebang line,
exactly as on Unix. (If there is no activated virtualenv, and no
``python.exe`` on ``PATH``, the launcher will look for a default
Python exactly as if the shebang line had said ``#!python``).


Exclusions
==========

The PEP makes no attempt to promote the use of the launcher for
running Python on Windows.  Most existing documentation assumes the
user of ``python`` as the command to run Python, and (for example)
``pip`` to run an installed Python command.  This documentation is not
expected to change, and users who choose to manage their ``PATH``
environment variable can continue to use this form.  The focus of this
PEP is purely on allowing users who prefer to use the launcher when
dealing with their system Python installations, to be able to continue
to do so when using virtual environments.


Reference Implementation
========================

A patch implementing the proposed behaviour is available at
http://bugs.python.org/issue23465


References
==========

.. [1] https://virtualenv.pypa.io/


Copyright
=========

This document has been placed in the public domain.


id: pep-0451.rst
content:
PEP: 451
Title: A ModuleSpec Type for the Import System
Author: Eric Snow <ericsnowcurrently@gmail.com>
BDFL-Delegate: Brett Cannon <brett@python.org>, Alyssa Coghlan <ncoghlan@gmail.com>
Discussions-To: import-sig@python.org
Status: Final
Type: Standards Track
Created: 08-Aug-2013
Python-Version: 3.4
Post-History: 08-Aug-2013, 28-Aug-2013, 18-Sep-2013, 24-Sep-2013, 04-Oct-2013
Resolution: https://mail.python.org/pipermail/python-dev/2013-November/130104.html


Abstract
========

This PEP proposes to add a new class to importlib.machinery called
"ModuleSpec".  It will provide all the import-related information used
to load a module and will be available without needing to load the
module first.  Finders will directly provide a module's spec instead of
a loader (which they will continue to provide indirectly).  The import
machinery will be adjusted to take advantage of module specs, including
using them to load modules.


Terms and Concepts
==================

The changes in this proposal are an opportunity to make several
existing terms and concepts more clear, whereas currently they are
(unfortunately) ambiguous.  New concepts are also introduced in this
proposal.  Finally, it's worth explaining a few other existing terms
with which people may not be so familiar.  For the sake of context, here
is a brief summary of all three groups of terms and concepts.  A more
detailed explanation of the import system is found at
[#import_system_docs]_.

name
----

In this proposal, a module's "name" refers to its fully-qualified name,
meaning the fully-qualified name of the module's parent (if any) joined
to the simple name of the module by a period.

finder
------

A "finder" is an object that identifies the loader that the import
system should use to load a module.  Currently this is accomplished by
calling the finder's find_module() method, which returns the loader.

Finders are strictly responsible for providing the loader, which they do
through their find_module() method. The import system then uses that
loader to load the module.

loader
------

A "loader" is an object that is used to load a module during import.
Currently this is done by calling the loader's load_module() method.  A
loader may also provide APIs for getting information about the modules
it can load, as well as about data from sources associated with such a
module.

Right now loaders (via load_module()) are responsible for certain
boilerplate, import-related operations.  These are:

1. Perform some (module-related) validation
2. Create the module object
3. Set import-related attributes on the module
4. "Register" the module to sys.modules
5. Exec the module
6. Clean up in the event of failure while loading the module

This all takes place during the import system's call to
Loader.load_module().

origin
------

This is a new term and concept.  The idea of it exists subtly in the
import system already, but this proposal makes the concept explicit.

"origin" in an import context means the system (or resource within a
system) from which a module originates.  For the purposes of this
proposal, "origin" is also a string which identifies such a resource or
system.  "origin" is applicable to all modules.

For example, the origin for built-in and frozen modules is the
interpreter itself.  The import system already identifies this origin as
"built-in" and "frozen", respectively.  This is demonstrated in the
following module repr: "<module 'sys' (built-in)>".

In fact, the module repr is already a relatively reliable, though
implicit, indicator of a module's origin.  Other modules also indicate
their origin through other means, as described in the entry for
"location".

It is up to the loader to decide on how to interpret and use a module's
origin, if at all.

location
--------

This is a new term.  However the concept already exists clearly in the
import system, as associated with the ``__file__`` and ``__path__``
attributes of modules, as well as the name/term "path" elsewhere.

A "location" is a resource or "place", rather than a system at large,
from which a module is loaded.  It qualifies as an "origin".  Examples
of locations include filesystem paths and URLs.  A location is
identified by the name of the resource, but may not necessarily identify
the system to which the resource pertains.  In such cases the loader
would have to identify the system itself.

In contrast to other kinds of module origin, a location cannot be
inferred by the loader just by the module name.  Instead, the loader
must be provided with a string to identify the location, usually by the
finder that generates the loader.  The loader then uses this information
to locate the resource from which it will load the module.  In theory
you could load the module at a given location under various names.

The most common example of locations in the import system are the
files from which source and extension modules are loaded.  For these
modules the location is identified by the string in the ``__file__``
attribute.  Although ``__file__`` isn't particularly accurate for some
modules (e.g. zipped), it is currently the only way that the import
system indicates that a module has a location.

A module that has a location may be called "locatable".

cache
-----

The import system stores compiled modules in the __pycache__ directory
as an optimization.  This module cache that we use today was provided by
:pep:`3147`.  For this proposal, the relevant API for module caching is the
``__cache__`` attribute of modules and the cache_from_source() function
in importlib.util.  Loaders are responsible for putting modules into the
cache (and loading out of the cache).   Currently the cache is only used
for compiled source modules.  However, loaders may take advantage of
the module cache for other kinds of modules.

package
-------

The concept does not change, nor does the term.  However, the
distinction between modules and packages is mostly superficial.
Packages *are* modules.  They simply have a ``__path__`` attribute and
import may add attributes bound to submodules.  The typically perceived
difference is a source of confusion.  This proposal explicitly
de-emphasizes the distinction between packages and modules where it
makes sense to do so.


Motivation
==========

The import system has evolved over the lifetime of Python.  In late 2002
:pep:`302` introduced standardized import hooks via finders and
loaders and sys.meta_path.  The importlib module, introduced
with Python 3.1, now exposes a pure Python implementation of the APIs
described by :pep:`302`, as well as of the full import system.  It is now
much easier to understand and extend the import system.  While a benefit
to the Python community, this greater accessibility also presents a
challenge.

As more developers come to understand and customize the import system,
any weaknesses in the finder and loader APIs will be more impactful.  So
the sooner we can address any such weaknesses the import system, the
better...and there are a couple we hope to take care of with this proposal.

Firstly, any time the import system needs to save information about a
module we end up with more attributes on module objects that are
generally only meaningful to the import system.  It would be nice to
have a per-module namespace in which to put future import-related
information and to pass around within the import system.  Secondly,
there's an API void between finders and loaders that causes undue
complexity when encountered.  The :pep:`420` (namespace packages)
implementation had to work around this.  The complexity surfaced again
during recent efforts on a separate proposal. [#ref_files_pep]_

The `finder`_ and `loader`_ sections above detail current responsibility
of both.  Notably, loaders are not required to provide any of the
functionality of their load_module() method through other methods.  Thus,
though the import-related information about a module is likely available
without loading the module, it is n


id: pep-0337.rst
content:
PEP: 337
Title: Logging Usage in the Standard Library
Author: Michael P. Dubner <dubnerm@mindless.com>
Status: Deferred
Type: Standards Track
Created: 02-Oct-2004
Python-Version: 2.5
Post-History: 10-Nov-2004


Abstract
========

This PEP defines a standard for using the logging system (:pep:`282`) in the
standard library.

Implementing this PEP will simplify development of daemon
applications.  As a downside this PEP requires slight
modifications (however in a back-portable way) to a large number
of standard modules.

After implementing this PEP one can use following filtering
scheme::

    logging.getLogger('py.BaseHTTPServer').setLevel(logging.FATAL)


PEP Deferral
============

Further exploration of the concepts covered in this PEP has been deferred
for lack of a current champion interested in promoting the goals of the
PEP and collecting and incorporating feedback, and with sufficient
available time to do so effectively.


Rationale
=========

There are a couple of situations when output to stdout or stderr
is impractical:

- Daemon applications where the framework doesn't allow the
  redirection of standard output to some file, but assumes use of
  some other form of logging.  Examples are syslog under \*nix'es
  and EventLog under WinNT+.

- GUI applications which want to output every new log entry in
  separate pop-up window (i.e. fading OSD).

Also sometimes applications want to filter output entries based on
their source or severity.  This requirement can't be implemented
using simple redirection.

Finally sometimes output needs to be marked with event timestamps,
which can be accomplished with ease using the logging system.


Proposal
========

Every module usable for daemon and GUI applications should be
rewritten to use the logging system instead of ``print`` or
``sys.stdout.write``.

There should be code like this included in the beginning of every
modified module::

    import logging

    _log = logging.getLogger('py.<module-name>')

A prefix of ``py.`` [2]_ must be used by all modules included in the
standard library distributed along with Python, and only by such
modules (unverifiable).  The use of ``_log`` is intentional as we
don't want to auto-export it.  For modules that use log only in
one class a logger can be created inside the class definition as
follows::

    class XXX:

        __log = logging.getLogger('py.<module-name>')

Then this class can create access methods to log to this private
logger.

So ``print`` and ``sys.std{out|err}.write`` statements should be
replaced with ``_log.{debug|info}``, and ``traceback.print_exception``
with ``_log.exception`` or sometimes ``_log.debug('...', exc_info=1)``.


Module List
===========

Here is a (possibly incomplete) list of modules to be reworked:

- asyncore (dispatcher.log, dispatcher.log_info)

- BaseHTTPServer (BaseHTTPRequestHandler.log_request,
  BaseHTTPRequestHandler.log_error,
  BaseHTTPRequestHandler.log_message)

- cgi (possibly - is cgi.log used by somebody?)

- ftplib (if FTP.debugging)

- gopherlib (get_directory)

- httplib (HTTPResponse, HTTPConnection)

- ihooks (_Verbose)

- imaplib (IMAP4._mesg)

- mhlib (MH.error)

- nntplib (NNTP)

- pipes (Template.makepipeline)

- pkgutil (extend_path)

- platform (_syscmd_ver)

- poplib (if POP3._debugging)

- profile (if Profile.verbose)

- robotparser (_debug)

- smtplib (if SGMLParser.verbose)

- shlex (if shlex.debug)

- smtpd (SMTPChannel/PureProxy where print >> DEBUGSTREAM)

- smtplib (if SMTP.debuglevel)

- SocketServer (BaseServer.handle_error)

- telnetlib (if Telnet.debuglevel)

- threading? (_Verbose._note, Thread.__bootstrap)

- timeit (Timer.print_exc)

- trace

- uu (decode)

Additionally there are a couple of modules with commented debug
output or modules where debug output should be added.  For
example:

- urllib

Finally possibly some modules should be extended to provide more
debug information.


Doubtful Modules
================

Listed here are modules that the community will propose for
addition to the module list and modules that the community say
should be removed from the module list.

- tabnanny (check)


Guidelines for Logging Usage
============================

Also we can provide some recommendation to authors of library
modules so they all follow the same format of naming loggers.  I
propose that non-standard library modules should use loggers named
after their full names, so a module "spam" in sub-package "junk"
of package "dummy" will be named "dummy.junk.spam" and, of course,
the ``__init__`` module of the same sub-package will have the logger
name "dummy.junk".


References
==========

.. [2] https://mail.python.org/pipermail/python-dev/2004-October/049282.html


Copyright
=========

This document has been placed in the public domain.


id: pep-0323.rst
content:
PEP: 323
Title: Copyable Iterators
Author: Alex Martelli <aleaxit@gmail.com>
Status: Deferred
Type: Standards Track
Created: 25-Oct-2003
Python-Version: 2.5
Post-History: 29-Oct-2003


Deferral
========

This PEP has been deferred. Copyable iterators are a nice idea, but after
four years, no implementation or widespread interest has emerged.


Abstract
========

This PEP suggests that some iterator types should support shallow
copies of their instances by exposing a ``__copy__`` method which meets
some specific requirements, and indicates how code using an iterator
might exploit such a ``__copy__`` method when present.


Update and Comments
===================

Support for ``__copy__`` was included in Py2.4's ``itertools.tee()``.

Adding ``__copy__`` methods to existing iterators will change the
behavior under ``tee()``.  Currently, the copied iterators remain
tied to the original iterator.  If the original advances, then
so do all of the copies.  Good practice is to overwrite the
original so that anomalies don't result:  ``a,b=tee(a)``.
Code that doesn't follow that practice may observe a semantic
change if a ``__copy__`` method is added to an iterator.

Motivation
==========

In Python up to 2.3, most built-in iterator types don't let the user
copy their instances.  User-coded iterators that do let their clients
call copy.copy on their instances may, or may not, happen to return,
as a result of the copy, a separate iterator object that may be
iterated upon independently from the original.

Currently, "support" for copy.copy in a user-coded iterator type is
almost invariably "accidental" -- i.e., the standard machinery of the
copy method in Python's standard library's copy module does build and
return a copy.  However, the copy will be independently iterable with
respect to the original only if calling ``.next()`` on an instance of that
class happens to change instance state solely by rebinding some
attributes to new values, and not by mutating some attributes'
existing values.

For example, an iterator whose "index" state is held as an integer
attribute will probably give usable copies, since (integers being
immutable) ``.next()`` presumably just rebinds that attribute.  On the
other hand, another iterator whose "index" state is held as a list
attribute will probably mutate the same list object when ``.next()``
executes, and therefore copies of such an iterator will not be
iterable separately and independently from the original.

Given this existing situation, ``copy.copy(it)`` on some iterator object
isn't very useful, nor, therefore, is it at all widely used.  However,
there are many cases in which being able to get a "snapshot" of an
iterator, as a "bookmark", so as to be able to keep iterating along
the sequence but later iterate again on the same sequence from the
bookmark onwards, is useful.  To support such "bookmarking", module
itertools, in 2.4, has grown a 'tee' function, to be used as::

    it, bookmark = itertools.tee(it)

The previous value of 'it' must not be used again, which is why this
typical usage idiom rebinds the name.  After this call, 'it' and
'bookmark' are independently-iterable iterators on the same underlying
sequence as the original value of 'it': this satisfies application
needs for "iterator copying".

However, when itertools.tee can make no hypotheses about the nature of
the iterator it is passed as an argument, it must save in memory all
items through which one of the two 'teed' iterators, but not yet both,
have stepped.  This can be quite costly in terms of memory, if the two
iterators get very far from each other in their stepping; indeed, in
some cases it may be preferable to make a list from the iterator so as
to be able to step repeatedly through the subsequence, or, if that is
too costy in terms of memory, save items to disk, again in order to be
able to iterate through them repeatedly.

This PEP proposes another idea that will, in some important cases,
allow ``itertools.tee`` to do its job with minimal cost in terms of
memory; user code may also occasionally be able to exploit the idea in
order to decide whether to copy an iterator, make a list from it, or
use an auxiliary disk file.

The key consideration is that some important iterators, such as those
which built-in function iter builds over sequences, would be
intrinsically easy to copy: just get another reference to the same
sequence, and a copy of the integer index.  However, in Python 2.3,
those iterators don't expose the state, and don't support ``copy.copy``.

The purpose of this PEP, therefore, is to have those iterator types
expose a suitable ``__copy__`` method.  Similarly, user-coded iterator
types that can provide copies of their instances, suitable for
separate and independent iteration, with limited costs in time and
space, should also expose a suitable ``__copy__`` method.  While
copy.copy also supports other ways to let a type control the way
its instances are copied, it is suggested, for simplicity, that
iterator types that support copying always do so by exposing a
``__copy__`` method, and not in the other ways ``copy.copy`` supports.

Having iterators expose a suitable ``__copy__`` when feasible will afford
easy optimization of itertools.tee and similar user code, as in::

    def tee(it):
        it = iter(it)
        try: copier = it.__copy__
        except AttributeError:
            # non-copyable iterator, do all the needed hard work
            # [snipped!]
        else:
            return it, copier()

Note that this function does NOT call "copy.copy(it)", which (even
after this PEP is implemented) might well still "just happen to
succeed". for some iterator type that is implemented as a user-coded
class. without really supplying an adequate "independently iterable"
copy object as its result.


Specification
=============

Any iterator type X may expose a method ``__copy__`` that is callable
without arguments on any instance x of X.  The method should be
exposed if and only if the iterator type can provide copyability with
reasonably little computational and memory effort.  Furthermore, the
new object y returned by method ``__copy__`` should be a new instance
of X that is iterable independently and separately from x, stepping
along the same "underlying sequence" of items.

For example, suppose a class Iter essentially duplicated the
functionality of the iter builtin for iterating on a sequence::

    class Iter(object):

        def __init__(self, sequence):
            self.sequence = sequence
            self.index = 0

        def __iter__(self):
            return self

        def next(self):
            try: result = self.sequence[self.index]
            except IndexError: raise StopIteration
            self.index += 1
            return result

To make this Iter class compliant with this PEP, the following
addition to the body of class Iter would suffice::

        def __copy__(self):
            result = self.__class__(self.sequence)
            result.index = self.index
            return result

Note that ``__copy__``, in this case, does not even try to copy the
sequence; if the sequence is altered while either or both of the
original and copied iterators are still stepping on it, the iteration
behavior is quite likely to go awry anyway -- it is not ``__copy__``'s
responsibility to change this normal Python behavior for iterators
which iterate on mutable sequences (that might, perhaps, be the
specification for a ``__deepcopy__`` method of iterators, which, however,
this PEP does not deal with).

Consider also a "random iterator", which provides a nonterminating
sequence of results from some method of a random instance, called
with given arguments::

    class RandomIterator(object):

        def __init__(self, bound_method, *args):
            self.call = bound_method
            self.args = args

        def __iter__(self):
            return self

        def next(self):
            return self.call(*self.args)

     


id: pep-0445.rst
content:
PEP: 445
Title: Add new APIs to customize Python memory allocators
Author: Victor Stinner <vstinner@python.org>
BDFL-Delegate: Antoine Pitrou <solipsis@pitrou.net>
Status: Final
Type: Standards Track
Created: 15-Jun-2013
Python-Version: 3.4
Resolution: https://mail.python.org/pipermail/python-dev/2013-July/127222.html

Abstract
========

This PEP proposes new Application Programming Interfaces (API) to customize
Python memory allocators.  The only implementation required to conform to
this PEP is CPython, but other implementations may choose to be compatible,
or to re-use a similar scheme.


Rationale
=========

Use cases:

* Applications embedding Python which want to isolate Python memory from
  the memory of the application, or want to use a different memory
  allocator optimized for its Python usage
* Python running on embedded devices with low memory and slow CPU.
  A custom memory allocator can be used for efficiency and/or to get
  access all the memory of the device.
* Debug tools for memory allocators:

  - track the memory usage (find memory leaks)
  - get the location of a memory allocation: Python filename and line
    number, and the size of a memory block
  - detect buffer underflow, buffer overflow and misuse of Python
    allocator APIs (see `Redesign Debug Checks on Memory Block
    Allocators as Hooks`_)
  - force memory allocations to fail to test handling of the
    ``MemoryError`` exception


Proposal
========

New Functions and Structures
----------------------------

* Add a new GIL-free (no need to hold the GIL) memory allocator:

  - ``void* PyMem_RawMalloc(size_t size)``
  - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)``
  - ``void PyMem_RawFree(void *ptr)``
  - The newly allocated memory will not have been initialized in any
    way.
  - Requesting zero bytes returns a distinct non-*NULL* pointer if
    possible, as if ``PyMem_Malloc(1)`` had been called instead.

* Add a new ``PyMemAllocator`` structure::

    typedef struct {
        /* user context passed as the first argument to the 3 functions */
        void *ctx;

        /* allocate a memory block */
        void* (*malloc) (void *ctx, size_t size);

        /* allocate or resize a memory block */
        void* (*realloc) (void *ctx, void *ptr, size_t new_size);

        /* release a memory block */
        void (*free) (void *ctx, void *ptr);
    } PyMemAllocator;

* Add a new ``PyMemAllocatorDomain`` enum to choose the Python
  allocator domain. Domains:

  - ``PYMEM_DOMAIN_RAW``: ``PyMem_RawMalloc()``, ``PyMem_RawRealloc()``
    and ``PyMem_RawFree()``

  - ``PYMEM_DOMAIN_MEM``: ``PyMem_Malloc()``, ``PyMem_Realloc()`` and
    ``PyMem_Free()``

  - ``PYMEM_DOMAIN_OBJ``: ``PyObject_Malloc()``, ``PyObject_Realloc()``
    and ``PyObject_Free()``

* Add new functions to get and set memory block allocators:

  - ``void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)``
  - ``void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)``
  - The new allocator must return a distinct non-*NULL* pointer when
    requesting zero bytes
  - For the ``PYMEM_DOMAIN_RAW`` domain, the allocator must be
    thread-safe: the GIL is not held when the allocator is called.

* Add a new ``PyObjectArenaAllocator`` structure::

    typedef struct {
        /* user context passed as the first argument to the 2 functions */
        void *ctx;

        /* allocate an arena */
        void* (*alloc) (void *ctx, size_t size);

        /* release an arena */
        void (*free) (void *ctx, void *ptr, size_t size);
    } PyObjectArenaAllocator;

* Add new functions to get and set the arena allocator used by
  *pymalloc*:

  - ``void PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)``
  - ``void PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)``

* Add a new function to reinstall the debug checks on memory allocators when
  a memory allocator is replaced with ``PyMem_SetAllocator()``:

  - ``void PyMem_SetupDebugHooks(void)``
  - Install the debug hooks on all memory block allocators. The function can be
    called more than once, hooks are only installed once.
  - The function does nothing is Python is not compiled in debug mode.

* Memory block allocators always return *NULL* if *size* is greater than
  ``PY_SSIZE_T_MAX``. The check is done before calling the inner
  function.

.. note::
    The *pymalloc* allocator is optimized for objects smaller than 512 bytes
    with a short lifetime. It uses memory mappings with a fixed size of 256
    KB called "arenas".

Here is how the allocators are set up by default:

* ``PYMEM_DOMAIN_RAW``, ``PYMEM_DOMAIN_MEM``: ``malloc()``,
  ``realloc()`` and ``free()``; call ``malloc(1)`` when requesting zero
  bytes
* ``PYMEM_DOMAIN_OBJ``: *pymalloc* allocator which falls back on
  ``PyMem_Malloc()`` for allocations larger than 512 bytes
* *pymalloc* arena allocator: ``VirtualAlloc()`` and ``VirtualFree()`` on
  Windows, ``mmap()`` and ``munmap()`` when available, or ``malloc()``
  and ``free()``


Redesign Debug Checks on Memory Block Allocators as Hooks
---------------------------------------------------------

Since Python 2.3, Python implements different checks on memory
allocators in debug mode:

* Newly allocated memory is filled with the byte ``0xCB``, freed memory
  is filled with the byte ``0xDB``.
* Detect API violations, ex: ``PyObject_Free()`` called on a memory
  block allocated by ``PyMem_Malloc()``
* Detect write before the start of the buffer (buffer underflow)
* Detect write after the end of the buffer (buffer overflow)

In Python 3.3, the checks are installed by replacing ``PyMem_Malloc()``,
``PyMem_Realloc()``, ``PyMem_Free()``, ``PyObject_Malloc()``,
``PyObject_Realloc()`` and ``PyObject_Free()`` using macros. The new
allocator allocates a larger buffer and writes a pattern to detect buffer
underflow, buffer overflow and use after free (by filling the buffer with
the byte ``0xDB``). It uses the original ``PyObject_Malloc()``
function to allocate memory. So ``PyMem_Malloc()`` and
``PyMem_Realloc()`` indirectly call ``PyObject_Malloc()`` and
``PyObject_Realloc()``.

This PEP redesigns the debug checks as hooks on the existing allocators
in debug mode. Examples of call traces without the hooks:

* ``PyMem_RawMalloc()`` => ``_PyMem_RawMalloc()`` => ``malloc()``
* ``PyMem_Realloc()`` => ``_PyMem_RawRealloc()`` => ``realloc()``
* ``PyObject_Free()`` => ``_PyObject_Free()``

Call traces when the hooks are installed (debug mode):

* ``PyMem_RawMalloc()`` => ``_PyMem_DebugMalloc()``
  => ``_PyMem_RawMalloc()`` => ``malloc()``
* ``PyMem_Realloc()`` => ``_PyMem_DebugRealloc()``
  => ``_PyMem_RawRealloc()`` => ``realloc()``
* ``PyObject_Free()`` => ``_PyMem_DebugFree()``
  => ``_PyObject_Free()``

As a result, ``PyMem_Malloc()`` and ``PyMem_Realloc()`` now call
``malloc()`` and ``realloc()`` in both release mode and debug mode,
instead of calling ``PyObject_Malloc()`` and ``PyObject_Realloc()`` in
debug mode.

When at least one memory allocator is replaced with
``PyMem_SetAllocator()``, the ``PyMem_SetupDebugHooks()`` function must
be called to reinstall the debug hooks on top on the new allocator.


Don't call malloc() directly anymore
------------------------------------

``PyObject_Malloc()`` falls back on ``PyMem_Malloc()`` instead of
``malloc()`` if size is greater or equal than 512 bytes, and
``PyObject_Realloc()`` falls back on ``PyMem_Realloc()`` instead of
``realloc()``

Direct calls to ``malloc()`` are replaced with ``PyMem_Malloc()``, or
``PyMem_RawMalloc()`` if the GIL is not held.

External libraries like zlib or OpenSSL can be configured to allocate memory
using ``PyMem_Malloc()`` or ``PyMem_RawMalloc()``. If the allocator of a
library can only be replaced globally (rather than on an object-by-object
basis), it shouldn't be replaced when Python is embedded in an application.

For the "track memory usage" use case, it is important to track memory
allocated in 


id: pep-0479.rst
content:
PEP: 479
Title: Change StopIteration handling inside generators
Author: Chris Angelico <rosuav@gmail.com>, Guido van Rossum <guido@python.org>
Status: Final
Type: Standards Track
Created: 15-Nov-2014
Python-Version: 3.5
Post-History: 15-Nov-2014, 19-Nov-2014, 05-Dec-2014


Abstract
========

This PEP proposes a change to generators: when ``StopIteration`` is
raised inside a generator, it is replaced with ``RuntimeError``.
(More precisely, this happens when the exception is about to bubble
out of the generator's stack frame.)  Because the change is backwards
incompatible, the feature is initially introduced using a
``__future__`` statement.


Acceptance
==========

This PEP was accepted by the BDFL on November 22.  Because of the
exceptionally short period from first draft to acceptance, the main
objections brought up after acceptance were carefully considered and
have been reflected in the "Alternate proposals" section below.
However, none of the discussion changed the BDFL's mind and the PEP's
acceptance is now final.  (Suggestions for clarifying edits are still
welcome -- unlike IETF RFCs, the text of a PEP is not cast in stone
after its acceptance, although the core design/plan/specification
should not change after acceptance.)


Rationale
=========

The interaction of generators and ``StopIteration`` is currently
somewhat surprising, and can conceal obscure bugs.  An unexpected
exception should not result in subtly altered behaviour, but should
cause a noisy and easily-debugged traceback.  Currently,
``StopIteration`` raised accidentally inside a generator function will
be interpreted as the end of the iteration by the loop construct
driving the generator.

The main goal of the proposal is to ease debugging in the situation
where an unguarded ``next()`` call (perhaps several stack frames deep)
raises ``StopIteration`` and causes the iteration controlled by the
generator to terminate silently.  (Whereas, when some other exception
is raised, a traceback is printed pinpointing the cause of the
problem.)

This is particularly pernicious in combination with the ``yield from``
construct of :pep:`380`, as it breaks the abstraction that a
subgenerator may be factored out of a generator.  That PEP notes this
limitation, but notes that "use cases for these [are] rare to
non-existent".  Unfortunately while intentional use is rare, it is
easy to stumble on these cases by accident::

    import contextlib

    @contextlib.contextmanager
    def transaction():
        print('begin')
        try:
            yield from do_it()
        except:
            print('rollback')
            raise
        else:
            print('commit')

    def do_it():
        print('Refactored initial setup')
        yield # Body of with-statement is executed here
        print('Refactored finalization of successful transaction')

    def gene():
        for i in range(2):
            with transaction():
                yield i
                # return
                raise StopIteration  # This is wrong
            print('Should not be reached')

    for i in gene():
        print('main: i =', i)

Here factoring out ``do_it`` into a subgenerator has introduced a
subtle bug: if the wrapped block raises ``StopIteration``, under the
current behavior this exception will be swallowed by the context
manager; and, worse, the finalization is silently skipped!  Similarly
problematic behavior occurs when an ``asyncio`` coroutine raises
``StopIteration``, causing it to terminate silently, or when ``next``
is used to take the first result from an iterator that unexpectedly
turns out to be empty, for example::

    # using the same context manager as above
    import pathlib

    with transaction():
        print('commit file {}'.format(
            # I can never remember what the README extension is
            next(pathlib.Path('/some/dir').glob('README*'))))

In both cases, the refactoring abstraction of ``yield from`` breaks
in the presence of bugs in client code.

Additionally, the proposal reduces the difference between list
comprehensions and generator expressions, preventing surprises such as
the one that started this discussion [2]_.  Henceforth, the following
statements will produce the same result if either produces a result at
all::

    a = list(F(x) for x in xs if P(x))
    a = [F(x) for x in xs if P(x)]

With the current state of affairs, it is possible to write a function
``F(x)`` or a predicate ``P(x)`` that causes the first form to produce
a (truncated) result, while the second form raises an exception
(namely, ``StopIteration``).  With the proposed change, both forms
will raise an exception at this point (albeit ``RuntimeError`` in the
first case and ``StopIteration`` in the second).

Finally, the proposal also clears up the confusion about how to
terminate a generator: the proper way is ``return``, not
``raise StopIteration``.

As an added bonus, the above changes bring generator functions much
more in line with regular functions.  If you wish to take a piece of
code presented as a generator and turn it into something else, you
can usually do this fairly simply, by replacing every ``yield`` with
a call to ``print()`` or ``list.append()``; however, if there are any
bare ``next()`` calls in the code, you have to be aware of them.  If
the code was originally written without relying on ``StopIteration``
terminating the function, the transformation would be that much
easier.


Background information
======================

When a generator frame is (re)started as a result of a ``__next__()``
(or ``send()`` or ``throw()``) call, one of three outcomes can occur:

* A yield point is reached, and the yielded value is returned.
* The frame is returned from; ``StopIteration`` is raised.
* An exception is raised, which bubbles out.

In the latter two cases the frame is abandoned (and the generator
object's ``gi_frame`` attribute is set to None).


Proposal
========

If a ``StopIteration`` is about to bubble out of a generator frame, it
is replaced with ``RuntimeError``, which causes the ``next()`` call
(which invoked the generator) to fail, passing that exception out.
From then on it's just like any old exception. [3]_

This affects the third outcome listed above, without altering any
other effects.  Furthermore, it only affects this outcome when the
exception raised is ``StopIteration`` (or a subclass thereof).

Note that the proposed replacement happens at the point where the
exception is about to bubble out of the frame, i.e. after any
``except`` or ``finally`` blocks that could affect it have been
exited.  The ``StopIteration`` raised by returning from the frame is
not affected (the point being that ``StopIteration`` means that the
generator terminated "normally", i.e. it did not raise an exception).

A subtle issue is what will happen if the caller, having caught the
``RuntimeError``, calls the generator object's ``__next__()`` method
again.  The answer is that from this point on it will raise
``StopIteration`` -- the behavior is the same as when any other
exception was raised by the generator.

Another logical consequence of the proposal: if someone uses
``g.throw(StopIteration)`` to throw a ``StopIteration`` exception into
a generator, if the generator doesn't catch it (which it could do
using a ``try/except`` around the ``yield``), it will be transformed
into ``RuntimeError``.

During the transition phase, the new feature must be enabled
per-module using::

    from __future__ import generator_stop

Any generator function constructed under the influence of this
directive will have the ``REPLACE_STOPITERATION`` flag set on its code
object, and generators with the flag set will behave according to this
proposal.  Once the feature becomes standard, the flag may be dropped;
code should not inspect generators for it.

A proof-of-concept patch has been created to facilitate testing. [4]_


Consequences for existing code
==============================

This change will affect exist


id: pep-0690.rst
content:
PEP: 690
Title: Lazy Imports
Author: GermÃ¡n MÃ©ndez Bravo <german.mb@gmail.com>, Carl Meyer <carl@oddbird.net>
Sponsor: Barry Warsaw <barry@python.org>
Discussions-To: https://discuss.python.org/t/pep-690-lazy-imports/15474
Status: Rejected
Type: Standards Track
Created: 29-Apr-2022
Python-Version: 3.12
Post-History: `03-May-2022 <https://discuss.python.org/t/pep-690-lazy-imports/15474>`__,
              `03-May-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/IHOSWMIBKCXVB46FI7NGOC2F34RUYZ5Z/>`__
Resolution: https://discuss.python.org/t/pep-690-lazy-imports-again/19661/26


Abstract
========

This PEP proposes a feature to transparently defer the finding and execution of
imported modules until the moment when an imported object is first used.  Since
Python programs commonly import many more modules than a single invocation of
the program is likely to use in practice, lazy imports can greatly reduce the
overall number of modules loaded, improving startup time and memory usage. Lazy
imports also mostly eliminate the risk of import cycles.


Motivation
==========

Common Python code style :pep:`prefers <8#imports>` imports at module
level, so they don't have to be repeated within each scope the imported object
is used in, and to avoid the inefficiency of repeated execution of the import
system at runtime. This means that importing the main module of a program
typically results in an immediate cascade of imports of most or all of the
modules that may ever be needed by the program.

Consider the example of a Python command line program (CLI) with a number of
subcommands. Each subcommand may perform different tasks, requiring the import
of different dependencies. But a given invocation of the program will only
execute a single subcommand, or possibly none (i.e. if just ``--help`` usage
info is requested). Top-level eager imports in such a program will result in the
import of many modules that will never be used at all; the time spent (possibly
compiling and) executing these modules is pure waste.

To improve startup time, some large Python CLIs make imports lazy by manually
placing imports inline into functions to delay imports of expensive subsystems.
This manual approach is labor-intensive and fragile; one misplaced import or
refactor can easily undo painstaking optimization work.

The Python standard library already includes built-in support for lazy imports,
via `importlib.util.LazyLoader
<https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader>`_.
There are also third-party packages such as `demandimport
<https://github.com/bwesterb/py-demandimport/>`_. These provide a "lazy module
object" which delays its own import until first attribute access. This is not
sufficient to make all imports lazy: imports such as ``from foo import a, b``
will still eagerly import the module ``foo`` since they immediately access an
attribute from it. It also imposes noticeable runtime overhead on every module
attribute access, since it requires a Python-level ``__getattr__`` or
``__getattribute__`` implementation.

Authors of scientific Python packages have also made extensive use of lazy
imports to allow users to write e.g. ``import scipy as sp`` and then easily
access many different submodules with e.g. ``sp.linalg``, without requiring all
the many submodules to be imported up-front. `SPEC 1
<https://scientific-python.org/specs/spec-0001/>`_ codifies this practice in the
form of a ``lazy_loader`` library that can be used explicitly in a package
``__init__.py`` to provide lazily accessible submodules.

Users of static typing also have to import names for use in type annotations
that may never be used at runtime (if :pep:`563` or possibly in future
:pep:`649` are used to avoid eager runtime evaluation of annotations). Lazy
imports are very attractive in this scenario to avoid overhead of unneeded
imports.

This PEP proposes a more general and comprehensive solution for lazy imports
that can encompass all of the above use cases and does not impose detectable
overhead in real-world use. The implementation in this PEP has already
`demonstrated
<https://github.com/facebookincubator/cinder/blob/cinder/3.8/CinderDoc/lazy_imports.rst>`_
startup time improvements up to 70% and memory-use reductions up to 40% on
real-world Python CLIs.

Lazy imports also eliminate most import cycles. With eager imports, "false
cycles" can easily occur which are fixed by simply moving an import to the
bottom of a module or inline into a function, or switching from ``from foo
import bar`` to ``import foo``. With lazy imports, these "cycles" just work.
The only cycles which will remain are those where two modules actually each use
a name from the other at module level; these "true" cycles are only fixable by
refactoring the classes or functions involved.


Rationale
=========

The aim of this feature is to make imports transparently lazy. "Lazy" means
that the import of a module (execution of the module body and addition of the
module object to ``sys.modules``) should not occur until the module (or a name
imported from it) is actually referenced during execution. "Transparent" means
that besides the delayed import (and necessarily observable effects of that,
such as delayed import side effects and changes to ``sys.modules``), there is
no other observable change in behavior: the imported object is present in the
module namespace as normal and is transparently loaded whenever first used: its
status as a "lazy imported object" is not directly observable from Python or
from C extension code.

The requirement that the imported object be present in the module namespace as
usual, even before the import has actually occurred, means that we need some
kind of "lazy object" placeholder to represent the not-yet-imported object.
The transparency requirement dictates that this placeholder must never be
visible to Python code; any reference to it must trigger the import and replace
it with the real imported object.

Given the possibility that Python (or C extension) code may pull objects
directly out of a module ``__dict__``, the only way to reliably prevent
accidental leakage of lazy objects is to have the dictionary itself be
responsible to ensure resolution of lazy objects on lookup.

When a lookup finds that the key references a lazy object, it resolves the lazy
object immediately before returning it. To avoid side effects mutating
dictionaries midway through iteration, all lazy objects in a dictionary are
resolved prior to starting an iteration; this could incur a performance penalty
when using bulk iterations (``iter(dict)``, ``reversed(dict)``,
``dict.__reversed__()``, ``dict.keys()``, ``iter(dict.keys())`` and
``reversed(dict.keys())``). To avoid this performance penalty on the vast
majority of dictionaries, which never contain any lazy objects, we steal a bit
from the ``dk_kind`` field for a new ``dk_lazy_imports`` flag to keep track of
whether a dictionary may contain lazy objects or not.

This implementation comprehensively prevents leakage of lazy objects, ensuring
they are always resolved to the real imported object before anyone can get hold
of them for any use, while avoiding any significant performance impact on
dictionaries in general.


Specification
=============

Lazy imports are opt-in, and they can be globally enabled either via a new
``-L`` flag to the Python interpreter, or via a call to a new
``importlib.set_lazy_imports()`` function. This function takes two arguments, a
boolean ``enabled`` and an ``excluding`` container. If ``enabled`` is true, lazy
imports will be turned on from that point forward. If it is false, they will be
turned off from that point forward. (Use of the ``excluding`` keyword is
discussed below under "Per-module opt-out.")

When the flag ``-L`` is passed to the Python interpreter, a new
``sys.flags.lazy_imports`` is set to ``True``, otherwise it exists as ``False``.
This flag is used to propagate ``-L`` to new Python subpr


id: pep-0684.rst
content:
PEP: 684
Title: A Per-Interpreter GIL
Author: Eric Snow <ericsnowcurrently@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-684-a-per-interpreter-gil/19583
Status: Final
Type: Standards Track
Requires: 683
Created: 08-Mar-2022
Python-Version: 3.12
Post-History: `08-Mar-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/CF7B7FMACFYDAHU6NPBEVEY6TOSGICXU/>`__,
              `29-Sep-2022 <https://discuss.python.org/t/pep-684-a-per-interpreter-gil/19583>`__,
              `28-Oct-2022 <https://discuss.python.org/t/pep-684-a-per-interpreter-gil/19583/19/>`__,
Resolution: https://discuss.python.org/t/19583/42


Abstract
========

Since Python 1.5 (1997), CPython users can run multiple interpreters
in the same process.  However, interpreters in the same process
have always shared a significant
amount of global state.  This is a source of bugs, with a growing
impact as more and more people use the feature.  Furthermore,
sufficient isolation would facilitate true multi-core parallelism,
where interpreters no longer share the GIL.  The changes outlined in
this proposal will result in that level of interpreter isolation.


High-Level Summary
==================

At a high level, this proposal changes CPython in the following ways:

* stops sharing the GIL between interpreters, given sufficient isolation
* adds several new interpreter config options for isolation settings
* keeps incompatible extensions from causing problems

The GIL
-------

The GIL protects concurrent access to most of CPython's runtime state.
So all that GIL-protected global state must move to each interpreter
before the GIL can.

(In a handful of cases, other mechanisms can be used to ensure
thread-safe sharing instead, such as locks or "immortal" objects.)

CPython Runtime State
---------------------

Properly isolating interpreters requires that most of CPython's
runtime state be stored in the ``PyInterpreterState`` struct.  Currently,
only a portion of it is; the rest is found either in C global variables
or in ``_PyRuntimeState``.  Most of that will have to be moved.

This directly coincides with an ongoing effort (of many years) to greatly
reduce internal use of global variables and consolidate the runtime
state into ``_PyRuntimeState`` and ``PyInterpreterState``.
(See `Consolidating Runtime Global State`_ below.)  That project has
`significant merit on its own <Benefits to Consolidation_>`_
and has faced little controversy.  So, while a per-interpreter GIL
relies on the completion of that effort, that project should not be
considered a part of this proposal--only a dependency.

Other Isolation Considerations
------------------------------

CPython's interpreters must be strictly isolated from each other, with
few exceptions.  To a large extent they already are.  Each interpreter
has its own copy of all modules, classes, functions, and variables.
The CPython C-API docs `explain further <caveats_>`_.

.. _caveats: https://docs.python.org/3/c-api/init.html#bugs-and-caveats

However, aside from what has already been mentioned (e.g. the GIL),
there are a couple of ways in which interpreters still share some state.

First of all, some process-global resources (e.g. memory,
file descriptors, environment variables) are shared.  There are no
plans to change this.

Second, some isolation is faulty due to bugs or implementations that
did not take multiple interpreters into account.  This includes
CPython's runtime and the stdlib, as well as extension modules that
rely on global variables.  Bugs should be opened in these cases,
as some already have been.

Depending on Immortal Objects
-----------------------------

:pep:`683` introduces immortal objects as a CPython-internal feature.
With immortal objects, we can share any otherwise immutable global
objects between all interpreters.  Consequently, this PEP does not
need to address how to deal with the various objects
`exposed in the public C-API <capi objects_>`_.
It also simplifies the question of what to do about the builtin
static types.  (See `Global Objects`_ below.)

Both issues have alternate solutions, but everything is simpler with
immortal objects.  If PEP 683 is not accepted then this one will be
updated with the alternatives.  This lets us reduce noise in this
proposal.


Motivation
==========

The fundamental problem we're solving here is a lack of true multi-core
parallelism (for Python code) in the CPython runtime.  The GIL is the
cause.  While it usually isn't a problem in practice, at the very least
it makes Python's multi-core story murky, which makes the GIL
a consistent distraction.

Isolated interpreters are also an effective mechanism to support
certain concurrency models.  :pep:`554` discusses this in more detail.

Indirect Benefits
-----------------

Most of the effort needed for a per-interpreter GIL has benefits that
make those tasks worth doing anyway:

* makes multiple-interpreter behavior more reliable
* has led to fixes for long-standing runtime bugs that otherwise
  hadn't been prioritized
* has been exposing (and inspiring fixes for) previously unknown runtime bugs
* has driven cleaner runtime initialization (:pep:`432`, :pep:`587`)
* has driven cleaner and more complete runtime finalization
* led to structural layering of the C-API (e.g. ``Include/internal``)
* also see `Benefits to Consolidation`_ below

.. XXX Add links to example GitHub issues?

Furthermore, much of that work benefits other CPython-related projects:

* performance improvements ("`faster-cpython`_")
* pre-fork application deployment (e.g. `Instagram server`_)
* extension module isolation (see :pep:`630`, etc.)
* embedding CPython

.. _faster-cpython: https://github.com/faster-cpython/ideas

.. _Instagram server: https://instagram-engineering.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf

Existing Use of Multiple Interpreters
-------------------------------------

The C-API for multiple interpreters has been used for many years.
However, until relatively recently the feature wasn't widely known,
nor extensively used (with the exception of mod_wsgi).

In the last few years use of multiple interpreters has been increasing.
Here are some of the public projects using the feature currently:

* `mod_wsgi <https://github.com/GrahamDumpleton/mod_wsgi>`_
* `OpenStack Ceph <https://github.com/ceph/ceph/pull/14971>`_
* `JEP <https://github.com/ninia/jep>`_
* `Kodi <https://github.com/xbmc/xbmc>`_

Note that, with :pep:`554`, multiple interpreter usage would likely
grow significantly (via Python code rather than the C-API).

PEP 554 (Multiple Interpreters in the Stdlib)
---------------------------------------------

:pep:`554` is strictly about providing a minimal stdlib module
to give users access to multiple interpreters from Python code.
In fact, it specifically avoids proposing any changes related to
the GIL.  Consider, however, that users of that module would benefit
from a per-interpreter GIL, which makes PEP 554 more appealing.


Rationale
=========

During initial investigations in 2014, a variety of possible solutions
for multi-core Python were explored, but each had its drawbacks
without simple solutions:

* the existing practice of releasing the GIL in extension modules

  * doesn't help with Python code

* other Python implementations (e.g. Jython, IronPython)

  * CPython dominates the community

* remove the GIL (e.g. gilectomy, "no-gil")

  * too much technical risk (at the time)

* Trent Nelson's "PyParallel" project

  * incomplete; Windows-only at the time

* ``multiprocessing``

  * too much work to make it effective enough;
    high penalties in some situations (at large scale, Windows)

* other parallelism tools (e.g. dask, ray, MPI)

  * not a fit for the runtime/stdlib

* give up on multi-core (e.g. async, do nothing)

  * this can only end in tears

Even in 2014, it was fairly clear that a solution using isolated
interpreters did not have a high level of technical risk and that
most 


id: pep-0653.rst
content:
PEP: 653
Title: Precise Semantics for Pattern Matching
Author: Mark Shannon <mark@hotpy.org>
Status: Draft
Type: Standards Track
Created: 09-Feb-2021
Post-History: 18-Feb-2021


Abstract
========

This PEP proposes a semantics for pattern matching that respects the general concept of :pep:`634`,
but is more precise, easier to reason about, and should be faster.

The object model will be extended with two special (dunder) attributes, ``__match_container__`` and
``__match_class__``, in addition to the ``__match_args__`` attribute from :pep:`634`, to support pattern matching.
Both of these new attributes must be integers and ``__match_args__`` is required to be a tuple of unique strings.

With this PEP:

* The semantics of pattern matching will be clearer, so that patterns are easier to reason about.
* It will be possible to implement pattern matching in a more efficient fashion.
* Pattern matching will be more usable for complex classes, by allowing classes some more control over which patterns they match.

Motivation
==========

Pattern matching in Python, as described in :pep:`634`, is to be added to Python 3.10.
Unfortunately, :pep:`634` is not as precise about the semantics as it could be,
nor does it allow classes sufficient control over how they match patterns.

Precise semantics
-----------------

:pep:`634` explicitly includes a section on undefined behavior.
Large amounts of undefined behavior may be acceptable in a language like C,
but in Python it should be kept to a minimum.
Pattern matching in Python can be defined more precisely without losing expressiveness or performance.

Improved control over class matching
------------------------------------

:pep:`634` delegates the decision over whether a class is a sequence or mapping to ``collections.abc``.
Not all classes that could be considered sequences are registered as subclasses of ``collections.abc.Sequence``.
This PEP allows them to match sequence patterns, without the full ``collections.abc.Sequence`` machinery.

:pep:`634` privileges some builtin classes with a special form of matching, the "self" match.
For example the pattern ``list(x)`` matches a list and assigns the list to ``x``.
By allowing classes to choose which kinds of pattern they match, other classes can use this form as well.

For example, using ``sympy``, we might want to write::

    # a*a == a**2
    case Mul(args=[Symbol(a), Symbol(b)]) if a == b:
        return Pow(a, 2)

Which requires the sympy class ``Symbol`` to "self" match.
For ``sympy`` to support this pattern with :pep:`634` is possible, but a bit tricky.
With this PEP it can be implemented very easily [1]_.

Robustness
----------

With this PEP, access to attributes during pattern matching becomes well defined and deterministic.
This makes pattern matching less error prone when matching objects with hidden side effects, such as object-relational mappers.
Objects will have more control over their own deconstruction, which can help prevent unintended consequences should attribute access have side-effects.

:pep:`634` relies on the ``collections.abc`` module when determining which patterns a value can match, implicitly importing it if necessary.
This PEP will eliminate surprising import errors and misleading audit events from those imports.


Efficient implementation
------------------------

The semantics proposed in this PEP will allow efficient implementation, partly as a result of having precise semantics
and partly from using the object model.

With precise semantics, it is possible to reason about what code transformations are correct,
and thus apply optimizations effectively.

Because the object model is a core part of Python, implementations already handle special attribute lookup efficiently.
Looking up a special attribute is much faster than performing a subclass test on an abstract base class.

Rationale
=========

The object model and special methods are at the core of the Python language. Consequently,
implementations support them well.
Using special attributes for pattern matching allows pattern matching to be implemented in a way that
integrates well with the rest of the implementation, and is thus easier to maintain and is likely to perform better.

A match statement performs a sequence of pattern matches. In general, matching a pattern has three parts:

1. Can the value match this kind of pattern?
2. When deconstructed, does the value match this particular pattern?
3. Is the guard true?

To determine whether a value can match a particular kind of pattern, we add the ``__match_container__``
and ``__match_class__`` attributes.
This allows the kind of a value to be determined in a efficient fashion.

Specification
=============

Additions to the object model
-----------------------------

The ``__match_container__`` and ``__match_class__`` attributes will be added to ``object``.
``__match_container__`` should be overridden by classes that want to match mapping or sequence patterns.
``__match_class__`` should be overridden by classes that want to change the default behavior when matching class patterns.

``__match_container__`` must be an integer and should be exactly one of these::

  0
  MATCH_SEQUENCE = 1
  MATCH_MAPPING = 2

``MATCH_SEQUENCE`` is used to indicate that instances of the class can match sequence patterns.

``MATCH_MAPPING`` is used to indicate that instances of the class can match mapping patterns.

``__match_class__`` must be an integer and should be exactly one of these::

  0
  MATCH_SELF = 8

``MATCH_SELF`` is used to indicate that for a single positional argument class pattern, the subject will be used and not deconstructed.

.. note::
    In the rest of this document, we will refer to the above values by name only.
    Symbolic constants will be provided both for Python and C, and the values will
    never be changed.

``object`` will have the following values for the special attributes::

  __match_container__ = 0
  __match_class__= 0
  __match_args__ = ()

These special attributes will be inherited as normal.

If ``__match_args__`` is overridden, then it is required to hold a tuple of unique strings. It may be empty.

.. note::
    ``__match_args__`` will be automatically generated for dataclasses and named tuples, as specified in :pep:`634`.

The pattern matching implementation is *not* required to check that any of these attributes behave as specified.
If the value of ``__match_container__``, ``__match_class__`` or ``__match_args__`` is not as specified, then
the implementation may raise any exception, or match the wrong pattern.
Of course, implementations are free to check these properties and provide meaningful error messages if they can do so efficiently.

Semantics of the matching process
---------------------------------

In the following, all variables of the form ``$var`` are temporary variables and are not visible to the Python program.
They may be visible via introspection, but that is an implementation detail and should not be relied on.
The pseudo-statement ``FAIL`` is used to signify that matching failed for this pattern and that matching should move to the next pattern.
If control reaches the end of the translation without reaching a ``FAIL``, then it has matched, and following patterns are ignored.

Variables of the form ``$ALL_CAPS`` are meta-variables holding a syntactic element, they are not normal variables.
So, ``$VARS = $items`` is not an assignment of ``$items`` to ``$VARS``,
but an unpacking of ``$items`` into the variables that ``$VARS`` holds.
For example, with the abstract syntax ``case [$VARS]:``, and the concrete syntax ``case[a, b]:`` then ``$VARS`` would hold the variables ``(a, b)``,
not the values of those variables.

The pseudo-function ``QUOTE`` takes a variable and returns the name of that variable.
For example, if the meta-variable ``$VAR`` held the variable ``foo`` then ``QUOTE($VAR) == "foo"``.

All additional code listed below that is not present in the original source will not tri


id: pep-0647.rst
content:
PEP: 647
Title: User-Defined Type Guards
Author: Eric Traut <erictr at microsoft.com>
Sponsor: Guido van Rossum <guido@python.org>
Discussions-To: typing-sig@python.org
Status: Final
Type: Standards Track
Topic: Typing
Created: 07-Oct-2020
Python-Version: 3.10
Post-History: 28-Dec-2020, 09-Apr-2021
Resolution: https://mail.python.org/archives/list/python-dev@python.org/thread/2ME6F6YUVKHOQYKSHTVQQU5WD4CVAZU4/

.. canonical-typing-spec:: :ref:`typing:typeguard` and
                           :py:data:`typing.TypeGuard`

Abstract
========

This PEP specifies a way for programs to influence conditional type narrowing
employed by a type checker based on runtime checks.


Motivation
==========

Static type checkers commonly employ a technique called "type narrowing" to
determine a more precise type of an expression within a program's code flow.
When type narrowing is applied within a block of code based on a conditional
code flow statement (such as ``if`` and ``while`` statements), the conditional
expression is sometimes referred to as a "type guard". Python type checkers
typically support various forms of type guards expressions.

::

    def func(val: Optional[str]):
        # "is None" type guard
        if val is not None:
            # Type of val is narrowed to str
            ...
        else:
            # Type of val is narrowed to None
            ...

    def func(val: Optional[str]):
        # Truthy type guard
        if val:
            # Type of val is narrowed to str
            ...
        else:
            # Type of val remains Optional[str]
            ...

    def func(val: Union[str, float]):
        # "isinstance" type guard
        if isinstance(val, str):
            # Type of val is narrowed to str
            ...
        else:
            # Type of val is narrowed to float
            ...

    def func(val: Literal[1, 2]):
        # Comparison type guard
        if val == 1:
            # Type of val is narrowed to Literal[1]
            ...
        else:
            # Type of val is narrowed to Literal[2]
            ...

There are cases where type narrowing cannot be applied based on static
information only. Consider the following example:

::

    def is_str_list(val: List[object]) -> bool:
        """Determines whether all objects in the list are strings"""
        return all(isinstance(x, str) for x in val)

    def func1(val: List[object]):
        if is_str_list(val):
            print(" ".join(val)) # Error: invalid type


This code is correct, but a type checker will report a type error because
the value ``val`` passed to the ``join`` method is understood to be of type
``List[object]``. The type checker does not have enough information to
statically verify that the type of ``val`` is ``List[str]`` at this point.

This PEP introduces a way for a function like ``is_str_list`` to be defined as
a "user-defined type guard". This allows code to extend the type guards that
are supported by type checkers.

Using this new mechanism, the ``is_str_list`` function in the above example
would be modified slightly. Its return type would be changed from ``bool``
to ``TypeGuard[List[str]]``.  This promises not merely that the return value
is boolean, but that a true indicates the input to the function was of the
specified type.

::

    from typing import TypeGuard

    def is_str_list(val: List[object]) -> TypeGuard[List[str]]:
        """Determines whether all objects in the list are strings"""
        return all(isinstance(x, str) for x in val)


User-defined type guards can also be used to determine whether a dictionary
conforms to the type requirements of a TypedDict.

::

    class Person(TypedDict):
        name: str
        age: int

    def is_person(val: dict) -> "TypeGuard[Person]":
        try:
            return isinstance(val["name"], str) and isinstance(val["age"], int)
        except KeyError:
            return False

    def print_age(val: dict):
        if is_person(val):
            print(f"Age: {val['age']}")
        else:
            print("Not a person!")



Specification
=============

TypeGuard Type
--------------

This PEP introduces the symbol ``TypeGuard`` exported from the ``typing``
module. ``TypeGuard`` is a special form that accepts a single type argument.
It is used to annotate the return type of a user-defined type guard function.
Return statements within a type guard function should return bool values,
and type checkers should verify that all return paths return a bool.

In all other respects, TypeGuard is a distinct type from bool. It is not a
subtype of bool. Therefore, ``Callable[..., TypeGuard[int]]`` is not assignable
to ``Callable[..., bool]``.

When ``TypeGuard`` is used to annotate the return type of a function or
method that accepts at least one parameter, that function or method is
treated by type checkers as a user-defined type guard. The type argument
provided for ``TypeGuard`` indicates the type that has been validated by
the function.

User-defined type guards can be generic functions, as shown in this example:

::

    _T = TypeVar("_T")

    def is_two_element_tuple(val: Tuple[_T, ...]) -> TypeGuard[Tuple[_T, _T]]:
        return len(val) == 2

    def func(names: Tuple[str, ...]):
        if is_two_element_tuple(names):
            reveal_type(names)  # Tuple[str, str]
        else:
            reveal_type(names)  # Tuple[str, ...]


Type checkers should assume that type narrowing should be applied to the
expression that is passed as the first positional argument to a user-defined
type guard. If the type guard function accepts more than one argument, no
type narrowing is applied to those additional argument expressions.

If a type guard function is implemented as an instance method or class method,
the first positional argument maps to the second parameter (after "self" or
"cls").

Here are some examples of user-defined type guard functions that accept more
than one argument:

::

    def is_str_list(val: List[object], allow_empty: bool) -> TypeGuard[List[str]]:
        if len(val) == 0:
            return allow_empty
        return all(isinstance(x, str) for x in val)

    _T = TypeVar("_T")

    def is_set_of(val: Set[Any], type: Type[_T]) -> TypeGuard[Set[_T]]:
        return all(isinstance(x, type) for x in val)


The return type of a user-defined type guard function will normally refer to
a type that is strictly "narrower" than the type of the first argument (that
is, it's a more specific type that can be assigned to the more general type).
However, it is not required that the return type be strictly narrower. This
allows for cases like the example above where ``List[str]`` is not assignable
to ``List[object]``.

When a conditional statement includes a call to a user-defined type guard
function, and that function returns true, the expression passed as the first
positional argument to the type guard function should be assumed by a static
type checker to take on the type specified in the TypeGuard return type,
unless and until it is further narrowed within the conditional code block.

Some built-in type guards provide narrowing for both positive and negative
tests (in both the ``if`` and ``else`` clauses). For example, consider the
type guard for an expression of the form ``x is None``. If ``x`` has a type that
is a union of None and some other type, it will be narrowed to ``None`` in the
positive case and the other type in the negative case. User-defined type
guards apply narrowing only in the positive case (the ``if`` clause). The type
is not narrowed in the negative case.

::

    OneOrTwoStrs = Union[Tuple[str], Tuple[str, str]]
    def func(val: OneOrTwoStrs):
        if is_two_element_tuple(val):
            reveal_type(val)  # Tuple[str, str]
            ...
        else:
            reveal_type(val)   # OneOrTwoStrs
            ...

        if not is_two_element_tuple(val):
            reveal_type(val)   # OneOrTwoStrs
            ...
        else:
            reveal_type(v


id: pep-0727.rst
content:
PEP: 727
Title: Documentation in Annotated Metadata
Author: SebastiÃ¡n RamÃ­rez <tiangolo@gmail.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://discuss.python.org/t/32566
Status: Withdrawn
Type: Standards Track
Topic: Typing
Created: 28-Aug-2023
Python-Version: 3.13
Post-History: `30-Aug-2023 <https://discuss.python.org/t/32566>`__


Abstract
========

This PEP proposes a standardized way to provide documentation strings for Python
symbols defined with :py:class:`~typing.Annotated` using a new class
``typing.Doc``.

PEP Withdrawal
==============

The reception of this PEP was mostly negative, with concerns raised about
verbosity and readability. As a result, this PEP has been withdrawn.


Motivation
==========

There's already a well-defined way to provide documentation for classes,
functions, class methods, and modules: using docstrings.

Currently there is no formalized standard to provide documentation strings for other
types of symbols: parameters, return values, class-scoped variables (class variables
and instance variables), local variables, and type aliases.

Nevertheless, to allow documenting most of these additional symbols, several
conventions have been created as microsyntaxes inside of docstrings, and are
currently commonly used: Sphinx, numpydoc, Google, Keras, etc.

There are two scenarios in which these conventions would be supported by tools: for
authors, while **editing** the contents of the documentation strings and for users,
while **rendering** that content in some way (in documentation sites, tooltips
in editors, etc).

Because each of these conventions uses a microsyntax inside a string, when
**editing** those docstrings, editors can't easily provide support for autocompletion,
inline errors for broken syntax, etc. Any type of **editing** support for these
conventions would be on top of the support for editing standard Python syntax.

When documenting parameters with current conventions, because the docstring is in
a different place in the code than the actual parameters and it requires
duplication of information (the parameter name) the information about
a parameter is easily in a place in the code quite far away from the
declaration of the actual parameter and it is disconnected from it.
This means it's easy to refactor a function, remove a parameter, and forget to
remove its docs. The same happens when adding a new parameter: it's easy to forget
to add the docstring for it.

And because of this same duplication of information (the parameter name) editors and
other tools need complex custom logic to check or ensure the consistency of the
parameters in the signature and in their docstring, or they simply don't
fully support that.

As these existing conventions are different types of microsyntaxes inside of
strings, robustly parsing them for **rendering** requires complex logic that
needs to be implemented by the tools supporting them. Additionally, libraries
and tools don't have a straightforward way to obtain the documentation for
each individual parameter or variable at runtime, without depending on a
specific docstring convention parser. Accessing the parameter documentation
strings at runtime would be useful, for example, for testing the contents
of each parameter's documentation, to ensure consistency across several
similar functions, or to extract and expose that same parameter
documentation in some other way (e.g. an API with FastAPI, a CLI with Typer, etc).

Some of these previous formats tried to account for the lack of type annotations
in older Python versions by including typing information in the docstrings (e.g.
`Sphinx <https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists>`__,
`numpydoc <https://numpydoc.readthedocs.io/en/latest/format.html#parameters>`__)
but now that information doesn't need to be in docstrings as there is now an official
:pep:`syntax for type annotations <484>`.


Rationale
=========

This proposal intends to address these shortcomings by extending and complementing the
information in docstrings, keeping backwards compatibility with existing docstrings
(it doesn't deprecate them), and doing it in a way that leverages the Python
language and structure, via type annotations with :py:class:`~typing.Annotated`, and
a new class ``Doc`` in ``typing``.

The reason why this would belong in the standard Python library instead of an
external package is because although the implementation would be quite trivial,
the actual power and benefit from it would come from being a standard, to facilitate
its usage from library authors and to provide a default way to document Python
symbols using :py:class:`~typing.Annotated`. Some tool providers (at least VS Code
and PyCharm) have shown they would consider implementing support for this only if
it was a standard.

This doesn't deprecate current usage of docstrings, docstrings should be considered
the preferred documentation method when available (not available in type aliases,
parameters, etc).
And docstrings would be complemented by this proposal for documentation specific to
the symbols that can be declared with :py:class:`~typing.Annotated`
(currently only covered by the several available microsyntax conventions).

This should be relatively transparent to common developers (library users) unless
they manually open the source files from libraries adopting it.

It should be considered opt-in for library authors who would like to adopt it and
they should be free to decide to use it or not.

It would be only useful for libraries that are willing to use optional type hints.


Summary
-------

Here's a short summary of the features of this proposal in contrast to current
conventions:

* **Editing** would be already fully supported by default by any editor (current
  or future) supporting Python syntax, including syntax errors, syntax
  highlighting, etc.

* **Rendering** would be relatively straightforward to implement by static tools
  (tools that don't need runtime execution), as the information can be extracted
  from the AST they normally already create.

* Deduplication of information: the name of a parameter would be defined in a single
  place, not duplicated inside of a docstring.

* Elimination of the possibility of having inconsistencies when removing a parameter
  or class variable and forgetting to remove its documentation.

* Minimization of the probability of adding a new parameter or class variable
  and forgetting to add its documentation.

* Elimination of the possibility of having inconsistencies between the name of a
  parameter in the signature and the name in the docstring when it is renamed.

* Access to the documentation string for each symbol at runtime, including existing
  (older) Python versions.

* A more formalized way to document other symbols, like type aliases, that could
  use :py:class:`~typing.Annotated`.

* No microsyntax to learn for newcomers, it's just Python syntax.

* Parameter documentation inheritance for functions captured
  by :py:class:`~typing.ParamSpec`.


Specification
=============

The main proposal is to introduce a new class, ``typing.Doc``.
This class should only be used within :py:class:`~typing.Annotated` annotations.
It takes a single positional-only string argument. It should be used to
document the intended meaning and use of the symbol declared using
:py:class:`~typing.Annotated`.

For example:

.. code:: python

   from typing import Annotated, Doc

   class User:
       name: Annotated[str, Doc("The user's name")]
       age: Annotated[int, Doc("The user's age")]

       ...

:py:class:`~typing.Annotated` is normally used as a type annotation, in those cases,
any ``typing.Doc`` inside of it would document the symbol being annotated.

When :py:class:`~typing.Annotated` is used to declare a type alias, ``typing.Doc``
would then document the type alias symbol.

For example:

.. code:: python

   from typing import Annotated, Doc, TypeAlias


