# embeddings

id: pep-0734.rst
content:
PEP: 734
Title: Multiple Interpreters in the Stdlib
Author: Eric Snow <ericsnowcurrently@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-734-multiple-interpreters-in-the-stdlib/41147
Status: Deferred
Type: Standards Track
Created: 06-Nov-2023
Python-Version: 3.13
Post-History: `14-Dec-2023 <https://discuss.python.org/t/pep-734-multiple-interpreters-in-the-stdlib/41147/>`__,
Replaces: 554
Resolution: https://discuss.python.org/t/pep-734-multiple-interpreters-in-the-stdlib/41147/24


.. note::
   This PEP is essentially a continuation of :pep:`554`.  That document
   had grown a lot of ancillary information across 7 years of discussion.
   This PEP is a reduction back to the essential information.  Much of
   that extra information is still valid and useful, just not in the
   immediate context of the specific proposal here.


Abstract
========

This PEP proposes to add a new module, ``interpreters``, to support
inspecting, creating, and running code in multiple interpreters in the
current process.  This includes ``Interpreter`` objects that represent
the underlying interpreters.  The module will also provide a basic
``Queue`` class for communication between interpreters.
Finally, we will add a new ``concurrent.futures.InterpreterPoolExecutor``
based on the ``interpreters`` module.


Introduction
============

Fundamentally, an "interpreter" is the collection of (essentially)
all runtime state which Python threads must share.  So, let's first
look at threads.  Then we'll circle back to interpreters.

Threads and Thread States
-------------------------

A Python process will have one or more OS threads running Python code
(or otherwise interacting with the C API).  Each of these threads
interacts with the CPython runtime using its own thread state
(``PyThreadState``), which holds all the runtime state unique to that
thread.  There is also some runtime state that is shared between
multiple OS threads.

Any OS thread may switch which thread state it is currently using, as
long as it isn't one that another OS thread is already using (or has
been using).  This "current" thread state is stored by the runtime
in a thread-local variable, and may be looked up explicitly with
``PyThreadState_Get()``.  It gets set automatically for the initial
("main") OS thread and for ``threading.Thread`` objects.  From the
C API it is set (and cleared) by ``PyThreadState_Swap()`` and may
be set by ``PyGILState_Ensure()``.  Most of the C API requires that
there be a current thread state, either looked up implicitly
or passed in as an argument.

The relationship between OS threads and thread states is one-to-many.
Each thread state is associated with at most a single OS thread and
records its thread ID.  A thread state is never used for more than one
OS thread.  In the other direction, however, an OS thread may have more
than one thread state associated with it, though, again, only one
may be current.

When there's more than one thread state for an OS thread,
``PyThreadState_Swap()`` is used in that OS thread to switch
between them, with the requested thread state becoming the current one.
Whatever was running in the thread using the old thread state is
effectively paused until that thread state is swapped back in.

Interpreter States
------------------

As noted earlier, there is some runtime state that multiple OS threads
share.  Some of it is exposed by the ``sys`` module, though much is
used internally and not exposed explicitly or only through the C API.

This shared state is called the interpreter state
(``PyInterpreterState``).  We'll sometimes refer to it here as just
"interpreter", though that is also sometimes used to refer to the
``python`` executable, to the Python implementation, and to the
bytecode interpreter (i.e. ``exec()``/``eval()``).

CPython has supported multiple interpreters in the same process (AKA
"subinterpreters") since version 1.5 (1997).  The feature has been
available via the :ref:`C API <python:sub-interpreter-support>`.

Interpreters and Threads
------------------------

Thread states are related to interpreter states in much the same way
that OS threads and processes are related (at a high level).  To
begin with, the relationship is one-to-many.
A thread state belongs to a single interpreter (and stores
a pointer to it).  That thread state is never used for a different
interpreter.  In the other direction, however, an interpreter may have
zero or more thread states associated with it.  The interpreter is only
considered active in OS threads where one of its thread states
is current.

Interpreters are created via the C API using
``Py_NewInterpreterFromConfig()`` (or ``Py_NewInterpreter()``, which
is a light wrapper around ``Py_NewInterpreterFromConfig()``).
That function does the following:

1. create a new interpreter state
2. create a new thread state
3. set the thread state as current
   (a current tstate is needed for interpreter init)
4. initialize the interpreter state using that thread state
5. return the thread state (still current)

Note that the returned thread state may be immediately discarded.
There is no requirement that an interpreter have any thread states,
except as soon as the interpreter is meant to actually be used.
At that point it must be made active in the current OS thread.

To make an existing interpreter active in the current OS thread,
the C API user first makes sure that interpreter has a corresponding
thread state.  Then ``PyThreadState_Swap()`` is called like normal
using that thread state.  If the thread state for another interpreter
was already current then it gets swapped out like normal and execution
of that interpreter in the OS thread is thus effectively paused until
it is swapped back in.

Once an interpreter is active in the current OS thread like that, the
thread can call any of the C API, such as ``PyEval_EvalCode()``
(i.e. ``exec()``).  This works by using the current thread state as
the runtime context.

The "Main" Interpreter
----------------------

When a Python process starts, it creates a single interpreter state
(the "main" interpreter) with a single thread state for the current
OS thread.  The Python runtime is then initialized using them.

After initialization, the script or module or REPL is executed using
them.  That execution happens in the interpreter's ``__main__`` module.

When the process finishes running the requested Python code or REPL,
in the main OS thread, the Python runtime is finalized in that thread
using the main interpreter.

Runtime finalization has only a slight, indirect effect on still-running
Python threads, whether in the main interpreter or in subinterpreters.
That's because right away it waits indefinitely for all non-daemon
Python threads to finish.

While the C API may be queried, there is no mechanism by which any
Python thread is directly alerted that finalization has begun,
other than perhaps with "atexit" functions that may be been
registered using ``threading._register_atexit()``.

Any remaining subinterpreters are themselves finalized later,
but at that point they aren't current in any OS threads.

Interpreter Isolation
---------------------

CPython's interpreters are intended to be strictly isolated from each
other.  That means interpreters never share objects (except in very
specific cases with immortal, immutable builtin objects).  Each
interpreter has its own modules (``sys.modules``), classes, functions,
and variables.  Even where two interpreters define the same class,
each will have its own copy.  The same applies to state in C, including
in extension modules.  The CPython C API docs `explain more`_.

.. _explain more:
   https://docs.python.org/3/c-api/init.html#bugs-and-caveats

Notably, there is some process-global state that interpreters will
always share, some mutable and some immutable.  Sharing immutable
state presents few problems, while providing some benefits (mainly
performance).  However, all shared mutable state requires special
ma


id: pep-0708.rst
content:
PEP: 708
Title: Extending the Repository API to Mitigate Dependency Confusion Attacks
Author: Donald Stufft <donald@stufft.io>
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/24179
Status: Provisional
Type: Standards Track
Topic: Packaging
Created: 20-Feb-2023
Post-History: `01-Feb-2023 <https://discuss.python.org/t/23414/>`__,
              `23-Feb-2023 <https://discuss.python.org/t/24179>`__
Resolution: https://discuss.python.org/t/24179/72


Provisional Acceptance
======================

This PEP has been **provisionally accepted**,
with the following required conditions before the PEP is made Final:

1. An implementation of the PEP in PyPI (Warehouse)
   including any necessary UI elements
   to allow project owners to set the tracking data.
2. An implementation of the PEP in at least one repository other than PyPI,
   as you canât really test merging indexes without at least two indexes.
3. An implementation of the PEP in pip,
   which supports the intended semantics and can be used to demonstrate
   that the expected security benefits are achieved.
   This implementation will need to be "off by default" initially,
   which means that users will have to opt in to testing it.
   Ideally, we should collect explicit positive reports from users
   (both project owners and project users)
   who have successfully tried out the new feature,
   rather than just assuming that "no news is good news".


Abstract
========

Dependency confusion attacks, in which a malicious package is installed instead
of the one the user expected, are an `increasingly common supply chain threat
<https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610>`__.
Most such attacks against Python dependencies, including the
`recent PyTorch incident <https://pytorch.org/blog/compromised-nightly-dependency/>`_,
occur with multiple package repositories, where a dependency expected to come
from one repository (e.g. a custom index) is installed from another (e.g. PyPI).

To help address this problem, this PEP proposes extending the
:ref:`Simple Repository API <packaging:simple-repository-api>`
to allow repository operators to indicate that a project found on their
repository "tracks" a project on different repositories, and allows projects to
extend their namespaces across multiple repositories.

These features will allow installers to determine when a project being made
available from a particular mix of repositories is expected and should be
allowed, and when it is not and should halt the install with an error to protect
the user.


Motivation
===========

There is a long-standing class of attacks that are called "dependency confusion"
attacks, which roughly boil down to an individual user expected to get package
``A``, but instead they got ``B``. In Python, this almost always happens due to
the configuration of multiple repositories (possibly including the default of
PyPI), where they expected package ``A`` to come from repository ``X``, but
someone is able to publish package ``B`` to repository ``Y`` under the same
name.

Dependency Confusion attacks have long been possible, but they've recently
gained press with
`public examples of cases where these attacks were successfully executed <https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610>`__.

A specific example of this is the recent case where the PyTorch project had an
internal package named ``torchtriton`` which was only ever intended to be
installed from their repositories located at ``https://download.pytorch.org/``,
but that repository was designed to be used in conjunction with PyPI, and
the name of ``torchtriton`` was not claimed on PyPI, which allowed the attacker
to use that name and publish a malicious version.

There are a number of ways to mitigate against these attacks today, but they all
require that the end user go out of their way to protect themselves, rather than
being protected by default. This means that for the vast bulk of users, they are
likely to remain vulnerable, even if they are ultimately aware of these types of
attacks.

Ultimately the underlying cause of these attacks come from the fact that there
is no globally unique namespace that all Python package names come from.
Instead, each repository is its own distinct namespace, and when given an
"abstract" name such as ``spam`` to install, an installer has to implicitly turn
that into a "concrete" name such as ``pypi.org:spam`` or ``example.com:spam``.
Currently the standard behavior in Python installation tools is to implicitly
flatten these multiple namespaces into one that contains the files from all
namespaces.

This assumption that collapsing the namespaces is what was expected means that
when packages with the same name in different repositories
are authored by different parties (such as in the ``torchtriton`` case)
dependency confusion attacks become possible.

This is made particularly tricky in that there is no "right" answer; there are
valid use cases both for wanting two repositories merged into one namespace
*and* for wanting two repositories to be treated as distinct namespaces. This
means that an installer needs some mechanism by which to determine when it
should merge the namespaces of multiple repositories and when it should not,
rather than a blanket always merge or never merge rule.

This functionality could be pushed directly to the end user, since ultimately
the end user is the person whose expectations of what gets installed from what
repository actually matters. However, by extending the repository specification
to allow a repository to indicate when it is safe, we can enable individual
projects and repositories to "work by default", even when their
project naturally spans multiple distinct namespaces, while maintaining the
ability for an installer to be secure by default.

On its own, this PEP does not solve dependency confusion attacks, but what it
does do is provide enough information so that installers can prevent them
without causing too much collateral damage to otherwise valid and safe use
cases.


Rationale
=========

There are two broad use cases for merging names across repositories that this
PEP seeks to enable.

The first use case is when one repository is not defining its own names, but
rather is extending names defined in other repositories. This commonly happens
in cases where a project is being mirrored from one repository to another (see
`Bandersnatch <https://pypi.org/project/bandersnatch/>`__) or when a repository
is providing supplementary artifacts for a specific platform (see
`Piwheels <https://www.piwheels.org/>`__).

In this case neither the repositories nor the projects that are being extended
may have any knowledge that they are being extended or by whom, so this cannot
rely on any information that isn't present in the "extending" repository itself.

The second use case is when the project wants to publish to one "main"
repository, but then have additional repositories that provide binaries for
additional platforms, GPUs, CPUs, etc. Currently wheel tags are not sufficiently
able to express these types of binary compatibility, so projects that wish to
rely on them are forced to set up multiple repositories and have their users
manually configure them to get the correct binaries for their platform, GPU,
CPU, etc.

This use case is similar to the first, but the important difference that makes
it a distinct use case on its own is who is providing the information and what
their level of trust is.

When a user configures a specific repository (or relies on the default) there
is no ambiguity as to what repository they mean. A repository is identified by
an URL, and through the domain system, URLs are globally unique identifiers.
This lack of ambiguity means that an installer can assume that the repository
operator is trustworthy and can trust metadata that they provide without needing
to validate it.

On the flip side, given an installer finds a n


id: pep-0654.rst
content:
PEP: 654
Title: Exception Groups and except*
Author: Irit Katriel <irit@python.org>,
        Yury Selivanov <yury@edgedb.com>,
        Guido van Rossum <guido@python.org>
Discussions-To: https://discuss.python.org/t/accepting-pep-654-exception-groups-and-except/10813
Status: Final
Type: Standards Track
Created: 22-Feb-2021
Python-Version: 3.11
Post-History: `22-Feb-2021 <https://mail.python.org/archives/list/python-dev@python.org/thread/L5Q27DVKOKZCDNCAWRIQVOZ5DZCZHLRM/>`__,
              `20-Mar-2021 <https://mail.python.org/archives/list/python-dev@python.org/thread/MQ2UCSQ2ZC4FIGT7KSVI6BJA4FCXSOCL/>`__,
              `03-Oct-2021 <https://mail.python.org/archives/list/python-dev@python.org/thread/4B256YKUPW5P2M44GG5H6FBL3PSV6ODP/>`__,
Resolution: https://discuss.python.org/t/accepting-pep-654-exception-groups-and-except/10813/1

.. canonical-doc:: :ref:`python:lib-exception-groups` and :ref:`python:except_star`

   See :ref:`python:tut-exception-groups` for a user-focused tutorial.

Abstract
========

This document proposes language extensions that allow programs to raise and handle
multiple unrelated exceptions simultaneously:

* A new standard exception type, the ``ExceptionGroup``, which represents a
  group of unrelated exceptions being propagated together.

* A new syntax ``except*`` for handling ``ExceptionGroups``.

Motivation
==========

The interpreter is currently able to propagate at most one exception at a
time. The chaining features introduced in :pep:`3134` link together
exceptions that are related to each other as the cause or context, but
there are situations where multiple unrelated exceptions need to be propagated
together as the stack unwinds. Several real world use cases are listed below.

* **Concurrent errors**. Libraries for async concurrency provide APIs to invoke
  multiple tasks and return their results in aggregate. There isn't currently
  a good way for such libraries to handle situations where multiple tasks
  raise exceptions. The Python standard library's ``asyncio.gather()`` [1]_
  function provides two options: raise the first exception, or return the
  exceptions in the results list.  The Trio [2]_
  library has a ``MultiError`` exception type which it raises to report a
  collection of errors. Work on this PEP was initially motivated by the
  difficulties in handling ``MultiErrors`` [9]_, which are detailed in a design
  document for an improved version, ``MultiError2`` [3]_.
  That document demonstrates how difficult it is to create an effective API
  for reporting and handling multiple errors without the language changes we
  are proposing (see also the `Programming Without 'except \*'`_ section.)

  Implementing a better task spawning API in asyncio, inspired by Trio
  nurseries [13]_, was the main motivation for this PEP.  That work is currently
  blocked on Python not having native language level support for exception
  groups.

* **Multiple failures when retrying an operation.** The Python standard
  library's ``socket.create_connection`` function may attempt to connect to
  different addresses, and if all attempts fail it needs to report that to the
  user. It is an open issue how to aggregate these errors, particularly when
  they are different (see issue 29980 [4]_.)

* **Multiple user callbacks fail.** Python's ``atexit.register()`` function
  allows users to register functions that are called on system exit. If any of
  them raise exceptions, only the last one is reraised, but it would be better
  to reraise all of them together (see ``atexit`` documentation [5]_.)
  Similarly, the pytest library allows users to register finalizers which
  are executed at teardown. If more than one of these finalizers raises an
  exception, only the first is reported to the user. This can be improved with
  ``ExceptionGroups``, as explained in this issue by pytest developer Ran Benita
  (see pytest issue 8217 [6]_.)

* **Multiple errors in a complex calculation.** The Hypothesis library performs
  automatic bug reduction (simplifying code that demonstrates a bug). In the
  process it may find variations that generate different errors, and
  (optionally) reports all of them (see the Hypothesis documentation [7]_.)
  An ``ExceptionGroup`` mechanism as we are proposing here can resolve some of
  the difficulties with debugging that are mentioned in the link above, and
  which are due to the loss of context/cause information (communicated
  by Hypothesis Core Developer Zac Hatfield-Dodds).

* **Errors in wrapper code.** The Python standard library's
  ``tempfile.TemporaryDirectory`` context manager
  had an issue where an exception raised during cleanup in ``__exit__``
  effectively masked an exception that the user's code raised inside the context
  manager scope. While the user's exception was chained as the context of the
  cleanup error, it was not caught by the user's except clause
  (see issue 40857 [8]_.)

  The issue was resolved by making the cleanup code ignore errors, thus
  sidestepping the multiple exception problem. With the features we propose
  here, it would be possible for ``__exit__`` to raise an ``ExceptionGroup``
  containing its own errors along with the user's errors, and this would allow
  the user to catch their own exceptions by their types.


Rationale
=========

Grouping several exceptions together can be done without changes to the
language, simply by creating a container exception type.
Trio [2]_ is an example of a library that has made use of this technique in its
``MultiError`` [9]_ type. However, such an approach requires calling code to catch
the container exception type, and then to inspect it to determine the types of
errors that had occurred, extract the ones it wants to handle, and reraise the
rest. Furthermore, exceptions in Python have important information attached to
their ``__traceback__``, ``__cause__`` and ``__context__`` fields, and
designing a container type that preserves the integrity of this information
requires care; it is not as simple as collecting exceptions into a set.

Changes to the language are required in order to extend support for
exception groups in the style of existing exception handling mechanisms. At
the very least we would like to be able to catch an exception group only if
it contains an exception of a type that we choose to handle. Exceptions of
other types in the same group need to be automatically reraised,
otherwise it is too easy for user code to inadvertently swallow exceptions
that it is not handling.

We considered whether it is possible to modify the semantics of ``except``
for this purpose, in a backwards-compatible manner, and found that it is not.
See the `Rejected Ideas`_ section for more on this.

The purpose of this PEP, then, is to add the ``ExceptionGroup`` builtin type
and the ``except*`` syntax for handling exception groups in the interpreter.
The desired semantics of ``except*`` are sufficiently different from the
current exception handling semantics that we are not proposing to modify the
behavior of the ``except`` keyword but rather to add the new ``except*``
syntax.

Our premise is that exception groups and ``except*`` will be used
selectively, only when they are needed. We do not expect them to become
the default mechanism for exception handling.  The decision to raise
exception groups from a library needs to be considered carefully and
regarded as an API-breaking change. We expect that this will normally be
done by introducing a new API rather than modifying an existing one.


Specification
=============

ExceptionGroup and BaseExceptionGroup
-------------------------------------

We propose to add two new builtin exception types:
``BaseExceptionGroup(BaseException)`` and
``ExceptionGroup(BaseExceptionGroup, Exception)``. They are assignable to
``Exception.__cause__`` and ``Exception.__context__``, and they can be
raised and handled as any exception with ``raise ExceptionGroup(...)`` and
``try: ... except ExceptionGroup: ...`` or


id: pep-0640.rst
content:
PEP: 640
Title: Unused variable syntax
Author: Thomas Wouters <thomas@python.org>
Status: Rejected
Type: Standards Track
Created: 04-Oct-2020
Python-Version: 3.10
Post-History: 19-Oct-2020
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/SQC2FTLFV5A7DV7RCEAR2I2IKJKGK7W3/

Rejection Note
==============

Rejected by the Steering Council:
https://mail.python.org/archives/list/python-dev@python.org/message/SQC2FTLFV5A7DV7RCEAR2I2IKJKGK7W3/

Abstract
========

This PEP proposes new syntax for *unused variables*, providing a pseudo-name
that can be assigned to but not otherwise used. The assignment doesn't
actually happen, and the value is discarded instead.

Motivation
==========

In Python it is somewhat common to need to do an assignment without actually
needing the result. Conventionally, people use either ``"_"`` or a name such
as ``"unused"`` (or with ``"unused"`` as a prefix) for this. It's most
common in *unpacking assignments*::

   x, unused, z = range(3)
   x, *unused, z = range(10)

It's also used in ``for`` loops and comprehensions::

   for unused in range(10): ...
   [ SpamObject() for unused in range(10) ]

The use of ``"_"`` in these cases is probably the most common, but it
potentially conflicts with the use of ``"_"`` in internationalization, where
a call like gettext.gettext() is bound to ``"_"`` and used to mark strings
for translation.

In the proposal to add Pattern Matching to Python (originally :pep:`622`, now
split into :pep:`634`, :pep:`635` and :pep:`636`), ``"_"`` has an *additional*
special meaning. It is a wildcard pattern, used in places where variables
could be assigned to, to indicate anything should be matched but not
assigned to anything. The choice of ``"_"`` there matches the use of ``"_"``
in other languages, but the semantic difference with ``"_"`` elsewhere in
Python is significant.

This PEP proposes to allow a special token, ``"?"``, to be used instead of
any valid name in assignment. This has most of the benefits of ``"_"``
without affecting other uses of that otherwise regular variable. Allowing
the use of the same wildcard pattern would make pattern matching and
unpacking assignment more consistent with each other.

Rationale
=========

Marking certain variables as unused is a useful tool, as it helps clarity of
purpose of the code. It makes it obvious to readers of the code as well as
automated linters, that a particular variable is *intentionally* unused.

However, despite the convention, ``"_"`` is not a special variable. The
value is still assigned to, the object it refers to is still kept alive
until the end of the scope, and it can still be used. Nor is the use of
``"_"`` for unused variables entirely ubiquitous, since it conflicts with
conventional internationalization, it isn't obvious that it is a regular
variable, and it isn't as obviously unused like a variable named
``"unused"``.

In the Pattern Matching proposal, the use of ``"_"`` for wildcard patterns
side-steps the problems of ``"_"`` for unused variables by virtue of it
being in a separate scope. The only conflict it has with
internationalization is one of potential confusion, it will not actually
interact with uses of a global variable called ``"_"``. However, the
special-casing of ``"_"`` for this wildcard pattern purpose is still
problematic: the different semantics *and meaning* of ``"_"`` inside pattern
matching and outside of it means a break in consistency in Python.

Introducing ``"?"`` as special syntax for unused variables *both inside and
outside pattern matching* allows us to retain that consistency. It avoids
the conflict with internationalization *or any other uses of _ as a
variable*. It makes unpacking assignment align more closely with pattern
matching, making it easier to explain pattern matching as an extension of
unpacking assignment.

In terms of code readability, using a special token makes it easier to find
out what it means (``"what does question mark in Python do"`` versus ``"why
is my _ variable not getting assigned to"``), and makes it more obvious that
the actual intent is for the value to be unused -- since it is entirely
impossible to use it.

Specification
=============

A new token is introduced, ``"?"``, or ``token.QMARK``.

The grammar is modified to allow ``"?"`` in assignment contexts
(``star_atom`` and ``t_atom`` in the current grammar), creating a ``Name``
AST node with identifier set to NULL.

The AST is modified to allow the ``Name`` expression's identifier to be
optional (it is currently required). The identifier being empty would only
be allowed in a ``STORE`` context.

In CPython, the bytecode compiler is modified to emit ``POP_TOP`` instead of
``STORE_NAME`` for ``Name`` nodes with no identifier. Other uses of the
``Name`` node are updated to handle the identifier being empty, as
appropriate.

The uses of the modified grammar nodes encompass at least the following
forms of assignment::

   ? = ...
   x, ?, z = ...
   x, *?, z = ...
   for ? in range(3): ...  # including comprehension forms
   for x, ?, z in matrix: ...  # including comprehension forms
   with open(f) as ?: ...
   with func() as (x, ?, z): ...

The use of a single ``"?"``, not in an unpacking context, is allowed in
normal assignment and the ``with`` statement. It doesn't really make sense
on its own, and it is possible to disallow those specific cases. However,
``for ? in range(3)`` clearly has its uses, so for consistency reasons if
nothing else it seems more sensible to allow the use of the single ``"?"``
in other cases.

Using ``"?"`` in augmented assignment (``? *= 2``) is not allowed, since
``"?"`` can only be used for assignment. Having multiple occurrences of
``"?"`` is valid, just like when assigning to names, and the assignments do
not interfere with each other.

Backwards Compatibility
=======================

Introducing a new token means there are no backward compatibility concerns.
No valid syntax changes meaning.

``"?"`` is not considered an identifier, so ``str.isidentifier()`` does not
change.

The AST does change in an incompatible way, as the identifier of a Name
token can now be empty. Code using the AST will have to be adjusted
accordingly.

How to Teach This
=================

``"?"`` can be introduced along with unpacking assignment, explaining it is
special syntax for 'unused' and mentioning that it can also be used in other
places. Alternatively, it could be introduced as part of an explanation on
assignment in ``for`` loops, showing an example where the loop variable is
unused.

:pep:`636` discusses how to teach ``"_"``, and can simply replace ``"_"`` with
``"?"``, perhaps noting that ``"?"`` is similarly usable in other contexts.

Reference Implementation
========================

A prototype implementation exists at
<https://github.com/Yhg1s/cpython/tree/nonassign>.

Rejected Ideas
==============


Open Issues
===========

Should ``"?"`` be allowed in the following contexts::

   # imports done for side-effect only.
   import os as ?
   from os import path as ?

   # Function defined for side-effects only (e.g. decorators)
   @register_my_func
   def ?(...): ...

   # Class defined for side-effects only (e.g. decorators, __init_subclass__)
   class ?(...): ...

   # Parameters defined for unused positional-only arguments:
   def f(a, ?, ?): ...
   lambda a, ?, ?: ...

   # Unused variables with type annotations:
   ?: int = f()

   # Exception handling:
   try: ...
   except Exception as ?: ...

   # With blocks:
   with open(f) as ?: ...

Some of these may seem to make sense from a consistency point of view, but
practical uses are limited and dubious. Type annotations on ``"?"`` and
using it with ``except`` and ``with`` do not seem to make any sense. In the
reference implementation, ``except`` is not supported (the existing syntax
only allows a name) but ``with`` is (by virtue of the existing syntax
supporting unpacking assignment).

Should this PEP be accepted even if patte


id: pep-0668.rst
content:
PEP: 668
Title: Marking Python base environments as âexternally managedâ
Author: Geoffrey Thomas <geofft@ldpreload.com>,
        Matthias Klose <doko@ubuntu.com>,
        Filipe LaÃ­ns <lains@python.org>,
        Donald Stufft <donald@stufft.io>,
        Tzu-ping Chung <uranusjr@gmail.com>,
        Stefano Rivera <stefanor@debian.org>,
        Elana Hashman <ehashman@debian.org>,
        Pradyun Gedam <pradyunsg@gmail.com>
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/10302
Status: Accepted
Type: Standards Track
Topic: Packaging
Created: 18-May-2021
Post-History: 28-May-2021
Resolution: https://discuss.python.org/t/10302/44

.. canonical-pypa-spec:: :ref:`externally-managed-environments`

Abstract
========

A long-standing practical problem for Python users has been conflicts
between OS package managers and Python-specific package management
tools like pip. These conflicts include both Python-level API
incompatibilities and conflicts over file ownership.

Historically, Python-specific package management tools have defaulted
to installing packages into an implicit global context. With the
standardization and popularity of virtual environments, a better
solution for most (but not all) use cases is to use Python-specific
package management tools only within a virtual environment.

This PEP proposes a mechanism for a Python installation to communicate
to tools like pip that its global package installation context is
managed by some means external to Python, such as an OS package
manager. It specifies that Python-specific package management tools
should neither install nor remove packages into the interpreter's
global context, by default, and should instead guide the end user
towards using a virtual environment.

It also standardizes an interpretation of the ``sysconfig`` schemes so
that, if a Python-specific package manager is about to install a
package in an interpreter-wide context, it can do so in a manner that
will avoid conflicting with the external package manager and reduces
the risk of breaking software shipped by the external package manager.

Terminology
===========

A few terms used in this PEP have multiple meanings in the contexts
that it spans. For clarity, this PEP uses the following terms in
specific ways:

distro
    Short for "distribution," a collection of various sorts of
    software, ideally designed to work properly together, including
    (in contexts relevant to this document) the Python interpreter
    itself, software written in Python, and software written in other
    languages. That is, this is the sense used in phrases such as
    "Linux distro" or "Berkeley Software Distribution."

    A distro can be an operating system (OS) of its own, such as
    Debian, Fedora, or FreeBSD. It can also be an overlay distribution
    that installs on top of an existing OS, such as Homebrew or
    MacPorts.

    This document uses the short term "distro," because the term
    "distribution" has another meaning in Python packaging contexts: a
    source or binary distribution package of a single piece of Python
    language software, that is, in the sense of
    ``setuptools.dist.Distribution`` or "sdist". To avoid confusion,
    this document does not use the plain term "distribution" at all.
    In the Python packaging sense, it uses the full phrase
    "distribution package" or just "package" (see below).

    The provider of a distro - the team or company that collects and
    publishes the software and makes any needed modifications - is its
    **distributor**.
package
    A unit of software that can be installed and used within Python.
    That is, this refers to what Python-specific packaging tools tend
    to call a "`distribution package`_" or simply a "distribution";
    the colloquial abbreviation "package" is used in the sense of the
    Python Package Index.

    .. _`distribution package`: https://packaging.python.org/glossary/#term-Distribution-Package

    This document does not use "package" in the sense of an importable
    name that contains Python modules, though in many cases, a
    distribution package consists of a single importable package of
    the same name.

    This document generally does not use the term "package" to refer
    to units of installation by a distro's package manager (such as
    ``.deb`` or ``.rpm`` files). When needed, it uses phrasing such as
    "a distro's package." (Again, in many cases, a Python package is
    shipped inside a distro's package named something like ``python-``
    plus the Python package name.)
Python-specific package manager
    A tool for installing, upgrading, and/or removing Python packages
    in a manner that conforms to Python packaging standards (such as
    :pep:`376` and :pep:`427`). The most popular Python-specific package
    manager is pip [#pip]_; other examples include the old Easy
    Install command [#easy-install]_ as well as direct usage of a
    ``setup.py`` command.

    (Conda [#conda]_ is a bit of a special case, as the ``conda``
    command can install much more than just Python packages, making it
    more like a distro package manager in some senses. Since the
    ``conda`` command generally only operates on Conda-created
    environments, most of the concerns in this document do not apply
    to ``conda`` when acting as a Python-specific package manager.)
distro package manager
    A tool for installing, upgrading, and/or removing a distro's
    packages in an installed instance of that distro, which is capable
    of installing Python packages as well as non-Python packages, and
    therefore generally has its own database of installed software
    unrelated to :pep:`376`. Examples include ``apt``, ``dpkg``, ``dnf``,
    ``rpm``, ``pacman``, and ``brew``. The salient feature is that if
    a package was installed by a distro package manager, removing or
    upgrading it in a way that would satisfy a Python-specific package
    manager will generally leave a distro package manager in an
    inconsistent state.

    This document also uses phrases like "external package manager" or
    "system's package manager" to refer to a distro package manager in
    certain contexts.
shadow
    To shadow an installed Python package is to cause some other
    package to be preferred for imports without removing any files
    from the shadowed package. This requires multiple entries on
    ``sys.path``: if package A 2.0 installs module ``a.py`` in one
    ``sys.path`` entry, and package A 1.0 installs module ``a.py`` in
    a later ``sys.path`` entry, then ``import a`` returns the module
    from the former, and we say that A 2.0 shadows A 1.0.

Motivation
==========

Thanks to Python's immense popularity, software distros (by which we
mean Linux and other OS distros as well as overlay distros like
Homebrew and MacPorts) generally ship Python for two purposes: as a
software package to be used in its own right by end users, and as a
language dependency for other software in the distro.

For example, Fedora and Debian (and their downstream distros, as well
as many others) ship a ``/usr/bin/python3`` binary which provides the
``python3`` command available to end users as well as the
``#!/usr/bin/python3`` shebang for Python-language software included
in the distro. Because there are no official binary releases of Python
for Linux/UNIX, almost all Python end users on these OSes use the
Python interpreter built and shipped with their distro.

The ``python3`` executable available to the users of the distro and
the ``python3`` executable available as a dependency for other
software in the distro are typically the same binary. This means that
if an end user installs a Python package using a tool like ``pip``
outside the context of a virtual environment, that package is visible
to Python-language software shipped by the distro. If the
newly-installed package (or one of its dependencies) is a newer,
backwards-incompat


id: pep-0697.rst
content:
PEP: 697
Title: Limited C API for Extending Opaque Types
Author: Petr Viktorin <encukou@gmail.com>
Discussions-To: https://discuss.python.org/t/19743
Status: Final
Type: Standards Track
Created: 23-Aug-2022
Python-Version: 3.12
Post-History: `24-May-2022 <https://mail.python.org/archives/list/capi-sig@python.org/thread/SIP3VP7JU4OBWP62KBOYGOYCVIOTXEFH/>`__,
              `06-Oct-2022 <https://discuss.python.org/t/19743>`__,
Resolution: https://discuss.python.org/t/19743/30


.. canonical-doc::
   :external+py3.12:c:member:`PyType_Spec.basicsize`,
   :external+py3.12:c:func:`PyObject_GetTypeData`,
   :external+py3.12:c:macro:`Py_TPFLAGS_ITEMS_AT_END`,
   :external+py3.12:c:macro:`Py_RELATIVE_OFFSET`,
   :external+py3.12:c:func:`PyObject_GetItemData`


Abstract
========

Add `Limited C API <https://docs.python.org/3.11/c-api/stable.html#stable-application-binary-interface>`__
support for extending some types with opaque data
by allowing code to only deal with data specific to a particular (sub)class.

This mechanism is required to be usable with ``PyHeapTypeObject``.

This PEP does not propose allowing to extend non-dynamically sized variable
sized objects such as ``tuple`` or ``int`` due to their different memory layout
and perceived lack of demand for doing so.  This PEP leaves room to do so in
the future via the same mechanism if ever desired.


Motivation
==========

The motivating problem this PEP solves is attaching C-level state
to custom types --- i.e. metaclasses (subclasses of
:py:class:`python:type`).

This is often needed in âwrappersâ that expose another type
system (e.g. C++, Java, Rust) as Python classes.
These typically need to attach information about the âwrappedâ non-Python
class to the Python type object.

This should be possible to do in the Limited API, so that the language wrappers
or code generators can be used to create Stable ABI extensions.
(See :pep:`652` for the benefits of providing a stable ABI.)

Extending ``type`` is an instance of a more general problem:
extending a class while maintaining loose coupling â that is,
not depending on the memory layout used by the superclass.
(That's a lot of jargon; see Rationale for a concrete example of extending
``list``.)


Rationale
=========

Extending opaque types
----------------------

In the Limited API, most ``struct``\ s are opaque: their size and memory layout
are not exposed, so they can be changed in new versions of CPython (or
alternate implementations of the C API).

This means that the usual subclassing pattern -- making the ``struct``
used for instances of the *base* type be the first element of the ``struct``
used for instances of the *derived* type -- does not work.
To illustrate with code, the `example from the tutorial <https://docs.python.org/3.11/extending/newtypes_tutorial.html#subclassing-other-types>`_
extends :external+python:c:type:`PyListObject` (:py:class:`python:list`)
using the following ``struct``:

.. code-block:: c

    typedef struct {
        PyListObject list;
        int state;
    } SubListObject;

This won't compile in the Limited API, since ``PyListObject`` is opaque (to
allow changes as features and optimizations are implemented).

Instead, this PEP proposes using a ``struct`` with only the state needed
in the subclass, that is:

.. code-block:: c

    typedef struct {
        int state;
    } SubListState;

    // (or just `typedef int SubListState;` in this case)

The subclass can now be completely decoupled from the memory layout (and size)
of the superclass.

This is possible today. To use such a struct:

* when creating the class, use ``PyListObject->tp_basicsize + sizeof(SubListState)``
  as ``PyType_Spec.basicsize``;
* when accessing the data, use ``PyListObject->tp_basicsize`` as the offset
  into the instance (``PyObject*``).

However, this has disadvantages:

* The base's ``basicsize`` may not be properly aligned, causing issues
  on some architectures if not mitigated. (These issues can be particularly
  nasty if alignment changes in a new release.)
* ``PyTypeObject.tp_basicsize`` is not exposed in the
  Limited API, so extensions that support Limited API need to
  use ``PyObject_GetAttrString(obj, "__basicsize__")``.
  This is cumbersome, and unsafe in edge cases (the Python attribute can
  be overridden).
* Variable-size objects are not handled
  (see `Extending variable-size objects`_ below).

To make this easy (and even *best practice* for projects that choose loose
coupling over maximum performance), this PEP proposes an API to:

1. During class creation, specify that ``SubListState``
   should be âappendedâ to ``PyListObject``, without passing any additional
   details about ``list``. (The interpreter itself gets all necessary info,
   like ``tp_basicsize``, from the base).

   This will be specified by a negative ``PyType_Spec.basicsize``:
   ``-sizeof(SubListState)``.

2. Given an instance, and the subclass ``PyTypeObject*``,
   get a pointer to the ``SubListState``.
   A new function, ``PyObject_GetTypeData``, will be added for this.

The base class is not limited to ``PyListObject``, of course: it can be used to
extend any base class whose instance ``struct`` is opaque, unstable across
releases, or not exposed at all -- including :py:class:`python:type`
(``PyHeapTypeObject``) or third-party extensions
(for example, NumPy arrays [#f1]_).

For cases where no additional state is needed, a zero ``basicsize`` will be
allowed: in that case, the base's ``tp_basicsize`` will be inherited.
(This currently works, but lacks explicit documentation and tests.)

The ``tp_basicsize`` of the new class will be set to the computed total size,
so code that inspects classes will continue working as before.


Extending variable-size objects
-------------------------------

Additional considerations are needed to subclass
:external+python:c:type:`variable-sized objects <PyVarObject>`
while maintaining loose coupling:
the variable-sized data can collide with subclass data (``SubListState`` in
the example above).

Currently, CPython doesn't provide a way to prevent such collisions.
So, the proposed mechanism of extending opaque classes (negative
``base->tp_itemsize``) will *fail* by default.

We could stop there, but since the motivating type --- ``PyHeapTypeObject`` ---
is variable sized, we need a safe way to allow subclassing it.
A bit of background first:

Variable-size layouts
.....................

There are two main memory layouts for variable-sized objects.

In types such as ``int`` or ``tuple``, the variable data is stored at a fixed
offset.
If subclasses need additional space, it must be added after any variable-sized
data::

   PyTupleObject:
   âââââââââââââââââââââ¬ââââ¬ââââ¬âââââ
   â PyObject_VAR_HEAD âvar. data   â
   âââââââââââââââââââââ´ââââ´ââââ´âââââ

   tuple subclass:
   âââââââââââââââââââââ¬ââââ¬ââââ¬âââââ¬ââââââââââââââ
   â PyObject_VAR_HEAD âvar. data   âsubclass dataâ
   âââââââââââââââââââââ´ââââ´ââââ´âââââ´ââââââââââââââ

In other types, like ``PyHeapTypeObject``, variable-sized data always lives at
the end of the instance's memory area::

   heap type:
   âââââââââââââââââââââ¬âââââââââââââââ¬ââââ¬ââââ¬âââââ
   â PyObject_VAR_HEAD âHeap type dataâvar. data   â
   âââââââââââââââââââââ´âââââââââââââââ´ââââ´ââââ´âââââ

   type subclass:
   âââââââââââââââââââââ¬âââââââââââââââ¬ââââââââââââââ¬


id: pep-0683.rst
content:
PEP: 683
Title: Immortal Objects, Using a Fixed Refcount
Author: Eric Snow <ericsnowcurrently@gmail.com>, Eddie Elizondo <eduardo.elizondorueda@gmail.com>
Discussions-To: https://discuss.python.org/t/18183
Status: Final
Type: Standards Track
Created: 10-Feb-2022
Python-Version: 3.12
Post-History: `16-Feb-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/TPLEYDCXFQ4AMTW6F6OQFINSIFYBRFCR/>`__,
              `19-Feb-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/KDAR6CCMPOX36GQJUDWHQBKRD5USNV3B/>`__,
              `28-Feb-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/MI22URMVKC63OFMZTALHFZKAKVGAT4UF/>`__,
              `12-Aug-2022 <https://discuss.python.org/t/18183>`__,
Resolution: https://discuss.python.org/t/18183/26

.. canonical-doc:: :term:`reference count`

PEP Acceptance Conditions
=========================

The PEP was accepted with conditions:

* the primary proposal in `Solutions for Accidental De-Immortalization`_
  (reset the immortal refcount in ``tp_dealloc()``) was applied
* types without this were not immortalized (in CPython's code)
* the PEP was updated with final benchmark results once
  the implementation is finalized (confirming the change is worthwhile)

Abstract
========

Currently the CPython runtime maintains a
`small amount of mutable state <Runtime Object State_>`_ in the
allocated memory of each object.  Because of this, otherwise immutable
objects are actually mutable.  This can have a large negative impact
on CPU and memory performance, especially for approaches to increasing
Python's scalability.

This proposal mandates that, internally, CPython will support marking
an object as one for which that runtime state will no longer change.
Consequently, such an object's refcount will never reach 0, and thus
the object will never be cleaned up (except when the runtime knows
it's safe to do so, like during runtime finalization).
We call these objects "immortal".  (Normally, only a relatively small
number of internal objects will ever be immortal.)
The fundamental improvement here is that now an object
can be truly immutable.

Scope
-----

Object immortality is meant to be an internal-only feature, so this
proposal does not include any changes to public API or behavior
(with one exception).  As usual, we may still add some private
(yet publicly accessible) API to do things like immortalize an object
or tell if one is immortal.  Any effort to expose this feature to users
would need to be proposed separately.

There is one exception to "no change in behavior": refcounting semantics
for immortal objects will differ in some cases from user expectations.
This exception, and the solution, are discussed below.

Most of this PEP focuses on an internal implementation that satisfies
the above mandate.  However, those implementation details are not meant
to be strictly proscriptive.  Instead, at the least they are included
to help illustrate the technical considerations required by the mandate.
The actual implementation may deviate somewhat as long as it satisfies
the constraints outlined below.  Furthermore, the acceptability of any
specific implementation detail described below does not depend on
the status of this PEP, unless explicitly specified.

For example, the particular details of:

* how to mark something as immortal
* how to recognize something as immortal
* which subset of functionally immortal objects are marked as immortal
* which memory-management activities are skipped or modified for immortal objects

are not only CPython-specific but are also private implementation
details that are expected to change in subsequent versions.

Implementation Summary
----------------------

Here's a high-level look at the implementation:

If an object's refcount matches a very specific value (defined below)
then that object is treated as immortal.  The CPython C-API and runtime
will not modify the refcount (or other runtime state) of an immortal
object.  The runtime will now be explicitly responsible for deallocating
all immortal objects during finalization, unless statically allocated.
(See `Object Cleanup`_ below.)

Aside from the change to refcounting semantics, there is one other
possible negative impact to consider.  The threshold for an "acceptable"
performance penalty for immortal objects is 2% (the consensus at the
2022 Language Summit).  A naive implementation of the approach described
below makes CPython roughly 4% slower.  However, the implementation
is ~performance-neutral~ once known mitigations are applied.

TODO: Update the performance impact for the latest branch
(both for GCC and for clang).


Motivation
==========

As noted above, currently all objects are effectively mutable.  That
includes "immutable" objects like ``str`` instances.  This is because
every object's refcount is frequently modified as the object is used
during execution.  This is especially significant for a number of
commonly used global (builtin) objects, e.g. ``None``.  Such objects
are used a lot, both in Python code and internally.  That adds up to
a consistent high volume of refcount changes.

The effective mutability of all Python objects has a concrete impact
on parts of the Python community, e.g. projects that aim for
scalability like Instagram or the effort to make the GIL
per-interpreter.  Below we describe several ways in which refcount
modification has a real negative effect on such projects.
None of that would happen for objects that are truly immutable.

Reducing CPU Cache Invalidation
-------------------------------

Every modification of a refcount causes the corresponding CPU cache
line to be invalidated.  This has a number of effects.

For one, the write must be propagated to other cache levels
and to main memory.  This has small effect on all Python programs.
Immortal objects would provide a slight relief in that regard.

On top of that, multi-core applications pay a price.  If two threads
(running simultaneously on distinct cores) are interacting with the
same object (e.g. ``None``)  then they will end up invalidating each
other's caches with each incref and decref.  This is true even for
otherwise immutable objects like ``True``, ``0``, and ``str`` instances.
CPython's GIL helps reduce this effect, since only one thread runs at a
time, but it doesn't completely eliminate the penalty.

Avoiding Data Races
-------------------

Speaking of multi-core, we are considering making the GIL
a per-interpreter lock, which would enable true multi-core parallelism.
Among other things, the GIL currently protects against races between
multiple concurrent threads that may incref or decref the same object.
Without a shared GIL, two running interpreters could not safely share
any objects, even otherwise immutable ones like ``None``.

This means that, to have a per-interpreter GIL, each interpreter must
have its own copy of *every* object.  That includes the singletons and
static types.  We have a viable strategy for that but it will require
a meaningful amount of extra effort and extra complexity.

The alternative is to ensure that all shared objects are truly immutable.
There would be no races because there would be no modification.  This
is something that the immortality proposed here would enable for
otherwise immutable objects.  With immortal objects,
support for a per-interpreter GIL
becomes much simpler.

Avoiding Copy-on-Write
----------------------

For some applications it makes sense to get the application into
a desired initial state and then fork the process for each worker.
This can result in a large performance improvement, especially
memory usage.  Several enterprise Python users (e.g. Instagram,
YouTube) have taken advantage of this.  However, the above
refcount semantics drastically reduce the benefits and
have led to some sub-optimal workarounds.

Also note that "fork" isn't the only operating system mechanism
that uses copy-on-write semantics.  Another example


id: pep-0456.rst
content:
PEP: 456
Title: Secure and interchangeable hash algorithm
Author: Christian Heimes <christian@python.org>
BDFL-Delegate: Alyssa Coghlan
Status: Final
Type: Standards Track
Created: 27-Sep-2013
Python-Version: 3.4
Post-History: 06-Oct-2013, 14-Nov-2013, 20-Nov-2013
Resolution: https://mail.python.org/pipermail/python-dev/2013-November/130400.html


Abstract
========

This PEP proposes SipHash as default string and bytes hash algorithm to properly
fix hash randomization once and for all. It also proposes modifications to
Python's C code in order to unify the hash code and to make it easily
interchangeable.


Rationale
=========

Despite the last attempt [issue13703]_ CPython is still vulnerable to hash
collision DoS attacks [29c3]_ [issue14621]_. The current hash algorithm and
its randomization is not resilient against attacks. Only a proper
cryptographic hash function prevents the extraction of secret randomization
keys. Although no practical attack against a Python-based service has been
seen yet, the weakness has to be fixed. Jean-Philippe Aumasson and Daniel
J. Bernstein have already shown how the seed for the current implementation
can be recovered [poc]_.

Furthermore, the current hash algorithm is hard-coded and implemented multiple
times for bytes and three different Unicode representations UCS1, UCS2 and
UCS4. This makes it impossible for embedders to replace it with a different
implementation without patching and recompiling large parts of the interpreter.
Embedders may want to choose a more suitable hash function.

Finally the current implementation code does not perform well. In the common
case it only processes one or two bytes per cycle. On a modern 64-bit processor
the code can easily be adjusted to deal with eight bytes at once.

This PEP proposes three major changes to the hash code for strings and bytes:

* SipHash [sip]_ is introduced as default hash algorithm. It is fast and small
  despite its cryptographic properties. Due to the fact that it was designed
  by well known security and crypto experts, it is safe to assume that its
  secure for the near future.

* The existing FNV code is kept for platforms without a 64-bit data type. The
  algorithm is optimized to process larger chunks per cycle.

* Calculation of the hash of strings and bytes is moved into a single API
  function instead of multiple specialized implementations in
  ``Objects/object.c`` and ``Objects/unicodeobject.c``. The function takes a
  void pointer plus length and returns the hash for it.

* The algorithm can be selected at compile time. FNV is guaranteed to exist
  on all platforms. SipHash is available on the majority of modern systems.


Requirements for a hash function
================================

* It MUST be able to hash arbitrarily large blocks of memory from 1 byte up
  to the maximum ``ssize_t`` value.

* It MUST produce at least 32 bits on 32-bit platforms and at least 64 bits
  on 64-bit platforms. (Note: Larger outputs can be compressed with e.g.
  ``v ^ (v >> 32)``.)

* It MUST support hashing of unaligned memory in order to support
  hash(memoryview).

* It is highly RECOMMENDED that the length of the input influences the
  outcome, so that ``hash(b'\00') != hash(b'\x00\x00')``.

The internal interface code between the hash function and the tp_hash slots
implements special cases for zero length input and a return value of ``-1``.
An input of length ``0`` is mapped to hash value ``0``. The output ``-1``
is mapped to ``-2``.


Current implementation with modified FNV
========================================

CPython currently uses a variant of the Fowler-Noll-Vo hash function
[fnv]_. The variant is has been modified to reduce the amount and cost of hash
collisions for common strings. The first character of the string is added
twice, the first time with a bit shift of 7. The length of the input
string is XOR-ed to the final value. Both deviations from the original FNV
algorithm reduce the amount of hash collisions for short strings.

Recently [issue13703]_ a random prefix and suffix were added as an attempt to
randomize the hash values. In order to protect the hash secret the code still
returns ``0`` for zero length input.

C code::

    Py_uhash_t x;
    Py_ssize_t len;
    /* p is either 1, 2 or 4 byte type */
    unsigned char *p;
    Py_UCS2 *p;
    Py_UCS4 *p;

    if (len == 0)
        return 0;
    x = (Py_uhash_t) _Py_HashSecret.prefix;
    x ^= (Py_uhash_t) *p << 7;
    for (i = 0; i < len; i++)
        x = (1000003 * x) ^ (Py_uhash_t) *p++;
    x ^= (Py_uhash_t) len;
    x ^= (Py_uhash_t) _Py_HashSecret.suffix;
    return x;


Which roughly translates to Python::

    def fnv(p):
        if len(p) == 0:
            return 0

        # bit mask, 2**32-1 or 2**64-1
        mask = 2 * sys.maxsize + 1

        x = hashsecret.prefix
        x = (x ^ (ord(p[0]) << 7)) & mask
        for c in p:
            x = ((1000003 * x) ^ ord(c)) & mask
        x = (x ^ len(p)) & mask
        x = (x ^ hashsecret.suffix) & mask

        if x == -1:
            x = -2

        return x


FNV is a simple multiply and XOR algorithm with no cryptographic properties.
The randomization was not part of the initial hash code, but was added as
counter measure against hash collision attacks as explained in oCERT-2011-003
[ocert]_. Because FNV is not a cryptographic hash algorithm and the dict
implementation is not fortified against side channel analysis, the
randomization secrets can be calculated by a remote attacker. The author of
this PEP strongly believes that the nature of a non-cryptographic hash
function makes it impossible to conceal the secrets.


Examined hashing algorithms
===========================

The author of this PEP has researched several hashing algorithms that are
considered modern, fast and state-of-the-art.

SipHash
-------

SipHash [sip]_ is a cryptographic pseudo random function with a 128-bit seed
and 64-bit output. It was designed by Jean-Philippe Aumasson and Daniel J.
Bernstein as a fast and secure keyed hash algorithm. It's used by Ruby, Perl,
OpenDNS, Rust, Redis, FreeBSD and more. The C reference implementation has
been released under CC0 license (public domain).

Quote from SipHash's site:

    SipHash is a family of pseudorandom functions (a.k.a. keyed hash
    functions) optimized for speed on short messages. Target applications
    include network traffic authentication and defense against hash-flooding
    DoS attacks.

siphash24 is the recommend variant with best performance. It uses 2 rounds per
message block and 4 finalization rounds. Besides the reference implementation
several other implementations are available. Some are single-shot functions,
others use a MerkleâDamgÃ¥rd construction-like approach with init, update and
finalize functions. Marek Majkowski C implementation csiphash [csiphash]_
defines the prototype of the function. (Note: ``k`` is split up into two
uint64_t)::

  uint64_t siphash24(const void *src, unsigned long src_sz, const char k[16])

SipHash requires a 64-bit data type and is not compatible with pure C89
platforms.


MurmurHash
----------

MurmurHash [murmur]_ is a family of non-cryptographic keyed hash function
developed by Austin Appleby. Murmur3 is the latest and fast variant of
MurmurHash. The C++ reference implementation has been released into public
domain. It features 32- or 128-bit output with a 32-bit seed. (Note: The out
parameter is a buffer with either 1 or 4 bytes.)

Murmur3's function prototypes are::

  void MurmurHash3_x86_32(const void *key, int len, uint32_t seed, void *out)

  void MurmurHash3_x86_128(const void *key, int len, uint32_t seed, void *out)

  void MurmurHash3_x64_128(const void *key, int len, uint32_t seed, void *out)

The 128-bit variants requires a 64-bit data type and are not compatible with
pure C89 platforms. The 32-bit variant is fully C89-compatible.

Aumasson, Bernstein and BoÃlet have shown [sip]_ [ocert-2012-001]_ that
Murmur3 is not resilient against ha


id: pep-0330.rst
content:
PEP: 330
Title: Python Bytecode Verification
Author: Michel Pelletier <michel@users.sourceforge.net>
Status: Rejected
Type: Standards Track
Created: 17-Jun-2004
Python-Version: 2.6
Post-History:


Abstract
========

If Python Virtual Machine (PVM) bytecode is not "well-formed" it
is possible to crash or exploit the PVM by causing various errors
such as under/overflowing the value stack or reading/writing into
arbitrary areas of the PVM program space.  Most of these kinds of
errors can be eliminated by verifying that PVM bytecode does not
violate a set of simple constraints before execution.

This PEP proposes a set of constraints on the format and structure
of Python Virtual Machine (PVM) bytecode and provides an
implementation in Python of this verification process.


Pronouncement
=============

Guido believes that a verification tool has some value.  If
someone wants to add it to ``Tools/scripts``, no PEP is required.

Such a tool may have value for validating the output from
"bytecodehacks" or from direct edits of PYC files.  As security
measure, its value is somewhat limited because perfectly valid
bytecode can still do horrible things.  That situation could
change if the concept of restricted execution were to be
successfully resurrected.


Motivation
==========

The Python Virtual Machine executes Python programs that have been
compiled from the Python language into a bytecode representation.
The PVM assumes that any bytecode being executed is "well-formed"
with regard to a number implicit constraints.  Some of these
constraints are checked at run-time, but most of them are not due
to the overhead they would create.

When running in debug mode the PVM does do several run-time checks
to ensure that any particular bytecode cannot violate these
constraints that, to a degree, prevent bytecode from crashing or
exploiting the interpreter.  These checks add a measurable
overhead to the interpreter, and are typically turned off in
common use.

Bytecode that is not well-formed and executed by a PVM not running
in debug mode may create a variety of fatal and non-fatal errors.
Typically, ill-formed code will cause the PVM to seg-fault and
cause the OS to immediately and abruptly terminate the
interpreter.

Conceivably, ill-formed bytecode could exploit the interpreter and
allow Python bytecode to execute arbitrary C-level machine
instructions or to modify private, internal data structures in the
interpreter.  If used cleverly this could subvert any form of
security policy an application may want to apply to its objects.

Practically, it would be difficult for a malicious user to
"inject" invalid bytecode into a PVM for the purposes of
exploitation, but not impossible.  Buffer overflow and memory
overwrite attacks are commonly understood, particularly when the
exploit payload is transmitted unencrypted over a network or when
a file or network security permission weakness is used as a
foothold for further attacks.

Ideally, no bytecode should ever be allowed to read or write
underlying C-level data structures to subvert the operation of the
PVM, whether the bytecode was maliciously crafted or not.  A
simple pre-execution verification step could ensure that bytecode
cannot over/underflow the value stack or access other sensitive
areas of PVM program space at run-time.

This PEP proposes several validation steps that should be taken on
Python bytecode before it is executed by the PVM so that it
compiles with static and structure constraints on its instructions
and their operands.  These steps are simple and catch a large
class of invalid bytecode that can cause crashes.  There is also
some possibility that some run-time checks can be eliminated up
front by a verification pass.

There is, of course, no way to verify that bytecode is "completely
safe", for every definition of complete and safe.  Even with
bytecode verification, Python programs can and most likely in the
future will seg-fault for a variety of reasons and continue to
cause many different classes of run-time errors, fatal or not.
The verification step proposed here simply plugs an easy hole that
can cause a large class of fatal and subtle errors at the bytecode
level.

Currently, the Java Virtual Machine (JVM) verifies Java bytecode
in a way very similar to what is proposed here.  The JVM
Specification version 2 [1]_, Sections 4.8 and 4.9 were therefore
used as a basis for some of the constraints explained below.  Any
Python bytecode verification implementation at a minimum must
enforce these constraints, but may not be limited to them.


Static Constraints on Bytecode Instructions
===========================================

1. The bytecode string must not be empty. (``len(co_code) > 0``).

2. The bytecode string cannot exceed a maximum size
   (``len(co_code) < sizeof(unsigned char) - 1``).

3. The first instruction in the bytecode string begins at index 0.

4. Only valid byte-codes with the correct number of operands can
   be in the bytecode string.


Static Constraints on Bytecode Instruction Operands
===================================================

1. The target of a jump instruction must be within the code
   boundaries and must fall on an instruction, never between an
   instruction and its operands.

2. The operand of a ``LOAD_*`` instruction must be a valid index into
   its corresponding data structure.

3. The operand of a ``STORE_*`` instruction must be a valid index
   into its corresponding data structure.


Structural Constraints between Bytecode Instructions
====================================================

1. Each instruction must only be executed with the appropriate
   number of arguments in the value stack, regardless of the
   execution path that leads to its invocation.

2. If an instruction can be executed along several different
   execution paths, the value stack must have the same depth prior
   to the execution of the instruction, regardless of the path
   taken.

3. At no point during execution can the value stack grow to a
   depth greater than that implied by ``co_stacksize``.

4. Execution never falls off the bottom of ``co_code``.


Implementation
==============

This PEP is the working document for a Python bytecode
verification implementation written in Python.  This
implementation is not used implicitly by the PVM before executing
any bytecode, but is to be used explicitly by users concerned
about possibly invalid bytecode with the following snippet::

    import verify
    verify.verify(object)

The ``verify`` module provides a ``verify`` function which accepts the
same kind of arguments as ``dis.dis``: classes, methods, functions,
or code objects.  It verifies that the object's bytecode is
well-formed according to the specifications of this PEP.

If the code is well-formed the call to ``verify`` returns silently
without error.  If an error is encountered, it throws a
``VerificationError`` whose argument indicates the cause of the
failure.  It is up to the programmer whether or not to handle the
error in some way or execute the invalid code regardless.

Phillip Eby has proposed a pseudo-code algorithm for bytecode
stack depth verification used by the reference implementation.


Verification Issues
===================

This PEP describes only a small number of verifications.  While
discussion and analysis will lead to many more, it is highly
possible that future verification may need to be done or custom,
project-specific verifications.  For this reason, it might be
desirable to add a verification registration interface to the test
implementation to register future verifiers.  The need for this is
minimal since custom verifiers can subclass and extend the current
implementation for added behavior.


Required Changes
================

Armin Rigo noted that several byte-codes will need modification in
order for their stack effect to be statically analyzed.  These are
``END_FINALLY``, ``POP_BLOCK``, and ``MAKE_CLOSURE``.  Armin and Guido have
already agr


id: pep-0324.rst
content:
PEP: 324
Title: subprocess - New process module
Author: Peter Astrand <astrand@lysator.liu.se>
Status: Final
Type: Standards Track
Created: 19-Nov-2003
Python-Version: 2.4
Post-History:


Abstract
========

This PEP describes a new module for starting and communicating
with processes.


Motivation
==========

Starting new processes is a common task in any programming
language, and very common in a high-level language like Python.
Good support for this task is needed, because:

- Inappropriate functions for starting processes could mean a
  security risk: If the program is started through the shell, and
  the arguments contain shell meta characters, the result can be
  disastrous. [1]_

- It makes Python an even better replacement language for
  over-complicated shell scripts.

Currently, Python has a large number of different functions for
process creation.  This makes it hard for developers to choose.

The subprocess module provides the following enhancements over
previous functions:

- One "unified" module provides all functionality from previous
  functions.

- Cross-process exceptions: Exceptions happening in the child
  before the new process has started to execute are re-raised in
  the parent.  This means that it's easy to handle ``exec()``
  failures, for example.  With popen2, for example, it's
  impossible to detect if the execution failed.

- A hook for executing custom code between fork and exec.  This
  can be used for, for example, changing uid.

- No implicit call of /bin/sh.  This means that there is no need
  for escaping dangerous shell meta characters.

- All combinations of file descriptor redirection is possible.
  For example, the "python-dialog" [2]_ needs to spawn a process
  and redirect stderr, but not stdout.  This is not possible with
  current functions, without using temporary files.

- With the subprocess module, it's possible to control if all open
  file descriptors should be closed before the new program is
  executed.

- Support for connecting several subprocesses (shell "pipe").

- Universal newline support.

- A ``communicate()`` method, which makes it easy to send stdin data
  and read stdout and stderr data, without risking deadlocks.
  Most people are aware of the flow control issues involved with
  child process communication, but not all have the patience or
  skills to write a fully correct and deadlock-free select loop.
  This means that many Python applications contain race
  conditions.  A ``communicate()`` method in the standard library
  solves this problem.


Rationale
=========

The following points summarizes the design:

- subprocess was based on popen2, which is tried-and-tested.

- The factory functions in popen2 have been removed, because I
  consider the class constructor equally easy to work with.

- popen2 contains several factory functions and classes for
  different combinations of redirection.  subprocess, however,
  contains one single class.  Since the subprocess module supports
  12 different combinations of redirection, providing a class or
  function for each of them would be cumbersome and not very
  intuitive.  Even with popen2, this is a readability problem.
  For example, many people cannot tell the difference between
  popen2.popen2 and popen2.popen4 without using the documentation.

- One small utility function is provided: ``subprocess.call()``. It
  aims to be an enhancement over ``os.system()``, while still very
  easy to use:

  - It does not use the Standard C function system(), which has
    limitations.

  - It does not call the shell implicitly.

  - No need for quoting; using an argument list.

  - The return value is easier to work with.

  The ``call()`` utility function accepts an 'args' argument, just
  like the ``Popen`` class constructor.  It waits for the command to
  complete, then returns the ``returncode`` attribute.  The
  implementation is very simple::

      def call(*args, **kwargs):
          return Popen(*args, **kwargs).wait()

  The motivation behind the ``call()`` function is simple: Starting a
  process and wait for it to finish is a common task.

  While ``Popen`` supports a wide range of options, many users have
  simple needs.  Many people are using ``os.system()`` today, mainly
  because it provides a simple interface.  Consider this example::

      os.system("stty sane -F " + device)

  With ``subprocess.call()``, this would look like::

      subprocess.call(["stty", "sane", "-F", device])

  or, if executing through the shell::

      subprocess.call("stty sane -F " + device, shell=True)

- The "preexec" functionality makes it possible to run arbitrary
  code between fork and exec.  One might ask why there are special
  arguments for setting the environment and current directory, but
  not for, for example, setting the uid.  The answer is:

  - Changing environment and working directory is considered
    fairly common.

  - Old functions like ``spawn()`` has support for an
    "env"-argument.

  - env and cwd are considered quite cross-platform: They make
    sense even on Windows.

- On POSIX platforms, no extension module is required: the module
  uses ``os.fork()``, ``os.execvp()`` etc.

- On Windows platforms, the module requires either Mark Hammond's
  Windows extensions [5]_, or a small extension module called
  _subprocess.


Specification
=============

This module defines one class called Popen::

    class Popen(args, bufsize=0, executable=None,
                stdin=None, stdout=None, stderr=None,
                preexec_fn=None, close_fds=False, shell=False,
                cwd=None, env=None, universal_newlines=False,
                startupinfo=None, creationflags=0):


Arguments are:

- ``args`` should be a string, or a sequence of program arguments.
  The program to execute is normally the first item in the args
  sequence or string, but can be explicitly set by using the
  executable argument.

  On UNIX, with ``shell=False`` (default): In this case, the ``Popen``
  class uses ``os.execvp()`` to execute the child program.  ``args``
  should normally be a sequence.  A string will be treated as a
  sequence with the string as the only item (the program to
  execute).

  On UNIX, with ``shell=True``: If ``args`` is a string, it specifies the
  command string to execute through the shell.  If ``args`` is a
  sequence, the first item specifies the command string, and any
  additional items will be treated as additional shell arguments.

  On Windows: the ``Popen`` class uses ``CreateProcess()`` to execute the
  child program, which operates on strings.  If ``args`` is a
  sequence, it will be converted to a string using the
  ``list2cmdline`` method.  Please note that not all MS Windows
  applications interpret the command line the same way: The
  ``list2cmdline`` is designed for applications using the same rules
  as the MS C runtime.

- ``bufsize``, if given, has the same meaning as the corresponding
  argument to the built-in ``open()`` function: 0 means unbuffered, 1
  means line buffered, any other positive value means use a buffer
  of (approximately) that size.  A negative ``bufsize`` means to use
  the system default, which usually means fully buffered.  The
  default value for ``bufsize`` is 0 (unbuffered).

- ``stdin``, ``stdout`` and ``stderr`` specify the executed programs' standard
  input, standard output and standard error file handles,
  respectively.  Valid values are ``PIPE``, an existing file
  descriptor (a positive integer), an existing file object, and
  ``None``.  ``PIPE`` indicates that a new pipe to the child should be
  created.  With ``None``, no redirection will occur; the child's file
  handles will be inherited from the parent.  Additionally, ``stderr``
  can be STDOUT, which indicates that the stderr data from the
  applications should be captured into the same file handle as for
  stdout.

- If ``preexec_fn`` is set to a callable object, this object will be
  called in the child process just befor


id: pep-0442.rst
content:
PEP: 442
Title: Safe object finalization
Author: Antoine Pitrou <solipsis@pitrou.net>
BDFL-Delegate: Benjamin Peterson <benjamin@python.org>
Status: Final
Type: Standards Track
Created: 18-May-2013
Python-Version: 3.4
Post-History: 18-May-2013
Resolution: https://mail.python.org/pipermail/python-dev/2013-June/126746.html


Abstract
========

This PEP proposes to deal with the current limitations of object
finalization.  The goal is to be able to define and run finalizers
for any object, regardless of their position in the object graph.

This PEP doesn't call for any change in Python code.  Objects
with existing finalizers will benefit automatically.


Definitions
===========

Reference
    A directional link from an object to another.  The target of the
    reference is kept alive by the reference, as long as the source is
    itself alive and the reference isn't cleared.

Weak reference
    A directional link from an object to another, which doesn't keep
    alive its target.  This PEP focuses on non-weak references.

Reference cycle
    A cyclic subgraph of directional links between objects, which keeps
    those objects from being collected in a pure reference-counting
    scheme.

Cyclic isolate (CI)
    A standalone subgraph of objects in which no object is referenced
    from the outside, containing one or several reference cycles, *and*
    whose objects are still in a usable, non-broken state: they can
    access each other from their respective finalizers.

Cyclic garbage collector (GC)
    A device able to detect cyclic isolates and turn them into cyclic
    trash.  Objects in cyclic trash are eventually disposed of by
    the natural effect of the references being cleared and their
    reference counts dropping to zero.

Cyclic trash (CT)
    A former cyclic isolate whose objects have started being cleared
    by the GC.  Objects in cyclic trash are potential zombies; if they
    are accessed by Python code, the symptoms can vary from weird
    AttributeErrors to crashes.

Zombie / broken object
    An object part of cyclic trash.  The term stresses that the object
    is not safe: its outgoing references may have been cleared, or one
    of the objects it references may be zombie.  Therefore,
    it should not be accessed by arbitrary code (such as finalizers).

Finalizer
    A function or method called when an object is intended to be
    disposed of.  The finalizer can access the object and release any
    resource held by the object (for example mutexes or file
    descriptors).  An example is a ``__del__`` method.

Resurrection
    The process by which a finalizer creates a new reference to an
    object in a CI.  This can happen as a quirky but supported
    side-effect of ``__del__`` methods.


Impact
======

While this PEP discusses CPython-specific implementation details, the
change in finalization semantics is expected to affect the Python
ecosystem as a whole.  In particular, this PEP obsoletes the current
guideline that "objects with a ``__del__`` method should not be part of a
reference cycle".


Benefits
========

The primary benefits of this PEP regard objects with finalizers, such
as objects with a ``__del__`` method and generators with a ``finally``
block.  Those objects can now be reclaimed when they are part of a
reference cycle.

The PEP also paves the way for further benefits:

* The module shutdown procedure may not need to set global variables to
  None anymore.  This could solve a well-known class of irritating issues.

The PEP doesn't change the semantics of:

* Weak references caught in reference cycles.

* C extension types with a custom ``tp_dealloc`` function.


Description
===========

Reference-counted disposal
--------------------------

In normal reference-counted disposal, an object's finalizer is called
just before the object is deallocated.  If the finalizer resurrects
the object, deallocation is aborted.

*However*, if the object was already finalized, then the finalizer isn't
called.  This prevents us from finalizing zombies (see below).

Disposal of cyclic isolates
---------------------------

Cyclic isolates are first detected by the garbage collector, and then
disposed of.  The detection phase doesn't change and won't be described
here.  Disposal of a CI traditionally works in the following order:

1. Weakrefs to CI objects are cleared, and their callbacks called. At
   this point, the objects are still safe to use.

2. The CI becomes a CT as the GC systematically breaks all
   known references inside it (using the ``tp_clear`` function).

3. Nothing.  All CT objects should have been disposed of in step 2
   (as a side-effect of clearing references); this collection is
   finished.

This PEP proposes to turn CI disposal into the following sequence (new
steps are in bold):

1. Weakrefs to CI objects are cleared, and their callbacks called. At
   this point, the objects are still safe to use.

2. **The finalizers of all CI objects are called.**

3. **The CI is traversed again to determine if it is still isolated.
   If it is determined that at least one object in CI is now reachable
   from outside the CI, this collection is aborted and the whole CI
   is resurrected.  Otherwise, proceed.**

4. The CI becomes a CT as the GC systematically breaks all
   known references inside it (using the ``tp_clear`` function).

5. Nothing.  All CT objects should have been disposed of in step 4
   (as a side-effect of clearing references); this collection is
   finished.

.. note::
    The GC doesn't recalculate the CI after step 2 above, hence the need
    for step 3 to check that the whole subgraph is still isolated.


C-level changes
===============

Type objects get a new ``tp_finalize`` slot to which ``__del__`` methods
are mapped (and reciprocally).  Generators are modified to use this slot,
rather than ``tp_del``.  A ``tp_finalize`` function is a normal C
function which will be called with a valid and alive ``PyObject`` as its
only argument.  It doesn't need to manipulate the object's reference count,
as this will be done by the caller.  However, it must ensure that the
original exception state is restored before returning to the caller.

For compatibility, ``tp_del`` is kept in the type structure.  Handling
of objects with a non-NULL ``tp_del`` is unchanged: when part of a CI,
they are not finalized and end up in ``gc.garbage``.  However, a non-NULL
``tp_del`` is not encountered anymore in the CPython source tree (except
for testing purposes).

Two new C API functions are provided to ease calling of ``tp_finalize``,
especially from custom deallocators.

On the internal side, a bit is reserved in the GC header for GC-managed
objects to signal that they were finalized.  This helps avoid finalizing
an object twice (and, especially, finalizing a CT object after it was
broken by the GC).

.. note::
    Objects which are not GC-enabled can also have a ``tp_finalize`` slot.
    They don't need the additional bit since their ``tp_finalize`` function
    can only be called from the deallocator: it therefore cannot be called
    twice, except when resurrected.


Discussion
==========

Predictability
--------------

Following this scheme, an object's finalizer is always called exactly
once, even if it was resurrected afterwards.

For CI objects, the order in which finalizers are called (step 2 above)
is undefined.

Safety
------

It is important to explain why the proposed change is safe.  There
are two aspects to be discussed:

* Can a finalizer access zombie objects (including the object being
  finalized)?

* What happens if a finalizer mutates the object graph so as to impact
  the CI?

Let's discuss the first issue.  We will divide possible cases in two
categories:

* If the object being finalized is part of the CI: by construction, no
  objects in CI are zombies yet, since CI finalizers are called before
  any reference breaking is done.  Therefore, the finalizer cannot
  access zombie objects, which don't exis


id: pep-0318.rst
content:
PEP: 318
Title: Decorators for Functions and Methods
Author: Kevin D. Smith <Kevin.Smith@theMorgue.org>, Jim J. Jewett, Skip Montanaro, Anthony Baxter
Status: Final
Type: Standards Track
Created: 05-Jun-2003
Python-Version: 2.4
Post-History: 09-Jun-2003, 10-Jun-2003, 27-Feb-2004, 23-Mar-2004, 30-Aug-2004,
              02-Sep-2004


WarningWarningWarning
=====================

This document is meant to describe the decorator syntax and the
process that resulted in the decisions that were made.  It does not
attempt to cover the huge number of potential alternative syntaxes,
nor is it an attempt to exhaustively list all the positives and
negatives of each form.


Abstract
========

The current method for transforming functions and methods (for instance,
declaring them as a class or static method) is awkward and can lead to
code that is difficult to understand.  Ideally, these transformations
should be made at the same point in the code where the declaration
itself is made.  This PEP introduces new syntax for transformations of a
function or method declaration.


Motivation
==========

The current method of applying a transformation to a function or method
places the actual transformation after the function body.  For large
functions this separates a key component of the function's behavior from
the definition of the rest of the function's external interface.  For
example::

    def foo(self):
        perform method operation
    foo = classmethod(foo)

This becomes less readable with longer methods.  It also seems less
than pythonic to name the function three times for what is conceptually
a single declaration.  A solution to this problem is to move the
transformation of the method closer to the method's own declaration.
The intent of the new syntax is to replace ::

    def foo(cls):
        pass
    foo = synchronized(lock)(foo)
    foo = classmethod(foo)

with an alternative that places the decoration in the function's
declaration::

    @classmethod
    @synchronized(lock)
    def foo(cls):
        pass

Modifying classes in this fashion is also possible, though the benefits
are not as immediately apparent.  Almost certainly, anything which could
be done with class decorators could be done using metaclasses, but
using metaclasses is sufficiently obscure that there is some attraction
to having an easier way to make simple modifications to classes.  For
Python 2.4, only function/method decorators are being added.

:pep:`3129` proposes to add class decorators as of Python 2.6.


Why Is This So Hard?
--------------------

Two decorators (``classmethod()`` and ``staticmethod()``) have been
available in Python since version 2.2.  It's been assumed since
approximately that time that some syntactic support for them would
eventually be added to the language.  Given this assumption, one might
wonder why it's been so difficult to arrive at a consensus.  Discussions
have raged off-and-on at times in both comp.lang.python and the
python-dev mailing list about how best to implement function decorators.
There is no one clear reason why this should be so, but a few problems
seem to be most divisive.

* Disagreement about where the "declaration of intent" belongs.
  Almost everyone agrees that decorating/transforming a function at the
  end of its definition is suboptimal.  Beyond that there seems to be no
  clear consensus where to place this information.

* Syntactic constraints.  Python is a syntactically simple language
  with fairly strong constraints on what can and can't be done without
  "messing things up" (both visually and with regards to the language
  parser).  There's no obvious way to structure this information so
  that people new to the concept will think, "Oh yeah, I know what
  you're doing."  The best that seems possible is to keep new users from
  creating a wildly incorrect mental model of what the syntax means.

* Overall unfamiliarity with the concept.  For people who have a
  passing acquaintance with algebra (or even basic arithmetic) or have
  used at least one other programming language, much of Python is
  intuitive.  Very few people will have had any experience with the
  decorator concept before encountering it in Python.  There's just no
  strong preexisting meme that captures the concept.

* Syntax discussions in general appear to cause more contention than
  almost anything else. Readers are pointed to the ternary operator
  discussions that were associated with :pep:`308` for another example of
  this.


Background
==========

There is general agreement that syntactic support is desirable to
the current state of affairs.  Guido mentioned `syntactic support
for decorators`_ in his DevDay keynote presentation at the `10th
Python Conference`_, though `he later said`_ it was only one of
several extensions he proposed there "semi-jokingly".  `Michael Hudson
raised the topic`_ on ``python-dev`` shortly after the conference,
attributing the initial bracketed syntax to an earlier proposal on
``comp.lang.python`` by `Gareth McCaughan`_.

.. _syntactic support for decorators:
   http://www.python.org/doc/essays/ppt/python10/py10keynote.pdf
.. _10th python conference:
   http://www.python.org/workshops/2002-02/
.. _michael hudson raised the topic:
   https://mail.python.org/pipermail/python-dev/2002-February/020005.html
.. _he later said:
   https://mail.python.org/pipermail/python-dev/2002-February/020017.html
.. _gareth mccaughan:
   http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=slrna40k88.2h9o.Gareth.McCaughan%40g.local

Class decorations seem like an obvious next step because class
definition and function definition are syntactically similar,
however Guido remains unconvinced, and class decorators will almost
certainly not be in Python 2.4.

The discussion continued on and off on python-dev from February
2002 through July 2004.  Hundreds and hundreds of posts were made,
with people proposing many possible syntax variations.  Guido took
a list of proposals to `EuroPython 2004`_, where a discussion took
place.  Subsequent to this, he decided that we'd have the `Java-style`_
@decorator syntax, and this appeared for the first time in 2.4a2.
Barry Warsaw named this the 'pie-decorator' syntax, in honor of the
Pie-thon Parrot shootout which occurred around the same time as
the decorator syntax, and because the @ looks a little like a pie.
Guido `outlined his case`_ on Python-dev, including `this piece`_
on some of the (many) rejected forms.

.. _EuroPython 2004:
    http://www.python.org/doc/essays/ppt/euro2004/euro2004.pdf
.. _outlined his case:
    https://mail.python.org/pipermail/python-dev/2004-August/author.html
.. _this piece:
    https://mail.python.org/pipermail/python-dev/2004-August/046672.html
..  _Java-style:
    http://java.sun.com/j2se/1.5.0/docs/guide/language/annotations.html


On the name 'Decorator'
=======================

There's been a number of complaints about the choice of the name
'decorator' for this feature.  The major one is that the name is not
consistent with its use in the `GoF book`_.  The name 'decorator'
probably owes more to its use in the compiler area -- a syntax tree is
walked and annotated.  It's quite possible that a better name may turn
up.

.. _GoF book:
    https://web.archive.org/web/20031204182047/http://patterndigest.com/patterns/Decorator.html


Design Goals
============

The new syntax should

* work for arbitrary wrappers, including user-defined callables and
  the existing builtins ``classmethod()`` and ``staticmethod()``.  This
  requirement also means that a decorator syntax must support passing
  arguments to the wrapper constructor

* work with multiple wrappers per definition

* make it obvious what is happening; at the very least it should be
  obvious that new users can safely ignore it when writing their own
  code

* be a syntax "that ... [is] easy to remember once explained"

* not make future extensions more difficult

* be easy to type; programs that use it are 


id: pep-0495.rst
content:
PEP: 495
Title: Local Time Disambiguation
Author: Alexander Belopolsky <alexander.belopolsky@gmail.com>, Tim Peters <tim.peters@gmail.com>
Discussions-To: datetime-sig@python.org
Status: Final
Type: Standards Track
Created: 02-Aug-2015
Python-Version: 3.6
Resolution: https://mail.python.org/pipermail/datetime-sig/2015-September/000900.html


Abstract
========

This PEP adds a new attribute ``fold`` to instances of the
``datetime.time`` and ``datetime.datetime`` classes that can be used
to differentiate between two moments in time for which local times are
the same.  The allowed values for the ``fold`` attribute will be 0 and 1
with 0 corresponding to the earlier and 1 to the later of the two
possible readings of an ambiguous local time.


Rationale
=========

In most world locations, there have been and will be times when
local clocks are moved back. [#]_ In those times, intervals are
introduced in which local clocks show the same time twice in the same
day.  In these situations, the information displayed on a local clock
(or stored in a Python datetime instance) is insufficient to identify
a particular moment in time.  The proposed solution is to add an
attribute to the ``datetime`` instances taking values of 0 and 1 that
will enumerate the two ambiguous times.

.. image:: pep-0495-daylightsavings.png
   :align: center
   :alt: A cartoon of a strong man struggling to stop the hands of a large clock.
         The caption reads: You can't stop time... but you can turn it back one
         hour at 2 a.m. Oct. 28 when daylight-saving time ends and standard time
         begins."
   :width: 30%


.. [#] People who live in locations observing the Daylight Saving
  Time (DST) move their clocks back (usually one hour) every Fall.

  It is less common, but occasionally clocks can be moved back for
  other reasons.  For example, Ukraine skipped the spring-forward
  transition in March 1990 and instead, moved their clocks back on
  July 1, 1990, switching from Moscow Time to Eastern European Time.
  In that case, standard (winter) time was in effect before and after
  the transition.

  Both DST and standard time changes may result in time shifts other
  than an hour.


Terminology
===========

When clocks are moved back, we say that a *fold* [#]_ is created in time.
When the clocks are moved forward, a *gap* is created.  A local time
that falls in the fold is called *ambiguous*.  A local time that falls
in the gap is called *missing*.

.. [#] The term "fall-backward fold" was invented in 1990s by Paul Eggert
  of UCLA who used it in various Internet discussions related to the C language
  standard that culminated in a `Defect Report #139`_.

.. _Defect Report #139: http://www.open-std.org/jtc1/sc22/wg14/docs/rr/dr_136.html



Proposal
========

The "fold" attribute
--------------------

We propose adding an attribute called ``fold`` to instances of the
``datetime.time`` and ``datetime.datetime`` classes.  This attribute
should have the value 0 for all instances except those that represent
the second (chronologically) moment in time in an ambiguous case. For
those instances, the value will be 1. [#]_

.. [#] An instance that has ``fold=1`` in a non-ambiguous case is
  said to represent an invalid time (or is invalid for short), but
  users are not prevented from creating invalid instances by passing
  ``fold=1`` to a constructor or to a ``replace()`` method.  This
  is similar to the current situation with the instances that fall in
  the spring-forward gap.  Such instances don't represent any valid
  time, but neither the constructors nor the ``replace()`` methods
  check whether the instances that they produce are valid.  Moreover,
  this PEP specifies how various functions should behave when given an
  invalid instance.


Affected APIs
-------------

Attributes
..........

Instances of ``datetime.time`` and ``datetime.datetime`` classes will
get a new attribute ``fold`` with two possible values: 0 and 1.

Constructors
............

The ``__new__`` methods of the ``datetime.time`` and
``datetime.datetime`` classes will get a new keyword-only argument
called ``fold`` with the default value 0.  The value of the
``fold`` argument will be used to initialize the value of the
``fold`` attribute in the returned instance.

Methods
.......

The ``replace()`` methods of the ``datetime.time`` and
``datetime.datetime`` classes will get a new keyword-only argument
called ``fold``.  It will behave similarly to the other ``replace()``
arguments: if the ``fold`` argument is specified and given a value 0
or 1, the new instance returned by ``replace()`` will have its
``fold`` attribute set to that value.  In CPython, any non-integer
value of ``fold`` will raise a ``TypeError``, but other
implementations may allow the value ``None`` to behave the same as
when ``fold`` is not given. [#]_  (This is
a nod to the existing difference in treatment of ``None`` arguments
in other positions of this method across Python implementations;
it is not intended to leave the door open for future alternative
interpretation of ``fold=None``.)  If the ``fold`` argument is not
specified, the original value of the ``fold`` attribute is copied to
the result.

.. [#] PyPy and pure Python implementation distributed with CPython
       already allow ``None`` to mean "no change to existing
       attribute" for all other attributes in ``replace()``.

C-API
.....

Access macros will be defined to extract the value of ``fold`` from
``PyDateTime_DateTime`` and ``PyDateTime_Time`` objects.

.. code::

  int PyDateTime_DATE_GET_FOLD(PyDateTime_DateTime *o)

Return the value of ``fold`` as a C ``int``.

.. code::

  int PyDateTime_TIME_GET_FOLD(PyDateTime_Time *o)

Return the value of ``fold`` as a C ``int``.

New constructors will be defined that will take an additional
argument to specify the value of ``fold`` in the created
instance:

.. code::

  PyObject* PyDateTime_FromDateAndTimeAndFold(
      int year, int month, int day, int hour, int minute,
      int second, int usecond, int fold)

Return a ``datetime.datetime`` object with the specified year, month,
day, hour, minute, second, microsecond and fold.

.. code::

  PyObject* PyTime_FromTimeAndFold(
      int hour, int minute, int second, int usecond, int fold)

Return a ``datetime.time`` object with the specified hour, minute,
second, microsecond and fold.


Affected Behaviors
------------------

What time is it?
................

The ``datetime.now()`` method called without arguments will set
``fold=1`` when returning the second of the two ambiguous times in a
system local time fold.  When called with a ``tzinfo`` argument, the
value of the ``fold`` will be determined by the ``tzinfo.fromutc()``
implementation.  When an instance of the ``datetime.timezone`` class
(the stdlib's fixed-offset ``tzinfo`` subclass,
*e.g.* ``datetime.timezone.utc``) is passed as ``tzinfo``, the
returned datetime instance will always have ``fold=0``.
The ``datetime.utcnow()`` method is unaffected.


Conversion from naive to aware
..............................

A new feature is proposed to facilitate conversion from naive datetime
instances to aware.

The ``astimezone()`` method will now work for naive ``self``.  The
system local timezone will be assumed in this case and the ``fold``
flag will be used to determine which local timezone is in effect
in the ambiguous case.

For example, on a system set to US/Eastern timezone::

  >>> dt = datetime(2014, 11, 2, 1, 30)
  >>> dt.astimezone().strftime('%D %T %Z%z')
  '11/02/14 01:30:00 EDT-0400'
  >>> dt.replace(fold=1).astimezone().strftime('%D %T %Z%z')
  '11/02/14 01:30:00 EST-0500'

An implication is that ``datetime.now(tz)`` is fully equivalent to
``datetime.now().astimezone(tz)`` (assuming ``tz`` is an instance of a
post-PEP ``tzinfo`` implementation, i.e. one that correctly handles
and sets ``fold``).


Conversion from POSIX seconds from EPOCH
........................................

The ``fromtimestamp()`` static method


id: pep-0481.rst
content:
PEP: 481
Title: Migrate CPython to Git, Github, and Phabricator
Author: Donald Stufft <donald@stufft.io>
Status: Withdrawn
Type: Process
Created: 29-Nov-2014
Post-History: 29-Nov-2014


Abstract
========

.. note:: This PEP has been withdrawn, if you're looking for the PEP
          documenting the move to Github, please refer to :pep:`512`.

This PEP proposes migrating the repository hosting of CPython and the
supporting repositories to Git and Github. It also proposes adding Phabricator
as an alternative to Github Pull Requests to handle reviewing changes. This
particular PEP is offered as an alternative to :pep:`474` and :pep:`462` which aims
to achieve the same overall benefits but restricts itself to tools that support
Mercurial and are completely Open Source.


Rationale
=========

CPython is an open source project which relies on a number of volunteers
donating their time. As an open source project it relies on attracting new
volunteers as well as retaining existing ones in order to continue to have
a healthy amount of manpower available. In addition to increasing the amount of
manpower that is available to the project, it also needs to allow for effective
use of what manpower *is* available.

The current toolchain of the CPython project is a custom and unique combination
of tools which mandates a workflow that is similar to one found in a lot of
older projects, but which is becoming less and less popular as time goes on.

The one-off nature of the CPython toolchain and workflow means that any new
contributor is going to need spend time learning the tools and workflow before
they can start contributing to CPython. Once a new contributor goes through
the process of learning the CPython workflow they also are unlikely to be able
to take that knowledge and apply it to future projects they wish to contribute
to. This acts as a barrier to contribution which will scare off potential new
contributors.

In addition the tooling that CPython uses is under-maintained, antiquated,
and it lacks important features that enable committers to more effectively use
their time when reviewing and approving changes. The fact that it is
under-maintained means that bugs are likely to last for longer, if they ever
get fixed, as well as it's more likely to go down for extended periods of time.
The fact that it is antiquated means that it doesn't effectively harness the
capabilities of the modern web platform. Finally the fact that it lacks several
important features such as a lack of pre-testing commits and the lack of an
automatic merge tool means that committers have to do needless busy work to
commit even the simplest of changes.


Version Control System
----------------------

The first decision that needs to be made is the VCS of the primary server side
repository. Currently the CPython repository, as well as a number of supporting
repositories, uses Mercurial. When evaluating the VCS we must consider the
capabilities of the VCS itself as well as the network effect and mindshare of
the community around that VCS.

There are really only two real options for this, Mercurial and Git. Between the
two of them the technical capabilities are largely equivalent. For this reason
this PEP will largely ignore the technical arguments about the VCS system and
will instead focus on the social aspects.

It is not possible to get exact numbers for the number of projects or people
which are using a particular VCS, however we can infer this by looking at
several sources of information for what VCS projects are using.

The Open Hub (previously Ohloh) statistics [#openhub-stats]_ show that 37% of
the repositories indexed by The Open Hub are using Git (second only to SVN
which has 48%) while Mercurial has just 2% (beating only bazaar which has 1%).
This has Git being just over 18 times as popular as Mercurial on The Open Hub.

Another source of information on the popular of the difference VCSs is PyPI
itself. This source is more targeted at the Python community itself since it
represents projects developed for Python. Unfortunately PyPI does not have a
standard location for representing this information, so this requires manual
processing. If we limit our search to the top 100 projects on PyPI (ordered
by download counts) we can see that 62% of them use Git while 22% of them use
Mercurial while 13% use something else. This has Git being just under 3 times
as popular as Mercurial for the top 100 projects on PyPI.

Obviously from these numbers Git is by far the more popular DVCS for open
source projects and choosing the more popular VCS has a number of positive
benefits.

For new contributors it increases the likelihood that they will have already
learned the basics of Git as part of working with another project or if they
are just now learning Git, that they'll be able to take that knowledge and
apply it to other projects. Additionally a larger community means more people
writing how to guides, answering questions, and writing articles about Git
which makes it easier for a new user to find answers and information about
the tool they are trying to learn.

Another benefit is that by nature of having a larger community, there will be
more tooling written *around* it. This increases options for everything from
GUI clients, helper scripts, repository hosting, etc.


Repository Hosting
------------------

This PEP proposes allowing GitHub Pull Requests to be submitted, however GitHub
does not have a way to submit Pull Requests against a repository that is not
hosted on GitHub. This PEP also proposes that in addition to GitHub Pull
Requests Phabricator's Differential app can also be used to submit proposed
changes and Phabricator *does* allow submitting changes against a repository
that is not hosted on Phabricator.

For this reason this PEP proposes using GitHub as the canonical location of
the repository with a read-only mirror located in Phabricator. If at some point
in the future GitHub is no longer desired, then repository hosting can easily
be moved to solely in Phabricator and the ability to accept GitHub Pull
Requests dropped.

In addition to hosting the repositories on Github, a read only copy of all
repositories will also be mirrored onto the PSF Infrastructure.


Code Review
-----------

Currently CPython uses a custom fork of Rietveld which has been modified to
not run on Google App Engine which is really only able to be maintained
currently by one person. In addition it is missing out on features that are
present in many modern code review tools.

This PEP proposes allowing both Github Pull Requests and Phabricator changes
to propose changes and review code. It suggests both so that contributors can
select which tool best enables them to submit changes, and reviewers can focus
on reviewing changes in the tooling they like best.


GitHub Pull Requests
~~~~~~~~~~~~~~~~~~~~

GitHub is a very popular code hosting site and is increasingly becoming the
primary place people look to contribute to a project. Enabling users to
contribute through GitHub is enabling contributors to contribute using tooling
that they are likely already familiar with and if they are not they are likely
to be able to apply to another project.

GitHub Pull Requests have a fairly major advantage over the older "submit a
patch to a bug tracker" model. It allows developers to work completely within
their VCS using standard VCS tooling so it does not require creating a patch
file and figuring out what the right location is to upload it to. This lowers
the barrier for sending a change to be reviewed.

On the reviewing side, GitHub Pull Requests are far easier to review, they have
nice syntax highlighted diffs which can operate in either unified or side by
side views. They allow expanding the context on a diff up to and including the
entire file. Finally they allow commenting inline and on the pull request as
a whole and they present that in a nice unified way which will also hide
comments which no longer apply. Gith


id: pep-0480.rst
content:
PEP: 480
Title: Surviving a Compromise of PyPI: End-to-end signing of packages
Author: Trishank Karthik Kuppusamy <karthik@trishank.com>,
        Vladimir Diaz <vladimir.diaz@nyu.edu>,
        Justin Cappos <jcappos@nyu.edu>, Marina Moore <mm9693@nyu.edu>
BDFL-Delegate: Donald Stufft <donald@stufft.io>
Discussions-To: https://discuss.python.org/t/5666
Status: Draft
Type: Standards Track
Topic: Packaging
Requires:  458
Created: 08-Oct-2014


Abstract
========

Proposed is an extension to :pep:`458` that adds support for end-to-end signing
and the maximum security model.  End-to-end signing allows both PyPI and
developers to sign for the distributions that are downloaded by clients.  The
minimum security model proposed by :pep:`458` supports continuous delivery of
distributions (because they are signed by online keys), but that model does not
protect distributions in the event that PyPI is compromised.  In the minimum
security model, attackers who have compromised the signing keys stored on PyPI
Infrastructure may sign for malicious distributions.   The maximum security model,
described in this PEP, retains the benefits of :pep:`458` (e.g., immediate
availability of distributions that are uploaded to PyPI), but additionally
ensures that end-users are not at risk of installing forged software if PyPI is
compromised.

This PEP requires some changes to the PyPI infrastructure, and some suggested
changes for developers who wish to participate in end-to-end signing. These
changes include updating the metadata layout from :pep:`458` to include delegations
to developer keys, adding a process to register developer keys with PyPI, and a
change in the upload workflow for developers who take advantage of end-to-end
signing. All of these changes are described in detail later in this PEP. Package
managers that wish to take advantage of end-to-end signing do not need to do any
additional work beyond what is required to consume metadata described in PEP
458.

This PEP discusses the changes made to :pep:`458` but excludes its informational
elements to primarily focus on the maximum security model. For example, an
overview of The Update Framework or the basic mechanisms in :pep:`458` are not
covered here. The changes to :pep:`458` include modifications to the snapshot
process, key compromise analysis, auditing snapshots, and the steps that should
be taken in the event of a PyPI compromise. The signing and key management
process that PyPI MAY RECOMMEND is discussed but not strictly defined. How the
release process should be implemented to manage keys and metadata is left to
the implementors of the signing tools. That is, this PEP delineates the
expected cryptographic key type and signature format included in metadata that
MUST be uploaded by developers in order to support end-to-end verification of
distributions.

PEP Status
==========

The community discussed this PEP from 2014 to 2018. Due to the amount
of work required to implement this PEP, discussion was deferred until
after approval for the precursor step in :pep:`458`. As of mid-2020 PEP
458 is approved and implementation is in progress, and the PEP authors
aim to gain approval so they can secure appropriate funding for
implementation.


Rationale
=========

:pep:`458` proposes how PyPI should be integrated with The Update Framework
(TUF) [2]_.  It explains how modern package managers like pip can be made more
secure, and the types of attacks that can be prevented if PyPI is modified on
the server side to include TUF metadata.  Package managers can reference the
TUF metadata available on PyPI to download distributions more securely.

:pep:`458` also describes the metadata layout of the PyPI repository and employs
the minimum security model, which supports continuous delivery of projects and
uses online cryptographic keys to sign the distributions uploaded by
developers.  Although the minimum security model guards against most attacks on
software updaters [5]_ [6]_, such as mix-and-match and extraneous dependencies
attacks, it can be improved to support end-to-end signing and to prohibit
forged distributions in the event that PyPI is compromised.

:pep:`480` builds on :pep:`458` by adding support for developer signing, and
reducing the reliance on online keys to prevent malicious distributions.
The main strength of :pep:`458` and the minimum security model is the automated
and simplified release process: developers may upload distributions and then
have PyPI sign for their distributions.  Much of the release process is handled
in an automated fashion by online roles and this approach requires storing
cryptographic signing keys on the PyPI infrastructure.  Unfortunately,
cryptographic keys that are stored online are vulnerable to theft.  The maximum
security model, proposed in this PEP, permits developers to sign for the
distributions that they make available to PyPI users, and does not put
end-users at risk of downloading malicious distributions if the online keys
stored on PyPI infrastructure are compromised.


Threat Model
============

The threat model assumes the following:

* Offline keys are safe and securely stored.

* Attackers can compromise at least one of PyPI's trusted keys that are stored
  online, and may do so at once or over a period of time.

* Attackers can respond to client requests.

* Attackers may control any number of developer keys for projects a client does
  not want to install.

Attackers are considered successful if they can cause a client to install (or
leave installed) something other than the most up-to-date version of the
software the client is updating. When an attacker is preventing the
installation of updates, the attacker's goal is that clients not realize that
anything is wrong.


Definitions
===========

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD",
"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be
interpreted as described in :rfc:`2119`.

This PEP focuses on integrating TUF with PyPI; however, the reader is
encouraged to read about TUF's design principles [2]_.  It is also RECOMMENDED
that the reader be familiar with the TUF specification [3]_, and :pep:`458`
(which this PEP is extending).

The following terms used in this PEP are defined in the Python Packaging
Glossary [4]_: *project*, *release*, *distribution*.

Terms used in this PEP are defined as follows:

* Distribution file: A versioned archive file that contains Python packages,
  modules, and other resource files that are used to distribute a release. The
  terms *distribution file*, *distribution package* [4]_, or simply
  *distribution* or *package* may be used interchangeably in this PEP.

* Simple index: The HTML page that contains internal links to distribution
  files.

* Target files: As a rule of thumb, target files are all files on PyPI whose
  integrity should be guaranteed with TUF. Typically, this includes
  distribution files, and PyPI metadata such as simple indices.

* Roles: Roles in TUF encompass the set of actions a party is authorized to
  perform, including what metadata they may sign and which packages they are
  responsible for. There is one *root* role in PyPI.  There are multiple roles
  whose responsibilities are delegated to them directly or indirectly by the
  *root* role. The term "top-level role" refers to the *root* role and any role
  delegated by the *root* role. Each role has a single metadata file that it is
  trusted to provide.

* Metadata: Metadata are files that describe roles, other metadata, and target
  files.

* Repository: A repository is a resource comprised of named metadata and target
  files.  Clients request metadata and target files stored on a repository.

* Consistent snapshot: A set of TUF metadata and target files that capture the
  complete state of all projects on PyPI as they existed at some fixed point in
  time.

* Developer: Either the owner or maintainer of a project who is allowed to
  update TUF


id: pep-0494.rst
content:
PEP: 494
Title: Python 3.6 Release Schedule
Author: Ned Deily <nad@python.org>
Status: Final
Type: Informational
Topic: Release
Created: 30-May-2015
Python-Version: 3.6


Abstract
========

This document describes the development and release schedule for
Python 3.6.  The schedule primarily concerns itself with PEP-sized
items.


Release Manager and Crew
========================

- 3.6 Release Manager: Ned Deily
- Windows installers: Steve Dower
- Mac installers: Ned Deily
- Documentation: Julien Palard, Georg Brandl


3.6 Lifespan
============

3.6 will receive `bugfix updates <https://devguide.python.org/devcycle/#maintenance-branches>`_
approximately every 3 months for about 24 months. Sometime after the release of
3.7.0 final, a final 3.6 bugfix update will be released.
After that, it is expected that
`security updates <https://devguide.python.org/devcycle/#security-branches>`_
(source only) will be released as needed until 5 years after
the release of 3.6 final, so until approximately 2021-12.

As of 2021-12-23, 3.6 has reached the
`end-of-life phase <https://devguide.python.org/devcycle/#end-of-life-branches>`_
of its release cycle. 3.6.15 was the final security release. The code base for
3.6 is now frozen and no further updates will be provided nor issues of any
kind will be accepted on the bug tracker.


Release Schedule
================

3.6.0 schedule
--------------

- 3.6 development begins: 2015-05-24
- 3.6.0 alpha 1: 2016-05-17
- 3.6.0 alpha 2: 2016-06-13
- 3.6.0 alpha 3: 2016-07-11
- 3.6.0 alpha 4: 2016-08-15
- 3.6.0 beta 1: 2016-09-12
  (No new features beyond this point.)
- 3.6.0 beta 2: 2016-10-10
- 3.6.0 beta 3: 2016-10-31
- 3.6.0 beta 4: 2016-11-21
- 3.6.0 candidate 1: 2016-12-06
- 3.6.0 candidate 2: 2016-12-16
- 3.6.0 final: 2016-12-23

3.6.1 schedule (first bugfix release)
-------------------------------------

- 3.6.1 candidate: 2017-03-05
- 3.6.1 final: 2017-03-21

3.6.2 schedule
--------------

- 3.6.2 candidate 1: 2017-06-17
- 3.6.2 candidate 2: 2017-07-07
- 3.6.2 final: 2017-07-17

3.6.3 schedule
--------------

- 3.6.3 candidate: 2017-09-19
- 3.6.3 final: 2017-10-03

3.6.4 schedule
--------------

- 3.6.4 candidate: 2017-12-05
- 3.6.4 final: 2017-12-19

3.6.5 schedule
--------------

- 3.6.5 candidate: 2018-03-13
- 3.6.5 final: 2018-03-28

3.6.6 schedule
--------------

- 3.6.6 candidate: 2018-06-12
- 3.6.6 final: 2018-06-27

3.6.7 schedule
--------------

- 3.6.7 candidate: 2018-09-26
- 3.6.7 candidate 2: 2018-10-13
- 3.6.7 final: 2018-10-20

3.6.8 schedule (last bugfix release)
------------------------------------

Last binary releases

- 3.6.8 candidate: 2018-12-11
- 3.6.8 final: 2018-12-24

3.6.9 schedule (first security-only release)
--------------------------------------------

Source only

- 3.6.9 candidate 1: 2019-06-18
- 3.6.9 final: 2019-07-02

3.6.10 schedule
---------------

- 3.6.10 candidate 1: 2019-12-11
- 3.6.10 final: 2019-12-18

3.6.11 schedule
---------------

- 3.6.11 candidate 1: 2020-06-15
- 3.6.11 final: 2020-06-27

3.6.12 schedule
---------------

- 3.6.12 final: 2020-08-17

3.6.13 schedule
---------------

- 3.6.13 final: 2021-02-15

3.6.14 schedule
---------------

- 3.6.14 final: 2021-06-28

3.6.15 schedule (last security-only release)
--------------------------------------------

- 3.6.15 final: 2021-09-04


Features for 3.6
================

Implemented changes for 3.6 (as of 3.6.0 beta 1):

* :pep:`468`, Preserving Keyword Argument Order
* :pep:`487`, Simpler customization of class creation
* :pep:`495`, Local Time Disambiguation
* :pep:`498`, Literal String Formatting
* :pep:`506`, Adding A Secrets Module To The Standard Library
* :pep:`509`, Add a private version to dict
* :pep:`515`, Underscores in Numeric Literals
* :pep:`519`, Adding a file system path protocol
* :pep:`520`, Preserving Class Attribute Definition Order
* :pep:`523`, Adding a frame evaluation API to CPython
* :pep:`524`, Make os.urandom() blocking on Linux (during system startup)
* :pep:`525`, Asynchronous Generators (provisional)
* :pep:`526`, Syntax for Variable Annotations (provisional)
* :pep:`528`, Change Windows console encoding to UTF-8 (provisional)
* :pep:`529`, Change Windows filesystem encoding to UTF-8 (provisional)
* :pep:`530`, Asynchronous Comprehensions


Copyright
=========

This document has been placed in the public domain.


id: pep-0319.rst
content:
PEP: 319
Title: Python Synchronize/Asynchronize Block
Author: Michel Pelletier <michel@users.sourceforge.net>
Status: Rejected
Type: Standards Track
Created: 24-Feb-2003
Python-Version: 2.4
Post-History:


Abstract
========

This PEP proposes adding two new keywords to Python, 'synchronize'
and 'asynchronize'.

Pronouncement
=============

This PEP is rejected in favor of :pep:`343`.

The 'synchronize' Keyword
    The concept of code synchronization in Python is too low-level.
    To synchronize code a programmer must be aware of the details of
    the following pseudo-code pattern::

        initialize_lock()

        ...

        acquire_lock()
        try:
            change_shared_data()
        finally:
            release_lock()

    This synchronized block pattern is not the only pattern (more
    discussed below) but it is very common.  This PEP proposes
    replacing the above code with the following equivalent::

        synchronize:
            change_shared_data()

    The advantages of this scheme are simpler syntax and less room for
    user error.  Currently users are required to write code about
    acquiring and releasing thread locks in 'try/finally' blocks;
    errors in this code can cause notoriously difficult concurrent
    thread locking issues.


The 'asynchronize' Keyword
    While executing a 'synchronize' block of code a programmer may
    want to "drop back" to running asynchronously momentarily to run
    blocking input/output routines or something else that might take an
    indeterminate amount of time and does not require synchronization.
    This code usually follows the pattern::

        initialize_lock()

        ...

        acquire_lock()
        try:
            change_shared_data()
            release_lock()             # become async
            do_blocking_io()
            acquire_lock()             # sync again
            change_shared_data2()

        finally:
            release_lock()

    The asynchronous section of the code is not very obvious visually,
    so it is marked up with comments.  Using the proposed
    'asynchronize' keyword this code becomes much cleaner, easier to
    understand, and less prone to error::

        synchronize:
            change_shared_data()

            asynchronize:
               do_blocking_io()

            change_shared_data2()

    Encountering an 'asynchronize' keyword inside a non-synchronized
    block can raise either an error or issue a warning (as all code
    blocks are implicitly asynchronous anyway).  It is important to
    note that the above example is **not** the same as::

        synchronize:
            change_shared_data()

        do_blocking_io()

        synchronize:
            change_shared_data2()

    Because both synchronized blocks of code may be running inside the
    same iteration of a loop, Consider::

        while in_main_loop():
            synchronize:
                change_shared_data()

                asynchronize:
                   do_blocking_io()

                change_shared_data2()

    Many threads may be looping through this code.  Without the
    'asynchronize' keyword one thread cannot stay in the loop and
    release the lock at the same time while blocking IO is going on.
    This pattern of releasing locks inside a main loop to do blocking
    IO is used extensively inside the CPython interpreter itself.


Synchronization Targets
=======================

As proposed the 'synchronize' and 'asynchronize' keywords
synchronize a block of code.  However programmers may want to
specify a target object that threads synchronize on.  Any object
can be a synchronization target.

Consider a two-way queue object: two different objects are used by
the same 'synchronize' code block to synchronize both queues
separately in the 'get' method::

    class TwoWayQueue:
        def __init__(self):
            self.front = []
            self.rear = []

        def putFront(self, item):
            self.put(item, self.front)

        def getFront(self):
            item = self.get(self.front)
            return item

        def putRear(self, item):
            self.put(item, self.rear)

        def getRear(self):
            item = self.get(self.rear)
            return item

        def put(self, item, queue):
            synchronize queue:
                queue.append(item)

        def get(self, queue):
            synchronize queue:
                item = queue[0]
                del queue[0]
                return item

Here is the equivalent code in Python as it is now without a
'synchronize' keyword::

    import thread

    class LockableQueue:

        def __init__(self):
            self.queue = []
            self.lock = thread.allocate_lock()

    class TwoWayQueue:
        def __init__(self):
            self.front = LockableQueue()
            self.rear = LockableQueue()

        def putFront(self, item):
            self.put(item, self.front)

        def getFront(self):
            item = self.get(self.front)
            return item

        def putRear(self, item):
            self.put(item, self.rear)

        def getRear(self):
            item = self.get(self.rear)
            return item

        def put(self, item, queue):
            queue.lock.acquire()
            try:
                queue.append(item)
            finally:
                queue.lock.release()

        def get(self, queue):
            queue.lock.acquire()
            try:
                item = queue[0]
                del queue[0]
                return item
            finally:
                queue.lock.release()

The last example had to define an extra class to associate a lock
with the queue where the first example the 'synchronize' keyword
does this association internally and transparently.


Other Patterns that Synchronize
===============================

There are some situations where the 'synchronize' and
'asynchronize' keywords cannot entirely replace the use of lock
methods like ``acquire`` and ``release``.  Some examples are if the
programmer wants to provide arguments for ``acquire`` or if a lock
is acquired in one code block but released in another, as shown
below.

Here is a class from Zope modified to use both the 'synchronize'
and 'asynchronize' keywords and also uses a pool of explicit locks
that are acquired and released in different code blocks and thus
don't use 'synchronize'::

    import thread
    from ZServerPublisher import ZServerPublisher

    class ZRendevous:

        def __init__(self, n=1):
            pool=[]
            self._lists=pool, [], []

            synchronize:
                while n > 0:
                    l=thread.allocate_lock()
                    l.acquire()
                    pool.append(l)
                    thread.start_new_thread(ZServerPublisher,
                                            (self.accept,))
                    n=n-1

        def accept(self):
            synchronize:
                pool, requests, ready = self._lists
                while not requests:
                    l=pool[-1]
                    del pool[-1]
                    ready.append(l)

                    asynchronize:
                        l.acquire()

                    pool.append(l)

                r=requests[0]
                del requests[0]
                return r

        def handle(self, name, request, response):
            synchronize:
                pool, requests, ready = self._lists
                requests.append((name, request, response))
                if ready:
                    l=ready[-1]
                    del ready[-1]
                    l.release()

Here is the original class as found in the
'Zope/ZServer/PubCore/ZRendevous.py' module.  The "convenience" of
the '_a' and '_r' shortcut names obscure the code::

    import thread
    from ZServerPublisher import ZServerPublisher

    class ZRendevous:

        def __init__(self, n=1):
            sync=thread.allocate_loc


id: pep-0325.rst
content:
PEP: 325
Title: Resource-Release Support for Generators
Author: Samuele Pedroni <pedronis@python.org>
Status: Rejected
Type: Standards Track
Created: 25-Aug-2003
Python-Version: 2.4
Post-History:


Abstract
========

Generators allow for natural coding and abstraction of traversal
over data.  Currently if external resources needing proper timely
release are involved, generators are unfortunately not adequate.
The typical idiom for timely release is not supported, a yield
statement is not allowed in the try clause of a try-finally
statement inside a generator.  The finally clause execution can be
neither guaranteed nor enforced.

This PEP proposes that the built-in generator type implement a
close method and destruction semantics, such that the restriction
on yield placement can be lifted, expanding the applicability of
generators.


Pronouncement
=============

Rejected in favor of :pep:`342` which includes substantially all of
the requested behavior in a more refined form.


Rationale
=========

Python generators allow for natural coding of many data traversal
scenarios.  Their instantiation produces iterators,
i.e. first-class objects abstracting traversal (with all the
advantages of first- classness).  In this respect they match in
power and offer some advantages over the approach using iterator
methods taking a (smalltalkish) block.  On the other hand, given
current limitations (no yield allowed in a try clause of a
try-finally inside a generator) the latter approach seems better
suited to encapsulating not only traversal but also exception
handling and proper resource acquisition and release.

Let's consider an example (for simplicity, files in read-mode are
used)::

    def all_lines(index_path):
        for path in file(index_path, "r"):
            for line in file(path.strip(), "r"):
                yield line

this is short and to the point, but the try-finally for timely
closing of the files cannot be added.  (While instead of a path, a
file, whose closing then would be responsibility of the caller,
could be passed in as argument, the same is not applicable for the
files opened depending on the contents of the index).

If we want timely release, we have to sacrifice the simplicity and
directness of the generator-only approach: (e.g.) ::

   class AllLines:

        def __init__(self, index_path):
            self.index_path = index_path
            self.index = None
            self.document = None

        def __iter__(self):
            self.index = file(self.index_path, "r")
            for path in self.index:
                self.document = file(path.strip(), "r")
            for line in self.document:
                yield line
            self.document.close()
            self.document = None

        def close(self):
            if self.index:
                self.index.close()
            if self.document:
                self.document.close()

to be used as::

    all_lines = AllLines("index.txt")
    try:
        for line in all_lines:
            ...
    finally:
        all_lines.close()

The more convoluted solution implementing timely release, seems
to offer a precious hint.  What we have done is encapsulate our
traversal in an object (iterator) with a close method.

This PEP proposes that generators should grow such a close method
with such semantics that the example could be rewritten as::

    # Today this is not valid Python: yield is not allowed between
    # try and finally, and generator type instances support no
    # close method.

    def all_lines(index_path):
        index = file(index_path, "r")
        try:
            for path in index:
                document = file(path.strip(), "r")
            try:
                for line in document:
                    yield line
            finally:
                document.close()
        finally:
            index.close()

    all = all_lines("index.txt")
    try:
        for line in all:
            ...
    finally:
        all.close() # close on generator

Currently :pep:`255` disallows yield inside a try clause of a
try-finally statement, because the execution of the finally clause
cannot be guaranteed as required by try-finally semantics.

The semantics of the proposed close method should be such that
while the finally clause execution still cannot be guaranteed, it
can be enforced when required.  Specifically, the close method
behavior should trigger the execution of the finally clauses
inside the generator, either by forcing a return in the generator
frame or by throwing an exception in it.  In situations requiring
timely resource release, close could then be explicitly invoked.

The semantics of generator destruction on the other hand should be
extended in order to implement a best-effort policy for the
general case.  Specifically, destruction should invoke ``close()``.
The best-effort limitation comes from the fact that the
destructor's execution is not guaranteed in the first place.

This seems to be a reasonable compromise, the resulting global
behavior being similar to that of files and closing.


Possible Semantics
==================

The built-in generator type should have a close method
implemented, which can then be invoked as::

    gen.close()

where ``gen`` is an instance of the built-in generator type.
Generator destruction should also invoke close method behavior.

If a generator is already terminated, close should be a no-op.

Otherwise, there are two alternative solutions, Return or
Exception Semantics:

A - Return Semantics: The generator should be resumed, generator
execution should continue as if the instruction at the re-entry
point is a return.  Consequently, finally clauses surrounding the
re-entry point would be executed, in the case of a then allowed
try-yield-finally pattern.

Issues: is it important to be able to distinguish forced
termination by close, normal termination, exception propagation
from generator or generator-called code?  In the normal case it
seems not, finally clauses should be there to work the same in all
these cases, still this semantics could make such a distinction
hard.

Except-clauses, like by a normal return, are not executed, such
clauses in legacy generators expect to be executed for exceptions
raised by the generator or by code called from it.  Not executing
them in the close case seems correct.

B - Exception Semantics: The generator should be resumed and
execution should continue as if a special-purpose exception
(e.g. CloseGenerator) has been raised at re-entry point.  Close
implementation should consume and not propagate further this
exception.

Issues: should ``StopIteration`` be reused for this purpose?  Probably
not.  We would like close to be a harmless operation for legacy
generators, which could contain code catching ``StopIteration`` to
deal with other generators/iterators.

In general, with exception semantics, it is unclear what to do if
the generator does not terminate or we do not receive the special
exception propagated back.  Other different exceptions should
probably be propagated, but consider this possible legacy
generator code::

    try:
        ...
        yield ...
        ...
    except: # or except Exception:, etc
        raise Exception("boom")

If close is invoked with the generator suspended after the yield,
the except clause would catch our special purpose exception, so we
would get a different exception propagated back, which in this
case ought to be reasonably consumed and ignored but in general
should be propagated, but separating these scenarios seems hard.

The exception approach has the advantage to let the generator
distinguish between termination cases and have more control.  On
the other hand, clear-cut semantics seem harder to define.


Remarks
=======

If this proposal is accepted, it should become common practice to
document whether a generator acquires resources, so that its close
method ought to be called.  If a generator is no longer used,
calling close should be


id: pep-0443.rst
content:
PEP: 443
Title: Single-dispatch generic functions
Author: Åukasz Langa <lukasz@python.org>
Discussions-To: python-dev@python.org
Status: Final
Type: Standards Track
Created: 22-May-2013
Python-Version: 3.4
Post-History: 22-May-2013, 25-May-2013, 31-May-2013
Replaces: 245, 246, 3124


Abstract
========

This PEP proposes a new mechanism in the ``functools`` standard library
module that provides a simple form of generic programming known as
single-dispatch generic functions.

A **generic function** is composed of multiple functions implementing
the same operation for different types. Which implementation should be
used during a call is determined by the dispatch algorithm. When the
implementation is chosen based on the type of a single argument, this is
known as **single dispatch**.


Rationale and Goals
===================

Python has always provided a variety of built-in and standard-library
generic functions, such as ``len()``, ``iter()``, ``pprint.pprint()``,
``copy.copy()``, and most of the functions in the ``operator`` module.
However, it currently:

1. does not have a simple or straightforward way for developers to
   create new generic functions,

2. does not have a standard way for methods to be added to existing
   generic functions (i.e., some are added using registration
   functions, others require defining ``__special__`` methods, possibly
   by monkeypatching).

In addition, it is currently a common anti-pattern for Python code to
inspect the types of received arguments, in order to decide what to do
with the objects.

For example, code may wish to accept either an object
of some type, or a sequence of objects of that type.
Currently, the "obvious way" to do this is by type inspection, but this
is brittle and closed to extension.

Abstract Base Classes make it easier
to discover present behaviour, but don't help adding new behaviour.
A developer using an already-written library may be unable to change how
their objects are treated by such code, especially if the objects they
are using were created by a third party.

Therefore, this PEP proposes a uniform API to address dynamic
overloading using decorators.


User API
========

To define a generic function, decorate it with the ``@singledispatch``
decorator. Note that the dispatch happens on the type of the first
argument. Create your function accordingly::

  >>> from functools import singledispatch
  >>> @singledispatch
  ... def fun(arg, verbose=False):
  ...     if verbose:
  ...         print("Let me just say,", end=" ")
  ...     print(arg)

To add overloaded implementations to the function, use the
``register()`` attribute of the generic function. This is a decorator,
taking a type parameter and decorating a function implementing the
operation for that type::

  >>> @fun.register(int)
  ... def _(arg, verbose=False):
  ...     if verbose:
  ...         print("Strength in numbers, eh?", end=" ")
  ...     print(arg)
  ...
  >>> @fun.register(list)
  ... def _(arg, verbose=False):
  ...     if verbose:
  ...         print("Enumerate this:")
  ...     for i, elem in enumerate(arg):
  ...         print(i, elem)

To enable registering lambdas and pre-existing functions, the
``register()`` attribute can be used in a functional form::

  >>> def nothing(arg, verbose=False):
  ...     print("Nothing.")
  ...
  >>> fun.register(type(None), nothing)

The ``register()`` attribute returns the undecorated function. This
enables decorator stacking, pickling, as well as creating unit tests for
each variant independently::

  >>> @fun.register(float)
  ... @fun.register(Decimal)
  ... def fun_num(arg, verbose=False):
  ...     if verbose:
  ...         print("Half of your number:", end=" ")
  ...     print(arg / 2)
  ...
  >>> fun_num is fun
  False

When called, the generic function dispatches on the type of the first
argument::

  >>> fun("Hello, world.")
  Hello, world.
  >>> fun("test.", verbose=True)
  Let me just say, test.
  >>> fun(42, verbose=True)
  Strength in numbers, eh? 42
  >>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True)
  Enumerate this:
  0 spam
  1 spam
  2 eggs
  3 spam
  >>> fun(None)
  Nothing.
  >>> fun(1.23)
  0.615

Where there is no registered implementation for a specific type, its
method resolution order is used to find a more generic implementation.
The original function decorated with ``@singledispatch`` is registered
for the base ``object`` type, which means it is used if no better
implementation is found.

To check which implementation will the generic function choose for
a given type, use the ``dispatch()`` attribute::

  >>> fun.dispatch(float)
  <function fun_num at 0x104319058>
  >>> fun.dispatch(dict)    # note: default implementation
  <function fun at 0x103fe0000>

To access all registered implementations, use the read-only ``registry``
attribute::

  >>> fun.registry.keys()
  dict_keys([<class 'NoneType'>, <class 'int'>, <class 'object'>,
            <class 'decimal.Decimal'>, <class 'list'>,
            <class 'float'>])
  >>> fun.registry[float]
  <function fun_num at 0x1035a2840>
  >>> fun.registry[object]
  <function fun at 0x103fe0000>

The proposed API is intentionally limited and opinionated, as to ensure
it is easy to explain and use, as well as to maintain consistency with
existing members in the ``functools`` module.


Implementation Notes
====================

The functionality described in this PEP is already implemented in the
``pkgutil`` standard library module as ``simplegeneric``. Because this
implementation is mature, the goal is to move it largely as-is. The
reference implementation is available on hg.python.org [#ref-impl]_.

The dispatch type is specified as a decorator argument. An alternative
form using function annotations was considered but its inclusion
has been rejected. As of May 2013, this usage pattern is out of scope
for the standard library [#pep-0008]_, and the best practices for
annotation usage are still debated.

Based on the current ``pkgutil.simplegeneric`` implementation, and
following the convention on registering virtual subclasses on Abstract
Base Classes, the dispatch registry will not be thread-safe.

Abstract Base Classes
---------------------

The ``pkgutil.simplegeneric`` implementation relied on several forms of
method resolution order (MRO). ``@singledispatch`` removes special
handling of old-style classes and Zope's ExtensionClasses. More
importantly, it introduces support for Abstract Base Classes (ABC).

When a generic function implementation is registered for an ABC, the
dispatch algorithm switches to an extended form of C3 linearization,
which includes the relevant ABCs in the MRO of the provided argument.
The algorithm inserts ABCs where their functionality is introduced, i.e.
``issubclass(cls, abc)`` returns ``True`` for the class itself but
returns ``False`` for all its direct base classes. Implicit ABCs for
a given class (either registered or inferred from the presence of
a special method like ``__len__()``) are inserted directly after the
last ABC explicitly listed in the MRO of said class.

In its most basic form, this linearization returns the MRO for the given
type::

  >>> _compose_mro(dict, [])
  [<class 'dict'>, <class 'object'>]

When the second argument contains ABCs that the specified type is
a subclass of, they are inserted in a predictable order::

  >>> _compose_mro(dict, [Sized, MutableMapping, str,
  ...                     Sequence, Iterable])
  [<class 'dict'>, <class 'collections.abc.MutableMapping'>,
   <class 'collections.abc.Mapping'>, <class 'collections.abc.Sized'>,
   <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>,
   <class 'object'>]

While this mode of operation is significantly slower, all dispatch
decisions are cached. The cache is invalidated on registering new
implementations on the generic function or when user code calls
``register()`` on an ABC to implicitly subclass it. In the latter case,
it is possible to create a situation with am


id: pep-0457.rst
content:
PEP: 457
Title: Notation For Positional-Only Parameters
Author: Larry Hastings <larry@hastings.org>
Discussions-To: python-dev@python.org
Status: Final
Type: Informational
Created: 08-Oct-2013


========
Overview
========

This PEP proposes a notation for positional-only parameters in Python.
Positional-only parameters are parameters without an externally-usable
name; when a function accepting positional-only parameters is called,
positional arguments are mapped to these parameters based solely on
their position.

This PEP is an Informational PEP describing the notation for use when
describing APIs that use positional-only parameters (e.g. in Argument
Clinic, or in the string representation of ``inspect.Signature``
objects). A separate PEP, :pep:`570`, proposes elevation of this notation
to full Python syntax.

=========
Rationale
=========

Python has always supported positional-only parameters.
Early versions of Python lacked the concept of specifying
parameters by name, so naturally all parameters were
positional-only.  This changed around Python 1.0, when
all parameters suddenly became positional-or-keyword.
But, even in current versions of Python, many CPython
"builtin" functions still only accept positional-only
arguments.

Functions implemented in modern Python can accept
an arbitrary number of positional-only arguments, via the
variadic ``*args`` parameter.  However, there is no Python
syntax to specify accepting a specific number of
positional-only parameters.  Put another way, there are
many builtin functions whose signatures are simply not
expressible with Python syntax.

This PEP proposes a notation for such signatures that could form the
basis of a backwards-compatible syntax that should permit implementing
any builtin in pure Python code (see :pep:`570` for that proposal).

-----------------------------------------------------
Positional-Only Parameter Semantics In Current Python
-----------------------------------------------------

There are many, many examples of builtins that only
accept positional-only parameters.  The resulting
semantics are easily experienced by the Python
programmer--just try calling one, specifying its
arguments by name::

    >>> pow(x=5, y=3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: pow() takes no keyword arguments

In addition, there are some functions with particularly
interesting semantics:

* ``range()``, which accepts an optional parameter
  to the *left* of its required parameter. [#RANGE]_

* ``dict()``, whose mapping/iterator parameter is optional and
  semantically must be positional-only.  Any externally
  visible name for this parameter would occlude
  that name going into the ``**kwarg`` keyword variadic
  parameter dict! [#DICT]_

Obviously one can simulate any of these in pure Python code
by accepting ``(*args, **kwargs)`` and parsing the arguments
by hand.  But this results in a disconnect between the
Python function's signature and what it actually accepts,
not to mention the work of implementing said argument parsing.

==========
Motivation
==========

This PEP does not propose we implement positional-only
parameters in Python.  The goal of this PEP is simply
to define the syntax, so that:

* Documentation can clearly, unambiguously, and
  consistently express exactly how the arguments
  for a function will be interpreted.

* The syntax is reserved for future use, in case
  the community decides someday to add positional-only
  parameters to the language.

* Argument Clinic can use a variant of the syntax
  as part of its input when defining
  the arguments for built-in functions.

=================================================================
The Current State Of Documentation For Positional-Only Parameters
=================================================================

The documentation for positional-only parameters is incomplete
and inconsistent:

* Some functions denote optional *groups* of positional-only arguments
  by enclosing them in nested square brackets. [#BORDER]_

* Some functions denote optional groups of positional-only arguments
  by presenting multiple prototypes with varying numbers of
  arguments. [#SENDFILE]_

* Some functions use *both* of the above approaches. [#RANGE]_ [#ADDCH]_

One more important idea to consider: currently in the documentation
there's no way to tell whether a function takes positional-only
parameters.  ``open()`` accepts keyword arguments, ``ord()`` does
not, but there is no way of telling just by reading the
documentation that this is true.

====================
Syntax And Semantics
====================

From the "ten-thousand foot view", and ignoring ``*args`` and ``**kwargs``
for now, the grammar for a function definition currently looks like this::

    def name(positional_or_keyword_parameters, *, keyword_only_parameters):

Building on that perspective, the new syntax for functions would look
like this::

    def name(positional_only_parameters, /, positional_or_keyword_parameters,
             *, keyword_only_parameters):

All parameters before the ``/`` are positional-only.  If ``/`` is
not specified in a function signature, that function does not
accept any positional-only parameters.

Positional-only parameters can have a default value, and if they
do they are optional.  Positional-only parameters that don't have
a default value are "required" positional-only parameters.

More semantics of positional-only parameters:

* Although positional-only parameter technically have names,
  these names are internal-only; positional-only parameters
  are *never* externally addressable by name.  (Similarly
  to ``*args`` and ``**kwargs``.)

* If there are arguments after the ``/``, then you must specify
  a comma after the ``/``, just as there is a comma
  after the ``*`` denoting the shift to keyword-only parameters.

* This syntax has no effect on ``*args`` or ``**kwargs``.

======================
Additional Limitations
======================

Argument Clinic uses a form of this syntax for specifying
builtins.  It imposes further limitations that are
theoretically unnecessary but make the implementation
easier.  Specifically:

* A function that has positional-only parameters currently
  cannot have any other kind of parameter.  (This will
  probably be relaxed slightly in the near future.)

* Argument Clinic supports an additional syntax called
  "optional groups".  An "optional group" is a sequential
  set of positional-only parameters that must be specified
  or not-specified as a group.  If, for example, you define
  a function in Argument Clinic that takes four parameters,
  and all of them are positional-only and in one optional
  group, then when calling the function you must specify
  either zero arguments or four arguments.  This is necessary
  to cover more of Python's legacy library, but is outside
  the scope of this PEP, and is not recommended for actual
  inclusion in the Python language.

==============================
Notes For A Future Implementor
==============================

If we decide to implement positional-only parameters in a future
version of Python, we'd have to do some additional work to preserve
their semantics.  The problem: how do we inform a parameter that
no value was passed in for it when the function was called?

The obvious solution: add a new singleton constant to Python
that is passed in when a parameter is not mapped to an argument.
I propose that the value be called ``undefined``,
and be a singleton of a special class called ``Undefined``.
If a positional-only parameter did not receive an argument
when called, its value would be set to ``undefined``.

But this raises a further problem.  How do can we tell the
difference between "this positional-only parameter did not
receive an argument" and "the caller passed in ``undefined``
for this parameter"?

It'd be nice to make it illegal to pass ``undefined`` in
as an argument to a function--to, say, raise


id: pep-0331.rst
content:
PEP: 331
Title: Locale-Independent Float/String Conversions
Author: Christian R. Reis <kiko at async.com.br>
Status: Final
Type: Standards Track
Created: 19-Jul-2003
Python-Version: 2.4
Post-History: 21-Jul-2003, 13-Aug-2003, 18-Jun-2004


Abstract
========

Support for the ``LC_NUMERIC`` locale category in Python 2.3 is
implemented only in Python-space.  This causes inconsistent
behavior and thread-safety issues for applications that use
extension modules and libraries implemented in C that parse and
generate floats from strings.  This document proposes a plan for
removing this inconsistency by providing and using substitute
locale-agnostic functions as necessary.


Introduction
============

Python provides generic localization services through the locale
module, which among other things allows localizing the display and
conversion process of numeric types.  Locale categories, such as
``LC_TIME`` and ``LC_COLLATE``, allow configuring precisely what aspects
of the application are to be localized.

The ``LC_NUMERIC`` category specifies formatting for non-monetary
numeric information, such as the decimal separator in float and
fixed-precision numbers.  Localization of the ``LC_NUMERIC`` category
is currently implemented only in Python-space; C libraries invoked
from the Python runtime are unaware of Python's ``LC_NUMERIC``
setting.  This is done to avoid changing the behavior of certain
low-level functions that are used by the Python parser and related
code [2]_.

However, this presents a problem for extension modules that wrap C
libraries.  Applications that use these extension modules will
inconsistently display and convert floating-point values.

James Henstridge, the author of PyGTK [3]_, has additionally
pointed out that the ``setlocale()`` function also presents
thread-safety issues, since a thread may call the C library
``setlocale()`` outside of the GIL, and cause Python to parse and
generate floats incorrectly.


Rationale
=========

The inconsistency between Python and C library localization for
``LC_NUMERIC`` is a problem for any localized application using C
extensions.  The exact nature of the problem will vary depending
on the application, but it will most likely occur when parsing or
formatting a floating-point value.


Example Problem
===============

The initial problem that motivated this PEP is related to the
GtkSpinButton [4]_ widget in the GTK+ UI toolkit, wrapped by the
PyGTK module.  The widget can be set to numeric mode, and when
this occurs, characters typed into it are evaluated as a number.

Problems occur when ``LC_NUMERIC`` is set to a locale with a float
separator that differs from the C locale's standard (for instance,
',' instead of '.' for the Brazilian locale pt_BR).  Because
``LC_NUMERIC`` is not set at the libc level, float values are
displayed incorrectly (using '.' as a separator) in the
spinbutton's text entry, and it is impossible to enter fractional
values using the ',' separator.

This small example demonstrates reduced usability for localized
applications using this toolkit when coded in Python.


Proposal
========

Martin v. LÃ¶wis commented on the initial constraints for an
acceptable solution to the problem on python-dev:

- ``LC_NUMERIC`` can be set at the C library level without
  breaking the parser.
- ``float()`` and ``str()`` stay locale-unaware.
- locale-aware ``str()`` and ``atof()`` stay in the locale module.

An analysis of the Python source suggests that the following
functions currently depend on ``LC_NUMERIC`` being set to the C
locale:

- ``Python/compile.c:parsenumber()``
- ``Python/marshal.c:r_object()``
- ``Objects/complexobject.c:complex_to_buf()``
- ``Objects/complexobject.c:complex_subtype_from_string()``
- ``Objects/floatobject.c:PyFloat_FromString()``
- ``Objects/floatobject.c:format_float()``
- ``Objects/stringobject.c:formatfloat()``
- ``Modules/stropmodule.c:strop_atof()``
- ``Modules/cPickle.c:load_float()``

The proposed approach is to implement ``LC_NUMERIC``-agnostic
functions for converting from (``strtod()``/``atof()``) and to
(``snprintf()``) float formats, using these functions where the
formatting should not vary according to the user-specified locale.

The locale module should also be changed to remove the
special-casing for ``LC_NUMERIC``.

This change should also solve the aforementioned thread-safety
problems.


Potential Code Contributions
============================

This problem was initially reported as a problem in the GTK+
libraries [5]_; since then it has been correctly diagnosed as an
inconsistency in Python's implementation.  However, in a fortunate
coincidence, the glib library (developed primarily for GTK+, not
to be confused with the GNU C library) implements a number of
``LC_NUMERIC``-agnostic functions (for an example, see [6]_) for
reasons similar to those presented in this paper.

In the same GTK+ problem report, Havoc Pennington suggested that
the glib authors would be willing to contribute this code to the
PSF, which would simplify implementation of this PEP considerably.
Alex Larsson, the original author of the glib code, submitted a
PSF Contributor Agreement [7]_ on 2003-08-20 [8]_ to ensure the code
could be safely integrated; this agreement has been received and
accepted.


Risks
=====

There may be cross-platform issues with the provided
locale-agnostic functions, though this risk is low given that the
code supplied simply reverses any locale-dependent changes made to
floating-point numbers.

Martin and Guido pointed out potential copyright issues with the
contributed code.  I believe we will have no problems in this area
as members of the GTK+ and glib teams have said they are fine with
relicensing the code, and a PSF contributor agreement has been
mailed in to ensure this safety.

Tim Peters has pointed out [9]_ that there are situations involving
threading in which the proposed change is insufficient to solve
the problem completely.  A complete solution, however, does not
currently exist.


Implementation
==============

An implementation was developed by Gustavo Carneiro <gjc at
inescporto.pt>, and attached to Sourceforge.net bug 774665 [10]_

The final patch [11]_ was integrated into Python CVS by Martin v.
LÃ¶wis on 2004-06-08, as stated in the bug report.


References
==========

.. [2] Python locale documentation for embedding,
       http://docs.python.org/library/locale.html

.. [3] PyGTK homepage, http://www.daa.com.au/~james/pygtk/

.. [4] GtkSpinButton screenshot (demonstrating problem),
       http://www.async.com.br/~kiko/spin.png

.. [5] GNOME bug report, http://bugzilla.gnome.org/show_bug.cgi?id=114132

.. [6] Code submission of g_ascii_strtod and g_ascii_dtostr (later
       renamed g_ascii_formatd) by Alex Larsson,
       http://mail.gnome.org/archives/gtk-devel-list/2001-October/msg00114.html

.. [7] PSF Contributor Agreement,
       https://www.python.org/psf/contrib/contrib-form/

.. [8] Alex Larsson's email confirming his agreement was mailed in,
       https://mail.python.org/pipermail/python-dev/2003-August/037755.html

.. [9] Tim Peters' email summarizing LC_NUMERIC trouble with Spambayes,
       https://mail.python.org/pipermail/python-dev/2003-September/037898.html

.. [10] Python bug report, https://bugs.python.org/issue774665

.. [11] Integrated LC_NUMERIC-agnostic patch,
        https://sourceforge.net/tracker/download.php?group_id=5470&atid=305470&file_id=89685&aid=774665


Copyright
=========

This document has been placed in the public domain.


id: pep-0682.rst
content:
PEP: 682
Title: Format Specifier for Signed Zero
Author: John Belmonte <john@neggie.net>
Sponsor: Mark Dickinson <dickinsm@gmail.com>
PEP-Delegate: Mark Dickinson
Discussions-To: https://discuss.python.org/t/pep-682-format-specifier-for-signed-zero/13596
Status: Final
Type: Standards Track
Created: 29-Jan-2022
Python-Version: 3.11
Post-History: 08-Feb-2022
Resolution: https://discuss.python.org/t/accepting-pep-682-format-specifier-for-signed-zero/14088


Abstract
========

Though ``float`` and ``Decimal`` types can represent `signed zero`_, in many
fields of mathematics negative zero is surprising or unwanted -- especially
in the context of displaying an (often rounded) numerical result.  This PEP
proposes an extension to the `string format specification`_ allowing negative
zero to be normalized to positive zero.

.. _`signed zero`: https://en.wikipedia.org/wiki/Signed_zero
.. _`string format specification`: https://docs.python.org/3/library/string.html#formatstrings


Motivation
==========

Here is negative zero::

    >>> x = -0.
    >>> x
    -0.0

When formatting a number, negative zero can result from rounding.  Assuming
the user's intention is truly to discard precision, the distinction between
negative and positive zero of the rounded result might be considered an
unwanted artifact::

    >>> for x in (.002, -.001, .060):
    ...     print(f'{x: .1f}')
     0.0
    -0.0
     0.1

There are various approaches to clearing the sign of a negative zero.  It
can be achieved without a conditional by adding positive zero::

    >>> x = -0.
    >>> x + 0.
    0.0

To normalize negative zero when formatting, it is necessary to perform
a redundant (and error-prone) pre-rounding of the input::

    >>> for x in (.002, -.001, .060):
    ...     print(f'{round(x, 1) + 0.: .1f}')
     0.0
     0.0
     0.1

There is ample evidence that, regardless of the language, programmers are
often looking for a way to suppress negative zero, and landing on a
variety of workarounds (pre-round, post-regex, etc.).  A sampling:

* `How to have negative zero always formatted as positive zero in a
  python string?`_ (Python, post-regex)
* `(Iron)Python formatting issue with modulo operator & "negative zero"`_
  (Python, pre-round)
* `Negative sign in case of zero in java`_ (Java, post-regex)
* `Prevent small negative numbers printing as "-0"`_ (Objective-C, custom
  number formatter)

What we would like instead is a first-class option to normalize negative
zero, on top of everything else that numerical string formatting already
offers.

.. _`How to have negative zero always formatted as positive zero in a python string?`: https://stackoverflow.com/questions/11010683/how-to-have-negative-zero-always-formatted-as-positive-zero-in-a-python-string/36604981#36604981
.. _`(Iron)Python formatting issue with modulo operator & "negative zero"`: https://stackoverflow.com/questions/41564311/ironpython-formatting-issue-with-modulo-operator-negative-zero/41564834#41564834
.. _`Negative sign in case of zero in java`: https://stackoverflow.com/questions/11929096/negative-sign-in-case-of-zero-in-java
.. _`Prevent small negative numbers printing as "-0"`: https://stackoverflow.com/questions/10969399/prevent-small-negative-numbers-printing-as-0


Rationale
=========

There are use cases where negative zero is unwanted in formatted number
output -- arguably, not wanting it is more common.  Expanding the format
specification is the best way to support this because number formatting
already incorporates rounding, and the normalization of negative zero must
happen after rounding.

While it is possible to pre-round and normalize a number before formatting,
it's tedious and prone to error if the rounding doesn't precisely match
that of the format spec.  Furthermore, functions that wrap formatting would
find themselves having to parse format specs to extract the precision
information.  For example, consider how this utility for formatting
one-dimensional numerical arrays would be complicated by such pre-rounding:

.. code-block:: python

    def format_vector(v, format_spec='8.2f'):
        """Format a vector (any iterable) using given per-term format string."""
        return f"[{','.join(f'{term:{format_spec}}' for term in v)}]"

To date, there doesn't appear to be any other widely-used language or library
providing a formatting option for negative zero.  However, the same ``z``
option syntax and semantics specified below have been `proposed for C++
std::format()`_.  While the proposal was withdrawn for C++20, a consensus
proposal is promised for C++23.  (The original `feature request`_ prompting
this PEP was argued without knowledge of the C++ proposal.)

When Rust developers debated whether to suppress negative zero in ``print``
output, they took a small `survey of other languages`_.  Notably, it didn't
mention any language providing an option for negative zero handling.

.. _`proposed for C++ std::format()`: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1496r2.pdf
.. _`feature request`: https://bugs.python.org/issue45995
.. _`survey of other languages`: https://github.com/rust-lang/rfcs/issues/1074#issuecomment-718243936


Specification
=============

An optional, literal ``z`` is added to the
`Format Specification Mini-Language`_ following ``sign``:

.. code-block:: text

    [[fill]align][sign][z][#][0][width][grouping_option][.precision][type]

where ``z`` is allowed for floating-point presentation types (``f``, ``g``,
etc.,  as defined by the format specification documentation).  Support for
``z`` is provided by the ``.__format__()`` method of each numeric type,
allowing the specifier to be used in f-strings, built-in ``format()``, and
``str.format()``.

When ``z`` is present, negative zero (whether the original value or the
result of rounding) will be normalized to positive zero.

Synopsis::

    >>> x = -.00001
    >>> f'{x:z.1f}'
    '0.0'

    >>> x = decimal.Decimal('-.00001')
    >>> '{:+z.1f}'.format(x)
    '+0.0'

.. _`Format Specification Mini-Language`: https://docs.python.org/3/library/string.html#format-specification-mini-language


Design Notes
------------
The solution must be opt-in, because we can't change the behavior of
programs that may be expecting or relying on negative zero when formatting
numbers.

The proposed extension is intentionally ``[sign][z]`` rather than
``[sign[z]]``.  The default for ``sign`` (``-``) is not widely known or
explicitly written, so this avoids everyone having to learn it just to use
the ``z`` option.

While f-strings, built-in ``format()``, and ``str.format()`` can access
the new option, %-formatting cannot.  There is already precedent for not
extending %-formatting with new options, as was the case for the
``,`` option (:pep:`378`).

C99 ``printf`` already uses the ``z`` option character for another
purpose:  qualifying the unsigned type (``u``) to match the length of
``size_t``.  However, since the signed zero option specifically disallows
``z`` for integer presentation types, it's possible to disambiguate the two
uses, should C want to adopt this new option.


Backwards Compatibility
=======================

The new formatting behavior is opt-in, so numerical formatting of existing
programs will not be affected.


How to Teach This
=================
A typical introductory Python course will not cover string formatting
in full detail.  For such a course, no adjustments would need to be made.
For a course that does go into details of the string format specification,
a single example demonstrating the effect of the ``z`` option on a negative
value that's rounded to zero by the formatting should be enough.  For an
independent developer encountering the feature in someone else's code,
reference to the `Format Specification Mini-Language`_ section of the
library reference manual should suffice.

.. _`Format Specification Mini-Language`: https://docs.python.org/3/library/string.html#format-specification-mini-language


Reference Implem


id: pep-0696.rst
content:
PEP: 696
Title: Type Defaults for Type Parameters
Author: James Hilton-Balfe <gobot1234yt@gmail.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-696-type-defaults-for-typevarlikes/22569
Status: Final
Type: Standards Track
Topic: Typing
Created: 14-Jul-2022
Python-Version: 3.13
Post-History: `22-Mar-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/7VWBZWXTCX6RAJO6GG67BAXUPFZ24NTC/>`__,
              `08-Jan-2023 <https://discuss.python.org/t/pep-696-type-defaults-for-typevarlikes/22569/>`__,
Resolution: https://discuss.python.org/t/pep-696-type-defaults-for-typevarlikes/22569/34

.. canonical-typing-spec:: :ref:`typing:type_parameter_defaults` and
                           :external+py3.13:ref:`type-params`

Abstract
--------

This PEP introduces the concept of type defaults for type parameters,
including ``TypeVar``, ``ParamSpec``, and ``TypeVarTuple``,
which act as defaults for type parameters for which no type is specified.

Default type argument support is available in some popular languages
such as C++, TypeScript, and Rust. A survey of type parameter syntax in
some common languages has been conducted by the author of :pep:`695`
and can be found in its
:pep:`Appendix A <695#appendix-a-survey-of-type-parameter-syntax>`.


Motivation
----------

::

   T = TypeVar("T", default=int)  # This means that if no type is specified T = int

   @dataclass
   class Box(Generic[T]):
       value: T | None = None

   reveal_type(Box())                      # type is Box[int]
   reveal_type(Box(value="Hello World!"))  # type is Box[str]

One place this `regularly comes
up <https://github.com/python/typing/issues/975>`__ is ``Generator``. I
propose changing the *stub definition* to something like::

   YieldT = TypeVar("YieldT")
   SendT = TypeVar("SendT", default=None)
   ReturnT = TypeVar("ReturnT", default=None)

   class Generator(Generic[YieldT, SendT, ReturnT]): ...

   Generator[int] == Generator[int, None] == Generator[int, None, None]

This is also useful for a ``Generic`` that is commonly over one type.

::

   class Bot: ...

   BotT = TypeVar("BotT", bound=Bot, default=Bot)

   class Context(Generic[BotT]):
       bot: BotT

   class MyBot(Bot): ...

   reveal_type(Context().bot)         # type is Bot  # notice this is not Any which is what it would be currently
   reveal_type(Context[MyBot]().bot)  # type is MyBot

Not only does this improve typing for those who explicitly use it, it
also helps non-typing users who rely on auto-complete to speed up their
development.

This design pattern is common in projects like:

- `discord.py <https://github.com/Rapptz/discord.py>`__ â where the
  example above was taken from.
- `NumPy <https://github.com/numpy/numpy>`__ â the default for types
  like ``ndarray``'s ``dtype`` would be ``float64``. Currently it's
  ``Unknown`` or ``Any``.
- `TensorFlow <https://github.com/tensorflow/tensorflow>`__ â this
  could be used for Tensor similarly to ``numpy.ndarray`` and would be
  useful to simplify the definition of ``Layer``.


Specification
-------------

Default Ordering and Subscription Rules
'''''''''''''''''''''''''''''''''''''''

The order for defaults should follow the standard function parameter
rules, so a type parameter with no ``default`` cannot follow one with
a ``default`` value. Doing so should ideally raise a ``TypeError`` in
``typing._GenericAlias``/``types.GenericAlias``, and a type checker
should flag this as an error.

::

   DefaultStrT = TypeVar("DefaultStrT", default=str)
   DefaultIntT = TypeVar("DefaultIntT", default=int)
   DefaultBoolT = TypeVar("DefaultBoolT", default=bool)
   T = TypeVar("T")
   T2 = TypeVar("T2")

   class NonDefaultFollowsDefault(Generic[DefaultStrT, T]): ...  # Invalid: non-default TypeVars cannot follow ones with defaults


   class NoNonDefaults(Generic[DefaultStrT, DefaultIntT]): ...

   (
       NoNoneDefaults ==
       NoNoneDefaults[str] ==
       NoNoneDefaults[str, int]
   )  # All valid


   class OneDefault(Generic[T, DefaultBoolT]): ...

   OneDefault[float] == OneDefault[float, bool]  # Valid
   reveal_type(OneDefault)          # type is type[OneDefault[T, DefaultBoolT = bool]]
   reveal_type(OneDefault[float]()) # type is OneDefault[float, bool]


   class AllTheDefaults(Generic[T1, T2, DefaultStrT, DefaultIntT, DefaultBoolT]): ...

   reveal_type(AllTheDefaults)                  # type is type[AllTheDefaults[T1, T2, DefaultStrT = str, DefaultIntT = int, DefaultBoolT = bool]]
   reveal_type(AllTheDefaults[int, complex]())  # type is AllTheDefaults[int, complex, str, int, bool]
   AllTheDefaults[int]  # Invalid: expected 2 arguments to AllTheDefaults
   (
       AllTheDefaults[int, complex] ==
       AllTheDefaults[int, complex, str] ==
       AllTheDefaults[int, complex, str, int] ==
       AllTheDefaults[int, complex, str, int, bool]
   )  # All valid

With the new Python 3.12 syntax for generics (introduced by :pep:`695`), this can
be enforced at compile time::

   type Alias[DefaultT = int, T] = tuple[DefaultT, T]  # SyntaxError: non-default TypeVars cannot follow ones with defaults

   def generic_func[DefaultT = int, T](x: DefaultT, y: T) -> None: ...  # SyntaxError: non-default TypeVars cannot follow ones with defaults

   class GenericClass[DefaultT = int, T]: ...  # SyntaxError: non-default TypeVars cannot follow ones with defaults

``ParamSpec`` Defaults
''''''''''''''''''''''

``ParamSpec`` defaults are defined using the same syntax as
``TypeVar`` \ s but use a ``list`` of types or an ellipsis
literal "``...``" or another in-scope ``ParamSpec`` (see `Scoping Rules`_).

::

   DefaultP = ParamSpec("DefaultP", default=[str, int])

   class Foo(Generic[DefaultP]): ...

   reveal_type(Foo)                  # type is type[Foo[DefaultP = [str, int]]]
   reveal_type(Foo())                # type is Foo[[str, int]]
   reveal_type(Foo[[bool, bool]]())  # type is Foo[[bool, bool]]

``TypeVarTuple`` Defaults
'''''''''''''''''''''''''

``TypeVarTuple`` defaults are defined using the same syntax as
``TypeVar`` \ s but use an unpacked tuple of types instead of a single type
or another in-scope ``TypeVarTuple`` (see `Scoping Rules`_).

::

   DefaultTs = TypeVarTuple("DefaultTs", default=Unpack[tuple[str, int]])

   class Foo(Generic[*DefaultTs]): ...

   reveal_type(Foo)               # type is type[Foo[DefaultTs = *tuple[str, int]]]
   reveal_type(Foo())             # type is Foo[str, int]
   reveal_type(Foo[int, bool]())  # type is Foo[int, bool]

Using Another Type Parameter as ``default``
''''''''''''''''''''''''''''''''''''''''''''

This allows for a value to be used again when the type parameter to a
generic is missing but another type parameter is specified.

To use another type parameter as a default the ``default`` and the
type parameter must be the same type (a ``TypeVar``'s default must be
a ``TypeVar``, etc.).

`This could be used on builtins.slice <https://github.com/python/typing/issues/159>`__
where the ``start`` parameter should default to ``int``, ``stop``
default to the type of ``start`` and step default to ``int | None``.

::

   StartT = TypeVar("StartT", default=int)
   StopT = TypeVar("StopT", default=StartT)
   StepT = TypeVar("StepT", default=int | None)

   class slice(Generic[StartT, StopT, StepT]): ...

   reveal_type(slice)  # type is type[slice[StartT = int, StopT = StartT, StepT = int | None]]
   reveal_type(slice())                        # type is slice[int, int, int | None]
   reveal_type(slice[str]())                   # type is slice[str, str, int | None]
   reveal_type(slice[str, bool, timedelta]())  # type is slice[str, bool, timedelta]

   T2 = TypeVar("T2", default=DefaultStrT)

   class Foo(Generic[DefaultStrT, T2]):
       def __init__(self, a: DefaultStrT, b: T2) -> None: ...

   reveal_type(Foo(1, ""))  # type is Foo[int, str]
   Foo[int](1, "")          # Invalid: Foo[int, str] cannot be assigned to self: Foo[int, int] in Foo.__ini


id: pep-0669.rst
content:
PEP: 669
Title: Low Impact Monitoring for CPython
Author: Mark Shannon <mark@hotpy.org>
Discussions-To: https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/
Status: Final
Type: Standards Track
Created: 18-Aug-2021
Python-Version: 3.12
Post-History: `07-Dec-2021 <https://mail.python.org/archives/list/python-dev@python.org/thread/VNSD4TSAM2BM64FJNIQPAOPNEGNX4MDX/>`__,
              `10-Jan-2022 <https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018>`__,
Resolution: https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/42

.. canonical-doc:: :mod:`python:sys.monitoring`

Abstract
========

Using a profiler or debugger in CPython can have a severe impact on
performance. Slowdowns by an order of magnitude are common.

This PEP proposes an API for monitoring Python programs running
on CPython that will enable monitoring at low cost.

Although this PEP does not specify an implementation, it is expected that
it will be implemented using the quickening step of
:pep:`659`.

A ``sys.monitoring`` namespace will be added, which will contain
the relevant functions and constants.


Motivation
==========

Developers should not have to pay an unreasonable cost to use debuggers,
profilers and other similar tools.

C++ and Java developers expect to be able to run a program at full speed
(or very close to it) under a debugger.
Python developers should expect that too.

Rationale
=========

The quickening mechanism provided by :pep:`659` provides a way to dynamically
modify executing Python bytecode. These modifications have little cost beyond
the parts of the code that are modified and a relatively low cost to those
parts that are modified. We can leverage this to provide an efficient
mechanism for monitoring that was not possible in 3.10 or earlier.

By using quickening, we expect that code run under a debugger on 3.12
should outperform code run without a debugger on 3.11.
Profiling will still slow down execution, but by much less than in 3.11.


Specification
=============

Monitoring of Python programs is done by registering callback functions
for events and by activating a set of events.

Activating events and registering callback functions are independent of each other.

Both registering callbacks and activating events are done on a per-tool basis.
It is possible to have multiple tools that respond to different sets of events.

Note that, unlike ``sys.settrace()``, events and callbacks are per interpreter, not per thread.

Events
------

As a code object executes various events occur that might be of interest
to tools. By activating events and by registering callback functions
tools can respond to these events in any way that suits them.
Events can be set globally, or for individual code objects.

For 3.12, CPython will support the following events:

* PY_START: Start of a Python function (occurs immediately after the call, the callee's frame will be on the stack)
* PY_RESUME: Resumption of a Python function (for generator and coroutine functions), except for throw() calls.
* PY_THROW: A Python function is resumed by a throw() call.
* PY_RETURN: Return from a Python function (occurs immediately before the return, the callee's frame will be on the stack).
* PY_YIELD: Yield from a Python function (occurs immediately before the yield, the callee's frame will be on the stack).
* PY_UNWIND:  Exit from a Python function during exception unwinding.
* CALL: A call in Python code (event occurs before the call).
* C_RETURN: Return from any callable, except Python functions (event occurs after the return).
* C_RAISE: Exception raised from any callable, except Python functions (event occurs after the exit).
* RAISE: An exception is raised, except those that cause a ``STOP_ITERATION`` event.
* EXCEPTION_HANDLED: An exception is handled.
* LINE: An instruction is about to be executed that has a different line number from the preceding instruction.
* INSTRUCTION -- A VM instruction is about to be executed.
* JUMP -- An unconditional jump in the control flow graph is made.
* BRANCH -- A conditional branch is taken (or not).
* STOP_ITERATION -- An artificial ``StopIteration`` is raised;
  see `the STOP_ITERATION event`_.

More events may be added in the future.

All events will be attributes of the ``events`` namespace in ``sys.monitoring``.
All events will represented by a power of two integer, so that they can be combined
with the ``|`` operator.

Events are divided into three groups:

Local events
''''''''''''

Local events are associated with normal execution of the program and happen
at clearly defined locations. All local events can be disabled.
The local events are:

* PY_START
* PY_RESUME
* PY_RETURN
* PY_YIELD
* CALL
* LINE
* INSTRUCTION
* JUMP
* BRANCH
* STOP_ITERATION

Ancilliary events
'''''''''''''''''

Ancillary events can be monitored like other events, but are controlled
by another event:

* C_RAISE
* C_RETURN

The ``C_RETURN`` and ``C_RAISE`` events are are controlled by the ``CALL``
event. ``C_RETURN`` and ``C_RAISE`` events will only be seen if the
corresponding ``CALL`` event is being monitored.

Other events
''''''''''''

Other events are not necessarily tied to a specific location in the
program and cannot be individually disabled.

The other events that can be monitored are:

* PY_THROW
* PY_UNWIND
* RAISE
* EXCEPTION_HANDLED


The STOP_ITERATION event
''''''''''''''''''''''''

:pep:`PEP 380 <380#use-of-stopiteration-to-return-values>`
specifies that a ``StopIteration`` exception is raised when returning a value
from a generator or coroutine. However, this is a very inefficient way to
return a value, so some Python implementations, notably CPython 3.12+, do not
raise an exception unless it would be visible to other code.

To allow tools to monitor for real exceptions without slowing down generators
and coroutines, the ``STOP_ITERATION`` event is provided.
``STOP_ITERATION`` can be locally disabled, unlike ``RAISE``.

Tool identifiers
----------------

The VM can support up to 6 tools at once.
Before registering or activating events, a tool should choose an identifier.
Identifiers are integers in the range 0 to 5.

::

  sys.monitoring.use_tool_id(id, name:str) -> None
  sys.monitoring.free_tool_id(id) -> None
  sys.monitoring.get_tool(id) ->  str | None

``sys.monitoring.use_tool_id`` raises a ``ValueError`` if ``id`` is in use.
``sys.monitoring.get_tool`` returns the name of the tool if ``id`` is in use,
otherwise it returns ``None``.

All IDs are treated the same by the VM with regard to events, but the
following IDs are pre-defined to make co-operation of tools easier::

  sys.monitoring.DEBUGGER_ID = 0
  sys.monitoring.COVERAGE_ID = 1
  sys.monitoring.PROFILER_ID = 2
  sys.monitoring.OPTIMIZER_ID = 5

There is no obligation to set an ID, nor is there anything preventing a tool
from using an ID even it is already in use.
However, tools are encouraged to use a unique ID and respect other tools.

For example, if a debugger were attached and ``DEBUGGER_ID`` were in use, it
should report an error, rather than carrying on regardless.

The ``OPTIMIZER_ID`` is provided for tools like Cinder or PyTorch
that want to optimize Python code, but need to decide what to
optimize in a way that depends on some wider context.

Setting events globally
-----------------------

Events can be controlled globally by modifying the set of events being monitored:

* ``sys.monitoring.get_events(tool_id:int)->int``
  Returns the ``int`` representing all the active events.

* ``sys.monitoring.set_events(tool_id:int, event_set: int)``
  Activates all events which are set in ``event_set``.
  Raises a ``ValueError`` if ``tool_id`` is not in use.

No events are active by default.

Per code object events
----------------------

Events can also be controlled on a per code object basis:

* ``sys.monitoring.get_local_events(tool_id:int, code: CodeType)->int``
  Returns all the local events for ``code``

* ``sys.monitorin


id: pep-0641.rst
content:
PEP: 641
Title: Using an underscore in the version portion of Python 3.10 compatibility tags
Author: Brett Cannon <brett@python.org>,
        Steve Dower <steve.dower@python.org>,
        Barry Warsaw <barry@python.org>
PEP-Delegate: Pablo Galindo <pablogsal@python.org>
Discussions-To: https://discuss.python.org/t/pep-641-using-an-underscore-in-the-version-portion-of-python-3-10-compatibility-tags/5513
Status: Rejected
Type: Standards Track
Created: 20-Oct-2020
Python-Version: 3.10
Post-History: 21-Oct-2020
Resolution: https://discuss.python.org/t/pep-641-using-an-underscore-in-the-version-portion-of-python-3-10-compatibility-tags/5513/42

Abstract
========

.. note::
   This PEP was rejected due to potential breakage in the community.

Using the tag system outlined in :pep:`425` (primarily used for wheel
file names), each release of Python specifies compatibility tags
(e.g. ``cp39``, ``py39`` for CPython 3.9). For CPython 3.10, this PEP
proposes using ``3_10`` as the version portion of the tags
(instead of ``310``).


Motivation
==========

Up to this point, the version portion of compatibility tags used in
e.g. wheel file names has been a straight concatenation of the major
and minor versions of Python, both for the CPython interpreter tag and
the generic, interpreter-agnostic interpreter tag (e.g. ``cp39`` and
``py39``, respectively). This also applies to the ABI tag
(e.g. ``cp39``). Thanks to both the major and minor versions being
single digits, it has been unambiguous what which digit in e.g. ``39``
represented.

But starting with Python 3.10, ambiguity comes up as ``310`` does not
clearly delineate whether the Python version is ``3.10``, ``31.0``, or
``310`` as the major-only version of Python. Thus using ``3_10`` to
separate major/minor portions as allowed by :pep:`425` disambiguates
the Python version being supported.


Rationale
=========

Using ``3_10`` instead of another proposed separator is a restriction
of :pep:`425`, thus the only options are ``3_10`` or ``310``.


Specification
=============

The ``SOABI`` configure variable and
``sysconfig.get_config_var('py_version_nodot')`` will be updated to
use ``3_10`` appropriately.


Backwards Compatibility
=======================

Tools relying on the 'packaging' project [2]_ already expect a
version specification of ``3_10`` for Python 3.10. Keeping the version
specifier as ``310`` would require backing that change out and
updating dependent projects (e.g. pip).

Switching to ``3_10`` will impact any tools that implicitly rely on
the convention that the minor version is a single digit. However,
these are broken regardless of any change here.

For tools assuming the major version is only the first digit, they
will require updating if we switch to ``3_10``.

In non-locale ASCII, ``_`` sorts after any digit, so lexicographic
sorting matching a sort by Python version of a wheel file name will be
kept.

Since :pep:`515` (Python 3.6), underscores in numeric literals are ignored.
This means that ``int("3_10")`` and ``int("310")`` produce the same result,
and ordering based on conversion to an integer will be preserved.
**However**, this is still a bad way to sort tags, and the point is raised
here simply to show that this proposal does not make things worse.

Security Implications
=====================

There are no known security concerns.


How to Teach This
=================

As use of the interpreter tag is mostly machine-based and this PEP
disambiguates, there should not be any special teaching consideration
required.


Reference Implementation
========================

A pull request [1]_ already exists adding support to CPython 3.10.
Support for reading wheel files with this proposed PEP is already
implemented.


Rejected Ideas
==============

Not making the change
---------------------
It was considered to not change the tag and stay with ``310``. The
argument was it's less work and it won't break any existing
tooling. But in the end it was thought that the disambiguation is
better to have.


Open Issues
===========

How far should we take this?
----------------------------
Other places where the major and minor version are used could be
updated to use an underscore as well (e.g. ``.pyc`` files, the import
path to the zip file for the stdlib). It is not known how useful it
would be to make this pervasive.

Standardizing on double digit minor version numbers
---------------------------------------------------
An alternative suggestion has been made to disambiguate where the
major and minor versions start/stop by forcing the minor version to
always be two digits, padding with a ``0`` as required. The advantages
of this is it makes the current ``cp310`` interpreter tag accurate,
thus minimizing breakage. It also does differentiate going forward.

There are a couple of drawbacks, though. One is the disambiguation
only exists *if* you know that the minor version number is two digits;
compare that to ``cp3_10`` which is unambiguous regardless of your
base knowledge. The potential for a three digit minor version number
is also not addressed by this two digit requirement.

There is also the issue of other interpreters not following the
practice in the past, present, or future. For instance, it is
unknown if other people have used a three digit version portion of the
interpreter tag previously for another interpreter where this rule
would be incorrect. This change would also suggest that interpreters which
currently have a single digit minor version -- e.g. PyPy 7.3 -- to
change from ``pp73`` to ``pp703`` or make the switch from their next
minor release onward (e.g. 7.4 or 8.0). Otherwise this would make this
rule exclusive to the ``cp`` interpreter type which would make it more
confusing for people.


References
==========

.. [1] Reference implementation
   (https://github.com/python/cpython/pull/20333)

.. [2] The 'packaging' project
   (https://pypi.org/project/packaging/)


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0655.rst
content:
PEP: 655
Title: Marking individual TypedDict items as required or potentially-missing
Author: David Foster <david at dafoster.net>
Sponsor: Guido van Rossum <guido at python.org>
Discussions-To: https://mail.python.org/archives/list/typing-sig@python.org/thread/53XVOD5ZUKJ263MWA6AUPEA6J7LBBLNV/
Status: Final
Type: Standards Track
Topic: Typing
Created: 30-Jan-2021
Python-Version: 3.11
Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021, 17-Jan-2022, 28-Jan-2022
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/AJEDNVC3FXM5QXNNW5CR4UCT4KI5XVUE/

.. canonical-typing-spec:: :ref:`typing:required-notrequired`,
                           :py:data:`typing.Required` and
                           :py:data:`typing.NotRequired`

Abstract
========

:pep:`589` defines notation
for declaring a TypedDict with all required keys and notation for defining
a TypedDict with :pep:`all potentially-missing keys <589#totality>`, however it
does not provide a mechanism to declare some keys as required and others
as potentially-missing. This PEP introduces two new notations:
``Required[]``, which can be used on individual items of a
TypedDict to mark them as required, and
``NotRequired[]``, which can be used on individual items
to mark them as potentially-missing.

This PEP makes no Python grammar changes. Correct usage
of required and potentially-missing keys of TypedDicts is intended to be
enforced only by static type checkers and need not be enforced by
Python itself at runtime.


Motivation
==========

It is not uncommon to want to define a TypedDict with some keys that are
required and others that are potentially-missing. Currently the only way
to define such a TypedDict is to declare one TypedDict with one value
for ``total`` and then inherit it from another TypedDict with a
different value for ``total``:

::

   class _MovieBase(TypedDict):  # implicitly total=True
       title: str

   class Movie(_MovieBase, total=False):
       year: int

Having to declare two different TypedDict types for this purpose is
cumbersome.

This PEP introduces two new type qualifiers, ``typing.Required`` and
``typing.NotRequired``, which allow defining a *single* TypedDict with
a mix of both required and potentially-missing keys:

::

   class Movie(TypedDict):
       title: str
       year: NotRequired[int]

This PEP also makes it possible to define TypedDicts in the
:pep:`alternative functional syntax <589#alternative-syntax>`
with a mix of required and potentially-missing keys,
which is not currently possible at all because the alternative syntax does
not support inheritance:

::

   Actor = TypedDict('Actor', {
       'name': str,
       # "in" is a keyword, so the functional syntax is necessary
       'in': NotRequired[List[str]],
   })


Rationale
=========

One might think it unusual to propose notation that prioritizes marking
*required* keys rather than *potentially-missing* keys, as is
customary in other languages like TypeScript:

.. code-block:: typescript

   interface Movie {
       title: string;
       year?: number;  // ? marks potentially-missing keys
   }

The difficulty is that the best word for marking a potentially-missing
key, ``Optional[]``, is already used in Python for a completely
different purpose: marking values that could be either of a particular
type or ``None``. In particular the following does not work:

::

   class Movie(TypedDict):
       ...
       year: Optional[int]  # means int|None, not potentially-missing!

Attempting to use any synonym of âoptionalâ to mark potentially-missing
keys (like ``Missing[]``) would be too similar to ``Optional[]``
and be easy to confuse with it.

Thus it was decided to focus on positive-form phrasing for required keys
instead, which is straightforward to spell as ``Required[]``.

Nevertheless it is common for folks wanting to extend a regular
(``total=True``) TypedDict to only want to add a small number of
potentially-missing keys, which necessitates a way to mark keys that are
*not* required and potentially-missing, and so we also allow the
``NotRequired[]`` form for that case.


Specification
=============

The ``typing.Required`` type qualifier is used to indicate that a
variable declared in a TypedDict definition is a required key:

::

   class Movie(TypedDict, total=False):
       title: Required[str]
       year: int

Additionally the ``typing.NotRequired`` type qualifier is used to
indicate that a variable declared in a TypedDict definition is a
potentially-missing key:

::

   class Movie(TypedDict):  # implicitly total=True
       title: str
       year: NotRequired[int]

It is an error to use ``Required[]`` or ``NotRequired[]`` in any
location that is not an item of a TypedDict.
Type checkers must enforce this restriction.

It is valid to use ``Required[]`` and ``NotRequired[]`` even for
items where it is redundant, to enable additional explicitness if desired:

::

   class Movie(TypedDict):
       title: Required[str]  # redundant
       year: NotRequired[int]

It is an error to use both ``Required[]`` and ``NotRequired[]`` at the
same time:

::

   class Movie(TypedDict):
       title: str
       year: NotRequired[Required[int]]  # ERROR

Type checkers must enforce this restriction.
The runtime implementations of ``Required[]`` and ``NotRequired[]``
may also enforce this restriction.

The :pep:`alternative functional syntax <589#alternative-syntax>`
for TypedDict also supports
``Required[]`` and ``NotRequired[]``:

::

   Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]})


Interaction with ``total=False``
--------------------------------

Any :pep:`589`-style TypedDict declared with ``total=False`` is equivalent
to a TypedDict with an implicit ``total=True`` definition with all of its
keys marked as ``NotRequired[]``.

Therefore:

::

   class _MovieBase(TypedDict):  # implicitly total=True
       title: str

   class Movie(_MovieBase, total=False):
       year: int


is equivalent to:

::

   class _MovieBase(TypedDict):
       title: str

   class Movie(_MovieBase):
       year: NotRequired[int]


Interaction with ``Annotated[]``
-----------------------------------

``Required[]`` and ``NotRequired[]`` can be used with ``Annotated[]``,
in any nesting order:

::

   class Movie(TypedDict):
       title: str
       year: NotRequired[Annotated[int, ValueRange(-9999, 9999)]]  # ok

::

   class Movie(TypedDict):
       title: str
       year: Annotated[NotRequired[int], ValueRange(-9999, 9999)]  # ok

In particular allowing ``Annotated[]`` to be the outermost annotation
for an item allows better interoperability with non-typing uses of
annotations, which may always want ``Annotated[]`` as the outermost annotation.
[3]_


Runtime behavior
----------------


Interaction with ``get_type_hints()``
'''''''''''''''''''''''''''''''''''''

``typing.get_type_hints(...)`` applied to a TypedDict will by default
strip out any ``Required[]`` or ``NotRequired[]`` type qualifiers,
since these qualifiers are expected to be inconvenient for code
casually introspecting type annotations.

``typing.get_type_hints(..., include_extras=True)`` however
*will* retain ``Required[]`` and ``NotRequired[]`` type qualifiers,
for advanced code introspecting type annotations that
wishes to preserve *all* annotations in the original source:

::

   class Movie(TypedDict):
       title: str
       year: NotRequired[int]

   assert get_type_hints(Movie) == \
       {'title': str, 'year': int}
   assert get_type_hints(Movie, include_extras=True) == \
       {'title': str, 'year': NotRequired[int]}


Interaction with ``get_origin()`` and ``get_args()``
''''''''''''''''''''''''''''''''''''''''''''''''''''

``typing.get_origin()`` and ``typing.get_args()`` will be updated to
recognize ``Required[]`` and ``NotRequired[]``:

::

   assert get_origin(Required[int]) is Required
   assert get_args(Required[int]) == (int,)

   assert get_origin(NotRequired[int]) is NotRequired
   ass


id: pep-0709.rst
content:
PEP: 709
Title: Inlined comprehensions
Author: Carl Meyer <carl@oddbird.net>
Sponsor: Guido van Rossum <guido@python.org>
Discussions-To: https://discuss.python.org/t/pep-709-inlined-comprehensions/24240
Status: Final
Type: Standards Track
Created: 24-Feb-2023
Python-Version: 3.12
Post-History: `25-Feb-2023 <https://discuss.python.org/t/pep-709-inlined-comprehensions/24240>`__
Resolution: https://discuss.python.org/t/pep-709-inlined-comprehensions/24240/36


Abstract
========

Comprehensions are currently compiled as nested functions, which provides
isolation of the comprehension's iteration variable, but is inefficient at
runtime. This PEP proposes to inline list, dictionary, and set comprehensions
into the code where they are defined, and provide the expected isolation by
pushing/popping clashing locals on the stack. This change makes comprehensions
much faster: up to 2x faster for a microbenchmark of a comprehension alone,
translating to an 11% speedup for one sample benchmark derived from real-world
code that makes heavy use of comprehensions in the context of doing actual work.


Motivation
==========

Comprehensions are a popular and widely-used feature of the Python language.
The nested-function compilation of comprehensions optimizes for compiler
simplicity at the expense of performance of user code. It is possible to
provide near-identical semantics (see `Backwards Compatibility`_) with much
better runtime performance for all users of comprehensions, with only a small
increase in compiler complexity.


Rationale
=========

Inlining is a common compiler optimization in many languages.  Generalized
inlining of function calls at compile time in Python is near-impossible, since
call targets may be patched at runtime. Comprehensions are a special case,
where we have a call target known statically in the compiler that can neither
be patched (barring undocumented and unsupported fiddling with bytecode
directly) nor escape.

Inlining also permits other compiler optimizations of bytecode to be more
effective, because they can now "see through" the comprehension bytecode,
instead of it being an opaque call.

Normally a performance improvement would not require a PEP. In this case, the
simplest and most efficient implementation results in some user-visible effects,
so this is not just a performance improvement, it is a (small) change to the
language.


Specification
=============

Given a simple comprehension::

  def f(lst):
      return [x for x in lst]

The compiler currently emits the following bytecode for the function ``f``:

.. code-block:: text

   1           0 RESUME                   0

   2           2 LOAD_CONST               1 (<code object <listcomp> at 0x...)
               4 MAKE_FUNCTION            0
               6 LOAD_FAST                0 (lst)
               8 GET_ITER
              10 CALL                     0
              20 RETURN_VALUE

   Disassembly of <code object <listcomp> at 0x...>:
   2           0 RESUME                   0
               2 BUILD_LIST               0
               4 LOAD_FAST                0 (.0)
         >>    6 FOR_ITER                 4 (to 18)
              10 STORE_FAST               1 (x)
              12 LOAD_FAST                1 (x)
              14 LIST_APPEND              2
              16 JUMP_BACKWARD            6 (to 6)
         >>   18 END_FOR
              20 RETURN_VALUE

The bytecode for the comprehension is in a separate code object. Each time
``f()`` is called, a new single-use function object is allocated (by
``MAKE_FUNCTION``), called (allocating and then destroying a new frame on the
Python stack), and then immediately thrown away.

Under this PEP, the compiler will emit the following bytecode for ``f()``
instead:

.. code-block:: text

  1           0 RESUME                   0

  2           2 LOAD_FAST                0 (lst)
              4 GET_ITER
              6 LOAD_FAST_AND_CLEAR      1 (x)
              8 SWAP                     2
             10 BUILD_LIST               0
             12 SWAP                     2
        >>   14 FOR_ITER                 4 (to 26)
             18 STORE_FAST               1 (x)
             20 LOAD_FAST                1 (x)
             22 LIST_APPEND              2
             24 JUMP_BACKWARD            6 (to 14)
        >>   26 END_FOR
             28 SWAP                     2
             30 STORE_FAST               1 (x)
             32 RETURN_VALUE

There is no longer a separate code object, nor creation of a single-use function
object, nor any need to create and destroy a Python frame.

Isolation of the ``x`` iteration variable is achieved by the combination of the
new ``LOAD_FAST_AND_CLEAR`` opcode at offset ``6``, which saves any outer value
of ``x`` on the stack before running the comprehension, and ``30 STORE_FAST``,
which restores the outer value of ``x`` (if any) after running the
comprehension.

If the comprehension accesses variables from the outer scope, inlining avoids
the need to place these variables in a cell, allowing the comprehension (and all
other code in the outer function) to access them as normal fast locals instead.
This provides further performance gains.

In some cases, the comprehension iteration variable may be a global or cellvar
or freevar, rather than a simple function local, in the outer scope. In these
cases, the compiler also internally pushes and pops the scope information for
the variable when entering/leaving the comprehension, so that semantics are
maintained. For example, if the variable is a global outside the comprehension,
``LOAD_GLOBAL`` will still be used where it is referenced outside the
comprehension, but ``LOAD_FAST`` / ``STORE_FAST`` will be used within the
comprehension. If it is a cellvar/freevar outside the comprehension, the
``LOAD_FAST_AND_CLEAR`` / ``STORE_FAST`` used to save/restore it do not change
(there is no ``LOAD_DEREF_AND_CLEAR``), meaning that the entire cell (not just
the value within it) is saved/restored, so the comprehension does not write to
the outer cell.

Comprehensions occurring in module or class scope are also inlined. In this
case, the comprehension will introduce usage of fast-locals (``LOAD_FAST`` /
``STORE_FAST``) for the comprehension iteration variable within the
comprehension only, in a scope where otherwise only ``LOAD_NAME`` /
``STORE_NAME`` would be used, maintaining isolation.

In effect, comprehensions introduce a sub-scope where local variables are fully
isolated, but without the performance cost or stack frame entry of a call.

Generator expressions are currently not inlined in the reference implementation
of this PEP. In the future, some generator expressions may be inlined, where the
returned generator object does not leak.

Asynchronous comprehensions are inlined the same as synchronous ones; no special
handling is needed.


Backwards Compatibility
=======================

Comprehension inlining will cause the following visible behavior changes. No
changes in the standard library or test suite were necessary to adapt to these
changes in the implementation, suggesting the impact in user code is likely to
be minimal.

Specialized tools depending on undocumented details of compiler bytecode output
may of course be affected in ways beyond the below, but these tools already must
adapt to bytecode changes in each Python version.

locals() includes outer variables
---------------------------------

Calling ``locals()`` within a comprehension will include all locals of the
function containing the comprehension. E.g. given the following function::

  def f(lst):
      return [locals() for x in lst]

Calling ``f([1])`` in current Python will return::

  [{'.0': <list_iterator object at 0x7f8d37170460>, 'x': 1}]

where ``.0`` is an internal implementation detail: the synthetic sole argument
to the comprehension "function".

Under this PEP, it will instead return::

  [{'lst': [1], 'x': 1}]

This now includes the outer ``lst`` variable as a lo


id: pep-0735.rst
content:
PEP: 735
Title: Dependency Groups in pyproject.toml
Author: Stephen Rosen <sirosen0@gmail.com>
Sponsor: Brett Cannon <brett@python.org>
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: https://discuss.python.org/t/39233
Status: Final
Type: Standards Track
Topic: Packaging
Created: 20-Nov-2023
Post-History: `14-Nov-2023 <https://discuss.python.org/t/29684>`__, `20-Nov-2023 <https://discuss.python.org/t/39233>`__
Resolution: `10-Oct-2024 <https://discuss.python.org/t/39233/312>`__

.. canonical-pypa-spec:: :ref:`packaging:dependency-groups`


Abstract
========

This PEP specifies a mechanism for storing package requirements in
``pyproject.toml`` files such that they are not included in any built distribution of
the project.

This is suitable for creating named groups of dependencies, similar to
``requirements.txt`` files, which launchers, IDEs, and other tools can find and
identify by name.

The feature defined here is referred to as "Dependency Groups".

Motivation
==========

There are two major use cases for which the Python community has no
standardized answer:

* How should development dependencies be defined for packages?

* How should dependencies be defined for projects which do not build
  distributions (non-package projects)?

In support of these two needs, there are two common solutions which are similar
to this proposal:

* ``requirements.txt`` files

* package `extras <https://packaging.python.org/en/latest/specifications/dependency-specifiers/#extras>`__

Both ``requirements.txt`` files and ``extras`` have limitations which this
standard seeks to overcome.

Note that the two use cases above describe two different types of projects
which this PEP seeks to support:

* Python packages, such as libraries

* non-package projects, such as data science projects

Several motivating use cases are defined in detail in the :ref:`Use Cases Appendix <use_cases>`.

Limitations of ``requirements.txt`` files
-----------------------------------------

Many projects may define one or more ``requirements.txt`` files,
and may arrange them either at the project root (e.g. ``requirements.txt`` and
``test-requirements.txt``) or else in a directory (e.g.
``requirements/base.txt`` and ``requirements/test.txt``). However, there are
major issues with the use of requirements files in this way:

* There is no standardized naming convention such that tools can discover or
  use these files by name.

* ``requirements.txt`` files are *not standardized*, but instead provide
  options to ``pip``.

As a result, it is difficult to define tool behaviors based on
``requirements.txt`` files. They are not trivial to discover or identify by
name, and their contents may contain a mix of package specifiers and additional
``pip`` options.

The lack of a standard for ``requirements.txt`` contents also means they are
not portable to any alternative tools which wish to process them other than
``pip``.

Additionally, ``requirements.txt`` files require a file per dependency list.
For some use-cases, this makes the marginal cost of dependency groupings high,
relative to their benefit.
A terser declaration is beneficial to projects with a number of small groups of
dependencies.

In contrast with this, Dependency Groups are defined at a well known location
in ``pyproject.toml`` with fully standardized contents. Not only will they have
immediate utility, but they will also serve as a starting point for future
standards.

Limitations of ``extras``
-------------------------

``extras`` are additional package metadata declared in the
``[project.optional-dependencies]`` table. They provide names for lists of
package specifiers which are published as part of a package's metadata, and
which a user can request under that name, as in ``pip install 'foo[bar]'`` to
install ``foo`` with the ``bar`` extra.

Because ``extras`` are package metadata, they are not guaranteed to be
statically defined and may require a build system to resolve.
Furthermore, definition of a ``[project.optional-dependencies]`` indicates to
many tools that a project is a package, and may drive tool behaviors such as
validation of the ``[project]`` table.

For projects which are packages, ``extras`` are a common solution for defining
development dependencies, but even under these circumstances they have
downsides:

* Because an ``extra`` defines optional *additional* dependencies, it is not
  possible to install an ``extra`` without installing the current package and
  its dependencies.

* Because they are user-installable, ``extras`` are part of the public interface
  for packages. Because ``extras`` are published, package developers often are
  concerned about ensuring that their development extras are not confused with
  user-facing extras.

Rationale
=========

This PEP defines the storage of requirements data in lists within a
``[dependency-groups]`` table.
This name was chosen to match the canonical name of the feature
("Dependency Groups").

This format should be as simple and learnable as possible, having a format
very similar to existing ``requirements.txt`` files for many cases. Each list
in ``[dependency-groups]`` is defined as a list of package specifiers. For
example:

.. code-block:: toml

    [dependency-groups]
    test = ["pytest>7", "coverage"]

There are a number of use cases for ``requirements.txt`` files which require
data which cannot be expressed in :pep:`508` dependency specifiers. Such
fields are not valid in Dependency Groups. Including many of the data and
fields which ``pip`` supports, such as index servers, hashes, and path
dependencies, requires new standards. This standard leaves room for new
standards and developments, but does not attempt to support all valid
``requirements.txt`` contents.

The only exception to this is the ``-r`` flag which ``requirements.txt`` files
use to include one file in another. Dependency Groups support an "include"
mechanism which is similar in meaning, allowing one dependency group to extend
another.

Dependency Groups have two additional features which are similar to
``requirements.txt`` files:

* they are not published as distinct metadata in any built distribution

* installation of a dependency group does not imply installation of a package's
  dependencies or the package itself

Use Cases
---------

The following use cases are considered important targets for this PEP. They are
defined in greater detail in the :ref:`Use Cases Appendix <use_cases>`.

* Web Applications deployed via a non-python-packaging build process
* Libraries with unpublished dev dependency groups
* Data science projects with groups of dependencies but no core package
* *Input data* to lockfile generation (Dependency Groups should generally not
  be used as a location for locked dependency data)
* Input data to an environment manager, such as tox, Nox, or Hatch
* Configurable IDE discovery of test and linter requirements

Regarding Poetry and PDM Dependency Groups
------------------------------------------

The existing Poetry and PDM tools already offer a feature which each calls
"Dependency Groups". However, absent any standard for specifying collections
of dependencies, each tool defines these in a tool-specific way, in the
relevant sections of the ``[tool]`` table.

(PDM also uses extras for some Dependency Groups, and overlaps the notion
heavily with extras.)

This PEP does not support all of the features of Poetry and PDM, which, like
``requirements.txt`` files for ``pip``, support several non-standard extensions
to common dependency specifiers.

It should be possible for such tools to use standardized Dependency Groups as
extensions of their own Dependency Group mechanisms.
However, defining a new data format which replaces the existing Poetry and PDM
solutions is a non-goal. Doing so would require standardizing several
additional features, such as path dependencies, which are supported by these
tools.

Dependency Groups are not Hidden Extras
------------------------------


id: pep-0721.rst
content:
PEP: 721
Title: Using tarfile.data_filter for source distribution extraction
Author: Petr Viktorin <encukou@gmail.com>
PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
Status: Final
Type: Standards Track
Topic: Packaging
Requires: 706
Created: 12-Jul-2023
Python-Version: 3.12
Post-History: `04-Jul-2023 <https://discuss.python.org/t/28928>`__,
Resolution: `02-Aug-2023 <https://discuss.python.org/t/28928/13>`__

.. canonical-pypa-spec:: :ref:`packaging:sdist-archive-features`

Abstract
========

Extracting a source distribution archive should normally use the ``data``
filter added in :pep:`706`.
We clarify details, and specify the behaviour for tools that cannot use the
filter directly.


Motivation
==========

The *source distribution* ``sdist`` is defined as a tar archive.

The ``tar`` format is designed to capture all metadata of Unix-like files.
Some of these are dangerous, unnecessary for source code, and/or
platform-dependent.
As explained in :pep:`706`, when extracting a tarball, one should always either
limit the allowed features, or explicitly give the tarball total control.


Rationale
=========

For source distributions, the ``data`` filter introduced in :pep:`706`
is enough. It allows slightly more features than ``git`` and ``zip`` (both
commonly used in packaging workflows).

However, not all tools can use the ``data`` filter,
so this PEP specifies an explicit set of expectations.
The aim is that the current behaviour of ``pip download``
and ``setuptools.archive_util.unpack_tarfile`` is valid,
except cases deemed too dangerous to allow.
Another consideration is ease of implementation for non-Python tools.


Unpatched versions of Python
----------------------------

Tools are allowed to ignore this PEP when running on Python without tarfile
filters.

The feature has been backported to all versions of Python supported by
``python.org``. Vendoring it in third-party libraries is tricky,
and we should not force all tools to do so.
This shifts the responsibility to keep up with security updates from the tools
to the users.


Permissions
-----------

Common tools (``git``, ``zip``) don't preserve Unix permissions (mode bits).
Telling users to not rely on them in *sdists*, and allowing tools to handle
them relatively freely, seems fair.

The only exception is the *executable* permission.
We recommend, but not require, that tools preserve it.
Given that scripts are generally platform-specific, it seems fitting to
say that keeping them executable is tool-specific behaviour.

Note that while ``git`` preserves executability, ``zip`` (and thus ``wheel``)
doesn't do it natively. (It is possible to encode it in âexternal attributesâ,
but Python's ``ZipFile.extract`` does not honour that.)


Specification
=============

The following will be added to `the PyPA source distribution format spec <https://packaging.python.org/en/latest/specifications/source-distribution-format/>`_
under a new heading, â*Source distribution archive features*â:

Because extracting tar files as-is is dangerous, and the results are
platform-specific, archive features of source distributions are limited.

Unpacking with the data filter
------------------------------

When extracting a source distribution, tools MUST either use
``tarfile.data_filter`` (e.g. ``TarFile.extractall(..., filter='data')``), OR
follow the *Unpacking without the data filter* section below.

As an exception, on Python interpreters without ``hasattr(tarfile, 'data_filter')``
(:pep:`706`), tools that normally use that filter (directly on indirectly)
MAY warn the user and ignore this specification.
The trade-off between usability (e.g. fully trusting the archive) and
security (e.g. refusing to unpack) is left up to the tool in this case.


Unpacking without the data filter
---------------------------------

Tools that do not use the ``data`` filter directly (e.g. for backwards
compatibility, allowing additional features, or not using Python) MUST follow
this section.
(At the time of this writing, the ``data`` filter also follows this section,
but it may get out of sync in the future.)

The following files are invalid in an ``sdist`` archive.
Upon encountering such an entry, tools SHOULD notify the user,
MUST NOT unpack the entry, and MAY abort with a failure:

- Files that would be placed outside the destination directory.
- Links (symbolic or hard) pointing outside the destination directory.
- Device files (including pipes).

The following are also invalid. Tools MAY treat them as above,
but are NOT REQUIRED to do so:

- Files with a ``..`` component in the filename or link target.
- Links pointing to a file that is not part of the archive.

Tools MAY unpack links (symbolic or hard) as regular files,
using content from the archive.

When extracting ``sdist`` archives:

- Leading slashes in file names MUST be dropped.
  (This is nowadays standard behaviour for ``tar`` unpacking.)
- For each ``mode`` (Unix permission) bit, tools MUST either:

  - use the platform's default for a new file/directory (respectively),
  - set the bit according to the archive, or
  - use the bit from ``rw-r--r--`` (``0o644``) for non-executable files or
    ``rwxr-xr-x`` (``0o755``) for executable files and directories.

- High ``mode`` bits (setuid, setgid, sticky) MUST be cleared.
- It is RECOMMENDED to preserve the user *executable* bit.


Further hints
-------------

Tool authors are encouraged to consider how *hints for further
verification* in ``tarfile`` documentation apply for their tool.


Backwards Compatibility
=======================

The existing behaviour is unspecified, and treated differently by different
tools.
This PEP makes the expectations explicit.

There is no known case of backwards incompatibility, but some project out there
probably does rely on details that aren't guaranteed.
This PEP bans the most dangerous of those features, and the rest is
made tool-specific.


Security Implications
=====================

The recommended ``data`` filter is believed safe against common exploits,
and is a single place to amend if flaws are found in the future.

The explicit specification includes protections from the ``data`` filter.


How to Teach This
=================

The PEP is aimed at authors of packaging tools, who should be fine with
a PEP and an updated packaging spec.


Reference Implementation
========================

TBD


Rejected Ideas
==============

None yet.


Open Issues
===========

None yet.


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0292.rst
content:
PEP: 292
Title: Simpler String Substitutions
Author: Barry Warsaw <barry@python.org>
Status: Final
Type: Standards Track
Created: 18-Jun-2002
Python-Version: 2.4
Post-History: 18-Jun-2002, 23-Mar-2004, 22-Aug-2004
Replaces: 215


Abstract
========

This PEP describes a simpler string substitution feature, also
known as string interpolation.  This PEP is "simpler" in two
respects:

1. Python's current string substitution feature
   (i.e. ``%``-substitution) is complicated and error prone.  This PEP
   is simpler at the cost of some expressiveness.

2. :pep:`215` proposed an alternative string interpolation feature,
   introducing a new ``$`` string prefix.  :pep:`292` is simpler than
   this because it involves no syntax changes and has much simpler
   rules for what substitutions can occur in the string.


Rationale
=========

Python currently supports a string substitution syntax based on
C's ``printf()`` '``%``' formatting character [1]_.  While quite rich,
``%``-formatting codes are also error prone, even for
experienced Python programmers.  A common mistake is to leave off
the trailing format character, e.g. the '``s``' in ``"%(name)s"``.

In addition, the rules for what can follow a ``%`` sign are fairly
complex, while the usual application rarely needs such complexity.
Most scripts need to do some string interpolation, but most of
those use simple 'stringification' formats, i.e. ``%s`` or ``%(name)s``
This form should be made simpler and less error prone.


A Simpler Proposal
==================

We propose the addition of a new class, called ``Template``, which
will live in the string module.  The ``Template`` class supports new
rules for string substitution; its value contains placeholders,
introduced with the ``$`` character.  The following rules for
``$``-placeholders apply:

1. ``$$`` is an escape; it is replaced with a single ``$``

2. ``$identifier`` names a substitution placeholder matching a mapping
   key of "identifier".  By default, "identifier" must spell a
   Python identifier as defined in [2]_.  The first non-identifier
   character after the ``$`` character terminates this placeholder
   specification.

3. ``${identifier}`` is equivalent to ``$identifier``.  It is required
   when valid identifier characters follow the placeholder but are
   not part of the placeholder, e.g. ``"${noun}ification"``.

If the ``$`` character appears at the end of the line, or is followed
by any other character than those described above, a ``ValueError``
will be raised at interpolation time.  Values in mapping are
converted automatically to strings.

No other characters have special meaning, however it is possible
to derive from the ``Template`` class to define different substitution
rules.  For example, a derived class could allow for periods in
the placeholder (e.g. to support a kind of dynamic namespace and
attribute path lookup), or could define a delimiter character
other than ``$``.

Once the ``Template`` has been created, substitutions can be performed
by calling one of two methods:

- ``substitute()``.  This method returns a new string which results
  when the values of a mapping are substituted for the
  placeholders in the ``Template``.  If there are placeholders which
  are not present in the mapping, a ``KeyError`` will be raised.

- ``safe_substitute()``.  This is similar to the ``substitute()`` method,
  except that ``KeyErrors`` are never raised (due to placeholders
  missing from the mapping).  When a placeholder is missing, the
  original placeholder will appear in the resulting string.

  Here are some examples::


    >>> from string import Template
    >>> s = Template('${name} was born in ${country}')
    >>> print s.substitute(name='Guido', country='the Netherlands')
    Guido was born in the Netherlands
    >>> print s.substitute(name='Guido')
    Traceback (most recent call last):
    [...]
    KeyError: 'country'
    >>> print s.safe_substitute(name='Guido')
    Guido was born in ${country}

The signature of ``substitute()`` and ``safe_substitute()`` allows for
passing the mapping of placeholders to values, either as a single
dictionary-like object in the first positional argument, or as
keyword arguments as shown above.  The exact details and
signatures of these two methods is reserved for the standard
library documentation.


Why ``$`` and Braces?
=====================

The BDFL said it best [3]_: "The ``$`` means "substitution" in so many
languages besides Perl that I wonder where you've been. [...]
We're copying this from the shell."

Thus the substitution rules are chosen because of the similarity
with so many other languages.  This makes the substitution rules
easier to teach, learn, and remember.


Comparison to PEP 215
=====================

:pep:`215` describes an alternate proposal for string interpolation.
Unlike that PEP, this one does not propose any new syntax for
Python.  All the proposed new features are embodied in a new
library module.  :pep:`215` proposes a new string prefix
representation such as ``$""`` which signal to Python that a new type
of string is present.  ``$``-strings would have to interact with the
existing r-prefixes and u-prefixes, essentially doubling the
number of string prefix combinations.

:pep:`215` also allows for arbitrary Python expressions inside the
``$``-strings, so that you could do things like::

    import sys
    print $"sys = $sys, sys = $sys.modules['sys']"

which would return::

   sys = <module 'sys' (built-in)>, sys = <module 'sys' (built-in)>

It's generally accepted that the rules in :pep:`215` are safe in the
sense that they introduce no new security issues (see :pep:`215`,
"Security Issues" for details).  However, the rules are still
quite complex, and make it more difficult to see the substitution
placeholder in the original ``$``-string.

The interesting thing is that the ``Template`` class defined in this
PEP is designed for inheritance and, with a little extra work,
it's possible to support :pep:`215`'s functionality using existing
Python syntax.

For example, one could define subclasses of ``Template`` and dict that
allowed for a more complex placeholder syntax and a mapping that
evaluated those placeholders.


Internationalization
====================

The implementation supports internationalization by recording the
original template string in the ``Template`` instance's ``template``
attribute.  This attribute would serve as the lookup key in an
gettext-based catalog.  It is up to the application to turn the
resulting string back into a ``Template`` for substitution.

However, the ``Template`` class was designed to work more intuitively
in an internationalized application, by supporting the mixing-in
of ``Template`` and unicode subclasses.  Thus an internationalized
application could create an application-specific subclass,
multiply inheriting from ``Template`` and unicode, and using instances
of that subclass as the gettext catalog key.  Further, the
subclass could alias the special ``__mod__()`` method to either
``.substitute()`` or ``.safe_substitute()`` to provide a more traditional
string/unicode like ``%``-operator substitution syntax.


Reference Implementation
========================

The implementation [4]_ has been committed to the Python 2.4 source tree.

References
==========

.. [1] String Formatting Operations
       https://docs.python.org/release/2.6/library/stdtypes.html#string-formatting-operations

.. [2] Identifiers and Keywords
       https://docs.python.org/release/2.6/reference/lexical_analysis.html#identifiers-and-keywords

.. [3] https://mail.python.org/pipermail/python-dev/2002-June/025652.html

.. [4] Reference Implementation
       http://sourceforge.net/tracker/index.php?func=detail&aid=1014055&group_id=5470&atid=305470

Copyright
=========

This document has been placed in the public domain.


id: pep-0286.rst
content:
PEP: 286
Title: Enhanced Argument Tuples
Author: Martin von LÃ¶wis <martin@v.loewis.de>
Status: Deferred
Type: Standards Track
Created: 03-Mar-2002
Python-Version: 2.3
Post-History:


Abstract
========

``PyArg_ParseTuple`` is confronted with difficult memory management if
an argument converter creates new memory.  To deal with these
cases, a specialized argument type is proposed.


PEP Deferral
============

Further exploration of the concepts covered in this PEP has been deferred
for lack of a current champion interested in promoting the goals of the
PEP and collecting and incorporating feedback, and with sufficient
available time to do so effectively.

The resolution of this PEP may also be affected by the resolution of
:pep:`426`, which proposes the use of a preprocessing step to generate
some aspects of C API interface code.


Problem description
===================

Today, argument tuples keep references to the function arguments,
which are guaranteed to live as long as the argument tuple exists
which is at least as long as the function call is being executed.

In some cases, parsing an argument will allocate new memory, which
is then to be released by the caller.  This has two problems:

1. In case of failure, the application cannot know what memory to
   release; most callers don't even know that they have the
   responsibility to release that memory.  Example for this are
   the ``N`` converter (bug #416288 [1]_) and the ``es#`` converter (bug
   #501716 [2]_).

2. Even for successful argument parsing, it is still inconvenient
   for the caller to be responsible for releasing the memory.  In
   some cases, this is unnecessarily inefficient.  For example,
   the ``es`` converter copies the conversion result into memory, even
   though there already is a string object that has the right
   contents.


Proposed solution
=================

A new type 'argument tuple' is introduced.  This type derives from
tuple, adding an ``__dict__`` member (at ``tp_dictoffset`` -4).  Instances
of this type might get the following attributes:

- 'failobjects', a list of objects which need to be deallocated
  in case of success

- 'okobjects', a list of object which will be released when the
  argument tuple is released

To manage this type, the following functions will be added, and
used appropriately in ``ceval.c`` and ``getargs.c``:

- ``PyArgTuple_New(int);``
- ``PyArgTuple_AddFailObject(PyObject*, PyObject*);``
- ``PyArgTuple_AddFailMemory(PyObject*, void*);``
- ``PyArgTuple_AddOkObject(PyObject*, PyObject*);``
- ``PyArgTuple_AddOkMemory(PyObject*, void*);``
- ``PyArgTuple_ClearFailed(PyObject*);``

When argument parsing fails, all fail objects will be released
through ``Py_DECREF``, and all fail memory will be released through
``PyMem_Free``.  If parsing succeeds, the references to the fail
objects and fail memory are dropped, without releasing anything.

When the argument tuple is released, all ok objects and memory
will be released.

If those functions are called with an object of a different type,
a warning is issued and no further action is taken; usage of the
affected converters without using argument tuples is deprecated.


Affected converters
===================

The following converters will add fail memory and fail objects: ``N``,
``es``, ``et``, ``es#``, ``et#`` (unless memory is passed into the converter)


New converters
==============

To simplify Unicode conversion, the ``e*`` converters are duplicated
as ``E*`` converters (``Es``, ``Et``, ``Es#``, ``Et#``).  The usage of the ``E*``
converters is identical to that of the ``e*`` converters, except that
the application will not need to manage the resulting memory.
This will be implemented through registration of Ok objects with
the argument tuple.  The ``e*`` converters are deprecated.


References
==========

.. [1] infrequent memory leak in pyexpat
   (http://bugs.python.org/issue416288)

.. [2] "es#" parser marker leaks memory
   (http://bugs.python.org/issue501716)


Copyright
=========

This document has been placed in the public domain.


id: pep-0279.rst
content:
PEP: 279
Title: The enumerate() built-in function
Author: Raymond Hettinger <python@rcn.com>
Status: Final
Type: Standards Track
Created: 30-Jan-2002
Python-Version: 2.3
Post-History:


Abstract
========

This PEP introduces a new built-in function, ``enumerate()`` to
simplify a commonly used looping idiom.  It provides all iterable
collections with the same advantage that ``iteritems()`` affords to
dictionaries -- a compact, readable, reliable index notation.


Rationale
=========

Python 2.2 introduced the concept of an iterable interface as
proposed in :pep:`234`.  The ``iter()`` factory function was provided
as common calling convention and deep changes were made to use
iterators as a unifying theme throughout Python.  The unification
came in the form of establishing a common iterable interface for
mappings, sequences, and file objects.

Generators, as proposed in :pep:`255`, were introduced as a means
for making it easier to create iterators, especially ones with
complex internal execution or variable states.  The availability
of generators makes it possible to improve on the loop counter
ideas in :pep:`212`.  Those ideas provided a clean syntax for
iteration with indices and values, but did not apply to all
iterable objects.  Also, that approach did not have the memory
friendly benefit provided by generators which do not evaluate the
entire sequence all at once.

The new proposal is to add a built-in function, ``enumerate()`` which
was made possible once iterators and generators became available.
It provides all iterables with the same advantage that ``iteritems()``
affords to dictionaries -- a compact, readable, reliable index
notation.  Like ``zip()``, it is expected to become a commonly used
looping idiom.

This suggestion is designed to take advantage of the existing
implementation and require little additional effort to
incorporate.  It is backwards compatible and requires no new
keywords.  The proposal will go into Python 2.3 when generators
become final and are not imported from ``__future__``.


BDFL Pronouncements
===================

The new built-in function is ACCEPTED.


Specification for a new built-in
================================

::

    def enumerate(collection):
       'Generates an indexed series:  (0,coll[0]), (1,coll[1]) ...'
       i = 0
       it = iter(collection)
       while 1:
          yield (i, it.next())
          i += 1

Note A: :pep:`212` Loop Counter Iteration discussed several
proposals for achieving indexing.  Some of the proposals only work
for lists unlike the above function which works for any generator,
xrange, sequence, or iterable object.  Also, those proposals were
presented and evaluated in the world prior to Python 2.2 which did
not include generators.  As a result, the non-generator version in
:pep:`212` had the disadvantage of consuming memory with a giant list
of tuples.  The generator version presented here is fast and
light, works with all iterables, and allows users to abandon the
sequence in mid-stream with no loss of computation effort.

There are other PEPs which touch on related issues: integer
iterators, integer for-loops, and one for modifying the arguments
to ``range`` and ``xrange``.  The ``enumerate()`` proposal does not preclude
the other proposals and it still meets an important need even if
those are adopted -- the need to count items in any iterable.  The
other proposals give a means of producing an index but not the
corresponding value.  This is especially problematic if a sequence
is given which doesn't support random access such as a file
object, generator, or sequence defined with ``__getitem__``.

Note B: Almost all of the PEP reviewers welcomed the function but
were divided as to whether there should be any built-ins.  The
main argument for a separate module was to slow the rate of
language inflation.  The main argument for a built-in was that the
function is destined to be part of a core programming style,
applicable to any object with an iterable interface.  Just as
``zip()`` solves the problem of looping over multiple sequences, the
``enumerate()`` function solves the loop counter problem.

If only one built-in is allowed, then ``enumerate()`` is the most
important general purpose tool, solving the broadest class of
problems while improving program brevity, clarity and reliability.

Note C:  Various alternative names were discussed:

=================  =============================================================
``iterindexed()``  five syllables is a mouthful
``index()``        nice verb but could be confused the ``.index()`` method
``indexed()``      widely liked however adjectives should be avoided
``indexer()``      noun did not read well in a for-loop
``count()``        direct and explicit but often used in other contexts
``itercount()``    direct, explicit and hated by more than one person
``iteritems()``    conflicts with key:value concept for dictionaries
``itemize()``      confusing because ``amap.items()`` != ``list(itemize(amap))``
``enum()``         pithy; less clear than enumerate; too similar to enum
                   in other languages where it has a different meaning
=================  =============================================================

All of the names involving 'count' had the further disadvantage of
implying that the count would begin from one instead of zero.

All of the names involving 'index' clashed with usage in database
languages where indexing implies a sorting operation rather than
linear sequencing.

Note D: This function was originally proposed with optional start
and stop arguments.  GvR pointed out that the function call
``enumerate(seqn,4,6)`` had an alternate, plausible interpretation as
a slice that would return the fourth and fifth elements of the
sequence.  To avoid the ambiguity, the optional arguments were
dropped even though it meant losing flexibility as a loop counter.
That flexibility was most important for the common case of
counting from one, as in::

    for linenum, line in enumerate(source,1):  print linenum, line


Comments from GvR:
    ``filter`` and ``map`` should die and be subsumed into list
    comprehensions, not grow more variants. I'd rather introduce
    built-ins that do iterator algebra (e.g. the iterzip that I've
    often used as an example).

    I like the idea of having some way to iterate over a sequence
    and its index set in parallel.  It's fine for this to be a
    built-in.

    I don't like the name "indexed"; adjectives do not make good
    function names.  Maybe ``iterindexed()``?


Comments from Ka-Ping Yee:
    I'm also quite happy with everything  you
    proposed ... and the extra built-ins (really 'indexed' in
    particular) are things I have wanted for a long time.


Comments from Neil Schemenauer:
    The new built-ins sound okay.  Guido
    may be concerned with increasing the number of built-ins too
    much.  You might be better off selling them as part of a
    module.  If you use a module then you can add lots of useful
    functions (Haskell has lots of them that we could steal).


Comments for Magnus Lie Hetland:
    I think indexed would be a useful and
    natural built-in function. I would certainly use it a lot.  I
    like ``indexed()`` a lot; +1. I'm quite happy to have it make PEP
    281 obsolete. Adding a separate module for iterator utilities
    seems like a good idea.


Comments from the Community:
    The response to the ``enumerate()`` proposal
    has been close to 100% favorable.  Almost everyone loves the
    idea.


Author response:
    Prior to these comments, four built-ins were proposed.
    After the comments, ``xmap``, ``xfilter`` and ``xzip`` were withdrawn.  The
    one that remains is vital for the language and is proposed by
    itself.  ``Indexed()`` is trivially easy to implement and can be
    documented in minutes.  More importantly, it is useful in
    everyday programming which does not otherwise involve explicit
    use of generators.


id: pep-0251.rst
content:
PEP: 251
Title: Python 2.2 Release Schedule
Author: Barry Warsaw <barry@python.org>, Guido van Rossum <guido@python.org>
Status: Final
Type: Informational
Topic: Release
Created: 17-Apr-2001
Python-Version: 2.2
Post-History: 14-Aug-2001


Abstract
========

This document describes the Python 2.2 development and release
schedule.  The schedule primarily concerns itself with PEP-sized
items.  Small bug fixes and changes will occur up until the first
beta release.

The schedule below represents the actual release dates of Python
2.2.  Note that any subsequent maintenance releases of Python 2.2
should be covered by separate PEPs.


Release Schedule
================

Tentative future release dates.  Note that we've slipped this
compared to the schedule posted around the release of 2.2a1.

* 21-Dec-2001: 2.2   [Released] (final release)
* 14-Dec-2001: 2.2c1 [Released]
* 14-Nov-2001: 2.2b2 [Released]
* 19-Oct-2001: 2.2b1 [Released]
* 28-Sep-2001: 2.2a4 [Released]
* 7-Sep-2001: 2.2a3 [Released]
* 22-Aug-2001: 2.2a2 [Released]
* 18-Jul-2001: 2.2a1 [Released]


Release Manager
===============

Barry Warsaw was the Python 2.2 release manager.


Release Mechanics
=================

We experimented with a new mechanism for releases: a week before
every alpha, beta or other release, we forked off a branch which
became the release.  Changes to the branch are limited to the
release manager and his designated 'bots.  This experiment was
deemed a success and should be observed for future releases.  See
:pep:`101` for the actual release mechanics.


New features for Python 2.2
===========================

The following new features are introduced in Python 2.2.  For a
more detailed account, see Misc/NEWS [2]_ in the Python
distribution, or Andrew Kuchling's "What's New in Python 2.2"
document [3]_.

- iterators (:pep:`234`)
- generators (:pep:`255`)
- unifying long ints and plain ints (:pep:`237`)
- division (:pep:`238`)
- unification of types and classes (:pep:`252`, :pep:`253`)


References
==========

.. [2] Misc/NEWS file from CVS
       http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Misc/NEWS?rev=1.337.2.4&content-type=text/vnd.viewcvs-markup

.. [3] Andrew Kuchling, What's New in Python 2.2
       http://www.python.org/doc/2.2.1/whatsnew/whatsnew22.html


Copyright
=========

This document has been placed in the public domain.


id: pep-0537.rst
content:
PEP: 537
Title: Python 3.7 Release Schedule
Author: Ned Deily <nad@python.org>
Status: Final
Type: Informational
Topic: Release
Created: 23-Dec-2016
Python-Version: 3.7


Abstract
========

This document describes the development and release schedule for
Python 3.7.  The schedule primarily concerns itself with PEP-sized
items.


Release Manager and Crew
========================

- 3.7 Release Manager: Ned Deily
- Windows installers: Steve Dower
- Mac installers: Ned Deily
- Documentation: Julien Palard


3.7 Lifespan
============

3.7 will receive `bugfix updates <https://devguide.python.org/devcycle/#maintenance-branches>`_
approximately every 3 months for about 24 months. Sometime after the release of
3.8.0 final, a final 3.7 bugfix update will be released.
After that, it is expected that
`security updates <https://devguide.python.org/devcycle/#security-branches>`_
(source only) will be released as needed until 5 years after
the release of 3.7 final, so until approximately 2023-06.

As of 2023-06-27, 3.7 has reached the
`end-of-life phase <https://devguide.python.org/devcycle/#end-of-life-branches>`_
of its release cycle. 3.7.17 was the final security release. The code base for
3.7 is now frozen and no further updates will be provided nor issues of any
kind will be accepted on the bug tracker.


Release Schedule
================

3.7.0 schedule
--------------

- 3.7 development begins: 2016-09-12
- 3.7.0 alpha 1: 2017-09-19
- 3.7.0 alpha 2: 2017-10-17
- 3.7.0 alpha 3: 2017-12-05
- 3.7.0 alpha 4: 2018-01-09
- 3.7.0 beta 1: 2018-01-31
  (No new features beyond this point.)
- 3.7.0 beta 2: 2018-02-27
- 3.7.0 beta 3: 2018-03-29
- 3.7.0 beta 4: 2018-05-02
- 3.7.0 beta 5: 2018-05-30
- 3.7.0 candidate 1: 2018-06-12
- 3.7.0 final: 2018-06-27

3.7.1 schedule (first bugfix release)
-------------------------------------

- 3.7.1 candidate 1: 2018-09-26
- 3.7.1 candidate 2: 2018-10-13
- 3.7.1 final: 2018-10-20

3.7.2 schedule
--------------

- 3.7.2 candidate 1: 2018-12-11
- 3.7.2 final: 2018-12-24

3.7.3 schedule
--------------

- 3.7.3 candidate 1: 2019-03-12
- 3.7.3 final: 2019-03-25

3.7.4 schedule
--------------

- 3.7.4 candidate 1: 2019-06-18
- 3.7.4 candidate 2: 2019-07-02
- 3.7.4 final: 2019-07-08

3.7.5 schedule
--------------

- 3.7.5 candidate 1: 2019-10-02
- 3.7.5 final: 2019-10-15

3.7.6 schedule
--------------

- 3.7.6 candidate 1: 2019-12-11
- 3.7.6 final: 2019-12-18

3.7.7 schedule
--------------

- 3.7.7 candidate 1: 2020-03-04
- 3.7.7 final: 2020-03-10

3.7.8 schedule (last bugfix release)
------------------------------------

Last planned release of binaries

- 3.7.8 candidate 1: 2020-06-15
- 3.7.8 final: 2020-06-27

3.7.9 schedule (security/binary release)
----------------------------------------

Security fixes plus updated binary installers
to address 3.7.8 issues; no further binary
releases are planned.

- 3.7.9 final: 2020-08-17

3.7.10 schedule
---------------

- 3.7.10 final: 2021-02-15

3.7.11 schedule
---------------

- 3.7.11 final: 2021-06-28

3.7.12 schedule
---------------

- 3.7.12 final: 2021-09-04

3.7.13 schedule
---------------

- 3.7.13 final: 2022-03-16

3.7.14 schedule
---------------

- 3.7.14 final: 2022-09-06

3.7.15 schedule
---------------

- 3.7.15 final: 2022-10-11

3.7.16 schedule
---------------

- 3.7.16 final: 2022-12-06

3.7.17 schedule (last security-only release)
--------------------------------------------

- 3.7.17 final: 2023-06-06


Features for 3.7
================

Implemented PEPs for 3.7 (as of 3.7.0 beta 1):

* :pep:`538`, Coercing the legacy C locale to a UTF-8 based locale
* :pep:`539`, A New C-API for Thread-Local Storage in CPython
* :pep:`540`, ``UTF-8`` mode
* :pep:`552`, Deterministic ``pyc``
* :pep:`553`, Built-in breakpoint()
* :pep:`557`, Data Classes
* :pep:`560`, Core support for typing module and generic types
* :pep:`562`, Module ``__getattr__`` and ``__dir__``
* :pep:`563`, Postponed Evaluation of Annotations
* :pep:`564`, Time functions with nanosecond resolution
* :pep:`565`, Show DeprecationWarning in __main__
* :pep:`567`, Context Variables

Copyright
=========

This document has been placed in the public domain.


id: pep-0523.rst
content:
PEP: 523
Title: Adding a frame evaluation API to CPython
Author: Brett Cannon <brett@python.org>,
        Dino Viehland <dinov@microsoft.com>
Status: Final
Type: Standards Track
Created: 16-May-2016
Python-Version: 3.6
Post-History: 16-May-2016
Resolution: https://mail.python.org/pipermail/python-dev/2016-August/145937.html


Abstract
========

This PEP proposes to expand CPython's C API [#c-api]_ to allow for
the specification of a per-interpreter function pointer to handle the
evaluation of frames [#pyeval_evalframeex]_. This proposal also
suggests adding a new field to code objects [#pycodeobject]_ to store
arbitrary data for use by the frame evaluation function.


Rationale
=========

One place where flexibility has been lacking in Python is in the direct
execution of Python code. While CPython's C API [#c-api]_ allows for
constructing the data going into a frame object and then evaluating it
via ``PyEval_EvalFrameEx()`` [#pyeval_evalframeex]_, control over the
execution of Python code comes down to individual objects instead of a
holistic control of execution at the frame level.

While wanting to have influence over frame evaluation may seem a bit
too low-level, it does open the possibility for things such as a
method-level JIT to be introduced into CPython without CPython itself
having to provide one. By allowing external C code to control frame
evaluation, a JIT can participate in the execution of Python code at
the key point where evaluation occurs. This then allows for a JIT to
conditionally recompile Python bytecode to machine code as desired
while still allowing for executing regular CPython bytecode when
running the JIT is not desired. This can be accomplished by allowing
interpreters to specify what function to call to evaluate a frame. And
by placing the API at the frame evaluation level it allows for a
complete view of the execution environment of the code for the JIT.

This ability to specify a frame evaluation function also allows for
other use-cases beyond just opening CPython up to a JIT. For instance,
it would not be difficult to implement a tracing or profiling function
at the call level with this API. While CPython does provide the
ability to set a tracing or profiling function at the Python level,
this would be able to match the data collection of the profiler and
quite possibly be faster for tracing by simply skipping per-line
tracing support.

It also opens up the possibility of debugging where the frame
evaluation function only performs special debugging work when it
detects it is about to execute a specific code object. In that
instance the bytecode could be theoretically rewritten in-place to
inject a breakpoint function call at the proper point for help in
debugging while not having to do a heavy-handed approach as
required by ``sys.settrace()``.

To help facilitate these use-cases, we are also proposing the adding
of a "scratch space" on code objects via a new field. This will allow
per-code object data to be stored with the code object itself for easy
retrieval by the frame evaluation function as necessary. The field
itself will simply be a ``PyObject *`` type so that any data stored in
the field will participate in normal object memory management.


Proposal
========

All proposed C API changes below will not be part of the stable ABI.


Expanding ``PyCodeObject``
--------------------------

One field is to be added to the ``PyCodeObject`` struct
[#pycodeobject]_::

  typedef struct {
     ...
     void *co_extra;  /* "Scratch space" for the code object. */
  } PyCodeObject;

The ``co_extra`` will be ``NULL`` by default and only filled in as
needed. Values stored in the field are expected to not be required
in order for the code object to function, allowing the loss of the
data of the field to be acceptable.

A private API has been introduced to work with the field::

  PyAPI_FUNC(Py_ssize_t) _PyEval_RequestCodeExtraIndex(freefunc);
  PyAPI_FUNC(int) _PyCode_GetExtra(PyObject *code, Py_ssize_t index,
                                   void **extra);
  PyAPI_FUNC(int) _PyCode_SetExtra(PyObject *code, Py_ssize_t index,
                                   void *extra);

Users of the field are expected to call
``_PyEval_RequestCodeExtraIndex()`` to receive (what should be
considered) an opaque index value to adding data into ``co-extra``.
With that index, users can set data using ``_PyCode_SetExtra()`` and
later retrieve the data with ``_PyCode_GetExtra()``. The API is
purposefully listed as private to communicate the fact that there are
no semantic guarantees of the API between Python releases.

Using a list and tuple were considered but was found to be less
performant, and with a key use-case being JIT usage the performance
consideration won out for using a custom struct instead of a Python
object.

A dict was also considered, but once again performance was more
important. While a dict will have constant overhead in looking up
data, the overhead for the common case of a single object being stored
in the data structure leads to a tuple having better performance
characteristics (i.e. iterating a tuple of length 1 is faster than
the overhead of hashing and looking up an object in a dict).


Expanding ``PyInterpreterState``
--------------------------------

The entrypoint for the frame evaluation function is per-interpreter::

  // Same type signature as PyEval_EvalFrameEx().
  typedef PyObject* (*_PyFrameEvalFunction)(PyFrameObject*, int);

  typedef struct {
      ...
      _PyFrameEvalFunction eval_frame;
  } PyInterpreterState;

By default, the ``eval_frame`` field will be initialized to a function
pointer that represents what ``PyEval_EvalFrameEx()`` currently is
(called ``_PyEval_EvalFrameDefault()``, discussed later in this PEP).
Third-party code may then set their own frame evaluation function
instead to control the execution of Python code. A pointer comparison
can be used to detect if the field is set to
``_PyEval_EvalFrameDefault()`` and thus has not been mutated yet.


Changes to ``Python/ceval.c``
-----------------------------

``PyEval_EvalFrameEx()`` [#pyeval_evalframeex]_ as it currently stands
will be renamed to ``_PyEval_EvalFrameDefault()``. The new
``PyEval_EvalFrameEx()`` will then become::

    PyObject *
    PyEval_EvalFrameEx(PyFrameObject *frame, int throwflag)
    {
        PyThreadState *tstate = PyThreadState_GET();
        return tstate->interp->eval_frame(frame, throwflag);
    }

This allows third-party code to place themselves directly in the path
of Python code execution while being backwards-compatible with code
already using the pre-existing C API.


Updating ``python-gdb.py``
--------------------------

The generated ``python-gdb.py`` file used for Python support in GDB
makes some hard-coded assumptions about ``PyEval_EvalFrameEx()``, e.g.
the names of local variables. It will need to be updated to work with
the proposed changes.


Performance impact
==================

As this PEP is proposing an API to add pluggability, performance
impact is considered only in the case where no third-party code has
made any changes.

Several runs of pybench [#pybench]_ consistently showed no performance
cost from the API change alone.

A run of the Python benchmark suite [#py-benchmarks]_ showed no
measurable cost in performance.

In terms of memory impact, since there are typically not many CPython
interpreters executing in a single process that means the impact of
``co_extra`` being added to ``PyCodeObject`` is the only worry.
According to [#code-object-count]_, a run of the Python test suite
results in about 72,395 code objects being created. On a 64-bit
CPU that would result in 579,160 bytes of extra memory being used if
all code objects were alive at once and had nothing set in their
``co_extra`` fields.


Example Usage
=============

A JIT for CPython
-----------------

Pyjion
''''''

The Pyjion project [#pyjion]_ has used this proposed API to implement
a JIT for CPython using the CoreCLR's JIT [#


id: pep-0245.rst
content:
PEP: 245
Title: Python Interface Syntax
Author: Michel Pelletier <michel@users.sourceforge.net>
Status: Rejected
Type: Standards Track
Created: 11-Jan-2001
Python-Version: 2.2
Post-History: 21-Mar-2001


.. note::

   The no-longer-available Zope interfaces wiki page
   (``https://www.zope.org/Wikis/Interfaces``) originally linked here,
   containing links to further resources for this PEP,
   can be `found on the Wayback Machine archive
   <https://web.archive.org/web/20050327013919/http://www.zope.org/Wikis/Interfaces/FrontPage>`__.
   Also, the Interface-Dev Zope mailing list on which this PEP was discussed
   was shut down, but `its archives remain available
   <https://mail.zope.dev/pipermail/interface-dev/>`__.


Rejection Notice
================

I'm rejecting this PEP.  It's been five years now.  While at some
point I expect that Python will have interfaces, it would be naive
to expect it to resemble the syntax in this PEP.  Also, :pep:`246` is
being rejected in favor of something completely different; interfaces
won't play a role in adaptation or whatever will replace it.  GvR.


Introduction
============

This PEP describes a proposed syntax for creating interface
objects in Python.


Overview
========

In addition to thinking about adding a static type system to
Python, the Types-SIG was also charged to devise an interface
system for Python.  In December of 1998, Jim Fulton released a
prototype interfaces system based on discussions from the SIG.
Many of the issues and background information on this discussion
and prototype can be found in the SIG archives [1]_.

Around the end of 2000, Digital Creations began thinking about
better component model designs for Zope [2]_.  Zope's future
component model relies heavily on interface objects.  This led to
further development of Jim's "Scarecrow" interfaces prototype.
Starting with version 2.3, Zope comes with an Interface package as
standard software.  Zope's Interface package is used as the
reference implementation for this PEP.

The syntax proposed by this PEP relies on syntax enhancements
describe in :pep:`232` and describes an underlying framework
which :pep:`233` could be based upon.  There is some work being
done with regard to interface objects and Proxy objects, so for
those optional parts of this PEP you may want to see [3]_.


The Problem
===========

Interfaces are important because they solve a number of problems
that arise while developing software:

- There are many implied interfaces in Python, commonly referred
  to as "protocols".  Currently determining those protocols is
  based on implementation introspection, but often that also
  fails.  For example, defining ``__getitem__`` implies both a
  sequence and a mapping (the former with sequential, integer
  keys).  There is no way for the developer to be explicit about
  which protocols the object intends to implement.

- Python is limited, from the developer's point of view, by the
  split between types and classes.  When types are expected, the
  consumer uses code like 'type(foo) == type("")' to determine if
  'foo' is a string.  When instances of classes are expected, the
  consumer uses 'isinstance(foo, MyString)' to determine if 'foo'
  is an instance of the 'MyString' class.  There is no unified
  model for determining if an object can be used in a certain,
  valid way.

- Python's dynamic typing is very flexible and powerful, but it
  does not have the advantage of static typed languages that
  provide type checking.  Static typed languages provide you with
  much more type safety, but are often overly verbose because
  objects can only be generalized by common subclassing and used
  specifically with casting (for example, in Java).

There are also a number of documentation problems that interfaces
try to solve.

- Developers waste a lot of time looking at the source code of
  your system to figure out how objects work.

- Developers who are new to your system may misunderstand how your
  objects work, causing, and possibly propagating, usage errors.

- Because a lack of interfaces means usage is inferred from the
  source, developers may end up using methods and attributes that
  are meant for "internal use only".

- Code inspection can be hard, and very discouraging to novice
  programmers trying to properly understand code written by gurus.

- A lot of time is wasted when many people try very hard to
  understand obscurity (like undocumented software).  Effort spend
  up front documenting interfaces will save much of this time in
  the end.

Interfaces try to solve these problems by providing a way for you
to specify a contractual obligation for your object, documentation
on how to use an object, and a built-in mechanism for discovering
the contract and the documentation.

Python has very useful introspection features.  It is well known
that this makes exploring concepts in the interactive interpreter
easier, because Python gives you the ability to look at all kinds
of information about the objects: the type, doc strings, instance
dictionaries, base classes, unbound methods and more.

Many of these features are oriented toward introspecting, using
and changing the implementation of software, and one of them ("doc
strings") is oriented toward providing documentation.  This
proposal describes an extension to this natural introspection
framework that describes an object's interface.


Overview of the Interface Syntax
================================

For the most part, the syntax of interfaces is very much like the
syntax of classes, but future needs, or needs brought up in
discussion, may define new possibilities for interface syntax.

A formal BNF description of the syntax is givena later in the PEP,
for the purposes of illustration, here is an example of two
different interfaces created with the proposed syntax::

    interface CountFishInterface:
        "Fish counting interface"

        def oneFish():
            "Increments the fish count by one"

        def twoFish():
            "Increments the fish count by two"

        def getFishCount():
            "Returns the fish count"

    interface ColorFishInterface:
        "Fish coloring interface"

        def redFish():
            "Sets the current fish color to red"

        def blueFish():
            "Sets the current fish color to blue"

        def getFishColor():
            "This returns the current fish color"

This code, when evaluated, will create two interfaces called
``CountFishInterface`` and ``ColorFishInterface``. These interfaces
are defined by the ``interface`` statement.

The prose documentation for the interfaces and their methods come
from doc strings.  The method signature information comes from the
signatures of the ``def`` statements.  Notice how there is no body
for the def statements.  The interface does not implement a
service to anything; it merely describes one.  Documentation
strings on interfaces and interface methods are mandatory, a
'pass' statement cannot be provided.  The interface equivalent of
a pass statement is an empty doc string.

You can also create interfaces that "extend" other interfaces.
Here, you can see a new type of Interface that extends the
CountFishInterface and ColorFishInterface::

    interface FishMarketInterface(CountFishInterface, ColorFishInterface):
        "This is the documentation for the FishMarketInterface"

        def getFishMonger():
            "Returns the fish monger you can interact with"

        def hireNewFishMonger(name):
            "Hire a new fish monger"

        def buySomeFish(quantity=1):
            "Buy some fish at the market"

The FishMarketInterface extends upon the CountFishInterface and
ColorfishInterface.


Interface Assertion
===================

The next step is to put classes and interfaces together by
creating a concrete Python class that asserts that it implements
an interface.  Here is an example FishMarket component that might
do this::

    class FishError(Error):


id: pep-0269.rst
content:
PEP: 269
Title: Pgen Module for Python
Author: Jonathan Riehl <jriehl@spaceship.com>
Status: Deferred
Type: Standards Track
Created: 24-Aug-2001
Python-Version: 2.2
Post-History:


Abstract
========

Much like the ``parser`` module exposes the Python parser, this PEP
proposes that the parser generator used to create the Python
parser, ``pgen``, be exposed as a module in Python.


Rationale
=========

Through the course of Pythonic history, there have been numerous
discussions about the creation of a Python compiler [1]_.  These
have resulted in several implementations of Python parsers, most
notably the ``parser`` module currently provided in the Python
standard library [2]_ and Jeremy Hylton's ``compiler`` module [3]_.
However, while multiple language changes have been proposed
[4]_ [5]_, experimentation with the Python syntax has lacked the
benefit of a Python binding to the actual parser generator used to
build Python.

By providing a Python wrapper analogous to Fred Drake Jr.'s parser
wrapper, but targeted at the ``pgen`` library, the following
assertions are made:

1. Reference implementations of syntax changes will be easier to
   develop.  Currently, a reference implementation of a syntax
   change would require the developer to use the ``pgen`` tool from
   the command line.  The resulting parser data structure would
   then either have to be reworked to interface with a custom
   CPython implementation, or wrapped as a C extension module.

2. Reference implementations of syntax changes will be easier to
   distribute.  Since the parser generator will be available in
   Python, it should follow that the resulting parser will
   accessible from Python.  Therefore, reference implementations
   should be available as pure Python code, versus using custom
   versions of the existing CPython distribution, or as compilable
   extension modules.

3. Reference implementations of syntax changes will be easier to
   discuss with a larger audience.  This somewhat falls out of the
   second assertion, since the community of Python users is most
   likely larger than the community of CPython developers.

4. Development of small languages in Python will be further
   enhanced, since the additional module will be a fully
   functional LL(1) parser generator.


Specification
=============

The proposed module will be called ``pgen``.  The ``pgen`` module will
contain the following functions:


``parseGrammarFile (fileName) -> AST``
--------------------------------------

The ``parseGrammarFile()`` function will read the file pointed to
by fileName and create an AST object.  The AST nodes will
contain the nonterminal, numeric values of the parser
generator meta-grammar.  The output AST will be an instance of
the AST extension class as provided by the ``parser`` module.
Syntax errors in the input file will cause the SyntaxError
exception to be raised.


``parseGrammarString (text) -> AST``
------------------------------------

The ``parseGrammarString()`` function will follow the semantics of
the ``parseGrammarFile()``, but accept the grammar text as a
string for input, as opposed to the file name.


``buildParser (grammarAst) -> DFA``
-----------------------------------

The ``buildParser()`` function will accept an AST object for input
and return a DFA (deterministic finite automaton) data
structure.  The DFA data structure will be a C extension
class, much like the AST structure is provided in the ``parser``
module.  If the input AST does not conform to the nonterminal
codes defined for the ``pgen`` meta-grammar, ``buildParser()`` will
throw a ``ValueError`` exception.


``parseFile (fileName, dfa, start) -> AST``
-------------------------------------------

The ``parseFile()`` function will essentially be a wrapper for the
``PyParser_ParseFile()`` C API function.  The wrapper code will
accept the DFA C extension class, and the file name.  An AST
instance that conforms to the lexical values in the ``token``
module and the nonterminal values contained in the DFA will be
output.


``parseString (text, dfa, start) -> AST``
-----------------------------------------

The ``parseString()`` function will operate in a similar fashion
to the ``parseFile()`` function, but accept the parse text as an
argument.  Much like ``parseFile()`` will wrap the
``PyParser_ParseFile()`` C API function, ``parseString()`` will wrap
the ``PyParser_ParseString()`` function.


``symbolToStringMap (dfa) -> dict``
-----------------------------------

The ``symbolToStringMap()`` function will accept a DFA instance
and return a dictionary object that maps from the DFA's
numeric values for its nonterminals to the string names of the
nonterminals as found in the original grammar specification
for the DFA.


``stringToSymbolMap (dfa) -> dict``
-----------------------------------

The ``stringToSymbolMap()`` function output a dictionary mapping
the nonterminal names of the input DFA to their corresponding
numeric values.


Extra credit will be awarded if the map generation functions and
parsing functions are also methods of the DFA extension class.


Implementation Plan
===================

A cunning plan has been devised to accomplish this enhancement:

1. Rename the ``pgen`` functions to conform to the CPython naming
   standards.  This action may involve adding some header files to
   the ``Include`` subdirectory.

2. Move the ``pgen`` C modules in the Makefile.pre.in from unique ``pgen``
   elements to the Python C library.

3. Make any needed changes to the ``parser`` module so the AST
   extension class understands that there are AST types it may not
   understand.  Cursory examination of the AST extension class
   shows that it keeps track of whether the tree is a suite or an
   expression.

3. Code an additional C module in the ``Modules`` directory.  The C
   extension module will implement the DFA extension class and the
   functions outlined in the previous section.

4. Add the new module to the build process.  Black magic, indeed.


Limitations
===========

Under this proposal, would be designers of Python 3000 will still
be constrained to Python's lexical conventions.  The addition,
subtraction or modification of the Python lexer is outside the
scope of this PEP.


Reference Implementation
========================

No reference implementation is currently provided. A patch
was provided at some point in
http://sourceforge.net/tracker/index.php?func=detail&aid=599331&group_id=5470&atid=305470
but that patch is no longer maintained.


References
==========

.. [1] The (defunct) Python Compiler-SIG
       http://www.python.org/sigs/compiler-sig/

.. [2] Parser Module Documentation
       http://docs.python.org/library/parser.html

.. [3] Hylton, Jeremy.
       http://docs.python.org/library/compiler.html

.. [4] Pelletier, Michel. "Python Interface Syntax", :pep:`245`

.. [5] The Python Types-SIG
       http://www.python.org/sigs/types-sig/


Copyright
=========

This document has been placed in the public domain.


id: pep-0241.rst
content:
PEP: 241
Title: Metadata for Python Software Packages
Author: A.M. Kuchling <amk@amk.ca>
Discussions-To: distutils-sig@python.org
Status: Superseded
Type: Standards Track
Topic: Packaging
Created: 12-Mar-2001
Post-History: `19-Mar-2001 <https://mail.python.org/archives/list/distutils-sig@python.org/thread/46XPDHQHI3XAAJHEZAMAMKZYAI6K7NB6/>`__
Superseded-By: 314

.. canonical-pypa-spec:: :ref:`packaging:core-metadata`


Introduction
============

This PEP describes a mechanism for adding metadata to Python
packages.  It includes specifics of the field names, and their
semantics and usage.


Including Metadata in Packages
==============================

The Distutils 'sdist' command will be modified to extract the
metadata fields from the arguments and write them to a file in the
generated zipfile or tarball.  This file will be named PKG-INFO
and will be placed in the top directory of the source
distribution (where the README, INSTALL, and other files usually
go).

Developers may not provide their own PKG-INFO file.  The "sdist"
command will, if it detects an existing PKG-INFO file, terminate
with an appropriate error message.  This should prevent confusion
caused by the PKG-INFO and setup.py files being out of sync.

The PKG-INFO file format is a single set of :rfc:`822` headers
parseable by the rfc822.py module.  The field names listed in the
following section are used as the header names.  There's no
extension mechanism in this simple format; the Catalog and Distutils
SIGs will aim at getting a more flexible format ready for Python 2.2.


Fields
======

This section specifies the names and semantics of each of the
supported metadata fields.

Fields marked with "(Multiple use)" may be specified multiple
times in a single PKG-INFO file.  Other fields may only occur
once in a PKG-INFO file.  Fields marked with "(optional)" are
not required to appear in a valid PKG-INFO file, all other
fields must be present.

Metadata-Version
----------------

Version of the file format; currently "1.0" is the only
legal value here.

Example::

    Metadata-Version: 1.0

Name
----

The name of the package.

Example::

    Name: BeagleVote

Version
-------

A string containing the package's version number.  This
field should be parseable by one of the Version classes
(StrictVersion or LooseVersion) in the distutils.version
module.

Example::

    Version: 1.0a2

Platform (multiple use)
-----------------------

A comma-separated list of platform specifications, summarizing
the operating systems supported by the package.  The major
supported platforms are listed below, but this list is
necessarily incomplete.

::

    POSIX, MacOS, Windows, BeOS, Palm OS.

Binary distributions will use the Supported-Platform field in
their metadata to specify the OS and CPU for which the binary
package was compiled.  The semantics of the Supported-Platform
are not specified in this PEP.

Example::

    Platform: POSIX, Windows

Summary
-------

A one-line summary of what the package does.

Example::

    Summary: A module for collecting votes from beagles.

Description (optional)
----------------------

A longer description of the package that can run to several
paragraphs.  (Software that deals with metadata should not
assume any maximum size for this field, though one hopes that
people won't include their instruction manual as the
long-description.)

Example::

    Description: This module collects votes from beagles
                 in order to determine their electoral wishes.
                 Do NOT try to use this module with basset hounds;
                 it makes them grumpy.

Keywords (optional)
-------------------

A list of additional keywords to be used to assist searching
for the package in a larger catalog.

Example::

    Keywords: dog puppy voting election

Home-page (optional)
--------------------

A string containing the URL for the package's home page.

Example::

    Home-page: http://www.example.com/~cschultz/bvote/

Author (optional)
-----------------

A string containing at a minimum the author's name.  Contact
information can also be added, separating each line with
newlines.

Example::

    Author: C. Schultz
            Universal Features Syndicate
            Los Angeles, CA

Author-email
------------

A string containing the author's e-mail address.  It can contain
a name and e-mail address in the legal forms for a :rfc:`822`
'From:' header.  It's not optional because cataloging systems
can use the e-mail portion of this field as a unique key
representing the author.  A catalog might provide authors the
ability to store their GPG key, personal home page, and other
additional metadata *about the author*, and optionally the
ability to associate several e-mail addresses with the same
person.  Author-related metadata fields are not covered by this
PEP.

Example::

    Author-email: "C. Schultz" <cschultz@example.com>

License
-------

A string selected from a short list of choices, specifying the
license covering the package.  Some licenses result in the
software being freely redistributable, so packagers and
resellers can automatically know that they're free to
redistribute the software.  Other licenses will require
a careful reading by a human to determine how the software can be
repackaged and resold.

The choices are::

    Artistic, BSD, DFSG, GNU GPL, GNU LGPL, "MIT",
    Mozilla PL, "public domain", Python, Qt PL, Zope PL, unknown,
    nocommercial, nosell, nosource, shareware, other

Definitions of some of the licenses are:

=============  ===================================================
DFSG           The license conforms to the Debian Free Software
               Guidelines, but does not use one of the other
               DFSG conforming licenses listed here.
               More information is available at:
               http://www.debian.org/social_contract#guidelines

Python         Python 1.6 or higher license.  Version 1.5.2 and
               earlier are under the MIT license.

public domain  Software is public domain, not copyrighted.

unknown        Status is not known

nocommercial   Free private use but commercial use not permitted

nosell         Free use but distribution for profit by arrangement

nosource       Freely distributable but no source code

shareware      Payment is requested if software is used

other          General category for other non-DFSG licenses
=============  ===================================================

Some of these licenses can be interpreted to mean the software is
freely redistributable.  The list of redistributable licenses is::

    Artistic, BSD, DFSG, GNU GPL, GNU LGPL, "MIT",
    Mozilla PL, "public domain", Python, Qt PL, Zope PL,
    nosource, shareware

Note that being redistributable does not mean a package
qualifies as free software, 'nosource' and 'shareware' being
examples.

Example::

    License: MIT


Acknowledgements
================

Many changes and rewrites to this document were suggested by the
readers of the Distutils SIG.  In particular, Sean Reifschneider
often contributed actual text for inclusion in this PEP.

The list of licenses was compiled using the SourceForge license
list and the CTAN license list compiled by Graham Williams; Carey
Evans also offered several useful suggestions on this list.


Copyright
=========

This document has been placed in the public domain.


id: pep-0527.rst
content:
PEP: 527
Title: Removing Un(der)used file types/extensions on PyPI
Author: Donald Stufft <donald@stufft.io>
BDFL-Delegate: Alyssa Coghlan <ncoghlan@gmail.com>
Discussions-To: distutils-sig@python.org
Status: Final
Type: Standards Track
Topic: Packaging
Created: 23-Aug-2016
Post-History: 23-Aug-2016
Resolution: https://mail.python.org/pipermail/distutils-sig/2016-September/029624.html


Abstract
========

This PEP recommends deprecating, and ultimately removing, support for uploading
certain unused or under used file types and extensions to PyPI. In particular
it recommends disallowing further uploads of any files of the types
``bdist_dumb``, ``bdist_rpm``, ``bdist_dmg``, ``bdist_msi``, and
``bdist_wininst``, leaving PyPI to only accept new uploads of the ``sdist``,
``bdist_wheel``, and ``bdist_egg`` file types.

In addition, this PEP proposes removing support for new uploads of sdists using
the ``.tar``, ``.tar.bz2``, ``.tar.xz``, ``.tar.Z``, ``.tgz``, ``.tbz``, and
any other extension besides ``.tar.gz`` and ``.zip``.

Finally, this PEP also proposes limiting the number of allowed sdist uploads
for each individual release of a project on PyPI to one instead of one for each
allowed extension.


Rationale
=========

File Formats
------------

Currently PyPI supports the following file types:

* ``sdist``
* ``bdist_wheel``
* ``bdist_egg``
* ``bdist_wininst``
* ``bdist_msi``
* ``bdist_dmg``
* ``bdist_rpm``
* ``bdist_dumb``

However, these different types of files have varying amounts of usefulness or
general use in the ecosystem. Continuing to support them adds a maintenance
burden on PyPI as well as tool authors and incurs a cost in both bandwidth and
disk space not only on PyPI itself, but also on any mirrors of PyPI.


Python packaging is a multi-level ecosystem where PyPI is primarily suited and
used to distribute virtual environment compatible packages directly from their
respective project owners. These packages are then consumed either directly
by end-users, or by downstream distributors that take these packages and turn
them into their respective system level packages (such as RPM, deb, MSI, etc).

While PyPI itself only directly works with these Python specific but platform
agnostic packages, we encourage community-driven and commercial conversions of
these packages to downstream formats for particular target environments, like:

* The conda cross-platform data analysis ecosystem (conda-forge)
* The deb based Linux ecosystem (Debian, Ubuntu, etc)
* The RPM based Linux ecosystem (Fedora, openSuSE, Mageia, etc)
* The homebrew, MacPorts and fink ecosystems for Mac OS X
* The Windows Package Management ecosystem (NuGet, Chocolatey, etc)
* 3rd party creation of Windows MSIs and installers (e.g. Christoph Gohlke's
  work at http://www.lfd.uci.edu/~gohlke/pythonlibs/ )
* other commercial redistribution formats (ActiveState's PyPM, Enthought
  Canopy, etc)
* other open source community redistribution formats (Nix, Gentoo, Arch, \*BSD,
  etc)

It is the belief of this PEP that the entire ecosystem is best supported by
keeping PyPI focused on the platform agnostic formats, where the limited amount
of time by volunteers can be best used instead of spreading the available time
out amongst several platforms. Further more, this PEP believes that the people
best positioned to provide well integrated packages for a particular platform
are people focused on that platform, and not across all possible platforms.


bdist_dumb
~~~~~~~~~~

As it's name implies, ``bdist_dumb`` is not a very complex format, however it
is so simple as to be worthless for actual usage.

For instance, if you're using something like pyenv on macOS and you're building
a library using Python 3.5, then ``bdist_dumb`` will produce a ``.tar.gz`` file
named something like ``exampleproject-1.0.macosx-10.11-x86_64.tar.gz``. Right
off the bat this file name is somewhat difficult to differentiate from an
``sdist`` since they both use the same file extension (and with the legacy pre
:pep:`440` versions, ``1.0-macosx-10.11-x86_64`` is a valid, although quite silly,
version number). However, once you open up the created ``.tar.gz``, you'd find
that there is no metadata inside that could be used for things like dependency
discovery and in fact, it is quite simply a tarball containing hardcoded paths
to wherever files would have been installed on the computer creating the
``bdist_dumb``. Going back to our pyenv on macOS example, this means that if I
created it, it would contain files like:

``Users/dstufft/.pyenv/versions/3.5.2/lib/python3.5/site-packages/example.py``


bdist_rpm
~~~~~~~~~

The ``bdist_rpm`` format on PyPI allows people to upload ``.rpm`` files for
end users to manually download by hand and then manually install by hand.
However, the common usage of ``rpm`` is with a specially designed repository
that allows automatic installation of dependencies, upgrades, etc which PyPI
does not provide. Thus, it is a type of file that is barely being used on PyPI
with only ~460 files of this type having been uploaded to PyPI (out a total of
662,544).

In addition, services like `COPR <https://copr.fedorainfracloud.org/>`_ provide
a better supported mechanism for publishing and using RPM files than we're ever
likely to get on PyPI.


bdist_dmg, bdist_msi, and bdist_wininst
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``bdist_dmg``, ``bdist_msi``, and ``bdist_winist`` formats are similar in
that they are an OS specific installer that will only install a library into an
environment and are not designed for real user facing installs of applications
(which would require things like bundling a Python interpreter and the like).

Out of these three, the usage for ``bdist_dmg`` and ``bdist_msi`` is very low,
with only ~500 ``bdist_msi`` files and ~50 ``bdist_dmg`` files having been
uploaded to PyPI. The ``bdist_wininst`` format has more use, with ~14,000 files
having ever been uploaded to PyPI.

It's quite easy to look at the low usage of ``bdist_dmg`` and ``bdist_msi`` and
conclude that removing them will be fairly low impact, however
``bdist_wininst`` has several orders of magnitude more usage. This is somewhat
misleading though, because although it has more people *uploading* those files
the actual usage of those uploaded files is fairly low. Taking a look at the
previous 30 days, we can see that 90% of all downloads of ``bdist_winist``
files from PyPI were generated by the mirroring infrastructure and 7% of them
were generated by setuptools (which can currently be better covered by
``bdist_egg`` files).

Given the small number of files uploaded for ``bdist_dmg`` and ``bdist_msi``
and that ``bdist_wininst`` is largely existing to either consume bandwidth and
disk space via the mirroring infrastructure *or* could be trivially replaced
with ``bdist_egg``, this PEP proposes to include these three formats in the
list of those to be disallowed.


File Extensions
---------------

Currently ``sdist`` supports a wide variety of file extensions like ``.tar.gz``,
``.tar``, ``.tar.bz2``, ``.tar.xz``, ``.zip``, ``.tar.Z``, ``.tgz``, and
``.tbz``. However, of those the only extensions which get anything more than
negligible usage is ``.tar.gz`` with 444,338 sdists currently, ``.zip`` with
58,774 sdists currently, and ``.tar.bz2`` with 3,265 sdists currently.

Having multiple formats accepted requires tooling both within PyPI and outside
of PyPI to handle all of the various extensions that *might* be used (even if
nobody is currently using them). This doesn't only affect PyPI, but ripples out
throughout the ecosystem. In addition, the different formats all have different
requirements for what optional C libraries Python was linked against and
different requirements for what versions of Python they support. In addition,
multiple formats also create a weird situation where there may be two
``sdist`` files for a particular project/release with subtly different content.

It's easy to advocate that anything outside of ``.t


id: pep-0533.rst
content:
PEP: 533
Title: Deterministic cleanup for iterators
Author: Nathaniel J. Smith
BDFL-Delegate: Yury Selivanov <yury@edgedb.com>
Status: Deferred
Type: Standards Track
Created: 18-Oct-2016
Post-History: 18-Oct-2016

Abstract
========

We propose to extend the iterator protocol with a new
``__(a)iterclose__`` slot, which is called automatically on exit from
``(async) for`` loops, regardless of how they exit. This allows for
convenient, deterministic cleanup of resources held by iterators
without reliance on the garbage collector. This is especially valuable
for asynchronous generators.


Note on timing
==============

In practical terms, the proposal here is divided into two separate
parts: the handling of async iterators, which should ideally be
implemented ASAP, and the handling of regular iterators, which is a
larger but more relaxed project that can't start until 3.7 at the
earliest. But since the changes are closely related, and we probably
don't want to end up with async iterators and regular iterators
diverging in the long run, it seems useful to look at them together.


Background and motivation
=========================

Python iterables often hold resources which require cleanup. For
example: ``file`` objects need to be closed; the :pep:`WSGI spec
<333>` adds a ``close`` method
on top of the regular iterator protocol and demands that consumers
call it at the appropriate time (though forgetting to do so is a
`frequent source of bugs
<http://blog.dscpl.com.au/2012/10/obligations-for-calling-close-on.html>`_);
and :pep:`342` (based on :pep:`325`) extended generator objects to add a
``close`` method to allow generators to clean up after themselves.

Generally, objects that need to clean up after themselves also define
a ``__del__`` method to ensure that this cleanup will happen
eventually, when the object is garbage collected. However, relying on
the garbage collector for cleanup like this causes serious problems in
several cases:

- In Python implementations that do not use reference counting
  (e.g. PyPy, Jython), calls to ``__del__`` may be arbitrarily delayed
  -- yet many situations require *prompt* cleanup of
  resources. Delayed cleanup produces problems like crashes due to
  file descriptor exhaustion, or WSGI timing middleware that collects
  bogus times.

- Async generators (:pep:`525`) can only perform cleanup under the
  supervision of the appropriate coroutine runner. ``__del__`` doesn't
  have access to the coroutine runner; indeed, the coroutine runner
  might be garbage collected before the generator object. So relying
  on the garbage collector is effectively impossible without some kind
  of language extension. (:pep:`525` does provide such an extension, but
  it has a number of limitations that this proposal fixes; see the
  "alternatives" section below for discussion.)

.. XX add discussion of:

  - Causality preservation, context preservation

  - Exception swallowing

Fortunately, Python provides a standard tool for doing resource
cleanup in a more structured way: ``with`` blocks. For example, this
code opens a file but relies on the garbage collector to close it::

  def read_newline_separated_json(path):
      for line in open(path):
          yield json.loads(line)

  for document in read_newline_separated_json(path):
      ...

and recent versions of CPython will point this out by issuing a
``ResourceWarning``, nudging us to fix it by adding a ``with`` block::

  def read_newline_separated_json(path):
      with open(path) as file_handle:      # <-- with block
          for line in file_handle:
              yield json.loads(line)

  for document in read_newline_separated_json(path):  # <-- outer for loop
      ...

But there's a subtlety here, caused by the interaction of ``with``
blocks and generators. ``with`` blocks are Python's main tool for
managing cleanup, and they're a powerful one, because they pin the
lifetime of a resource to the lifetime of a stack frame. But this
assumes that someone will take care of cleaning up the stack
frame... and for generators, this requires that someone ``close``
them.

In this case, adding the ``with`` block *is* enough to shut up the
``ResourceWarning``, but this is misleading -- the file object cleanup
here is still dependent on the garbage collector. The ``with`` block
will only be unwound when the ``read_newline_separated_json``
generator is closed. If the outer ``for`` loop runs to completion then
the cleanup will happen immediately; but if this loop is terminated
early by a ``break`` or an exception, then the ``with`` block won't
fire until the generator object is garbage collected.

The correct solution requires that all *users* of this API wrap every
``for`` loop in its own ``with`` block::

  with closing(read_newline_separated_json(path)) as genobj:
      for document in genobj:
          ...

This gets even worse if we consider the idiom of decomposing a complex
pipeline into multiple nested generators::

  def read_users(path):
      with closing(read_newline_separated_json(path)) as gen:
          for document in gen:
              yield User.from_json(document)

  def users_in_group(path, group):
      with closing(read_users(path)) as gen:
          for user in gen:
              if user.group == group:
                  yield user

In general if you have N nested generators then you need N+1 ``with``
blocks to clean up 1 file. And good defensive programming would
suggest that any time we use a generator, we should assume the
possibility that there could be at least one ``with`` block somewhere
in its (potentially transitive) call stack, either now or in the
future, and thus always wrap it in a ``with``. But in practice,
basically nobody does this, because programmers would rather write
buggy code than tiresome repetitive code. In simple cases like this
there are some workarounds that good Python developers know (e.g. in
this simple case it would be idiomatic to pass in a file handle
instead of a path and move the resource management to the top level),
but in general we cannot avoid the use of ``with``/``finally`` inside
of generators, and thus dealing with this problem one way or
another. When beauty and correctness fight then beauty tends to win,
so it's important to make correct code beautiful.

Still, is this worth fixing? Until async generators came along I would
have argued yes, but that it was a low priority, since everyone seems
to be muddling along okay -- but async generators make it much more
urgent. Async generators cannot do cleanup *at all* without some
mechanism for deterministic cleanup that people will actually use, and
async generators are particularly likely to hold resources like file
descriptors. (After all, if they weren't doing I/O, they'd be
generators, not async generators.) So we have to do something, and it
might as well be a comprehensive fix to the underlying problem. And
it's much easier to fix this now when async generators are first
rolling out, than it will be to fix it later.

The proposal itself is simple in concept: add a ``__(a)iterclose__``
method to the iterator protocol, and have (async) ``for`` loops call
it when the loop is exited, even if this occurs via ``break`` or
exception unwinding. Effectively, we're taking the current cumbersome
idiom (``with`` block + ``for`` loop) and merging them together into a
fancier ``for``. This may seem non-orthogonal, but makes sense when
you consider that the existence of generators means that ``with``
blocks actually depend on iterator cleanup to work reliably, plus
experience showing that iterator cleanup is often a desirable feature
in its own right.


Alternatives
============

PEP 525 asyncgen hooks
----------------------

:pep:`PEP 525 proposes a set of global thread-local hooks
<525#finalization>`
managed by new ``sys.{get/set}_asyncgen_hooks()`` functions, which
allow event loops to integrate with the garbage collector to run
cleanup for async generators. In principle, this proposa


id: pep-0255.rst
content:
PEP: 255
Title: Simple Generators
Author: Neil Schemenauer <nas@arctrix.com>,
        Tim Peters <tim.peters@gmail.com>,
        Magnus Lie Hetland <magnus@hetland.org>
Status: Final
Type: Standards Track
Requires: 234
Created: 18-May-2001
Python-Version: 2.2
Post-History: 14-Jun-2001, 23-Jun-2001


Abstract
========

This PEP introduces the concept of generators to Python, as well as a new
statement used in conjunction with them, the ``yield`` statement.


Motivation
==========

When a producer function has a hard enough job that it requires maintaining
state between values produced, most programming languages offer no pleasant and
efficient solution beyond adding a callback function to the producer's argument
list, to be called with each value produced.

For example, ``tokenize.py`` in the standard library takes this approach: the
caller must pass a *tokeneater* function to ``tokenize()``, called whenever
``tokenize()`` finds the next token.  This allows tokenize to be coded in a
natural way, but programs calling tokenize are typically convoluted by the need
to remember between callbacks which token(s) were seen last.  The *tokeneater*
function in ``tabnanny.py`` is a good example of that, maintaining a state
machine in global variables, to remember across callbacks what it has already
seen and what it hopes to see next.  This was difficult to get working
correctly, and is still difficult for people to understand.  Unfortunately,
that's typical of this approach.

An alternative would have been for tokenize to produce an entire parse of the
Python program at once, in a large list.  Then tokenize clients could be
written in a natural way, using local variables and local control flow (such as
loops and nested if statements) to keep track of their state.  But this isn't
practical:  programs can be very large, so no a priori bound can be placed on
the memory needed to materialize the whole parse; and some tokenize clients
only want to see whether something specific appears early in the program (e.g.,
a future statement, or, as is done in IDLE, just the first indented statement),
and then parsing the whole program first is a severe waste of time.

Another alternative would be to make tokenize an :pep:`iterator <234>`,
delivering the
next token whenever its ``.next()`` method is invoked.  This is pleasant for the
caller in the same way a large list of results would be, but without the memory
and "what if I want to get out early?" drawbacks.  However, this shifts the
burden on tokenize to remember *its* state between ``.next()`` invocations, and
the reader need only glance at ``tokenize.tokenize_loop()`` to realize what a
horrid chore that would be.  Or picture a recursive algorithm for producing the
nodes of a general tree structure:  to cast that into an iterator framework
requires removing the recursion manually and maintaining the state of the
traversal by hand.

A fourth option is to run the producer and consumer in separate threads.  This
allows both to maintain their states in natural ways, and so is pleasant for
both.  Indeed, Demo/threads/Generator.py in the Python source distribution
provides a usable synchronized-communication class for doing that in a general
way.  This doesn't work on platforms without threads, though, and is very slow
on platforms that do (compared to what is achievable without threads).

A final option is to use the Stackless [1]_ (:pep:`219`) variant implementation of Python
instead, which supports lightweight coroutines.  This has much the same
programmatic benefits as the thread option, but is much more efficient.
However, Stackless is a controversial rethinking of the Python core, and it may
not be possible for Jython to implement the same semantics.  This PEP isn't the
place to debate that, so suffice it to say here that generators provide a
useful subset of Stackless functionality in a way that fits easily into the
current CPython implementation, and is believed to be relatively
straightforward for other Python implementations.

That exhausts the current alternatives.  Some other high-level languages
provide pleasant solutions, notably iterators in Sather [2]_, which were
inspired by iterators in CLU; and generators in Icon [3]_, a novel language
where every expression *is a generator*.  There are differences among these,
but the basic idea is the same:  provide a kind of function that can return an
intermediate result ("the next value") to its caller, but maintaining the
function's local state so that the function can be resumed again right where it
left off.  A very simple example::

    def fib():
        a, b = 0, 1
        while 1:
           yield b
           a, b = b, a+b

When ``fib()`` is first invoked, it sets *a* to 0 and *b* to 1, then yields *b*
back to its caller.  The caller sees 1.  When ``fib`` is resumed, from its
point of view the ``yield`` statement is really the same as, say, a ``print``
statement:  ``fib`` continues after the yield with all local state intact. *a*
and *b* then become 1 and 1, and ``fib`` loops back to the ``yield``, yielding
1 to its invoker.  And so on.  From ``fib``'s point of view it's just
delivering a sequence of results, as if via callback.  But from its caller's
point of view, the ``fib`` invocation is an iterable object that can be resumed
at will.  As in the thread approach, this allows both sides to be coded in the
most natural ways; but unlike the thread approach, this can be done efficiently
and on all platforms.  Indeed, resuming a generator should be no more expensive
than a function call.

The same kind of approach applies to many producer/consumer functions. For
example, ``tokenize.py`` could yield the next token instead of invoking a
callback function with it as argument, and tokenize clients could iterate over
the tokens in a natural way:  a Python generator is a kind of Python
:pep:`iterator <234>`, but of an especially powerful kind.


Specification:  Yield
=====================

A new statement is introduced::

    yield_stmt:    "yield" expression_list

``yield`` is a new keyword, so a ``future`` statement (:pep:`236`) is needed to phase
this in: in the initial release, a module desiring to use generators must
include the line::

    from __future__ import generators

near the top (see :pep:`236`) for details).  Modules using the identifier
``yield`` without a ``future`` statement will trigger warnings. In the
following release, ``yield`` will be a language keyword and the ``future``
statement will no longer be needed.

The ``yield`` statement may only be used inside functions.  A function that
contains a ``yield`` statement is called a generator function.  A generator
function is an ordinary function object in all respects, but has the new
``CO_GENERATOR`` flag set in the code object's co_flags member.

When a generator function is called, the actual arguments are bound to
function-local formal argument names in the usual way, but no code in the body
of the function is executed.  Instead a generator-iterator object is returned;
this conforms to the :pep:`iterator protocol <234>`, so in particular can be used in
for-loops in a natural way.  Note that when the intent is clear from context,
the unqualified name "generator" may be used to refer either to a
generator-function or a generator-iterator.

Each time the ``.next()`` method of a generator-iterator is invoked, the code
in the body of the generator-function is executed until a ``yield`` or
``return`` statement (see below) is encountered, or until the end of the body
is reached.

If a ``yield`` statement is encountered, the state of the function is frozen,
and the value of *expression_list* is returned to ``.next()``'s caller.  By
"frozen" we mean that all local state is retained, including the current
bindings of local variables, the instruction pointer, and the internal
evaluation stack:  enough information is saved so that the next time
``.next()`` is invoked, the function can proceed exactly as if t


id: pep-0282.rst
content:
PEP: 282
Title: A Logging System
Author: Vinay Sajip <vinay_sajip at red-dove.com>,
        Trent Mick <trentm@activestate.com>
Status: Final
Type: Standards Track
Created: 04-Feb-2002
Python-Version: 2.3
Post-History:


Abstract
========

This PEP describes a proposed logging package for Python's
standard library.

Basically the system involves the user creating one or more logger
objects on which methods are called to log debugging notes,
general information, warnings, errors etc.  Different logging
'levels' can be used to distinguish important messages from less
important ones.

A registry of named singleton logger objects is maintained so that

1) different logical logging streams (or 'channels') exist
   (say, one for 'zope.zodb' stuff and another for
   'mywebsite'-specific stuff)

2) one does not have to pass logger object references around.

The system is configurable at runtime.  This configuration
mechanism allows one to tune the level and type of logging done
while not touching the application itself.


Motivation
==========

If a single logging mechanism is enshrined in the standard
library, 1) logging is more likely to be done 'well', and 2)
multiple libraries will be able to be integrated into larger
applications which can be logged reasonably coherently.


Influences
==========

This proposal was put together after having studied the
following logging packages:

* java.util.logging in JDK 1.4 (a.k.a. JSR047) [1]_
* log4j [2]_
* the Syslog package from the Protomatter project [3]_
* MAL's mx.Log package [4]_


Simple Example
==============

This shows a very simple example of how the logging package can be
used to generate simple logging output on stderr.

.. code-block:: python

    --------- mymodule.py -------------------------------
    import logging
    log = logging.getLogger("MyModule")

    def doIt():
            log.debug("Doin' stuff...")
            #do stuff...
            raise TypeError, "Bogus type error for testing"
    -----------------------------------------------------

.. code-block:: python

    --------- myapp.py ----------------------------------
    import mymodule, logging

    logging.basicConfig()

    log = logging.getLogger("MyApp")

    log.info("Starting my app")
    try:
            mymodule.doIt()
    except Exception, e:
            log.exception("There was a problem.")
    log.info("Ending my app")
    -----------------------------------------------------

.. code-block:: console

    $ python myapp.py

    INFO:MyApp: Starting my app
    DEBUG:MyModule: Doin' stuff...
    ERROR:MyApp: There was a problem.
    Traceback (most recent call last):
            File "myapp.py", line 9, in ?
                    mymodule.doIt()
            File "mymodule.py", line 7, in doIt
                    raise TypeError, "Bogus type error for testing"
    TypeError: Bogus type error for testing

    INFO:MyApp: Ending my app

The above example shows the default output format.  All
aspects of the output format should be configurable, so that
you could have output formatted like this:

.. code-block:: text

    2002-04-19 07:56:58,174 MyModule   DEBUG - Doin' stuff...

    or just

    Doin' stuff...


Control Flow
============

Applications make logging calls on **Logger** objects.  Loggers are
organized in a hierarchical namespace and child Loggers inherit
some logging properties from their parents in the namespace.

Logger names fit into a "dotted name" namespace, with dots
(periods) indicating sub-namespaces.  The namespace of logger
objects therefore corresponds to a single tree data structure.

* ``""`` is the root of the namespace
* ``"Zope"`` would be a child node of the root
* ``"Zope.ZODB"`` would be a child node of ``"Zope"``

These Logger objects create **LogRecord** objects which are passed
to **Handler** objects for output.  Both Loggers and Handlers may
use logging **levels** and (optionally) **Filters** to decide if they
are interested in a particular LogRecord.  When it is necessary to
output a LogRecord externally, a Handler can (optionally) use a
**Formatter** to localize and format the message before sending it
to an I/O stream.

Each Logger keeps track of a set of output Handlers.  By default
all Loggers also send their output to all Handlers of their
ancestor Loggers.  Loggers may, however, also be configured to
ignore Handlers higher up the tree.

The APIs are structured so that calls on the Logger APIs can be
cheap when logging is disabled.  If logging is disabled for a
given log level, then the Logger can make a cheap comparison test
and return.  If logging is enabled for a given log level, the
Logger is still careful to minimize costs before passing the
LogRecord into the Handlers.  In particular, localization and
formatting (which are relatively expensive) are deferred until the
Handler requests them.

The overall Logger hierarchy can also have a level associated with
it, which takes precedence over the levels of individual Loggers.
This is done through a module-level function::

    def disable(lvl):
        """
        Do not generate any LogRecords for requests with a severity less
        than 'lvl'.
        """
        ...


Levels
======

The logging levels, in increasing order of importance, are:

* DEBUG
* INFO
* WARN
* ERROR
* CRITICAL

The term CRITICAL is used in preference to FATAL, which is used by
log4j.  The levels are conceptually the same - that of a serious,
or very serious, error.  However, FATAL implies death, which in
Python implies a raised and uncaught exception, traceback, and
exit.  Since the logging module does not enforce such an outcome
from a FATAL-level log entry, it makes sense to use CRITICAL in
preference to FATAL.

These are just integer constants, to allow simple comparison of
importance.  Experience has shown that too many levels can be
confusing, as they lead to subjective interpretation of which
level should be applied to any particular log request.

Although the above levels are strongly recommended, the logging
system should not be prescriptive.  Users may define their own
levels, as well as the textual representation of any levels.  User
defined levels must, however, obey the constraints that they are
all positive integers and that they increase in order of
increasing severity.

User-defined logging levels are supported through two module-level
functions::

    def getLevelName(lvl):
            """Return the text for level 'lvl'."""
            ...

    def addLevelName(lvl, lvlName):
            """
            Add the level 'lvl' with associated text 'levelName', or
            set the textual representation of existing level 'lvl' to be
            'lvlName'."""
            ...

Loggers
=======

Each Logger object keeps track of a log level (or threshold) that
it is interested in, and discards log requests below that level.

A **Manager** class instance maintains the hierarchical namespace of
named Logger objects.  Generations are denoted with dot-separated
names: Logger "foo" is the parent of Loggers "foo.bar" and
"foo.baz".

The Manager class instance is a singleton and is not directly
exposed to users, who interact with it using various module-level
functions.

The general logging method is::

    class Logger:
        def log(self, lvl, msg, *args, **kwargs):
            """Log 'str(msg) % args' at logging level 'lvl'."""
            ...

However, convenience functions are defined for each logging level::

    class Logger:
        def debug(self, msg, *args, **kwargs): ...
        def info(self, msg, *args, **kwargs): ...
        def warn(self, msg, *args, **kwargs): ...
        def error(self, msg, *args, **kwargs): ...
        def critical(self, msg, *args, **kwargs): ...

Only one keyword argument is recognized at present - "exc_info".
If true, the caller wants exception information to be provided in
the logging output.  This mechanism is only needed if exception
information needs to be provided at **any** logging level.  In the
more com


id: pep-0296.rst
content:
PEP: 296
Title: Adding a bytes Object Type
Author: Scott Gilbert <xscottg at yahoo.com>
Status: Withdrawn
Type: Standards Track
Created: 12-Jul-2002
Python-Version: 2.3
Post-History:


Notice
=======

This PEP is withdrawn by the author (in favor of :pep:`358`).


Abstract
========

This PEP proposes the creation of a new standard type and builtin
constructor called 'bytes'.  The bytes object is an efficiently
stored array of bytes with some additional characteristics that
set it apart from several implementations that are similar.


Rationale
=========

Python currently has many objects that implement something akin to
the bytes object of this proposal.  For instance the standard
string, buffer, array, and mmap objects are all very similar in
some regards to the bytes object.  Additionally, several
significant third party extensions have created similar objects to
try and fill similar needs.  Frustratingly, each of these objects
is too narrow in scope and is missing critical features to make it
applicable to a wider category of problems.


Specification
=============

The bytes object has the following important characteristics:

1. Efficient underlying array storage via the standard C type "unsigned
   char".  This allows fine grain control over how much memory is
   allocated.  With the alignment restrictions designated in the next
   item, it is trivial for low level extensions to cast the pointer
   to a different type as needed.

   Also, since the object is implemented as an array of bytes, it is
   possible to pass the bytes object to the extensive library of
   routines already in the standard library that presently work with
   strings.  For instance, the bytes object in conjunction with the
   struct module could be used to provide a complete replacement for
   the array module using only Python script.

   If an unusual platform comes to light, one where there isn't a
   native unsigned 8 bit type, the object will do its best to
   represent itself at the Python script level as though it were an
   array of 8 bit unsigned values.  It is doubtful whether many
   extensions would handle this correctly, but Python script could be
   portable in these cases.

2. Alignment of the allocated byte array is whatever is promised by the
   platform implementation of malloc.  A bytes object created from an
   extension can be supplied that provides any arbitrary alignment as
   the extension author sees fit.

   This alignment restriction should allow the bytes object to be
   used as storage for all standard C types - including ``PyComplex``
   objects or other structs of standard C type types.  Further
   alignment restrictions can be provided by extensions as necessary.

3. The bytes object implements a subset of the sequence operations
   provided by string/array objects, but with slightly different
   semantics in some cases.  In particular, a slice always returns a
   new bytes object, but the underlying memory is shared between the
   two objects.  This type of slice behavior has been called creating
   a "view".  Additionally, repetition and concatenation are
   undefined for bytes objects and will raise an exception.

   As these objects are likely to find use in high performance
   applications, one motivation for the decision to use view slicing
   is that copying between bytes objects should be very efficient and
   not require the creation of temporary objects.  The following code
   illustrates this::

       # create two 10 Meg bytes objects
       b1 = bytes(10000000)
       b2 = bytes(10000000)

       # copy from part of one to another with out creating a 1 Meg temporary
       b1[2000000:3000000] = b2[4000000:5000000]

   Slice assignment where the rvalue is not the same length as the
   lvalue will raise an exception.  However, slice assignment will
   work correctly with overlapping slices (typically implemented with
   memmove).

4. The bytes object will be recognized as a native type by the ``pickle`` and
   ``cPickle`` modules for efficient serialization.  (In truth, this is
   the only requirement that can't be implemented via a third party
   extension.)

   Partial solutions to address the need to serialize the data stored
   in a bytes-like object without creating a temporary copy of the
   data into a string have been implemented in the past.  The tofile
   and fromfile methods of the array object are good examples of
   this.  The bytes object will support these methods too.  However,
   pickling is useful in other situations - such as in the shelve
   module, or implementing RPC of Python objects, and requiring the
   end user to use two different serialization mechanisms to get an
   efficient transfer of data is undesirable.

   XXX: Will try to implement pickling of the new bytes object in
   such a way that previous versions of Python will unpickle it as a
   string object.

   When unpickling, the bytes object will be created from memory
   allocated from Python (via ``malloc``).  As such, it will lose any
   additional properties that an extension supplied pointer might
   have provided (special alignment, or special types of memory).

   XXX: Will try to make it so that C subclasses of bytes type can
   supply the memory that will be unpickled into.  For instance, a
   derived class called PageAlignedBytes would unpickle to memory
   that is also page aligned.

   On any platform where an int is 32 bits (most of them), it is
   currently impossible to create a string with a length larger than
   can be represented in 31 bits.  As such, pickling to a string will
   raise an exception when the operation is not possible.

   At least on platforms supporting large files (many of them),
   pickling large bytes objects to files should be possible via
   repeated calls to the ``file.write()`` method.

5. The bytes type supports the ``PyBufferProcs`` interface, but a bytes object
   provides the additional guarantee that the pointer will not be
   deallocated or reallocated as long as a reference to the bytes
   object is held.  This implies that a bytes object is not resizable
   once it is created, but allows the global interpreter lock (GIL)
   to be released while a separate thread manipulates the memory
   pointed to if the ``PyBytes_Check(...)`` test passes.

   This characteristic of the bytes object allows it to be used in
   situations such as asynchronous file I/O or on multiprocessor
   machines where the pointer obtained by ``PyBufferProcs`` will be used
   independently of the global interpreter lock.

   Knowing that the pointer can not be reallocated or freed after the
   GIL is released gives extension authors the capability to get true
   concurrency and make use of additional processors for long running
   computations on the pointer.

6. In C/C++ extensions, the bytes object can be created from a supplied
   pointer and destructor function to free the memory when the
   reference count goes to zero.

   The special implementation of slicing for the bytes object allows
   multiple bytes objects to refer to the same pointer/destructor.
   As such, a refcount will be kept on the actual
   pointer/destructor.  This refcount is separate from the refcount
   typically associated with Python objects.

   XXX: It may be desirable to expose the inner refcounted object as an
   actual Python object.  If a good use case arises, it should be possible
   for this to be implemented later with no loss to backwards compatibility.

7. It is also possible to signify the bytes object as readonly, in this
   case it isn't actually mutable, but does provide the other features of a
   bytes object.

8. The bytes object keeps track of the length of its data with a Python
   ``LONG_LONG`` type.  Even though the current definition for ``PyBufferProcs``
   restricts the length to be the size of an int, this PEP does not propose
   to make any changes there.  Instead, extensions can work around this limit
   by making an expli


id: pep-0719.rst
content:
PEP: 719
Title: Python 3.13 Release Schedule
Author: Thomas Wouters <thomas@python.org>
Status: Active
Type: Informational
Topic: Release
Created: 26-May-2023
Python-Version: 3.13


Abstract
========

This document describes the development and release schedule for
Python 3.13.

Release Manager and Crew
========================

- 3.13 Release Manager: Thomas Wouters
- Windows installers: Steve Dower
- Mac installers: Ned Deily
- Documentation: Julien Palard


Release Schedule
================

3.13.0 schedule
---------------

Note: the dates below use a 17-month development period that results
in a 12-month release cadence between feature versions, as defined by
:pep:`602`.

Actual:

- 3.13 development begins: Monday, 2023-05-22
- 3.13.0 alpha 1: Friday, 2023-10-13
- 3.13.0 alpha 2: Wednesday, 2023-11-22
- 3.13.0 alpha 3: Wednesday, 2024-01-17
- 3.13.0 alpha 4: Thursday, 2024-02-15
- 3.13.0 alpha 5: Tuesday, 2024-03-12
- 3.13.0 alpha 6: Tuesday, 2024-04-09
- 3.13.0 beta 1: Wednesday, 2024-05-08
  (No new features beyond this point.)
- 3.13.0 beta 2: Wednesday, 2024-06-05
- 3.13.0 beta 3: Thursday, 2024-06-27
- 3.13.0 beta 4: Thursday, 2024-07-18
- 3.13.0 candidate 1: Thursday, 2024-08-01
- 3.13.0 candidate 2: Friday, 2024-09-06
- 3.13.0 candidate 3: Tuesday, 2024-10-01
- 3.13.0 final: Monday, 2024-10-07
- 3.13.1: Tuesday, 2024-12-03
- 3.13.2: Tuesday, 2025-02-04
- 3.13.3: Tuesday, 2025-04-08

Bugfix releases
---------------

Expected:

- 3.13.4: Tuesday, 2025-06-03
- 3.13.5: Tuesday, 2025-08-05
- 3.13.6: Tuesday, 2025-10-07
- 3.13.7: Tuesday, 2025-12-02
- 3.13.8: Tuesday, 2026-02-03
- 3.13.9: Tuesday, 2026-04-07
- 3.13.10: Tuesday, 2026-06-09
- 3.13.11: Tuesday, 2026-08-04
- 3.13.12: Tuesday, 2026-10-06


Source-only security fix releases
---------------------------------

Provided irregularly on an as-needed basis until October 2029.


3.13 Lifespan
-------------

3.13 will receive bugfix updates approximately every 2 months for
approximately 24 months. Around the time of the release of 3.15.0 final, the
final 3.13 bugfix update will be released. After that, it is expected that
security updates (source only) will be released until 5 years after the
release of 3.13.0 final, so until approximately October 2029.


Copyright
=========

This document is placed in the public domain or under the CC0-1.0-Universal
license, whichever is more permissive.


id: pep-0725.rst
content:
PEP: 725
Title: Specifying external dependencies in pyproject.toml
Author: Pradyun Gedam <pradyunsg@gmail.com>,
        Ralf Gommers <ralf.gommers@gmail.com>
Discussions-To: https://discuss.python.org/t/31888
Status: Draft
Type: Standards Track
Topic: Packaging
Created: 17-Aug-2023
Post-History: `18-Aug-2023 <https://discuss.python.org/t/31888>`__


Abstract
========

This PEP specifies how to write a project's external, or non-PyPI, build and
runtime dependencies in a ``pyproject.toml`` file for packaging-related tools
to consume.

This PEP proposes to add an ``[external]`` table to ``pyproject.toml`` with
three keys: "build-requires", "host-requires" and "dependencies". These
are for specifying three types of dependencies:

1. ``build-requires``, build tools to run on the build machine
2. ``host-requires``, build dependencies needed for host machine but also needed at build time.
3. ``dependencies``, needed at runtime on the host machine but not needed at build time.

Cross compilation is taken into account by distinguishing build and host dependencies.
Optional build-time and runtime dependencies are supported too, in a manner analogies
to how that is supported in the ``[project]`` table.


Motivation
==========

Python packages may have dependencies on build tools, libraries, command-line
tools, or other software that is not present on PyPI. Currently there is no way
to express those dependencies in standardized metadata
[#singular-vision-native-deps]_, [#pypacking-native-deps]_. Key motivators for
this PEP are to:

- Enable tools to automatically map external dependencies to packages in other
  packaging repositories,
- Make it possible to include needed dependencies in error messages emitting by
  Python package installers and build frontends,
- Provide a canonical place for package authors to record this dependency
  information.

Packaging ecosystems like Linux distros, Conda, Homebrew, Spack, and Nix need
full sets of dependencies for Python packages, and have tools like pyp2spec_
(Fedora), Grayskull_ (Conda), and dh_python_ (Debian) which attempt to
automatically generate dependency metadata for their own package managers from the metadata in
upstream Python packages. External dependencies are currently handled manually,
because there is no metadata for this in ``pyproject.toml`` or any other
standard location. Enabling automating this conversion is a key benefit of
this PEP, making packaging Python packages for distros easier and more reliable. In addition, the
authors envision other types of tools making use of this information, e.g.,
dependency analysis tools like Repology_, Dependabot_ and libraries.io_.
Software bill of materials (SBOM) generation tools may also be able to use this
information, e.g. for flagging that external dependencies listed in
``pyproject.toml`` but not contained in wheel metadata are likely vendored
within the wheel.

Packages with external dependencies are typically hard to build from source,
and error messages from build failures tend to be hard to decipher for end
users. Missing external dependencies on the end user's system are the most
likely cause of build failures. If installers can show the required external
dependencies as part of their error message, this may save users a lot of time.

At the moment, information on external dependencies is only captured in
installation documentation of individual packages. It is hard to maintain for
package authors and tends to go out of date. It's also hard for users and
distro packagers to find it. Having a canonical place to record this dependency
information will improve this situation.

This PEP is not trying to specify how the external dependencies should be used,
nor a mechanism to implement a name mapping from names of individual packages
that are canonical for Python projects published on PyPI to those of other
packaging ecosystems. Those topics should be addressed in separate PEPs.


Rationale
=========

Types of external dependencies
------------------------------

Multiple types of external dependencies can be distinguished:

- Concrete packages that can be identified by name and have a canonical
  location in another language-specific package repository. E.g., Rust
  packages on `crates.io <https://crates.io/>`__, R packages on
  `CRAN <https://cran.r-project.org/>`__, JavaScript packages on the
  `npm registry <https://www.npmjs.com/>`__.
- Concrete packages that can be identified by name but do not have a clear
  canonical location. This is typically the case for libraries and tools
  written in C, C++, Fortran, CUDA and other low-level languages. E.g.,
  Boost, OpenSSL, Protobuf, Intel MKL, GCC.
- "Virtual" packages, which are names for concepts, types of tools or
  interfaces. These typically have multiple implementations, which *are*
  concrete packages. E.g., a C++ compiler, BLAS, LAPACK, OpenMP, MPI.

Concrete packages are straightforward to understand, and are a concept present
in virtually every package management system. Virtual packages are a concept
also present in a number of packaging systems -- but not always, and the
details of their implementation varies.

Cross compilation
-----------------

Cross compilation is not yet (as of August 2023) well-supported by stdlib
modules and ``pyproject.toml`` metadata. It is however important when
translating external dependencies to those of other packaging systems (with
tools like ``pyp2spec``). Introducing support for cross compilation immediately
in this PEP is much easier than extending ``[external]`` in the future, hence
the authors choose to include this now.

Terminology
'''''''''''

This PEP uses the following terminology:

- *build machine*: the machine on which the package build process is being
  executed
- *host machine*: the machine on which the produced artifact will be installed
  and run
- *build dependency*: dependency for building the package that needs to be
  present at build time and itself was built for the build machine's OS and
  architecture
- *host dependency*: dependency for building the package that needs to be
  present at build time and itself was built for the host machine's OS and
  architecture

Note that this terminology is not consistent across build and packaging tools,
so care must be taken when comparing build/host dependencies in
``pyproject.toml`` to dependencies from other package managers.

Note that "target machine" or "target dependency" is not used in this PEP. That
is typically only relevant for cross-compiling compilers or other such advanced
scenarios [#gcc-cross-terminology]_, [#meson-cross]_ - this is out of scope for
this PEP.

Finally, note that while "dependency" is the term most widely used for packages
needed at build time, the existing key in ``pyproject.toml`` for PyPI
build-time dependencies is ``build-requires``. Hence this PEP uses the keys
``build-requires`` and ``host-requires`` under ``[external]`` for consistency.

Build and host dependencies
'''''''''''''''''''''''''''

Clear separation of metadata associated with the definition of build and target
platforms, rather than assuming that build and target platform will always be
the same, is important [#pypackaging-native-cross]_.

Build dependencies are typically run during the build process - they may be
compilers, code generators, or other such tools. In case the use of a build
dependency implies a runtime dependency, that runtime dependency does not have
to be declared explicitly. For example, when compiling Fortran code with
``gfortran`` into a Python extension module, the package likely incurs a
dependency on the ``libgfortran`` runtime library. The rationale for not
explicitly listing such runtime dependencies is two-fold: (1) it may depend on
compiler/linker flags or details of the build environment whether the
dependency is present, and (2) these runtime dependencies can be detected and
handled automatically by tools like ``auditwheel``.

Host dependencies are typically not run duri


id: pep-0731.rst
content:
PEP: 731
Title: C API Working Group Charter
Author: Guido van Rossum <guido@python.org>,
        Petr Viktorin <encukou@gmail.com>,
        Victor Stinner <vstinner@python.org>,
        Steve Dower <steve.dower@python.org>,
        Irit Katriel <irit@python.org>
Discussions-To: https://discuss.python.org/t/pep-731-c-api-working-group-charter/36117
Status: Active
Type: Process
Topic: Governance
Created: 11-Oct-2023
Post-History: `13-Oct-2023 <https://discuss.python.org/t/pep-731-c-api-working-group-charter/36117>`__,
              `23-May-2024 <https://discuss.python.org/t/c-api-working-group-vacancy/54015>`__,
              `19-Jun-2024 <https://discuss.python.org/t/c-api-working-group-vacancy-again/56167>`__
Resolution: https://discuss.python.org/t/pep-731-c-api-working-group-charter/36117/12

Abstract
========

This PEP proposes to establish the C API Working Group:
a small committee of Python core developers responsible for
overseeing and coordinating the development and maintenance of the Python C API.

The working group will maintain
documentation, test suites and tooling related to Python's C API.
As delegated by the Steering Council
it is the deciding body for changes to the C API,
from the addition or removal of individual API functions, types, etc.,
to the acceptance of new designs of a more or less radical nature.

The working group's mandate is to represent the interests of all Python users,
but especially all maintainers of code that uses Python's C API,
whether in the context of CPython, using an alternate Python implementation,
or using a binding framework for other programming languages (such as C++ and Rust).

The working group serves at the pleasure of the Python Steering Council.
This document serves as the working group's charter.

Epigraph
========

| KEEPER
| Stop!
  Who would cross the Bridge of Death must answer me these questions three,
  ere the other side he see.

#. What was Python named after?
#. What was Python 2's EOL?
#. What is the optimal strategy to evolve the CPython C API?

| LANCELOT
| Auuuuuuuugh!

Motivation
==========

Despite many discussions and in-person meetings
at core developer sprints and Language Summits,
and a thorough inventory of the problems and stakeholders of the C API,
no consensus has been reached about many contentious issues,
including, but not limited to:

- Conventions for designing new API functions;
- How to deal with compatibility;
- What's the best strategy for handling errors;
- The future of the Stable ABI and the Limited API;
- Whether to switch to a handle-based API convention (and how).

The general feeling is that there are too many stakeholders,
proposals, requirements, constraints, and conventions,
to make progress without having a small trusted group of deciders.

At the 2023 Language Summit in Salt Lake City it was decided to start work on
an `inventory of problems <https://github.com/capi-working-group/problems>`__.
At the 2023 core developer sprint in Brno this work is more or less finished
and after a discussion it appeared that the next step is to establish
a working group to ensure that we're not stymied forever.

The Steering Council has
`indicated <https://github.com/python/steering-council/issues/201#issuecomment-1648848155>`__
its desire to delegate decisions about the C API
to such a working group, anticipating its formal establishment.

Specification
=============

We propose the creation of a new group, the C API Working Group.
This group will be responsible for overseeing and coordinating the development and
maintenance of the Python C API.
It will do this by establishing the principles underpinning the work
and publishing guidelines that the core developers can refer to.

The "operations and process" section below describes
how the working group operates and how it is governed.

Members
-------

The members of the working group are:

- Erlend Aasland
- Michael Droettboom
- Petr Viktorin
- Serhiy Storchaka
- Steve Dower
- Victor Stinner


Mandate
-------

The working group's mandate is to ensure that the Python C API
is suitable for all users of and contributors to the API,
without unduly preferencing one group over another.
The working group will identify exemplar stakeholders,
their needs and preferences,
and will determine a plan for meeting these needs equitably and sustainably.
It will oversee execution of the plan.

Operations and process
----------------------

The working group has at least three members,
comprised of prominent Python core developers.
The members should consider the needs of the various stakeholders carefully.

The Steering Council appoints the initial working group.
There is no term limit for working group members.
Working group members may resign their position at any time, for any reason.
There is an expectation that the membership will change over time.

To determine replacements,
nominations will be collected from the core developer community.
Self-nominations are allowed.
The existing working group will then decide the replacement member(s)
from the nominees.
The expectation is that this will be done by fiat,
but the working group can choose a replacement by any means they see fit,
including a vote.

The working group remains accountable to the Steering Council.
At any point, for any reason, the Steering Council could
(publicly or privately) make a specific change
or request a non-specific change to the composition of the working group.

We acknowledge that this is not a particularly democratic structure
and puts a lot of faith in the working group.
However, the Python community has a long history of success
with structures that are not fully democratic!
We believe that self-governance, cycling of membership,
and accountability to the Steering Council will be sufficient
to ensure that the C API workgroup is meeting the needs of the community.

The working group may operate primarily through reviews of GitHub issues and PRs.
Regular meetings are likely not necessary,
but the working group may set up video calls,
a private chat, or whatever other mechanism they decide upon internally.

The working group should aim for transparency,
posting all decisions publicly on
`discuss.python.org <https://discuss.python.org>`__,
with a rationale if possible.
Before making a decision, the working group should give
all interested community members
(as examples of different categories of stakeholders)
a chance to weigh in.
There should be at least a week between the start of a discussion
and the working group's decision.

Relationship with the Steering Council
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Just like today, the Python Steering Council remains responsible
for the overall direction of the Python C API
and continues to decide on PEPs related to the C API,
using the standard PEP review process (community discussion, etc.).
The C API working group provides written opinions and
recommendations to the Steering Council on PEPs related to the C API.

However, the working group can make smaller C API changes directly.
The Steering Council may also choose to delegate decisions on some PEPs
to the working group (exactly as any other PEP delegation).

Amendments
----------

This PEP serves as a charter for the working group.
Changes to its operation can be made either through a new PEP
or through a change to this PEP.
In either case, the change will be decided upon
by the Steering Council after discussion in the community.

Contact
=======

To ask the C API Working Group for a decision,
community members may open an issue in the
`capi-workgroup/decisions <https://github.com/capi-workgroup/decisions/>`__
repository.

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0679.rst
content:
PEP: 679
Title: Allow parentheses in assert statements
Author: Pablo Galindo Salgado <pablogsal@python.org>
Discussions-To: https://discuss.python.org/t/pep-679-allow-parentheses-in-assert-statements/13003
Status: Draft
Type: Standards Track
Created: 07-Jan-2022
Python-Version: 3.12


Abstract
========

This PEP proposes to allow parentheses surrounding the two-argument form of
assert statements. This will cause the interpreter to reinterpret what before
would have been an assert with a two-element tuple that will always be True
(``assert (expression, message)``) to an assert statement with a subject and a
failure message, equivalent to the statement with the parentheses removed
(``assert expression, message``).


Motivation
==========

It is a common user mistake when using the form of the assert statement that includes
the error message to surround it with parentheses. Unfortunately, this mistake
passes undetected as the assert will always pass, because it is
interpreted as an assert statement where the expression is a two-tuple, which
always has truth-y value.

The mistake most often happens when extending the test or description beyond a
single line, as parentheses are the natural way to do that.

This is so common that a ``SyntaxWarning`` is `now emitted by the compiler
<https://bugs.python.org/issue35029>`_.

Additionally, some other statements in the language allow parenthesized forms
in one way or another like ``import`` statements (``from x import (a,b,c)``) and
``del`` statements (``del (a,b,c)``).

Allowing parentheses not only will remove the common mistake but also will allow
users and auto-formatters to format long assert statements over multiple lines
in what the authors of this document believe will be a more natural way.
Although is possible to currently format long ``assert`` statements over
multiple lines as::

  assert (
      very very long
      expression
  ), (
      "very very long "
      "message"
  )

the authors of this document believe the parenthesized form is more clear and more consistent with
the formatting of other grammar constructs::

  assert (
      very very long
      expression,

      "very very long "
      "message",
  )

This change has been originally discussed and proposed in [bpo-46167]_.

Rationale
=========

This change can be implemented in the parser or in the compiler. We have
selected implementing this change in the parser because doing it in the compiler
will require re-interpreting the AST of an assert statement with a two-tuple::

  Module(
      body=[
          Assert(
              test=Tuple(
                  elts=[
                      Name(id='x', ctx=Load()),
                      Name(id='y', ctx=Load())],
                  ctx=Load()))],
      type_ignores=[])

as the AST of an assert statement with an expression and a message::

  Module(
      body=[
          Assert(
              test=Name(id='x', ctx=Load()),
              msg=Name(id='y', ctx=Load()))],
      type_ignores=[])

The problem with this approach is that the AST of the first form will
technically be "incorrect" as we already have a specialized form for the AST of
an assert statement with a test and a message (the second one). This
means that many tools that deal with ASTs will need to be aware of this change
in semantics, which will be confusing as there is already a correct form that
better expresses the new meaning.

Specification
=============

This PEP proposes changing the grammar of the ``assert`` statement to: ::

    | 'assert' '(' expression ',' expression [','] ')' &(NEWLINE | ';')
    | 'assert' a=expression [',' expression ]

Where the first line is the new form of the assert statement that allows
parentheses. The lookahead is needed so statements like ``assert (a, b) <= c,
"something"`` are still parsed correctly and to prevent the parser to eagerly
capture the tuple as the full statement.

Optionally, new "invalid" rule can be added to produce custom syntax errors to
cover tuples with 0, 1, 3 or more elements.


Backwards Compatibility
=======================

The change is not technically backwards compatible, as parsing ``assert (x,y)``
is currently interpreted as an assert statement with a 2-tuple as the subject,
while after this change it will be interpreted as ``assert x,y``.

On the other hand, assert statements of this kind always pass, so they are
effectively not doing anything in user code. The authors of this document think
that this backwards incompatibility nature is beneficial, as it will highlight
these cases in user code while before they will have passed unnoticed (assuming that
these cases still exist because users are ignoring syntax warnings).

Security Implications
=====================

There are no security implications for this change.


How to Teach This
=================

The new form of the ``assert`` statement will be documented as part of the language
standard.

When teaching the form with error message of the ``assert`` statement to users,
now it can be noted that adding parentheses also work as expected, which allows to break
the statement over multiple lines.


Reference Implementation
========================

A proposed draft PR with the change exist in [GH-30247]_.


References
==========

.. [bpo-46167] https://bugs.python.org/issue46167
.. [GH-30247] https://github.com/python/cpython/pull/30247


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0651.rst
content:
PEP: 651
Title: Robust Stack Overflow Handling
Author: Mark Shannon <mark@hotpy.org>
Status: Rejected
Type: Standards Track
Created: 18-Jan-2021
Post-History: 19-Jan-2021


Rejection Notice
================

This PEP has been `rejected by the Python Steering Council
<https://mail.python.org/archives/list/python-dev@python.org/thread/75BFSBM5AJWXOF5OSPLMJQSTP3TDOKRP/>`_.


Abstract
========

This PEP proposes that Python should treat machine stack overflow differently from runaway recursion.

This would allow programs to set the maximum recursion depth to fit their needs
and provide additional safety guarantees.

If this PEP is accepted, then the following program will run safely to completion::

  sys.setrecursionlimit(1_000_000)

  def f(n):
      if n:
          f(n-1)

  f(500_000)

and the following program will raise a ``StackOverflow``, without causing a VM crash::

  sys.setrecursionlimit(1_000_000)

  class X:
      def __add__(self, other):
          return self + other

  X() + 1

Motivation
==========

CPython uses a single recursion depth counter to prevent both runaway recursion and C stack overflow.
However, runaway recursion and machine stack overflow are two different things.
Allowing machine stack overflow is a potential security vulnerability, but limiting recursion depth can prevent the
use of some algorithms in Python.

Currently, if a program needs to deeply recurse it must manage the maximum recursion depth allowed,
hopefully managing to set it in the region between the minimum needed to run correctly and the maximum that is safe
to avoid a memory protection error.

By separating the checks for C stack overflow from checks for recursion depth,
pure Python programs can run safely, using whatever level of recursion they require.

Rationale
=========

CPython currently relies on a single limit to guard against potentially dangerous stack overflow
in the virtual machine and to guard against run away recursion in the Python program.

This is a consequence of the implementation which couples the C and Python call stacks.
By breaking this coupling, we can improve both the usability of CPython and its safety.

The recursion limit exists to protect against runaway recursion, the integrity of the virtual machine should not depend on it.
Similarly, recursion should not be limited by implementation details.

Specification
=============

Two new exception classes will be added, ``StackOverflow`` and ``RecursionOverflow``, both of which will be
sub-classes of ``RecursionError``

StackOverflow exception
-----------------------

A ``StackOverflow`` exception will be raised whenever the interpreter or builtin module code
determines that the C stack is at or nearing a limit of safety.
``StackOverflow`` is a sub-class of ``RecursionError``,
so any code that handles ``RecursionError`` will handle ``StackOverflow``

RecursionOverflow exception
---------------------------

A ``RecursionOverflow`` exception will be raised when a call to a Python function
causes the recursion limit to be exceeded.
This is a slight change from current behavior which raises a ``RecursionError``.
``RecursionOverflow`` is a sub-class of ``RecursionError``,
so any code that handles ``RecursionError`` will continue to work as before.

Decoupling the Python stack from the C stack
--------------------------------------------

In order to provide the above guarantees and ensure that any program that worked previously
continues to do so, the Python and C stack will need to be separated.
That is, calls to Python functions from Python functions, should not consume space on the C stack.
Calls to and from builtin functions will continue to consume space on the C stack.

The size of the C stack will be implementation defined, and may vary from machine to machine.
It may even differ between threads. However, there is an expectation that any code that could run
with the recursion limit set to the previous default value, will continue to run.

Many operations in Python perform some sort of call at the C level.
Most of these will continue to consume C stack, and will result in a
``StackOverflow`` exception if uncontrolled recursion occurs.


Other Implementations
---------------------

Other implementations are required to fail safely regardless of what value the recursion limit is set to.

If the implementation couples the Python stack to the underlying VM or hardware stack,
then it should raise a ``RecursionOverflow`` exception when the recursion limit is exceeded,
but the underlying stack does not overflow.
If the underlying stack overflows, or is near to overflow,
then a ``StackOverflow`` exception should be raised.

C-API
-----

A new function, ``Py_CheckStackDepth()`` will be added, and the behavior of ``Py_EnterRecursiveCall()`` will be modified slightly.

Py_CheckStackDepth()
''''''''''''''''''''

``int Py_CheckStackDepth(const char *where)``
will return 0 if there is no immediate danger of C stack overflow.
It will return -1 and set an exception, if the C stack is near to overflowing.
The ``where`` parameter is used in the exception message, in the same fashion
as the ``where`` parameter of ``Py_EnterRecursiveCall()``.

Py_EnterRecursiveCall()
'''''''''''''''''''''''

``Py_EnterRecursiveCall()`` will be modified to call ``Py_CheckStackDepth()`` before performing its current function.

PyLeaveRecursiveCall()
''''''''''''''''''''''

``Py_LeaveRecursiveCall()`` will remain unchanged.

Backwards Compatibility
=======================

This feature is fully backwards compatible at the Python level.
Some low-level tools, such as machine-code debuggers, will need to be modified.
For example, the gdb scripts for Python will need to be aware that there may be more than one Python frame
per C frame.

C code that uses the ``Py_EnterRecursiveCall()``, ``PyLeaveRecursiveCall()`` pair of
functions will continue to work correctly. In addition, ``Py_EnterRecursiveCall()``
may raise a ``StackOverflow`` exception.

New code should use the ``Py_CheckStackDepth()`` function, unless the code wants to
count as a Python function call with regard to the recursion limit.

We recommend that "python-like" code, such as Cython-generated functions,
use ``Py_EnterRecursiveCall()``, but other code use ``Py_CheckStackDepth()``.


Security Implications
=====================

It will no longer be possible to crash the CPython virtual machine through recursion.

Performance Impact
==================

It is unlikely that the performance impact will be significant.

The additional logic required will probably have a very small negative impact on performance.
The improved locality of reference from reduced C stack use should have some small positive impact.

It is hard to predict whether the overall effect will be positive or negative,
but it is quite likely that the net effect will be too small to be measured.


Implementation
==============

Monitoring C stack consumption
------------------------------

Gauging whether a C stack overflow is imminent is difficult. So we need to be conservative.
We need to determine a safe bounds for the stack, which is not something possible in portable C code.

For major platforms, the platform specific API will be used to provide an accurate stack bounds.
However, for minor platforms some amount of guessing may be required.
While this might sound bad, it is no worse than the current situation, where we guess that the
size of the C stack is at least 1000 times the stack space required for the chain of calls from
``_PyEval_EvalFrameDefault`` to ``_PyEval_EvalFrameDefault``.

This means that in some cases the amount of recursion possible may be reduced.
In general, however, the amount of recursion possible should be increased, as many calls will use no C stack.

Our general approach to determining a limit for the C stack is to get an address within the current C frame,
as early as possible in the call chain. The limit can then be guessed by adding some constant to that.

Making Pyt


id: pep-0645.rst
content:
PEP: 645
Title: Allow writing optional types as ``x?``
Author: Maggie Moss <maggiebmoss@gmail.com>
Sponsor: Guido van Rossum <guido@python.org>
Status: Withdrawn
Type: Standards Track
Created: 25-Aug-2020
Resolution: https://mail.python.org/archives/list/typing-sig@python.org/message/E75SPV6DDHLEEFSA5MBN5HUOQWDMUQJ2/


Abstract
========
This PEP proposes adding a ``?`` operator for types to allow writing ``int?`` in place of ``Optional[int]``.


PEP Withdrawal
==============

The notation ``T|None`` introduced by :pep:`604` to write ``Optional[T]`` is a
fine alternative to ``T?`` and does not require new syntax.

Using ``T?`` to mean ``T|None`` is also inconsistent with TypeScript
where it roughly means ``NotRequired[T]``.
Such inconsistency would likely confuse folks coming from TypeScript to Python.

The above represents the consensus of
`typing-sig <https://mail.python.org/archives/list/typing-sig@python.org/>`_
and the sponsor of this PEP.


Motivation
==========
Types have become a valuable and powerful part of the Python language. However, many type annotations are verbose and add
considerable friction to using type annotations. By improving the typing syntax, adding types to Python code becomes simpler
and improves the development experience for Python users.

In a similar vein, a PEP to introduce short hand syntax for :pep:`Union types <604>` has
been approved and implemented.


Rationale
=========

Types in Python can be quite verbose, this can be a hindrance when working towards type adoption. Making types more ergonomic,
as was done with the Union type in :pep:`604` (e.g., int | str), would reduce the effort needed to add types to new and existing Python code.
The Optional annotation is used frequently in both partially and fully typed Python code bases. In a small sampling of `5 well-typed open
source projects, on average 7% of annotations
<https://gist.github.com/MaggieMoss/fd8dfe002b2702fae243dbf81a62624e>`_ included at least one optional type. This indicates
that updating the syntax has the potential to make types more concise, reduce code length and improve readability.

Simplifying the syntax for optionals has been `discussed previously <https://github.com/python/typing/issues/429>`_ within the typing community.
The consensus during these conversations has been that ``?`` is the preferred operator. There is no native support for unary ``?`` in Python and this will
need to be added to the runtime.

Adding the ? sigil to the Python grammar has been proposed previously in :pep:`505`, which is currently in a deferred state.
:pep:`505` proposes a:

  - "None coalescing" binary operator ``??``

  - "None-aware attribute access" operator ``?.`` ("maybe dot")

  - "None-aware indexing" operator ``?[]`` ("maybe subscript")


Should :pep:`505` be approved in the future, it would not interfere with the typing specific ``?`` proposed in this PEP. As well,
since all uses of the ``?`` would be conceptually related, it would not be confusing in terms of learning Python or a hindrance to quick visual comprehension.

The proposed syntax, with the postfix operator, mimics the optional syntax found in other typed languages, like C#, TypeScript and Swift.
The widespread adoption and popularity of these languages means that Python developers are likely already familiar with this syntax.

.. code:: text

   // Optional in Swift
   var example: String?

   // Optional in C#
   string? example;

Adding this syntax would also follow the often used pattern of using builtin types as annotations. For example, ``list``, ``dict`` and ``None``. This would allow more annotations to be
added to Python code without importing from ``typing``.


Specification
=============

The new optional syntax should be accepted for function, variable, attribute and parameter annotations.

.. code:: text

   # instead of
   # def foo(x: Optional[int], y: Optional[str], z: Optional[list[int]): ...
   def foo(x: int?, y: str?, x: list[int]?): ...

   # def bar(x: list[typing.Optional[int]]): ...
   def bar(x: list[int?]): ...

The new optional syntax should be equivalent to the existing typing.Optional syntax

.. code:: text

   typing.Optional[int] == int?

The new optional syntax should have the same identity as the existing typing.Optional syntax.

.. code:: text

   typing.Optional[int] is int?


It should also be equivalent to a Union with None.

.. code:: text

   # old syntax
   int? == typing.Union[int, None]

   # new syntax
   int? == int | None

Since the new Union syntax specified in :pep:`604` is supported in ``isinstance`` and ``issubclass``, the new optional syntax should be supported in both ``isinstance`` and ``issubclass``,

.. code:: text

   isinstance(1, int?) # true
   issubclass(Child, Super?) # true

A new dunder method will need to be implemented to allow the ``?`` operator to be overloaded for other functionality.


Backwards Compatibility
=======================

``?`` is currently unused in Python syntax, therefore this PEP is fully backwards compatible.

Reference Implementation
========================

A reference implementation can be found `here <https://github.com/python/cpython/compare/main...MaggieMoss:new-optional-syntax-postfix>`_.

Rejected Ideas
==============

Discussed alternatives were

* The ``~`` operator was considered in place of ``?``.
* A prefix operator (``?int``).


Copyright
=========

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.


id: pep-0692.rst
content:
PEP: 692
Title: Using TypedDict for more precise \*\*kwargs typing
Author: Franek Magiera <framagie@gmail.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-692-using-typeddict-for-more-precise-kwargs-typing/17314
Status: Final
Type: Standards Track
Topic: Typing
Created: 29-May-2022
Python-Version: 3.12
Post-History: `29-May-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/U42MJE6QZYWPVIFHJIGIT7OE52ZGIQV3/>`__,
              `12-Jul-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/PLCNW2XR4OOKAKHEZQM7R2AYVYUXPZGW/>`__,
              `12-Jul-2022 <https://discuss.python.org/t/pep-692-using-typeddict-for-more-precise-kwargs-typing/17314>`__,
Resolution: https://discuss.python.org/t/pep-692-using-typeddict-for-more-precise-kwargs-typing/17314/81

.. canonical-typing-spec:: :ref:`typing:unpack-kwargs`

Abstract
========

Currently ``**kwargs`` can be type hinted as long as all of the keyword
arguments specified by them are of the same type. However, that behaviour can
be very limiting. Therefore, in this PEP we propose a new way to enable more
precise ``**kwargs`` typing. The new approach revolves around using
``TypedDict`` to type ``**kwargs`` that comprise keyword arguments of different
types.

Motivation
==========

Currently annotating ``**kwargs`` with a type ``T`` means that the ``kwargs``
type is in fact ``dict[str, T]``. For example::

    def foo(**kwargs: str) -> None: ...

means that all keyword arguments in ``foo`` are strings (i.e., ``kwargs`` is
of type ``dict[str, str]``). This behaviour limits the ability to type
annotate ``**kwargs`` only to the cases where all of them are of the same type.
However, it is often the case that keyword arguments conveyed by ``**kwargs``
have different types that are dependent on the keyword's name. In those cases
type annotating ``**kwargs`` is not possible. This is especially a problem for
already existing codebases where the need of refactoring the code in order to
introduce proper type annotations may be considered not worth the effort. This
in turn prevents the project from getting all of the benefits that type hinting
can provide.

Moreover, ``**kwargs`` can be used to reduce the amount of code needed in
cases when there is a top-level function that is a part of a public API and it
calls a bunch of helper functions, all of which expect the same keyword
arguments. Unfortunately, if those helper functions were to use ``**kwargs``,
there is no way to properly type hint them if the keyword arguments they expect
are of different types. In addition, even if the keyword arguments are of the
same type, there is no way to check whether the function is being called with
keyword names that it actually expects.

As described in the `Intended Usage`_ section,
using ``**kwargs`` is not always the best tool for the job. Despite that, it is
still a widely used pattern. As a consequence, there has been a lot of
discussion around supporting more precise ``**kwargs`` typing and it became a
feature that would be valuable for a large part of the Python community. This
is best illustrated by the `mypy GitHub issue 4441 <mypyIssue4441_>`__ which
contains a lot of real world cases that could benefit from this propsal.

One more use case worth mentioning for which ``**kwargs`` are also convenient,
is when a function should accommodate optional keyword-only arguments that
don't have default values. A need for a pattern like that can arise when values
that are usually used as defaults to indicate no user input, such as ``None``,
can be passed in by a user and should result in a valid, non-default behavior.
For example, this issue `came up <httpxIssue1384_>`__ in the popular ``httpx`` library.

Rationale
=========

:pep:`589` introduced the ``TypedDict`` type constructor that supports dictionary
types consisting of string keys and values of potentially different types. A
function's keyword arguments represented by a formal parameter that begins with
double asterisk, such as ``**kwargs``, are received as a dictionary.
Additionally, such functions are often called using unpacked dictionaries to
provide keyword arguments. This makes ``TypedDict`` a perfect candidate to be
used for more precise ``**kwargs`` typing. In addition, with ``TypedDict``
keyword names can be taken into account during static type analysis. However,
specifying ``**kwargs`` type with a ``TypedDict`` means, as mentioned earlier,
that each keyword argument specified by ``**kwargs`` is a ``TypedDict`` itself.
For instance::

    class Movie(TypedDict):
        name: str
        year: int

    def foo(**kwargs: Movie) -> None: ...

means that each keyword argument in ``foo`` is itself a ``Movie`` dictionary
that has a ``name`` key with a string type value and a ``year`` key with an
integer type value. Therefore, in order to support specifying ``kwargs`` type
as a ``TypedDict`` without breaking current behaviour, a new construct has to
be introduced.

To support this use case, we propose reusing ``Unpack`` which
was initially introduced in :pep:`646`. There are several reasons for doing so:

* Its name is quite suitable and intuitive for the ``**kwargs`` typing use case
  as our intention is to "unpack" the keywords arguments from the supplied
  ``TypedDict``.
* The current way of typing ``*args`` would be extended to ``**kwargs``
  and those are supposed to behave similarly.
* There would be no need to introduce any new special forms.
* The use of ``Unpack`` for the purposes described in this PEP does not
  interfere with the use cases described in :pep:`646`.

Specification
=============

With ``Unpack`` we introduce a new way of annotating ``**kwargs``.
Continuing the previous example::

    def foo(**kwargs: Unpack[Movie]) -> None: ...

would mean that the ``**kwargs`` comprise two keyword arguments specified by
``Movie`` (i.e. a ``name`` keyword of type ``str`` and a ``year`` keyword of
type ``int``). This indicates that the function should be called as follows::

    kwargs: Movie = {"name": "Life of Brian", "year": 1979}

    foo(**kwargs)                               # OK!
    foo(name="The Meaning of Life", year=1983)  # OK!

When ``Unpack`` is used, type checkers treat ``kwargs`` inside the
function body as a ``TypedDict``::

    def foo(**kwargs: Unpack[Movie]) -> None:
        assert_type(kwargs, Movie)  # OK!


Using the new annotation will not have any runtime effect - it should only be
taken into account by type checkers. Any mention of errors in the following
sections relates to type checker errors.

Function calls with standard dictionaries
-----------------------------------------

Passing a dictionary of type ``dict[str, object]`` as a ``**kwargs`` argument
to a function that has ``**kwargs`` annotated with ``Unpack`` must generate a
type checker error. On the other hand, the behaviour for functions using
standard, untyped dictionaries can depend on the type checker. For example::

    def foo(**kwargs: Unpack[Movie]) -> None: ...

    movie: dict[str, object] = {"name": "Life of Brian", "year": 1979}
    foo(**movie)  # WRONG! Movie is of type dict[str, object]

    typed_movie: Movie = {"name": "The Meaning of Life", "year": 1983}
    foo(**typed_movie)  # OK!

    another_movie = {"name": "Life of Brian", "year": 1979}
    foo(**another_movie)  # Depends on the type checker.

Keyword collisions
------------------

A ``TypedDict`` that is used to type ``**kwargs`` could potentially contain
keys that are already defined in the function's signature. If the duplicate
name is a standard parameter, an error should be reported by type checkers.
If the duplicate name is a positional-only parameter, no errors should be
generated. For example::

    def foo(name, **kwargs: Unpack[Movie]) -> None: ...     # WRONG! "name" will
                                                            # always bind to the
                                              


id: pep-0686.rst
content:
PEP: 686
Title: Make UTF-8 mode default
Author: Inada Naoki <songofacandy@gmail.com>
Discussions-To: https://discuss.python.org/t/14737
Status: Accepted
Type: Standards Track
Created: 18-Mar-2022
Python-Version: 3.15
Post-History: `18-Mar-2022 <https://discuss.python.org/t/14435>`__,
              `31-Mar-2022 <https://discuss.python.org/t/14737>`__
Resolution: https://discuss.python.org/t/14737/9


Abstract
========

This PEP proposes enabling :pep:`UTF-8 mode <540>` by default.

With this change, Python consistently uses UTF-8 for default encoding of
files, stdio, and pipes.


Motivation
==========

UTF-8 becomes de facto standard text encoding.

* The default encoding of Python source files is UTF-8.
* JSON, TOML, YAML use UTF-8.
* Most text editors, including Visual Studio Code and Windows Notepad use
  UTF-8 by default.
* Most websites and text data on the internet use UTF-8.
* And many other popular programming languages, including Node.js, Go, Rust,
  and Java uses UTF-8 by default.

Changing the default encoding to UTF-8 makes it easier for Python to
interoperate with them.

Additionally, many Python developers using Unix forget that the default
encoding is platform dependent.
They omit to specify ``encoding="utf-8"`` when they read text files encoded
in UTF-8 (e.g. JSON, TOML, Markdown, and Python source files).
Inconsistent default encoding causes many bugs.


Specification
=============

Enable UTF-8 mode by default
----------------------------

Python will enable UTF-8 mode by default from Python 3.15.

Users can still disable UTF-8 mode by setting ``PYTHONUTF8=0`` or
``-X utf8=0``.


``locale.getencoding()``
------------------------

Since UTF-8 mode affects ``locale.getpreferredencoding(False)``,
we need an API to get locale encoding regardless of UTF-8 mode.

``locale.getencoding()`` will be added for this purpose.
It returns locale encoding too, but ignores UTF-8 mode.

When ``warn_default_encoding`` option is specified,
``locale.getpreferredencoding()`` will emit ``EncodingWarning`` like
``open()`` (see also :pep:`597`).

This API was added in Python 3.11.


Fixing ``encoding="locale"`` option
-----------------------------------

:pep:`597` added the ``encoding="locale"`` option to the ``TextIOWrapper``.
This option is used to specify the locale encoding explicitly.
``TextIOWrapper`` should use locale encoding when the option is specified,
regardless of default text encoding.

But ``TextIOWrapper`` uses ``"UTF-8"`` in UTF-8 mode even if
``encoding="locale"`` is specified for now.
This behavior is inconsistent with the :pep:`597` motivation.
It is because we didn't expect making UTF-8 mode default when Python
changes its default text encoding.

This inconsistency should be fixed before making UTF-8 mode default.
``TextIOWrapper`` should use locale encoding when ``encoding="locale"`` is
passed even in UTF-8 mode.

This issue was fixed in Python 3.11.


Backward Compatibility
======================

Most Unix systems use UTF-8 locale and Python enables UTF-8 mode when its
locale is C or POSIX.
So this change mostly affects Windows users.

When a Python program depends on the default encoding, this change may cause
``UnicodeError``, mojibake, or even silent data corruption.
So this change should be announced loudly.

This is the guideline to fix this backward compatibility issue:

1. Disable UTF-8 mode.
2. Use ``EncodingWarning`` (:pep:`597`) to find every places UTF-8 mode
   affects.

   * If ``encoding`` option is omitted, consider using ``encoding="utf-8"``
     or ``encoding="locale"``.
   * If ``locale.getpreferredencoding()`` is used, consider using
     ``"utf-8"`` or ``locale.getencoding()``.

3. Test the application with UTF-8 mode.


Preceding examples
==================

* Ruby `changed <Feature #16604_>`__ the default ``external_encoding``
  to UTF-8 on Windows in Ruby 3.0 (2020).
* Java `changed <JEP 400_>`__ the default text encoding
  to UTF-8 in JDK 18. (2022).

Both Ruby and Java have an option for backward compatibility.
They don't provide any warning like :pep:`597`'s ``EncodingWarning``
in Python for use of the default encoding.


Rejected Alternative
====================

Deprecate implicit encoding
---------------------------

Deprecating the use of the default encoding is considered.

But there are many cases that the default encoding is used for reading/writing
only ASCII text.
Additionally, such warnings are not useful for non-cross platform applications
run on Unix.

So forcing users to specify the ``encoding`` everywhere is too painful.
Emitting a lot of ``DeprecationWarning`` will lead users ignore warnings.

:pep:`387` requires adding a warning for backward incompatible changes.
But it doesn't require using ``DeprecationWarning``.
So using optional ``EncodingWarning`` doesn't violate the :pep:`387`.

Java also rejected this idea in `JEP 400`_.


Use ``PYTHONIOENCODING`` for PIPEs
----------------------------------

To ease backward compatibility issue, using ``PYTHONIOENCODING`` as the
default encoding of PIPEs in the ``subprocess`` module is considered.

With this idea, users can use legacy encoding for
``subprocess.Popen(text=True)`` even in UTF-8 mode.

But this idea makes "default encoding" complicated.
And this idea is also backward incompatible.

So this idea is rejected. Users can disable UTF-8 mode until they replace
``text=True`` with ``encoding="utf-8"`` or ``encoding="locale"``.


How to teach this
=================

For new users, this change reduces things that need to teach.
Users don't need to learn about text encoding in their first year.
They should learn it when they need to use non-UTF-8 text files.

For existing users, see the `Backward compatibility`_ section.


References
==========

.. _Feature #16604: https://bugs.ruby-lang.org/issues/16604

.. _JEP 400: https://openjdk.java.net/jeps/400


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0309.rst
content:
PEP: 309
Title: Partial Function Application
Author: Peter Harris <scav@blueyonder.co.uk>
Status: Final
Type: Standards Track
Created: 08-Feb-2003
Python-Version: 2.5
Post-History: 10-Feb-2003, 27-Feb-2003, 22-Feb-2004, 28-Apr-2006


Note
====

Following the acceptance of this PEP, further discussion on python-dev and
comp.lang.python revealed a desire for several tools that operated on
function objects, but were not related to functional programming.  Rather
than create a new module for these tools, it was agreed [1]_ that the
"functional" module be renamed to "functools" to reflect its newly-widened
focus.

References in this PEP to a "functional" module have been left in for
historical reasons.


Abstract
========

This proposal is for a function or callable class that allows a new
callable to be constructed from a callable and a partial argument list
(including positional and keyword arguments).

I propose a standard library module called "functional", to hold
useful higher-order functions, including the implementation of
partial().

An implementation has been submitted to SourceForge [2]_.


Acceptance
==========

Patch #941881 was accepted and applied in 2005 for Py2.5.  It is
essentially as outlined here, a partial() type constructor binding
leftmost positional arguments and any keywords.  The partial object has
three read-only attributes func, args, and keywords.  Calls to the partial
object can specify keywords that override those in the object itself.

There is a separate and continuing discussion of whether to modify the
partial implementation with a __get__ method to more closely emulate
the behavior of an equivalent function.

Motivation
==========

In functional programming, function currying is a way of implementing
multi-argument functions in terms of single-argument functions.  A
function with N arguments is really a function with 1 argument that
returns another function taking (N-1) arguments.  Function application
in languages like Haskell and ML works such that a function call::

    f x y z

actually means::

   (((f x) y) z)

This would be only an obscure theoretical issue except that in actual
programming it turns out to be very useful.  Expressing a function in
terms of partial application of arguments to another function can be
both elegant and powerful, and in functional languages it is heavily
used.

In some functional languages, (e.g. Miranda) you can use an expression
such as ``(+1)`` to mean the equivalent of Python's
``(lambda x: x + 1)``.

In general, languages like that are strongly typed, so the compiler
always knows the number of arguments expected and can do the right
thing when presented with a functor and less arguments than expected.

Python does not implement multi-argument functions by currying, so if
you want a function with partially-applied arguments you would
probably use a lambda as above, or define a named function for each
instance.

However, lambda syntax is not to everyone's taste, so say the least.
Furthermore, Python's flexible parameter passing using both positional
and keyword presents an opportunity to generalise the idea of partial
application and do things that lambda cannot.


Example Implementation
======================

Here is one way to do a create a callable with partially-applied
arguments in Python.  The implementation below is based on improvements
provided by Scott David Daniels::

    class partial(object):

        def __init__(*args, **kw):
            self = args[0]
            self.fn, self.args, self.kw = (args[1], args[2:], kw)

        def __call__(self, *args, **kw):
            if kw and self.kw:
                d = self.kw.copy()
                d.update(kw)
            else:
                d = kw or self.kw
            return self.fn(*(self.args + args), **d)

(A recipe similar to this has been in the Python Cookbook for some
time [3]_.)

Note that when the object is called as though it were a function,
positional arguments are appended to those provided to the
constructor, and keyword arguments override and augment those provided
to the constructor.

Positional arguments, keyword arguments or both can be supplied at
when creating the object and when calling it.


Examples of Use
===============

So ``partial(operator.add, 1)`` is a bit like ``(lambda x: 1 + x)``.
Not an example where you see the benefits, of course.

Note too, that you could wrap a class in the same way, since classes
themselves are callable factories for objects.  So in some cases,
rather than defining a subclass, you can specialise classes by partial
application of the arguments to the constructor.

For example, ``partial(Tkinter.Label, fg='blue')`` makes Tkinter
Labels that have a blue foreground by default.

Here's a simple example that uses partial application to construct
callbacks for Tkinter widgets on the fly::

    from Tkinter import Tk, Canvas, Button
    import sys
    from functional import partial

    win = Tk()
    c = Canvas(win,width=200,height=50)
    c.pack()

    for colour in sys.argv[1:]:
        b = Button(win, text=colour,
                   command=partial(c.config, bg=colour))
        b.pack(side='left')

    win.mainloop()


Abandoned Syntax Proposal
=========================

I originally suggested the syntax ``fn@(*args, **kw)``, meaning the
same as ``partial(fn, *args, **kw)``.

The @ sign is used in some assembly languages to imply register
indirection, and the use here is also a kind of indirection.
``f@(x)`` is not ``f(x)``, but a thing that becomes ``f(x)`` when you
call it.

It was not well-received, so I have withdrawn this part of the
proposal.  In any case, @ has been taken for the new decorator syntax.


Feedback from comp.lang.python and python-dev
=============================================

Among the opinions voiced were the following (which I summarise):

* Lambda is good enough.

* The @ syntax is ugly (unanimous).

* It's really a curry rather than a closure.  There is an almost
  identical implementation of a curry class on ActiveState's Python
  Cookbook.

* A curry class would indeed be a useful addition to the standard
  library.

* It isn't function currying, but partial application.  Hence the
  name is now proposed to be partial().

* It maybe isn't useful enough to be in the built-ins.

* The idea of a module called ``functional`` was well received, and
  there are other things that belong there (for example function
  composition).

* For completeness, another object that appends partial arguments
  after those supplied in the function call (maybe called
  ``rightcurry``) has been suggested.

I agree that lambda is usually good enough, just not always.  And I
want the possibility of useful introspection and subclassing.

I disagree that @ is particularly ugly, but it may be that I'm just
weird.  We have dictionary, list and tuple literals neatly
differentiated by special punctuation -- a way of directly expressing
partially-applied function literals is not such a stretch.  However,
not one single person has said they like it, so as far as I'm
concerned it's a dead parrot.

I concur with calling the class partial rather than curry or closure,
so I have amended the proposal in this PEP accordingly.  But not
throughout: some incorrect references to 'curry' have been left in
since that's where the discussion was at the time.

Partially applying arguments from the right, or inserting arguments at
arbitrary positions creates its own problems, but pending discovery of
a good implementation and non-confusing semantics, I don't think it
should be ruled out.

Carl Banks posted an implementation as a real functional closure::

    def curry(fn, *cargs, **ckwargs):
        def call_fn(*fargs, **fkwargs):
            d = ckwargs.copy()
            d.update(fkwargs)
            return fn(*(cargs + fargs), **d)
        return call_fn

which he assures me is more efficient.

I also coded the class in Pyrex, to estimate how the performance
might be improved by 


id: pep-0335.rst
content:
PEP: 335
Title: Overloadable Boolean Operators
Author: Gregory Ewing <greg.ewing@canterbury.ac.nz>
Status: Rejected
Type: Standards Track
Created: 29-Aug-2004
Python-Version: 3.3
Post-History: 05-Sep-2004, 30-Sep-2011, 25-Oct-2011

Rejection Notice
================

This PEP was rejected.
See https://mail.python.org/pipermail/python-dev/2012-March/117510.html

Abstract
========

This PEP proposes an extension to permit objects to define their own
meanings for the boolean operators 'and', 'or' and 'not', and suggests
an efficient strategy for implementation.  A prototype of this
implementation is available for download.


Background
==========

Python does not currently provide any '__xxx__' special methods
corresponding to the 'and', 'or' and 'not' boolean operators.  In the
case of 'and' and 'or', the most likely reason is that these operators
have short-circuiting semantics, i.e. the second operand is not
evaluated if the result can be determined from the first operand.  The
usual technique of providing special methods for these operators
therefore would not work.

There is no such difficulty in the case of 'not', however, and it
would be straightforward to provide a special method for this
operator.  The rest of this proposal will therefore concentrate mainly
on providing a way to overload 'and' and 'or'.


Motivation
==========

There are many applications in which it is natural to provide custom
meanings for Python operators, and in some of these, having boolean
operators excluded from those able to be customised can be
inconvenient.  Examples include:

1. NumPy, in which almost all the operators are defined on
   arrays so as to perform the appropriate operation between
   corresponding elements, and return an array of the results.  For
   consistency, one would expect a boolean operation between two
   arrays to return an array of booleans, but this is not currently
   possible.

   There is a precedent for an extension of this kind: comparison
   operators were originally restricted to returning boolean results,
   and rich comparisons were added so that comparisons of NumPy
   arrays could return arrays of booleans.

2. A symbolic algebra system, in which a Python expression is
   evaluated in an environment which results in it constructing a tree
   of objects corresponding to the structure of the expression.

3. A relational database interface, in which a Python expression is
   used to construct an SQL query.

A workaround often suggested is to use the bitwise operators '&', '|'
and '~' in place of 'and', 'or' and 'not', but this has some
drawbacks:

* The precedence of these is different in relation to the other operators,
  and they may already be in use for other purposes (as in example 1).

* It is aesthetically displeasing to force users to use something other
  than the most obvious syntax for what they are trying to express.  This
  would be particularly acute in the case of example 3, considering that
  boolean operations are a staple of SQL queries.

* Bitwise operators do not provide a solution to the problem of
  chained comparisons such as 'a < b < c' which involve an implicit
  'and' operation. Such expressions currently cannot be used at all
  on data types such as NumPy arrays where the result of a comparison
  cannot be treated as having normal boolean semantics; they must be
  expanded into something like (a < b) & (b < c), losing a considerable
  amount of clarity.


Rationale
=========

The requirements for a successful solution to the problem of allowing
boolean operators to be customised are:

1. In the default case (where there is no customisation), the existing
   short-circuiting semantics must be preserved.

2. There must not be any appreciable loss of speed in the default
   case.

3. Ideally, the customisation mechanism should allow the object to
   provide either short-circuiting or non-short-circuiting semantics,
   at its discretion.

One obvious strategy, that has been previously suggested, is to pass
into the special method the first argument and a function for
evaluating the second argument.  This would satisfy requirements 1 and
3, but not requirement 2, since it would incur the overhead of
constructing a function object and possibly a Python function call on
every boolean operation.  Therefore, it will not be considered further
here.

The following section proposes a strategy that addresses all three
requirements.  A `prototype implementation`_ of this strategy is
available for download.

.. _prototype implementation:
   http://www.cosc.canterbury.ac.nz/~greg/python/obo//Python_OBO.tar.gz


Specification
=============

Special Methods
---------------

At the Python level, objects may define the following special methods.

===============  =================  ========================
Unary            Binary, phase 1    Binary, phase 2
===============  =================  ========================
* __not__(self)  * __and1__(self)   * __and2__(self, other)
                 * __or1__(self)    * __or2__(self, other)
                                    * __rand2__(self, other)
                                    * __ror2__(self, other)
===============  =================  ========================

The __not__ method, if defined, implements the 'not' operator.  If it
is not defined, or it returns NotImplemented, existing semantics are
used.

To permit short-circuiting, processing of the 'and' and 'or' operators
is split into two phases.  Phase 1 occurs after evaluation of the first
operand but before the second.  If the first operand defines the
relevant phase 1 method, it is called with the first operand as
argument.  If that method can determine the result without needing the
second operand, it returns the result, and further processing is
skipped.

If the phase 1 method determines that the second operand is needed, it
returns the special value NeedOtherOperand.  This triggers the
evaluation of the second operand, and the calling of a relevant
phase 2 method. During phase 2, the __and2__/__rand2__ and
__or2__/__ror2__ method pairs work as for other binary operators.

Processing falls back to existing semantics if at any stage a relevant
special method is not found or returns NotImplemented.

As a special case, if the first operand defines a phase 2 method but
no corresponding phase 1 method, the second operand is always
evaluated and the phase 2 method called.  This allows an object which
does not want short-circuiting semantics to simply implement the
phase 2 methods and ignore phase 1.


Bytecodes
---------

The patch adds four new bytecodes, LOGICAL_AND_1, LOGICAL_AND_2,
LOGICAL_OR_1 and LOGICAL_OR_2.  As an example of their use, the
bytecode generated for an 'and' expression looks like this::

            .
            .
            .
            evaluate first operand
            LOGICAL_AND_1  L
            evaluate second operand
            LOGICAL_AND_2
       L:   .
            .
            .

The LOGICAL_AND_1 bytecode performs phase 1 processing.  If it
determines that the second operand is needed, it leaves the first
operand on the stack and continues with the following code.  Otherwise
it pops the first operand, pushes the result and branches to L.

The LOGICAL_AND_2 bytecode performs phase 2 processing, popping both
operands and pushing the result.


Type Slots
----------

At the C level, the new special methods are manifested as five new
slots in the type object.  In the patch, they are added to the
tp_as_number substructure, since this allows making use of some
existing code for dealing with unary and binary operators.  Their
existence is signalled by a new type flag,
Py_TPFLAGS_HAVE_BOOLEAN_OVERLOAD.

The new type slots are::

    unaryfunc nb_logical_not;
    unaryfunc nb_logical_and_1;
    unaryfunc nb_logical_or_1;
    binaryfunc nb_logical_and_2;
    binaryfunc nb_logical_or_2;


Python/C API Functions
----------------------

There are also five new Python/C API functions corresponding t


id: pep-0453.rst
content:
PEP: 453
Title: Explicit bootstrapping of pip in Python installations
Author: Donald Stufft <donald@stufft.io>,
        Alyssa Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: Martin von LÃ¶wis
Status: Final
Type: Standards Track
Created: 10-Aug-2013
Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013, 19-Sep-2013,
              23-Sep-2013, 29-Sep-2013, 13-Oct-2013, 20-Oct-2013
Resolution: https://mail.python.org/pipermail/python-dev/2013-October/129810.html


Abstract
========

This PEP proposes that the
`Installing Python Modules <http://docs.python.org/3/install>`__ guide in
Python 2.7, 3.3 and 3.4 be updated to officially recommend the use of ``pip``
as the default installer for Python packages, and that appropriate technical
changes be made in Python 3.4 to provide ``pip`` by default in support of
that recommendation.


PEP Acceptance
==============

This PEP was accepted for inclusion in Python 3.4 by Martin von LÃ¶wis on
Tuesday 22nd October, 2013.

`Issue 19347 <https://github.com/python/cpython/issues/63546>`__ has been created to
track the implementation of this PEP.


Rationale
=========

There are two related, but distinct rationales for the proposal in this
PEP. The first relates to the experience of new users, while the second
relates to better enabling the evolution of the broader Python packaging
ecosystem.


Improving the new user experience
---------------------------------

Currently, on systems without a platform package manager and repository,
installing a third-party Python package into a freshly installed Python
requires first identifying an appropriate package manager and then
installing it.

Even on systems that *do* have a platform package manager, it is unlikely to
include every package that is available on the Python Package Index, and
even when a desired third-party package is available, the correct name in
the platform package manager may not be clear.

This means that, to work effectively with the Python Package Index
ecosystem, users must know which package manager to install, where to get
it, and how to install it. The effect of this is that third-party Python
projects are currently required to choose from a variety of undesirable
alternatives:

* Assume the user already has a suitable cross-platform package manager
  installed.
* Duplicate the instructions and tell their users how to install the
  package manager.
* Completely forgo the use of dependencies to ease installation concerns
  for their users.

All of these available options have significant drawbacks.

If a project simply assumes a user already has the tooling then beginning
users may get a confusing error message when the installation command
doesn't work. Some operating systems may ease this pain by providing a
global hook that looks for commands that don't exist and suggest an OS
package they can install to make the command work, but that only works
on systems with platform package managers that include a package that
provides the relevant cross-platform installer command (such as many major
Linux distributions). No such assistance is available for Windows and
Mac OS X users, or more conservative Linux distributions. The challenges
of dealing with this problem for beginners (who are often also completely
new to programming, the use of command line tools and editing system
environment variables) are a regular feature of feedback the core Python
developers receive from professional educators and others introducing new
users to Python.

If a project chooses to duplicate the installation instructions and tell
their users how to install the package manager before telling them how to
install their own project then whenever these instructions need updates
they need updating by every project that has duplicated them. This is
particular problematic when there are multiple competing installation
tools available, and different projects recommend different tools.

This specific problem can be partially alleviated by strongly promoting
``pip`` as the default installer and recommending that other projects
reference `pip's own bootstrapping instructions
<http://www.pip-installer.org/en/latest/installing.html>`__ rather than
duplicating them. However the user experience created by this approach
still isn't particularly good (although there is an effort under way to
create a combined Windows installer for ``pip`` and its dependencies that
should improve matters on that platform, and Mac OS X and \*nix platforms
generally have ``wget`` and hence the ability to easily download and run the
bootstrap scripts from the command line).

The projects that have decided to forgo dependencies altogether are forced
to either duplicate the efforts of other projects by inventing their own
solutions to problems or are required to simply include the other projects
in their own source trees. Both of these options present their own problems
either in duplicating maintenance work across the ecosystem or potentially
leaving users vulnerable to security issues because the included code or
duplicated efforts are not automatically updated when upstream releases a new
version.

By officially recommending and providing by default a specific cross-platform
package manager it will be easier for users trying to install these
third-party packages as well as easier for the people distributing them as
they should now be able to safely assume that most users will have the
appropriate installation tools available (or access to clear instructions on
how to obtain them). This is expected to become more important in the future
as the :pep:`Wheel <427>` package format (deliberately) does not have a built in
"installer" in the form of ``setup.py`` so users wishing to install
from a wheel file will want an installer even in the simplest cases.

Reducing the burden of actually installing a third-party package should
also decrease the pressure to add every useful module to the standard
library. This will allow additions to the standard library to focus more
on why Python should have a particular tool out of the box, and why it
is reasonable for that package to adopt the standard library's 18-24 month
feature release cycle, instead of using the general difficulty of installing
third-party packages as justification for inclusion.

Providing a standard installation system also helps with bootstrapping
alternate build and installer systems, such as ``zc.buildout``, ``hashdist``
and ``conda``. So long as ``pip install <tool>`` works, then a standard
Python-specific installer provides a reasonably secure, cross platform
mechanism to get access to these utilities.


Enabling the evolution of the broader Python packaging ecosystem
----------------------------------------------------------------

As no new packaging standard can achieve widespread adoption without a
transition strategy that covers the versions of Python that are in
widespread *current* use (rather than merely future versions, like most
language features), the change proposed in this PEP is considered a
necessary step in the evolution of the Python packaging ecosystem

The broader community has embraced the Python Package Index as a mechanism
for distributing and installing Python software, but the different concerns
of language evolution and secure software distribution mean that a faster
feature release cycle that encompasses older versions is needed to properly
support the latter.

In addition, the core CPython development team have the luxury of
dropping support for earlier Python versions well before the rest of the
community, as downstream commercial redistributors pick up the task of
providing support for those versions to users that still need it, while
many third party libraries maintain compatibility with those versions as
long as they remain in widespread use.

This means that the current ``setup.py install`` based model for package
installation poses serious difficulties for the development and adoption
of new packaging standards, as, dependin


id: pep-0447.rst
content:
PEP: 447
Title: Add __getdescriptor__ method to metaclass
Author: Ronald Oussoren <ronaldoussoren@mac.com>
Status: Deferred
Type: Standards Track
Created: 12-Jun-2013
Post-History: 02-Jul-2013, 15-Jul-2013, 29-Jul-2013, 22-Jul-2015


Abstract
========

Currently ``object.__getattribute__`` and ``super.__getattribute__`` peek
in the ``__dict__`` of classes on the MRO for a class when looking for
an attribute. This PEP adds an optional ``__getdescriptor__`` method to
a metaclass that replaces this behavior and gives more control over attribute
lookup, especially when using a `super`_ object.

That is, the MRO walking loop in ``_PyType_Lookup`` and
``super.__getattribute__`` gets changed from::

     def lookup(mro_list, name):
         for cls in mro_list:
             if name in cls.__dict__:
                 return cls.__dict__

         return NotFound

to::

     def lookup(mro_list, name):
         for cls in mro_list:
             try:
                 return cls.__getdescriptor__(name)
             except AttributeError:
                 pass

         return NotFound

The default implementation of ``__getdescriptor__`` looks in the class
dictionary::

   class type:
      def __getdescriptor__(cls, name):
          try:
              return cls.__dict__[name]
          except KeyError:
              raise AttributeError(name) from None


PEP Status
==========

This PEP is deferred until someone has time to update this PEP and push it forward.


Rationale
=========

It is currently not possible to influence how the `super class`_ looks
up attributes (that is, ``super.__getattribute__`` unconditionally
peeks in the class ``__dict__``), and that can be problematic for
dynamic classes that can grow new methods on demand, for example dynamic
proxy classes.

The ``__getdescriptor__`` method makes it possible to dynamically add
attributes even when looking them up using the `super class`_.

The new method affects ``object.__getattribute__`` (and
`PyObject_GenericGetAttr`_) as well for consistency and to have a single
place to implement dynamic attribute resolution for classes.

Background
----------

The current behavior of ``super.__getattribute__`` causes problems for
classes that are dynamic proxies for other (non-Python) classes or types,
an example of which is `PyObjC`_. PyObjC creates a Python class for every
class in the Objective-C runtime, and looks up methods in the Objective-C
runtime when they are used. This works fine for normal access, but doesn't
work for access with `super`_ objects. Because of this PyObjC currently
includes a custom `super`_ that must be used with its classes, as well as
completely reimplementing `PyObject_GenericGetAttr`_ for normal attribute
access.

The API in this PEP makes it possible to remove the custom `super`_ and
simplifies the implementation because the custom lookup behavior can be
added in a central location.

.. note::

   `PyObjC`_ cannot precalculate the contents of the class ``__dict__``
   because Objective-C classes can grow new methods at runtime. Furthermore,
   Objective-C classes tend to contain a lot of methods while most Python
   code will only use a small subset of them, this makes precalculating
   unnecessarily expensive.


The superclass attribute lookup hook
====================================

Both ``super.__getattribute__`` and ``object.__getattribute__`` (or
`PyObject_GenericGetAttr`_ and in particular ``_PyType_Lookup`` in C code)
walk an object's MRO and currently peek in the class' ``__dict__`` to look up
attributes.

With this proposal both lookup methods no longer peek in the class ``__dict__``
but call the special method ``__getdescriptor__``, which is a slot defined
on the metaclass. The default implementation of that method looks
up the name the class ``__dict__``, which means that attribute lookup is
unchanged unless a metatype actually defines the new special method.

Aside: Attribute resolution algorithm in Python
-----------------------------------------------

The attribute resolution process as implemented by ``object.__getattribute__``
(or ``PyObject_GenericGetAttr`` in CPython's implementation) is fairly
straightforward, but not entirely so without reading C code.

The current CPython implementation of object.__getattribute__ is basically
equivalent to the following (pseudo-) Python code (excluding some house
keeping and speed tricks)::


    def _PyType_Lookup(tp, name):
        mro = tp.mro()
        assert isinstance(mro, tuple)

        for base in mro:
           assert isinstance(base, type)

           # PEP 447 will change these lines:
           try:
               return base.__dict__[name]
           except KeyError:
               pass

        return None


    class object:
        def __getattribute__(self, name):
            assert isinstance(name, str)

            tp = type(self)
            descr = _PyType_Lookup(tp, name)

            f = None
            if descr is not None:
                f = descr.__get__
                if f is not None and descr.__set__ is not None:
                    # Data descriptor
                    return f(descr, self, type(self))

            dict = self.__dict__
            if dict is not None:
                try:
                    return self.__dict__[name]
                except KeyError:
                    pass

            if f is not None:
                # Non-data descriptor
                return f(descr, self, type(self))

            if descr is not None:
                # Regular class attribute
                return descr

            raise AttributeError(name)


    class super:
        def __getattribute__(self, name):
           assert isinstance(name, unicode)

           if name != '__class__':
               starttype = self.__self_type__
               mro = startype.mro()

               try:
                   idx = mro.index(self.__thisclass__)

               except ValueError:
                   pass

               else:
                   for base in mro[idx+1:]:
                       # PEP 447 will change these lines:
                       try:
                           descr = base.__dict__[name]
                       except KeyError:
                           continue

                       f = descr.__get__
                       if f is not None:
                           return f(descr,
                               None if (self.__self__ is self.__self_type__) else self.__self__,
                               starttype)

                       else:
                           return descr

           return object.__getattribute__(self, name)


This PEP should change the dict lookup at the lines starting at "# PEP 447" with
a method call to perform the actual lookup, making is possible to affect that
lookup both for normal attribute access and access through the `super proxy`_.

Note that specific classes can already completely override the default
behaviour by implementing their own ``__getattribute__`` slot (with or without
calling the super class implementation).


In Python code
--------------

A meta type can define a method ``__getdescriptor__`` that is called during
attribute resolution by both ``super.__getattribute__``
and ``object.__getattribute``::

    class MetaType(type):
        def __getdescriptor__(cls, name):
            try:
                return cls.__dict__[name]
            except KeyError:
                raise AttributeError(name) from None

The ``__getdescriptor__`` method has as its arguments a class (which is an
instance of the meta type) and the name of the attribute that is looked up.
It should return the value of the attribute without invoking descriptors,
and should raise `AttributeError`_ when the name cannot be found.

The `type`_ class provides a default implementation for ``__getdescriptor__``,
that looks up the name in the class dictionary.

Example usage
.............

The code below implements a silly metaclass that redirects a


