# embeddings

id: pep-0733.rst
content:
PEP: 733
Title: An Evaluation of Python's Public C API
Author: Erlend Egeberg Aasland <erlend@python.org>,
        Domenico Andreoli <domenico.andreoli@linux.com>,
        Stefan Behnel <stefan_ml@behnel.de>,
        Carl Friedrich Bolz-Tereick <cfbolz@gmx.de>,
        Simon Cross <hodgestar@gmail.com>,
        Steve Dower <steve.dower@python.org>,
        Tim Felgentreff <tim.felgentreff@oracle.com>,
        David Hewitt <1939362+davidhewitt@users.noreply.github.com>,
        Shantanu Jain <hauntsaninja at gmail.com>,
        Wenzel Jakob <wenzel.jakob@epfl.ch>,
        Irit Katriel <irit@python.org>,
        Marc-Andre Lemburg <mal@lemburg.com>,
        Donghee Na <donghee.na@python.org>,
        Karl Nelson <nelson85@llnl.gov>,
        Ronald Oussoren <ronaldoussoren@mac.com>,
        Antoine Pitrou <solipsis@pitrou.net>,
        Neil Schemenauer <nas@arctrix.com>,
        Mark Shannon <mark@hotpy.org>,
        Stepan Sindelar <stepan.sindelar@oracle.com>,
        Gregory P. Smith <greg@krypto.org>,
        Eric Snow <ericsnowcurrently@gmail.com>,
        Victor Stinner <vstinner@python.org>,
        Guido van Rossum <guido@python.org>,
        Petr Viktorin <encukou@gmail.com>,
        Carol Willing <willingc@gmail.com>,
        William Woodruff <william@yossarian.net>,
        David Woods <dw-git@d-woods.co.uk>,
        Jelle Zijlstra <jelle.zijlstra@gmail.com>,
Status: Final
Type: Informational
Created: 16-Oct-2023
Post-History: `01-Nov-2023 <https://discuss.python.org/t/pep-733-an-evaluation-of-python-s-public-c-api/37618>`__


Abstract
========

This **informational** PEP describes our shared view of the public C API. The
document defines:

* purposes of the C API
* stakeholders and their particular use cases and requirements
* strengths of the C API
* problems of the C API categorized into nine areas of weakness

This document does not propose solutions to any of the identified problems. By
creating a shared list of C API issues, this document will help to guide
continuing discussion about change proposals and to identify evaluation
criteria.


Introduction
============

Python's C API was not designed for the different purposes it currently
fulfills.  It evolved from what was initially the internal API between
the C code of the interpreter and the Python language and libraries.
In its first incarnation, it was exposed to make it possible to embed
Python into C/C++ applications and to write extension modules in C/C++.
These capabilities were instrumental to the growth of Python's ecosystem.
Over the decades, the C API grew to provide different tiers of stability,
conventions changed, and new usage patterns have emerged, such as bindings
to languages other than C/C++. In the next few years, new developments
are expected to further test the C API, such as the removal of the GIL
and the development of a JIT compiler. However, this growth was not
supported by clearly documented guidelines, resulting in inconsistent
approaches to API design in different subsystems of CPython. In addition,
CPython is no longer the only implementation of Python, and some of the
design decisions made when it was, are difficult for alternative
implementations to work with
[`Issue 64 <https://github.com/capi-workgroup/problems/issues/64>`__].
In the meantime, lessons were learned and mistakes in both the design
and the implementation of the C API were identified.

Evolving the C API is hard due to the combination of backwards
compatibility constraints and its inherent complexity, both
technical and social. Different types of users bring different,
sometimes conflicting, requirements. The tradeoff between stability
and progress is an ongoing, highly contentious topic of discussion
when suggestions are made for incremental improvements.
Several proposals have been put forward for improvement, redesign
or replacement of the C API, each representing a deep analysis of
the problems.  At the 2023 Language Summit, three back-to-back
sessions were devoted to different aspects of the C API. There is
general agreement that a new design can remedy the problems that
the C API has accumulated over the last 30 years, while at the
same time updating it for use cases that it was not originally
designed for.

However, there was also a sense at the Language Summit that we are
trying to discuss solutions without a clear common understanding
of the problems that we are trying to solve. We decided that
we need to agree on the current problems with the C API, before
we are able to evaluate any of the proposed solutions. We
therefore created the
`capi-workgroup <https://github.com/capi-workgroup/problems/issues/>`__
repository on GitHub in order to collect everyone's ideas on that
question.

Over 60 different issues were created on that repository, each
describing a problem with the C API. We categorized them and
identified a number of recurring themes. The sections below
mostly correspond to these themes, and each contains a combined
description of the issues raised in that category, along with
links to the individual issues. In addition, we included a section
that aims to identify the different stakeholders of the C API,
and the particular requirements that each of them has.


C API Stakeholders
==================

As mentioned in the introduction, the C API was originally
created as the internal interface between CPython's
interpreter and the Python layer. It was later exposed as
a way for third-party developers to extend and embed Python
programs. Over the years, new types of stakeholders emerged,
with different requirements and areas of focus. This section
describes this complex state of affairs in terms of the
actions that different stakeholders need to perform through
the C API.

Common Actions for All Stakeholders
-----------------------------------

There are actions which are generic, and required by
all types of API users:

* Define functions and call them
* Define new types
* Create instances of builtin and user-defined types
* Perform operations on object instances
* Introspect objects, including types, instances, and functions
* Raise and handle exceptions
* Import modules
* Access to Python's OS interface

The following sections look at the unique requirements of various stakeholders.

Extension Writers
-----------------

Extension writers are the traditional users of the C API. Their requirements
are the common actions listed above. They also commonly need to:

* Create new modules
* Efficiently interface between modules at the C level


Authors of Embedded Python Applications
---------------------------------------

Applications with an embedded Python interpreter. Examples are
`Blender <https://docs.blender.org/api/current/info_overview.html>`__ and
`OBS <https://obsproject.com/wiki/Getting-Started-With-OBS-Scripting>`__.

They need to be able to:

* Configure the interpreter (import paths, inittab, ``sys.argv``, memory
  allocator, etc.).
* Interact with the execution model and program lifetime, including
  clean interpreter shutdown and restart.
* Represent complex data models in a way Python can use without
  having to create deep copies.
* Provide and import frozen modules.
* Run and manage multiple independent interpreters (in particular, when
  embedded in a library that wants to avoid global effects).

Python Implementations
----------------------

Python implementations such as
`CPython <https://www.python.org>`__,
`PyPy <https://www.pypy.org>`__,
`GraalPy <https://www.graalvm.org/python/>`__,
`IronPython <https://ironpython.net>`__,
`RustPython <https://github.com/RustPython/RustPython>`__,
`MicroPython <https://micropython.org>`__,
and `Jython <https://www.jython.org>`__), may take
very different approaches for the implementation of
different subsystems. They need:

* The API to be abstract and hide implementation details.
* A specification of the API, ideally with a test suite
  that ensures compatibility.
* It would be nice to have an ABI that can be shared
 


id: pep-0280.rst
content:
PEP: 280
Title: Optimizing access to globals
Author: Guido van Rossum <guido@python.org>
Status: Deferred
Type: Standards Track
Created: 10-Feb-2002
Python-Version: 2.3
Post-History:


Deferral
========

While this PEP is a nice idea, no-one has yet emerged to do the work of
hashing out the differences between this PEP, :pep:`266` and :pep:`267`.
Hence, it is being deferred.


Abstract
========

This PEP describes yet another approach to optimizing access to
module globals, providing an alternative to :pep:`266` (Optimizing
Global Variable/Attribute Access by Skip Montanaro) and :pep:`267`
(Optimized Access to Module Namespaces by Jeremy Hylton).

The expectation is that eventually one approach will be picked and
implemented; possibly multiple approaches will be prototyped
first.


Description
===========

(Note: Jason Orendorff writes: """I implemented this once, long
ago, for Python 1.5-ish, I believe.  I got it to the point where
it was only 15% slower than ordinary Python, then abandoned it.
;) In my implementation, "cells" were real first-class objects,
and "celldict" was a copy-and-hack version of dictionary.  I
forget how the rest worked."""  Reference:
https://mail.python.org/pipermail/python-dev/2002-February/019876.html)

Let a cell be a really simple Python object, containing a pointer
to a Python object and a pointer to a cell.  Both pointers may be
``NULL``.  A Python implementation could be::

    class cell(object):

        def __init__(self):
            self.objptr = NULL
            self.cellptr = NULL

The cellptr attribute is used for chaining cells together for
searching built-ins; this will be explained later.

Let a celldict be a mapping from strings (the names of a module's
globals) to objects (the values of those globals), implemented
using a dict of cells.  A Python implementation could be::

    class celldict(object):

        def __init__(self):
            self.__dict = {} # dict of cells

        def getcell(self, key):
            c = self.__dict.get(key)
            if c is None:
                c = cell()
                self.__dict[key] = c
            return c

        def cellkeys(self):
            return self.__dict.keys()

        def __getitem__(self, key):
            c = self.__dict.get(key)
            if c is None:
                raise KeyError, key
            value = c.objptr
            if value is NULL:
                raise KeyError, key
            else:
                return value

        def __setitem__(self, key, value):
            c = self.__dict.get(key)
            if c is None:
                c = cell()
                self.__dict[key] = c
            c.objptr = value

        def __delitem__(self, key):
            c = self.__dict.get(key)
            if c is None or c.objptr is NULL:
                raise KeyError, key
            c.objptr = NULL

        def keys(self):
            return [k for k, c in self.__dict.iteritems()
                    if c.objptr is not NULL]

        def items(self):
            return [k, c.objptr for k, c in self.__dict.iteritems()
                    if c.objptr is not NULL]

        def values(self):
            preturn [c.objptr for c in self.__dict.itervalues()
                    if c.objptr is not NULL]

        def clear(self):
            for c in self.__dict.values():
                c.objptr = NULL

        # Etc.

It is possible that a cell exists corresponding to a given key,
but the cell's objptr is ``NULL``; let's call such a cell empty.  When
the celldict is used as a mapping, it is as if empty cells don't
exist.  However, once added, a cell is never deleted from a
celldict, and it is possible to get at empty cells using the
``getcell()`` method.

The celldict implementation never uses the cellptr attribute of
cells.

We change the module implementation to use a celldict for its
``__dict__``.  The module's getattr, setattr and delattr operations
now map to getitem, setitem and delitem on the celldict.  The type
of ``<module>.__dict__`` and ``globals()`` is probably the only backwards
incompatibility.

When a module is initialized, its ``__builtins__`` is initialized from
the ``__builtin__`` module's ``__dict__``, which is itself a celldict.
For each cell in ``__builtins__``, the new module's ``__dict__`` adds a
cell with a ``NULL`` objptr, whose cellptr points to the corresponding
cell of ``__builtins__``.  Python pseudo-code (ignoring rexec)::

    import __builtin__

    class module(object):

        def __init__(self):
            self.__dict__ = d = celldict()
            d['__builtins__'] = bd = __builtin__.__dict__
            for k in bd.cellkeys():
                c = self.__dict__.getcell(k)
                c.cellptr = bd.getcell(k)

        def __getattr__(self, k):
            try:
                return self.__dict__[k]
            except KeyError:
                raise IndexError, k

        def __setattr__(self, k, v):
            self.__dict__[k] = v

        def __delattr__(self, k):
            del self.__dict__[k]

The compiler generates ``LOAD_GLOBAL_CELL <i>`` (and ``STORE_GLOBAL_CELL
<i>`` etc.) opcodes for references to globals, where ``<i>`` is a small
index with meaning only within one code object like the const
index in ``LOAD_CONST``.  The code object has a new tuple, ``co_globals``,
giving the names of the globals referenced by the code indexed by
``<i>``.  No new analysis is required to be able to do this.

When a function object is created from a code object and a celldict,
the function object creates an array of cell pointers by asking the
celldict for cells corresponding to the names in the code object's
``co_globals``.  If the celldict doesn't already have a cell for a
particular name, it creates and an empty one.  This array of cell
pointers is stored on the function object as ``func_cells``.  When a
function object is created from a regular dict instead of a
celldict, ``func_cells`` is a ``NULL`` pointer.

When the VM executes a ``LOAD_GLOBAL_CELL <i>`` instruction, it gets
cell number ``<i>`` from ``func_cells``.  It then looks in the cell's
``PyObject`` pointer, and if not ``NULL``, that's the global value.  If it
is ``NULL``, it follows the cell's cell pointer to the next cell, if it
is not ``NULL``, and looks in the ``PyObject`` pointer in that cell.  If
that's also ``NULL``, or if there is no second cell, ``NameError`` is
raised.  (It could follow the chain of cell pointers until a ``NULL``
cell pointer is found; but I have no use for this.)  Similar for
``STORE_GLOBAL_CELL <i>``, except it doesn't follow the cell pointer
chain -- it always stores in the first cell.

There are fallbacks in the VM for the case where the function's
globals aren't a celldict, and hence ``func_cells`` is ``NULL``.  In that
case, the code object's ``co_globals`` is indexed with ``<i>`` to find the
name of the corresponding global and this name is used to index the
function's globals dict.


Additional Ideas
================

- Never make ``func_cell`` a ``NULL`` pointer; instead, make up an array
  of empty cells, so that ``LOAD_GLOBAL_CELL`` can index ``func_cells``
  without a ``NULL`` check.

- Make ``c.cellptr`` equal to c when a cell is created, so that
  ``LOAD_GLOBAL_CELL`` can always dereference ``c.cellptr`` without a ``NULL``
  check.

  With these two additional ideas added, here's Python pseudo-code
  for ``LOAD_GLOBAL_CELL``::

      def LOAD_GLOBAL_CELL(self, i):
          # self is the frame
          c = self.func_cells[i]
          obj = c.objptr
          if obj is not NULL:
              return obj # Existing global
          return c.cellptr.objptr # Built-in or NULL

- Be more aggressive:  put the actual values of builtins into module
  dicts, not just pointers to cells containing the actual values.

  There are two points to this:  (1) Simplify and speed access, which
  is the most common operation.  (2) Support faithful emulation of
  extreme existing corner cases.

  WRT  #2, the set of builtins


id: pep-0294.rst
content:
PEP: 294
Title: Type Names in the types Module
Author: Oren Tirosh <oren at hishome.net>
Status: Rejected
Type: Standards Track
Created: 19-Jun-2002
Python-Version: 2.5
Post-History:



Abstract
========

This PEP proposes that symbols matching the type name should be added
to the types module for all basic Python types in the types module::

    types.IntegerType -> types.int
    types.FunctionType -> types.function
    types.TracebackType -> types.traceback
     ...

The long capitalized names currently in the types module will be
deprecated.

With this change the types module can serve as a replacement for the
new module.  The new module shall be deprecated and listed in :pep:`4`.


Pronouncement
=============

A centralized repository of type names was a mistake.  Neither the
"types" nor "new" modules should be carried forward to Python 3.0.

In the meantime, it does not make sense to make the proposed updates
to the modules.  This would cause disruption without any compensating
benefit.

Instead, the problem that some internal types (frames, functions,
etc.) don't live anywhere outside those modules may be addressed by
either adding them to ``__builtin__`` or sys.  This will provide a
smoother transition to Python 3.0.


Rationale
=========

Using two sets of names for the same objects is redundant and
confusing.

In Python versions prior to 2.2 the symbols matching many type names
were taken by the factory functions for those types.  Now all basic
types have been unified with their factory functions and therefore the
type names are available to be consistently used to refer to the type
object.

Most types are accessible as either builtins or in the new module but
some types such as traceback and generator are only accessible through
the types module under names which do not match the type name.  This
PEP provides a uniform way to access all basic types under a single
set of names.


Specification
=============

The types module shall pass the following test::

    import types
    for t in vars(types).values():
        if type(t) is type:
    assert getattr(types, t.__name__) is t

The types 'class', 'instance method' and 'dict-proxy' have already
been renamed to the valid Python identifiers 'classobj',
'instancemethod' and 'dictproxy', making this possible.


Backward compatibility
======================

Because of their widespread use it is not planned to actually remove
the long names from the types module in some future version.  However,
the long names should be changed in documentation and library sources
to discourage their use in new code.


Reference Implementation
========================

A reference implementation is available in
`issue #569328 <https://bugs.python.org/issue569328>`_.


Copyright
=========

This document has been placed in the public domain.


id: pep-0525.rst
content:
PEP: 525
Title: Asynchronous Generators
Author: Yury Selivanov <yury@edgedb.com>
Discussions-To: python-dev@python.org
Status: Final
Type: Standards Track
Created: 28-Jul-2016
Python-Version: 3.6
Post-History: 02-Aug-2016, 23-Aug-2016, 01-Sep-2016, 06-Sep-2016


Abstract
========

:pep:`492` introduced support for native coroutines and ``async``/``await``
syntax to Python 3.5.  It is proposed here to extend Python's
asynchronous capabilities by adding support for
*asynchronous generators*.


Rationale and Goals
===================

Regular generators (introduced in :pep:`255`) enabled an elegant way of
writing complex *data producers* and have them behave like an iterator.

However, currently there is no equivalent concept for the *asynchronous
iteration protocol* (``async for``).  This makes writing asynchronous
data producers unnecessarily complex, as one must define a class that
implements ``__aiter__`` and ``__anext__`` to be able to use it in
an ``async for`` statement.

Essentially, the goals and rationale for :pep:`255`, applied to the
asynchronous execution case, hold true for this proposal as well.

Performance is an additional point for this proposal: in our testing of
the reference implementation, asynchronous generators are **2x** faster
than an equivalent implemented as an asynchronous iterator.

As an illustration of the code quality improvement, consider the
following class that prints numbers with a given delay once iterated::

    class Ticker:
        """Yield numbers from 0 to `to` every `delay` seconds."""

        def __init__(self, delay, to):
            self.delay = delay
            self.i = 0
            self.to = to

        def __aiter__(self):
            return self

        async def __anext__(self):
            i = self.i
            if i >= self.to:
                raise StopAsyncIteration
            self.i += 1
            if i:
                await asyncio.sleep(self.delay)
            return i


The same can be implemented as a much simpler asynchronous generator::

    async def ticker(delay, to):
        """Yield numbers from 0 to `to` every `delay` seconds."""
        for i in range(to):
            yield i
            await asyncio.sleep(delay)


Specification
=============

This proposal introduces the concept of *asynchronous generators* to
Python.

This specification presumes knowledge of the implementation of
generators and coroutines in Python (:pep:`342`, :pep:`380` and :pep:`492`).


Asynchronous Generators
-----------------------

A Python *generator* is any function containing one or more ``yield``
expressions::

    def func():            # a function
        return

    def genfunc():         # a generator function
        yield

We propose to use the same approach to define
*asynchronous generators*::

    async def coro():      # a coroutine function
        await smth()

    async def asyncgen():  # an asynchronous generator function
        await smth()
        yield 42

The result of calling an *asynchronous generator function* is
an *asynchronous generator object*, which implements the asynchronous
iteration protocol defined in :pep:`492`.

It is a ``SyntaxError`` to have a non-empty ``return`` statement in an
asynchronous generator.


Support for Asynchronous Iteration Protocol
-------------------------------------------

The protocol requires two special methods to be implemented:

1. An ``__aiter__`` method returning an *asynchronous iterator*.
2. An ``__anext__`` method returning an *awaitable* object, which uses
   ``StopIteration`` exception to "yield" values, and
   ``StopAsyncIteration`` exception to signal the end of the iteration.

Asynchronous generators define both of these methods.  Let's manually
iterate over a simple asynchronous generator::

    async def genfunc():
        yield 1
        yield 2

    gen = genfunc()

    assert gen.__aiter__() is gen

    assert await gen.__anext__() == 1
    assert await gen.__anext__() == 2

    await gen.__anext__()  # This line will raise StopAsyncIteration.


Finalization
------------

:pep:`492` requires an event loop or a scheduler to run coroutines.
Because asynchronous generators are meant to be used from coroutines,
they also require an event loop to run and finalize them.

Asynchronous generators can have ``try..finally`` blocks, as well as
``async with``.  It is important to provide a guarantee that, even
when partially iterated, and then garbage collected, generators can
be safely finalized.  For example::

    async def square_series(con, to):
        async with con.transaction():
            cursor = con.cursor(
                'SELECT generate_series(0, $1) AS i', to)
            async for row in cursor:
                yield row['i'] ** 2

    async for i in square_series(con, 1000):
        if i == 100:
            break

The above code defines an asynchronous generator that uses
``async with`` to iterate over a database cursor in a transaction.
The generator is then iterated over with ``async for``, which interrupts
the iteration at some point.

The ``square_series()`` generator will then be garbage collected,
and without a mechanism to asynchronously close the generator, Python
interpreter would not be able to do anything.

To solve this problem we propose to do the following:

1. Implement an ``aclose`` method on asynchronous generators
   returning a special *awaitable*.  When awaited it
   throws a ``GeneratorExit`` into the suspended generator and
   iterates over it until either a ``GeneratorExit`` or
   a ``StopAsyncIteration`` occur.

   This is very similar to what the ``close()`` method does to regular
   Python generators, except that an event loop is required to execute
   ``aclose()``.

2. Raise a ``RuntimeError``, when an asynchronous generator executes
   a ``yield`` expression in its ``finally`` block (using ``await``
   is fine, though)::

        async def gen():
            try:
                yield
            finally:
                await asyncio.sleep(1)   # Can use 'await'.

                yield                    # Cannot use 'yield',
                                         # this line will trigger a
                                         # RuntimeError.

3. Add two new methods to the ``sys`` module:
   ``set_asyncgen_hooks()`` and ``get_asyncgen_hooks()``.

The idea behind ``sys.set_asyncgen_hooks()`` is to allow event
loops to intercept asynchronous generators iteration and finalization,
so that the end user does not need to care about the finalization
problem, and everything just works.

``sys.set_asyncgen_hooks()`` accepts two arguments:

* ``firstiter``: a callable which will be called when an asynchronous
  generator is iterated for the first time.

* ``finalizer``: a callable which will be called when an asynchronous
  generator is about to be GCed.

When an asynchronous generator is iterated for the first time,
it stores a reference to the current *finalizer*.

When an asynchronous generator is about to be garbage collected,
it calls its cached *finalizer*.  The assumption is that the finalizer
will schedule an ``aclose()`` call with the loop that was active
when the iteration started.

For instance, here is how asyncio is modified to allow safe
finalization of asynchronous generators::

   # asyncio/base_events.py

   class BaseEventLoop:

       def run_forever(self):
           ...
           old_hooks = sys.get_asyncgen_hooks()
           sys.set_asyncgen_hooks(finalizer=self._finalize_asyncgen)
           try:
               ...
           finally:
               sys.set_asyncgen_hooks(*old_hooks)
               ...

       def _finalize_asyncgen(self, gen):
           self.create_task(gen.aclose())

The second argument, ``firstiter``, allows event loops to maintain
a weak set of asynchronous generators instantiated under their control.
This makes it possible to implement "shutdown" mechanisms to safely
finalize all open generators and close the event loop.

``sys.set_asyncgen_h


id: pep-0243.rst
content:
PEP: 243
Title: Module Repository Upload Mechanism
Author: Sean Reifschneider <jafo-pep@tummy.com>
Discussions-To: distutils-sig@python.org
Status: Withdrawn
Type: Standards Track
Topic: Packaging
Created: 18-Mar-2001
Python-Version: 2.1
Post-History: 20-Mar-2001, 24-Mar-2001


Abstract
========

For a module repository system (such as Perl's CPAN) to be
successful, it must be as easy as possible for module authors to
submit their work.  An obvious place for this submit to happen is
in the Distutils tools after the distribution archive has been
successfully created.  For example, after a module author has
tested their software (verifying the results of ``setup.py sdist``),
they might type ``setup.py sdist --submit``.  This would flag
Distutils to submit the source distribution to the archive server
for inclusion and distribution to the mirrors.

This PEP only deals with the mechanism for submitting the software
distributions to the archive, and does not deal with the actual
archive/catalog server.


Upload Process
==============

The upload will include the Distutils ``PKG-INFO`` meta-data
information (as specified in :pep:`241`), the actual software
distribution, and other optional information.  This information
will be uploaded as a multi-part form encoded the same as a
regular HTML file upload request.  This form is posted using
``ENCTYPE="multipart/form-data"`` encoding (:rfc:`1867`).

The upload will be made to the host "www.python.org" on port
80/tcp (``POST http://www.python.org:80/pypi``).  The form
will consist of the following fields:

- ``distribution`` -- The file containing the module software (for
  example, a ``.tar.gz`` or ``.zip`` file).

- ``distmd5sum`` -- The MD5 hash of the uploaded distribution,
  encoded in ASCII representing the hexadecimal representation
  of the digest (``for byte in digest: s = s + ('%02x' %
  ord(byte))``).

- ``pkginfo`` (optional) -- The file containing the distribution
  meta-data (as specified in :pep:`241`).  Note that if this is
  not included, the distribution file is expected to be in ``.tar``
  format (gzipped and bzipped compressed are allowed) or ``.zip``
  format, with a ``PKG-INFO`` file in the top-level directory it
  extracts (``package-1.00/PKG-INFO``).

- ``infomd5sum`` (required if pkginfo field is present) -- The MD5 hash
  of the uploaded meta-data, encoded in ASCII representing the
  hexadecimal representation of the digest (``for byte in digest:
  s = s + ('%02x' % ord(byte))``).

- ``platform`` (optional) -- A string representing the target
  platform for this distribution.  This is only for binary
  distributions.  It is encoded as
  ``<os_name>-<os_version>-<platform architecture>-<python
  version>``.

- ``signature`` (optional) -- A :rfc:`OpenPGP-compatible <2440>` signature of
  the uploaded distribution as signed by the author.  This may
  be used by the cataloging system to automate acceptance of
  uploads.

- ``protocol_version`` -- A string indicating the protocol version that
  the client supports.  This document describes protocol version "1".


Return Data
===========

The status of the upload will be reported using HTTP non-standard
(``X-*``) headers.  The ``X-Swalow-Status`` header may have the following
values:

- ``SUCCESS`` -- Indicates that the upload has succeeded.

- ``FAILURE`` -- The upload is, for some reason, unable to be
  processed.

- ``TRYAGAIN`` -- The server is unable to accept the upload at this
  time, but the client should try again at a later time.
  Potential causes of this are resource shortages on the server,
  administrative down-time, etc...

Optionally, there may be a ``X-Swalow-Reason`` header which includes a
human-readable string which provides more detailed information about
the ``X-Swalow-Status``.

If there is no ``X-Swalow-Status`` header, or it does not contain one of
the three strings above, it should be treated as a temporary failure.

Example::

    >>> f = urllib.urlopen('http://www.python.org:80/pypi')
    >>> s = f.headers['x-swalow-status']
    >>> s = s + ': ' + f.headers.get('x-swalow-reason', '<None>')
    >>> print s
    FAILURE: Required field "distribution" missing.


Sample Form
===========

The upload client must submit the page in the same form as
Netscape Navigator version 4.76 for Linux produces when presented
with the following form::

    <H1>Upload file</H1>
    <FORM NAME="fileupload" METHOD="POST" ACTION="pypi"
        ENCTYPE="multipart/form-data">
    <INPUT TYPE="file" NAME="distribution"><BR>
    <INPUT TYPE="text" NAME="distmd5sum"><BR>
    <INPUT TYPE="file" NAME="pkginfo"><BR>
    <INPUT TYPE="text" NAME="infomd5sum"><BR>
    <INPUT TYPE="text" NAME="platform"><BR>
    <INPUT TYPE="text" NAME="signature"><BR>
    <INPUT TYPE="hidden" NAME="protocol_version" VALUE="1"><BR>
    <INPUT TYPE="SUBMIT" VALUE="Upload">
    </FORM>


Platforms
=========

The following are valid os names::

    aix beos debian dos freebsd hpux mac macos mandrake netbsd
    openbsd qnx redhat solaris suse windows yellowdog

The above include a number of different types of distributions of
Linux.  Because of versioning issues these must be split out, and
it is expected that when it makes sense for one system to use
distributions made on other similar systems, the download client
will make the distinction.

Version is the official version string specified by the vendor for
the particular release.  For example, "2000" and "nt" (Windows),
"9.04" (HP-UX), "7.0" (RedHat, Mandrake).

The following are valid architectures::

    alpha hppa ix86 powerpc sparc ultrasparc


Status
======

I currently have a proof-of-concept client and server implemented.
I plan to have the Distutils patches ready for the 2.1 release.
Combined with Andrew's :pep:`241` for specifying distribution
meta-data, I hope to have a platform which will allow us to gather
real-world data for finalizing the catalog system for the 2.2
release.


Copyright
=========

This document has been placed in the public domain.


id: pep-0257.rst
content:
PEP: 257
Title: Docstring Conventions
Author: David Goodger <goodger@python.org>,
        Guido van Rossum <guido@python.org>
Discussions-To: doc-sig@python.org
Status: Active
Type: Informational
Created: 29-May-2001
Post-History: 13-Jun-2001


Abstract
========

This PEP documents the semantics and conventions associated with
Python docstrings.


Rationale
=========

The aim of this PEP is to standardize the high-level structure of
docstrings: what they should contain, and how to say it (without
touching on any markup syntax within docstrings).  The PEP contains
conventions, not laws or syntax.

    "A universal convention supplies all of maintainability, clarity,
    consistency, and a foundation for good programming habits too.
    What it doesn't do is insist that you follow it against your will.
    That's Python!"

    -- Tim Peters on comp.lang.python, 2001-06-16

If you violate these conventions, the worst you'll get is some dirty
looks.  But some software (such as the Docutils_ docstring processing
system :pep:`256`, :pep:`258`) will be aware of the conventions, so following them
will get you the best results.


Specification
=============

What is a Docstring?
--------------------

A docstring is a string literal that occurs as the first statement in
a module, function, class, or method definition.  Such a docstring
becomes the ``__doc__`` special attribute of that object.

All modules should normally have docstrings, and all functions and
classes exported by a module should also have docstrings.  Public
methods (including the ``__init__`` constructor) should also have
docstrings.  A package may be documented in the module docstring of
the ``__init__.py`` file in the package directory.

String literals occurring elsewhere in Python code may also act as
documentation.  They are not recognized by the Python bytecode
compiler and are not accessible as runtime object attributes (i.e. not
assigned to ``__doc__``), but two types of extra docstrings may be
extracted by software tools:

1. String literals occurring immediately after a simple assignment at
   the top level of a module, class, or ``__init__`` method are called
   "attribute docstrings".

2. String literals occurring immediately after another docstring are
   called "additional docstrings".

Please see :pep:`258`, "Docutils Design Specification", for a
detailed description of attribute and additional docstrings.

For consistency, always use ``"""triple double quotes"""`` around
docstrings.  Use ``r"""raw triple double quotes"""`` if you use any
backslashes in your docstrings.

There are two forms of docstrings: one-liners and multi-line
docstrings.


One-line Docstrings
--------------------

One-liners are for really obvious cases.  They should really fit on
one line.  For example::

    def kos_root():
        """Return the pathname of the KOS root directory."""
        global _kos_root
        if _kos_root: return _kos_root
        ...

Notes:

- Triple quotes are used even though the string fits on one line.
  This makes it easy to later expand it.

- The closing quotes are on the same line as the opening quotes.  This
  looks better for one-liners.

- There's no blank line either before or after the docstring.

- The docstring is a phrase ending in a period.  It prescribes the
  function or method's effect as a command ("Do this", "Return that"),
  not as a description; e.g. don't write "Returns the pathname ...".

- The one-line docstring should NOT be a "signature" reiterating the
  function/method parameters (which can be obtained by introspection).
  Don't do::

      def function(a, b):
          """function(a, b) -> list"""

  This type of docstring is only appropriate for C functions (such as
  built-ins), where introspection is not possible.  However, the
  nature of the *return value* cannot be determined by introspection,
  so it should be mentioned.  The preferred form for such a docstring
  would be something like::

      def function(a, b):
          """Do X and return a list."""

  (Of course "Do X" should be replaced by a useful description!)


Multi-line Docstrings
----------------------

Multi-line docstrings consist of a summary line just like a one-line
docstring, followed by a blank line, followed by a more elaborate
description.  The summary line may be used by automatic indexing
tools; it is important that it fits on one line and is separated from
the rest of the docstring by a blank line.  The summary line may be on
the same line as the opening quotes or on the next line.  The entire
docstring is indented the same as the quotes at its first line (see
example below).

Insert a blank line after all docstrings (one-line or multi-line) that
document a class -- generally speaking, the class's methods are
separated from each other by a single blank line, and the docstring
needs to be offset from the first method by a blank line.

The docstring of a script (a stand-alone program) should be usable as
its "usage" message, printed when the script is invoked with incorrect
or missing arguments (or perhaps with a "-h" option, for "help").
Such a docstring should document the script's function and command
line syntax, environment variables, and files.  Usage messages can be
fairly elaborate (several screens full) and should be sufficient for a
new user to use the command properly, as well as a complete quick
reference to all options and arguments for the sophisticated user.

The docstring for a module should generally list the classes,
exceptions and functions (and any other objects) that are exported by
the module, with a one-line summary of each.  (These summaries
generally give less detail than the summary line in the object's
docstring.)  The docstring for a package (i.e., the docstring of the
package's ``__init__.py`` module) should also list the modules and
subpackages exported by the package.

The docstring for a function or method should summarize its behavior
and document its arguments, return value(s), side effects, exceptions
raised, and restrictions on when it can be called (all if applicable).
Optional arguments should be indicated.  It should be documented
whether keyword arguments are part of the interface.

The docstring for a class should summarize its behavior and list the
public methods and instance variables.  If the class is intended to be
subclassed, and has an additional interface for subclasses, this
interface should be listed separately (in the docstring).  The class
constructor should be documented in the docstring for its ``__init__``
method.  Individual methods should be documented by their own
docstring.

If a class subclasses another class and its behavior is mostly
inherited from that class, its docstring should mention this and
summarize the differences.  Use the verb "override" to indicate that a
subclass method replaces a superclass method and does not call the
superclass method; use the verb "extend" to indicate that a subclass
method calls the superclass method (in addition to its own behavior).

*Do not* use the Emacs convention of mentioning the arguments of
functions or methods in upper case in running text.  Python is case
sensitive and the argument names can be used for keyword arguments, so
the docstring should document the correct argument names.  It is best
to list each argument on a separate line.  For example::

    def complex(real=0.0, imag=0.0):
        """Form a complex number.

        Keyword arguments:
        real -- the real part (default 0.0)
        imag -- the imaginary part (default 0.0)
        """
        if imag == 0.0 and real == 0.0:
            return complex_zero
        ...

Unless the entire docstring fits on a line, place the closing quotes
on a line by themselves.  This way, Emacs' ``fill-paragraph`` command
can be used on it.


Handling Docstring Indentation
------------------------------

Docstring processing tools will strip a uniform amount of indentation
from the second and further lines of the


id: pep-0531.rst
content:
PEP: 531
Title: Existence checking operators
Author: Alyssa Coghlan <ncoghlan@gmail.com>
Status: Withdrawn
Type: Standards Track
Created: 25-Oct-2016
Python-Version: 3.7
Post-History: 28-Oct-2016

Abstract
========

Inspired by :pep:`505` and the related discussions, this PEP proposes the addition
of two new control flow operators to Python:

* Existence-checking precondition ("exists-then"): ``expr1 ?then expr2``
* Existence-checking fallback ("exists-else"): ``expr1 ?else expr2``

as well as the following abbreviations for common existence checking
expressions and statements:

* Existence-checking attribute access:
  ``obj?.attr`` (for ``obj ?then obj.attr``)
* Existence-checking subscripting:
  ``obj?[expr]`` (for ``obj ?then obj[expr]``)
* Existence-checking assignment:
  ``value ?= expr`` (for ``value = value ?else expr``)

The common ``?`` symbol in these new operator definitions indicates that they
use a new "existence checking" protocol rather than the established
truth-checking protocol used by if statements, while loops, comprehensions,
generator expressions, conditional expressions, logical conjunction, and
logical disjunction.

This new protocol would be made available as ``operator.exists``, with the
following characteristics:

* types can define a new ``__exists__`` magic method (Python) or
  ``tp_exists`` slot (C) to override the default behaviour. This optional
  method has the same signature and possible return values as ``__bool__``.
* ``operator.exists(None)`` returns ``False``
* ``operator.exists(NotImplemented)`` returns ``False``
* ``operator.exists(Ellipsis)`` returns ``False``
* ``float``, ``complex`` and ``decimal.Decimal`` will override the existence
  check such that ``NaN`` values return ``False`` and other values (including
  zero values) return ``True``
* for any other type, ``operator.exists(obj)`` returns True by default. Most
  importantly, values that evaluate to False in a truth checking context
  (zeroes, empty containers) will still evaluate to True in an existence
  checking context

PEP Withdrawal
==============

When posting this PEP for discussion on python-ideas [4]_, I asked reviewers to
consider 3 high level design questions before moving on to considering the
specifics of this particular syntactic proposal:

1. Do we collectively agree that "existence checking" is a useful
general concept that exists in software development and is distinct
from the concept of "truth checking"?
2. Do we collectively agree that the Python ecosystem would benefit
from an existence checking protocol that permits generalisation of
algorithms (especially short circuiting ones) across different "data
missing" indicators, including those defined in the language
definition, the standard library, and custom user code?
3. Do we collectively agree that it would be easier to use such a
protocol effectively if existence-checking equivalents to the
truth-checking "and" and "or" control flow operators were available?

While the answers to the first question were generally positive, it quickly
became clear that the answer to the second question is "No".

Steven D'Aprano articulated the counter-argument well in [5]_, but the general
idea is that when checking for "missing data" sentinels, we're almost always
looking for a *specific* sentinel value, rather than *any* sentinel value.

``NotImplemented`` exists, for example, due to ``None`` being a potentially
legitimate result from overloaded arithmetic operators and exception
handling imposing too much runtime overhead to be useful for operand coercion.

Similarly, ``Ellipsis`` exists for multi-dimensional slicing support due to
``None`` already have another meaning in a slicing context (indicating the use
of the default start or stop indices, or the default step size).

In mathematics, the value of ``NaN`` is that *programmatically* it behaves
like a normal value of its type (e.g. exposing all the usual attributes and
methods), while arithmetically it behaves according to the mathematical rules
for handling ``NaN`` values.

With that core design concept invalidated, the proposal as a whole doesn't
make sense, and it is accordingly withdrawn.

However, the discussion of the proposal did prompt consideration of a potential
protocol based approach to make the existing ``and``, ``or`` and ``if-else``
operators more flexible [6]_ without introducing any new syntax, so I'll be
writing that up as another possible alternative to :pep:`505`.


Relationship with other PEPs
============================

While this PEP was inspired by and builds on Mark Haase's excellent work in
putting together :pep:`505`, it ultimately competes with that PEP due to
significant differences in the specifics of the proposed syntax and semantics
for the feature.

It also presents a different perspective on the rationale for the change by
focusing on the benefits to existing Python users as the typical demands of
application and service development activities are genuinely changing. It
isn't an accident that similar features are now appearing in multiple
programming languages, and while it's a good idea for us to learn from how other
language designers are handling the problem, precedents being set elsewhere
are more relevant to *how* we would go about tackling this problem than they
are to whether or not we think it's a problem we should address in the first
place.


Rationale
=========

Existence checking expressions
------------------------------

An increasingly common requirement in modern software development is the need
to work with "semi-structured data": data where the structure of the data is
known in advance, but pieces of it may be missing at runtime, and the software
manipulating that data is expected to degrade gracefully (e.g. by omitting
results that depend on the missing data) rather than failing outright.

Some particularly common cases where this issue arises are:

* handling optional application configuration settings and function parameters
* handling external service failures in distributed systems
* handling data sets that include some partial records

It is the latter two cases that are the primary motivation for this PEP - while
needing to deal with optional configuration settings and parameters is a design
requirement at least as old as Python itself, the rise of public cloud
infrastructure, the development of software systems as collaborative networks
of distributed services, and the availability of large public and private data
sets for analysis means that the ability to degrade operations gracefully in
the face of partial service failures or partial data availability is becoming
an essential feature of modern programming environments.

At the moment, writing such software in Python can be genuinely awkward, as
your code ends up littered with expressions like:

* ``value1 = expr1.field.of.interest if expr1 is not None else None``
* ``value2 = expr2["field"]["of"]["interest"] if expr2 is not None else None``
* ``value3 = expr3 if expr3 is not None else expr4 if expr4 is not None else expr5``

If these are only occasional, then expanding out to full statement forms may
help improve readability, but if you have 4 or 5 of them in a row (which is a
fairly common situation in data transformation pipelines), then replacing them
with 16 or 20 lines of conditional logic really doesn't help matters.

Expanding the three examples above that way hopefully helps illustrate that::

    if expr1 is not None:
        value1 = expr1.field.of.interest
    else:
        value1 = None
    if expr2 is not None:
        value2 = expr2["field"]["of"]["interest"]
    else:
        value2 = None
    if expr3 is not None:
        value3 = expr3
    else:
        if expr4 is not None:
            value3 = expr4
        else:
            value3 = expr5

The combined impact of the proposals in this PEP is to allow the above sample
expressions to instead be written as:

* ``value1 = expr1?.field.of.interest``
* ``value2 =


id: pep-0519.rst
content:
PEP: 519
Title: Adding a file system path protocol
Author: Brett Cannon <brett@python.org>,
        Koos Zevenhoven <k7hoven@gmail.com>
Status: Final
Type: Standards Track
Created: 11-May-2016
Python-Version: 3.6
Post-History: 11-May-2016,
              12-May-2016,
              13-May-2016
Resolution: https://mail.python.org/pipermail/python-dev/2016-May/144646.html


Abstract
========

This PEP proposes a protocol for classes which represent a file system
path to be able to provide a ``str`` or ``bytes`` representation.
Changes to Python's standard library are also proposed to utilize this
protocol where appropriate to facilitate the use of path objects where
historically only ``str`` and/or ``bytes`` file system paths are
accepted. The goal is to facilitate the migration of users towards
rich path objects while providing an easy way to work with code
expecting ``str`` or ``bytes``.


Rationale
=========

Historically in Python, file system paths have been represented as
strings or bytes. This choice of representation has stemmed from C's
own decision to represent file system paths as
``const char *`` [#libc-open]_. While that is a totally serviceable
format to use for file system paths, it's not necessarily optimal. At
issue is the fact that while all file system paths can be represented
as strings or bytes, not all strings or bytes represent a file system
path. This can lead to issues where any e.g. string duck-types to a
file system path whether it actually represents a path or not.

To help elevate the representation of file system paths from their
representation as strings and bytes to a richer object representation,
the pathlib module [#pathlib]_ was provisionally introduced in
Python 3.4 through :pep:`428`. While considered by some as an improvement
over strings and bytes for file system paths, it has suffered from a
lack of adoption. Typically the key issue listed for the low adoption
rate has been the lack of support in the standard library. This lack
of support required users of pathlib to manually convert path objects
to strings by calling ``str(path)`` which many found error-prone.

One issue in converting path objects to strings comes from
the fact that the only generic way to get a string representation of
the path was to pass the object to ``str()``. This can pose a
problem when done blindly as nearly all Python objects have some
string representation whether they are a path or not, e.g.
``str(None)`` will give a result that
``builtins.open()`` [#builtins-open]_ will happily use to create a new
file.

Exacerbating this whole situation is the
``DirEntry`` object [#os-direntry]_. While path objects have a
representation that can be extracted using ``str()``, ``DirEntry``
objects expose a ``path`` attribute instead. Having no common
interface between path objects, ``DirEntry``, and any other
third-party path library has become an issue. A solution that allows
any path-representing object to declare that it is a path and a way
to extract a low-level representation that all path objects could
support is desired.

This PEP then proposes to introduce a new protocol to be followed by
objects which represent file system paths. Providing a protocol allows
for explicit signaling of what objects represent file system paths as
well as a way to extract a lower-level representation that can be used
with older APIs which only support strings or bytes.

Discussions regarding path objects that led to this PEP can be found
in multiple threads on the python-ideas mailing list archive
[#python-ideas-archive]_ for the months of March and April 2016 and on
the python-dev mailing list archives [#python-dev-archive]_ during
April 2016.


Proposal
========

This proposal is split into two parts. One part is the proposal of a
protocol for objects to declare and provide support for exposing a
file system path representation. The other part deals with changes to
Python's standard library to support the new protocol. These changes
will also lead to the pathlib module dropping its provisional status.

Protocol
--------

The following abstract base class defines the protocol for an object
to be considered a path object::

    import abc
    import typing as t


    class PathLike(abc.ABC):

        """Abstract base class for implementing the file system path protocol."""

        @abc.abstractmethod
        def __fspath__(self) -> t.Union[str, bytes]:
            """Return the file system path representation of the object."""
            raise NotImplementedError


Objects representing file system paths will implement the
``__fspath__()`` method which will return the ``str`` or ``bytes``
representation of the path. The ``str`` representation is the
preferred low-level path representation as it is human-readable and
what people historically represent paths as.


Standard library changes
------------------------

It is expected that most APIs in Python's standard library that
currently accept a file system path will be updated appropriately to
accept path objects (whether that requires code or simply an update
to documentation will vary). The modules mentioned below, though,
deserve specific details as they have either fundamental changes that
empower the ability to use path objects, or entail additions/removal
of APIs.


builtins
''''''''

``open()`` [#builtins-open]_ will be updated to accept path objects as
well as continue to accept ``str`` and ``bytes``.


os
'''

The ``fspath()`` function will be added with the following semantics::

    import typing as t


    def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]:
        """Return the string representation of the path.

        If str or bytes is passed in, it is returned unchanged. If __fspath__()
        returns something other than str or bytes then TypeError is raised. If
        this function is given something that is not str, bytes, or os.PathLike
        then TypeError is raised.
        """
        if isinstance(path, (str, bytes)):
            return path

        # Work from the object's type to match method resolution of other magic
        # methods.
        path_type = type(path)
        try:
            path = path_type.__fspath__(path)
        except AttributeError:
            if hasattr(path_type, '__fspath__'):
                raise
        else:
            if isinstance(path, (str, bytes)):
                return path
            else:
                raise TypeError("expected __fspath__() to return str or bytes, "
                                "not " + type(path).__name__)

        raise TypeError("expected str, bytes or os.PathLike object, not "
                        + path_type.__name__)

The ``os.fsencode()`` [#os-fsencode]_ and
``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept
path objects. As both functions coerce their arguments to
``bytes`` and ``str``, respectively, they will be updated to call
``__fspath__()`` if present to convert the path object to a ``str`` or
``bytes`` representation, and then perform their appropriate
coercion operations as if the return value from ``__fspath__()`` had
been the original argument to the coercion function in question.

The addition of ``os.fspath()``, the updates to
``os.fsencode()``/``os.fsdecode()``, and the current semantics of
``pathlib.PurePath`` provide the semantics necessary to
get the path representation one prefers. For a path object,
``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or
``bytes`` representation without any coercion, then ``os.fspath()``
can be used. If a ``str`` is desired and the encoding of ``bytes``
should be assumed to be the default file system encoding, then
``os.fsdecode()`` should be used. If a ``bytes`` representation is
desired and any strings should be encoded using the default file
system encoding, then ``os.fsencode()`` is used. This PEP recommends
using path objects when possible and falling back to string paths as
necessary and using ``bytes`


id: pep-0230.rst
content:
PEP: 230
Title: Warning Framework
Author: Guido van Rossum <guido@python.org>
Status: Final
Type: Standards Track
Created: 28-Nov-2000
Python-Version: 2.1
Post-History: 05-Nov-2000


Abstract
========

This PEP proposes a C and Python level API, as well as command
line flags, to issue warning messages and control what happens to
them.  This is mostly based on GvR's proposal posted to python-dev
on 05-Nov-2000, with some ideas (such as using classes to
categorize warnings) merged in from Paul Prescod's
counter-proposal posted on the same date.  Also, an attempt to
implement the proposal caused several small tweaks.


Motivation
==========

With Python 3000 looming, it is necessary to start issuing
warnings about the use of obsolete or deprecated features, in
addition to errors.  There are also lots of other reasons to be
able to issue warnings, both from C and from Python code, both at
compile time and at run time.

Warnings aren't fatal, and thus it's possible that a program
triggers the same warning many times during a single execution.
It would be annoying if a program emitted an endless stream of
identical warnings.  Therefore, a mechanism is needed that
suppresses multiple identical warnings.

It is also desirable to have user control over which warnings are
printed.  While in general it is useful to see all warnings all
the time, there may be times where it is impractical to fix the
code right away in a production program.  In this case, there
should be a way to suppress warnings.

It is also useful to be able to suppress specific warnings during
program development, e.g. when a warning is generated by a piece
of 3rd party code that cannot be fixed right away, or when there
is no way to fix the code (possibly a warning message is generated
for a perfectly fine piece of code).  It would be unwise to offer
to suppress all warnings in such cases: the developer would miss
warnings about the rest of the code.

On the other hand, there are also situations conceivable where
some or all warnings are better treated as errors.  For example,
it may be a local coding standard that a particular deprecated
feature should not be used.  In order to enforce this, it is
useful to be able to turn the warning about this particular
feature into an error, raising an exception (without necessarily
turning all warnings into errors).

Therefore, I propose to introduce a flexible "warning filter"
which can filter out warnings or change them into exceptions,
based on:

- Where in the code they are generated (per package, module, or
  function)

- The warning category (warning categories are discussed below)

- A specific warning message

The warning filter must be controllable both from the command line
and from Python code.


APIs For Issuing Warnings
=========================

- To issue a warning from Python::

      import warnings
      warnings.warn(message[, category[, stacklevel]])

  The category argument, if given, must be a warning category
  class (see below); it defaults to warnings.UserWarning.  This
  may raise an exception if the particular warning issued is
  changed into an error by the warnings filter.  The stacklevel
  can be used by wrapper functions written in Python, like this::

      def deprecation(message):
          warn(message, DeprecationWarning, level=2)

  This makes the warning refer to the deprecation()'s caller,
  rather than to the source of deprecation() itself (since the
  latter would defeat the purpose of the warning message).

- To issue a warning from C::

    int PyErr_Warn(PyObject *category, char *message);

  Return 0 normally, 1 if an exception is raised (either because
  the warning was transformed into an exception, or because of a
  malfunction in the implementation, such as running out of
  memory).  The category argument must be a warning category class
  (see below) or ``NULL``, in which case it defaults to
  ``PyExc_RuntimeWarning``.  When ``PyErr_Warn()`` function returns 1, the
  caller should do normal exception handling.

  The current C implementation of ``PyErr_Warn()`` imports the
  warnings module (implemented in Python) and calls its ``warn()``
  function.  This minimizes the amount of C code that needs to be
  added to implement the warning feature.

  [XXX Open Issue: what about issuing warnings during lexing or
  parsing, which don't have the exception machinery available?]


Warnings Categories
===================

There are a number of built-in exceptions that represent warning
categories.  This categorization is useful to be able to filter
out groups of warnings.  The following warnings category classes
are currently defined:

- ``Warning`` -- this is the base class of all warning category
  classes and it itself a subclass of Exception

- ``UserWarning`` -- the default category for ``warnings.warn()``

- ``DeprecationWarning`` -- base category for warnings about deprecated
  features

- ``SyntaxWarning`` -- base category for warnings about dubious
  syntactic features

- ``RuntimeWarning`` -- base category for warnings about dubious
  runtime features

[XXX: Other warning categories may be proposed during the review
period for this PEP.]

These standard warning categories are available from C as
``PyExc_Warning``, ``PyExc_UserWarning``, etc.  From Python, they are
available in the ``__builtin__`` module, so no import is necessary.

User code can define additional warning categories by subclassing
one of the standard warning categories.  A warning category must
always be a subclass of the Warning class.


The Warnings Filter
===================

The warnings filter control whether warnings are ignored,
displayed, or turned into errors (raising an exception).

There are three sides to the warnings filter:

- The data structures used to efficiently determine the
  disposition of a particular ``warnings.warn()`` or ``PyErr_Warn()``
  call.

- The API to control the filter from Python source code.

- The command line switches to control the filter.

The warnings filter works in several stages.  It is optimized for
the (expected to be common) case where the same warning is issued
from the same place in the code over and over.

First, the warning filter collects the module and line number
where the warning is issued; this information is readily available
through ``sys._getframe()``.

Conceptually, the warnings filter maintains an ordered list of
filter specifications; any specific warning is matched against
each filter specification in the list in turn until a match is
found; the match determines the disposition of the match.  Each
entry is a tuple as follows::

  (category, message, module, lineno, action)

- category is a class (a subclass of ``warnings.Warning``) of which
  the warning category must be a subclass in order to match

- message is a compiled regular expression that the warning
  message must match (the match is case-insensitive)

- module is a compiled regular expression that the module name
  must match

- lineno is an integer that the line number where the warning
  occurred must match, or 0 to match all line numbers

- action is one of the following strings:

  - "error" -- turn matching warnings into exceptions

  - "ignore" -- never print matching warnings

  - "always" -- always print matching warnings

  - "default" -- print the first occurrence of matching warnings
    for each location where the warning is issued

  - "module" -- print the first occurrence of matching warnings
    for each module where the warning is issued

  - "once" -- print only the first occurrence of matching
    warnings

Since the ``Warning`` class is derived from the built-in ``Exception``
class, to turn a warning into an error we simply raise
``category(message)``.


Warnings Output And Formatting Hooks
====================================

When the warnings filter decides to issue a warning (but not when
it decides to raise an exception), it passes the information about
the function ``warnings.showwarnin


id: pep-0556.rst
content:
PEP: 556
Title: Threaded garbage collection
Author: Antoine Pitrou <solipsis@pitrou.net>
Status: Deferred
Type: Standards Track
Created: 08-Sep-2017
Python-Version: 3.7
Post-History: 08-Sep-2017


Deferral Notice
===============

This PEP is currently not being actively worked on.  It may be revived
in the future.  The main missing steps are:

* polish the implementation, adapting the test suite where necessary;

* ensure setting threaded garbage collection does not disrupt existing
  code in unexpected ways (expected impact includes lengthening the
  lifetime of objects in reference cycles).


Abstract
========

This PEP proposes a new optional mode of operation for CPython's cyclic
garbage collector (GC) where implicit (i.e. opportunistic) collections
happen in a dedicated thread rather than synchronously.


Terminology
===========

An "implicit" GC run (or "implicit" collection) is one that is triggered
opportunistically based on a certain heuristic computed over allocation
statistics, whenever a new allocation is requested.  Details of the
heuristic are not relevant to this PEP, as it does not propose to change it.

An "explicit" GC run (or "explicit" collection) is one that is requested
programmatically by an API call such as ``gc.collect``.

"Threaded" refers to the fact that GC runs happen in a dedicated thread
separate from sequential execution of application code.  It does not mean
"concurrent" (the Global Interpreter Lock, or GIL, still serializes
execution among Python threads *including* the dedicated GC thread)
nor "parallel" (the GC is not able to distribute its work onto several
threads at once to lower wall-clock latencies of GC runs).


Rationale
=========

The mode of operation for the GC has always been to perform implicit
collections synchronously.  That is, whenever the aforementioned heuristic
is activated, execution of application code in the current thread is
suspended and the GC is launched in order to reclaim dead reference
cycles.

There is a catch, though.  Over the course of reclaiming dead reference
cycles (and any ancillary objects hanging at those cycles), the GC can
execute arbitrary finalization code in the form of ``__del__`` methods
and ``weakref`` callbacks.  Over the years, Python has been used for more
and more sophisticated purposes, and it is increasingly common for
finalization code to perform complex tasks, for example in distributed
systems where loss of an object may require notifying other (logical
or physical) nodes.

Interrupting application code at arbitrary points to execute finalization
code that may rely on a consistent internal state and/or on acquiring
synchronization primitives gives rise to reentrancy issues that even the
most seasoned experts have trouble fixing properly [#queue-reentrancy-bug]_.

This PEP bases itself on the observation that, despite the apparent
similarities, same-thread reentrancy is a fundamentally harder
problem than multi-thread synchronization.  Instead of letting each
developer or library author struggle with extremely hard reentrancy
issues, one by one, this PEP proposes to allow the GC to run in a
separate thread where well-known multi-thread synchronization practices
are sufficient.


Proposal
========

Under this PEP, the GC has two modes of operation:

* "serial", which is the default and legacy mode, where an implicit GC
  run is performed immediately in the thread that detects such an implicit
  run is desired (based on the aforementioned allocation heuristic).

* "threaded", which can be explicitly enabled at runtime on a per-process
  basis, where implicit GC runs are *scheduled* whenever the allocation
  heuristic is triggered, but run in a dedicated background thread.

Hard reentrancy problems which plague sophisticated uses of finalization
callbacks in the "serial" mode become relatively easy multi-thread
synchronization problems in the "threaded" mode of operation.

The GC also traditionally allows for explicit GC runs, using the Python
API ``gc.collect`` and the C API ``PyGC_Collect``.  The visible semantics
of these two APIs are left unchanged: they perform a GC run immediately
when called, and only return when the GC run is finished.


New public APIs
---------------

Two new Python APIs are added to the ``gc`` module:

* ``gc.set_mode(mode)`` sets the current mode of operation (either "serial"
  or "threaded").  If setting to "serial" and the current mode is
  "threaded", then the function also waits for the GC thread to end.

* ``gc.get_mode()`` returns the current mode of operation.

It is allowed to switch back and forth between modes of operation.


Intended use
------------

Given the per-process nature of the switch and its repercussions on
semantics of all finalization callbacks, it is recommended that it is
set at the beginning of an application's code (and/or in initializers
for child processes e.g. when using ``multiprocessing``).  Library functions
should probably not mess with this setting, just as they shouldn't call
``gc.enable`` or ``gc.disable``, but there's nothing to prevent them from
doing so.


Non-goals
---------

This PEP does not address reentrancy issues with other kinds of
asynchronous code execution (for example signal handlers registered
with the ``signal`` module).  The author believes that the overwhelming
majority of painful reentrancy issues occur with finalizers.  Most of the
time, signal handlers are able to set a single flag and/or wake up a
file descriptor for the main program to notice.  As for those signal
handlers which raise an exception, they *have* to execute in-thread.

This PEP also does not change the execution of finalization callbacks
when they are called as part of regular reference counting, i.e. when
releasing a visible reference drops an object's reference count to zero.
Since such execution happens at deterministic points in code, it is usually
not a problem.


Internal details
================

TODO: Update this section to conform to the current implementation.

``gc`` module
-------------

An internal flag ``gc_is_threaded`` is added, telling whether GC is serial
or threaded.

An internal structure ``gc_mutex`` is added to avoid two GC runs at once:

.. code-block::

   static struct {
       PyThread_type_lock lock;  /* taken when collecting */
       PyThreadState *owner;  /* whichever thread is currently collecting
                                 (NULL if no collection is taking place) */
   } gc_mutex;

An internal structure ``gc_thread`` is added to handle synchronization with
the GC thread:

.. code-block::

   static struct {
      PyThread_type_lock wakeup; /* acts as an event
                                    to wake up the GC thread */
      int collection_requested; /* non-zero if collection requested */
      PyThread_type_lock done; /* acts as an event signaling
                                  the GC thread has exited */
   } gc_thread;


``threading`` module
--------------------

Two private functions are added to the ``threading`` module:

* ``threading._ensure_dummy_thread(name)`` creates and registers a ``Thread``
  instance for the current thread with the given *name*, and returns it.

* ``threading._remove_dummy_thread(thread)`` removes the given *thread*
  (as returned by ``_ensure_dummy_thread``) from the threading module's
  internal state.

The purpose of these two functions is to improve debugging and introspection
by letting ``threading.current_thread()`` return a more meaningfully-named
object when called inside a finalization callback in the GC thread.


Pseudo-code
-----------

Here is a proposed pseudo-code for the main primitives, public and internal,
required for implementing this PEP.  All of them will be implemented in C
and live inside the ``gc`` module, unless otherwise noted:

.. code-block::

   def collect_with_callback(generation):
       """
       Collect up to the given *generation*.
       """
       # Same code as currently (see collect_with_callback(


id: pep-0542.rst
content:
PEP: 542
Title: Dot Notation Assignment In Function Header
Author: Markus Meskanen <markusmeskanen@gmail.com>
Status: Rejected
Type: Standards Track
Created: 10-Feb-2017
Resolution: https://mail.python.org/pipermail/python-dev/2019-March/156695.html


Abstract
========

Function definitions only allow simple function names to be used,
even though functions are assignable first class objects.

This PEP proposes adding support for assigning a function to
a class or instance attribute directly in the function
definition's header by using the dot notation to separate
the object from the function's name.

Although a similar feature, this PEP does not address general
assignment to anything that supports assignment, such as dict keys
and list indexes.

Rationale
=========

Currently if a function needs to be assigned to a class or instance
attribute, it requires an additional assignment statement to be made::

  class MyClass:
      ...

  my_instance = MyClass()

  def my_function(self):
      ...

  # Assign to class attribute
  MyClass.my_function = my_function

  # Or assign to instance attribute
  my_instance.my_function = my_function

While this isn't usually an inconvenience, using dot notation to
assign directly in the function's header would greatly simplify this::

  class MyClass:
      ...

  my_instance = MyClass()

  # Assign to class attribute
  def MyClass.my_function(self):
      ...

  # Or assign to instance attribute
  def my_instance.my_function(self):
      ...

There are multiple reasons to use this functionality over
a standard class method, for example when the class is referenced
inside the function's header (such as with decorators and typing).
This is also useful when an instance requires a callback attribute::

  class Menu:
      def __init__(self, items=None, select_callback=None):
          self.items = items if items is not None else []
          self.select_callback = select_callback

  my_menu = Menu([item1, item2])

  def my_menu.select_callback(item_index, menu):
      print(menu.items[item_index])

As opposed to::

  my_menu = Menu([item1, item2])

  def select_callback(item_index, menu):
      print(menu.items[item_index])
  my_menu.select_callback = select_callback

Or defining them in an "unnatural" order::

  def select_callback(item_index, menu):
      print(menu.items[item_index])

  my_menu = Menu([item1, item2], select_callback)

It reads better than the "unnatural" way, since you already know at
the time of the function definition what it's goig to be used for.
It also saves one line of code while removing visual complexity.

The feature would also avoid leaving the function's name into
the global namespace::

  eggs = 'something'

  def Spam.eggs(self):
      ...

  def Cheese.eggs(self):
      ...

  assert eggs == 'something'

Ideally this would be just syntastic sugar::

  def x.y():
      ...

  # Equals to

  def y():
      ...
  x.y = y

Similar to how decorators are syntastic sugar::

  @decorate
  def f():
      ...

  # Equals to

  def f():
      ...
  f = decorate(f)

Implementation
==============

The ``__name__`` would follow the principles of a normal function::

  class MyClass:
      def my_function1(self):
          ...

  def MyClass.my_function2(self):
      ...

  assert my_function1.__name__ == 'my_function1'
  assert my_function2.__name__ == 'my_function2'

The grammar would use ``dotted_name`` to support chaining of attributes::

  def Person.name.fset(self, value):
      self._name = value

Backwards Compatibility
=======================

This PEP is fully backwards compatible.

Copyright
=========

This document has been placed in the public domain.


id: pep-0224.rst
content:
PEP: 224
Title: Attribute Docstrings
Author: Marc-AndrÃ© Lemburg <mal@lemburg.com>
Status: Rejected
Type: Standards Track
Created: 23-Aug-2000
Python-Version: 2.1
Post-History:

.. rejected::

   See :ref:`224-rejection` for more information.

Introduction
============

This PEP describes the "attribute docstring" proposal for Python
2.0.  This PEP tracks the status and ownership of this feature.
It contains a description of the feature and outlines changes
necessary to support the feature.  The CVS revision history of
this file contains the definitive historical record.


Rationale
=========

This PEP proposes a small addition to the way Python currently
handles docstrings embedded in Python code.

Python currently only handles the case of docstrings which appear
directly after a class definition, a function definition or as
first string literal in a module.  The string literals are added
to the objects in question under the ``__doc__`` attribute and are
from then on available for introspection tools which can extract
the contained information for help, debugging and documentation
purposes.

Docstrings appearing in locations other than the ones mentioned
are simply ignored and don't result in any code generation.

Here is an example::

    class C:
        "class C doc-string"

        a = 1
        "attribute C.a doc-string (1)"

        b = 2
        "attribute C.b doc-string (2)"

The docstrings (1) and (2) are currently being ignored by the
Python byte code compiler, but could obviously be put to good use
for documenting the named assignments that precede them.

This PEP proposes to also make use of these cases by proposing
semantics for adding their content to the objects in which they
appear under new generated attribute names.

The original idea behind this approach which also inspired the
above example was to enable inline documentation of class
attributes, which can currently only be documented in the class's
docstring or using comments which are not available for
introspection.


Implementation
==============

Docstrings are handled by the byte code compiler as expressions.
The current implementation special cases the few locations
mentioned above to make use of these expressions, but otherwise
ignores the strings completely.

To enable use of these docstrings for documenting named
assignments (which is the natural way of defining e.g. class
attributes), the compiler will have to keep track of the last
assigned name and then use this name to assign the content of the
docstring to an attribute of the containing object by means of
storing it in as a constant which is then added to the object's
namespace during object construction time.

In order to preserve features like inheritance and hiding of
Python's special attributes (ones with leading and trailing double
underscores), a special name mangling has to be applied which
uniquely identifies the docstring as belonging to the name
assignment and allows finding the docstring later on by inspecting
the namespace.

The following name mangling scheme achieves all of the above::

    __doc_<attributename>__

To keep track of the last assigned name, the byte code compiler
stores this name in a variable of the compiling structure.  This
variable defaults to NULL.  When it sees a docstring, it then
checks the variable and uses the name as basis for the above name
mangling to produce an implicit assignment of the docstring to the
mangled name.  It then resets the variable to NULL to avoid
duplicate assignments.

If the variable does not point to a name (i.e. is NULL), no
assignments are made.  These will continue to be ignored like
before.  All classical docstrings fall under this case, so no
duplicate assignments are done.

In the above example this would result in the following new class
attributes to be created::

    C.__doc_a__ == "attribute C.a doc-string (1)"
    C.__doc_b__ == "attribute C.b doc-string (2)"

A patch to the current CVS version of Python 2.0 which implements
the above is available on SourceForge at [1]_.


Caveats of the Implementation
=============================

Since the implementation does not reset the compiling structure
variable when processing a non-expression, e.g. a function
definition, the last assigned name remains active until either the
next assignment or the next occurrence of a docstring.

This can lead to cases where the docstring and assignment may be
separated by other expressions::

   class C:
       "C doc string"

       b = 2

       def x(self):
           "C.x doc string"
            y = 3
            return 1

       "b's doc string"

Since the definition of method "x" currently does not reset the
used assignment name variable, it is still valid when the compiler
reaches the docstring "b's doc string" and thus assigns the string
to ``__doc_b__``.

A possible solution to this problem would be resetting the name
variable for all non-expression nodes in the compiler.


Possible Problems
=================

Even though highly unlikely, attribute docstrings could get
accidentally concatenated to the attribute's value::

   class C:
       x = "text" \
           "x's docstring"

The trailing slash would cause the Python compiler to concatenate
the attribute value and the docstring.

A modern syntax highlighting editor would easily make this
accident visible, though, and by simply inserting empty lines
between the attribute definition and the docstring you can avoid
the possible concatenation completely, so the problem is
negligible.

Another possible problem is that of using triple quoted strings as
a way to uncomment parts of your code.

If there happens to be an assignment just before the start of the
comment string, then the compiler will treat the comment as
docstring attribute and apply the above logic to it.

Besides generating a docstring for an otherwise undocumented
attribute there is no breakage.

.. _224-rejection:

Comments from our BDFL
======================

Early comments on the PEP from Guido:

    I "kinda" like the idea of having attribute docstrings (meaning
    it's not of great importance to me) but there are two things I
    don't like in your current proposal:

    1. The syntax you propose is too ambiguous: as you say,
       stand-alone string literal are used for other purposes and could
       suddenly become attribute docstrings.

    2. I don't like the access method either (``__doc_<attrname>__``).

The author's reply:

    ::

        > 1. The syntax you propose is too ambiguous: as you say, stand-alone
        >    string literal are used for other purposes and could suddenly
        >    become attribute docstrings.


    This can be fixed by introducing some extra checks in the
    compiler to reset the "doc attribute" flag in the compiler
    struct.

    ::

        > 2. I don't like the access method either (``__doc_<attrname>__``).

    Any other name will do. It will only have to match these
    criteria:

    * must start with two underscores (to match ``__doc__``)
    * must be extractable using some form of inspection (e.g. by using
      a naming convention which includes some fixed name part)
    * must be compatible with class inheritance (i.e. should be
      stored as attribute)

Later on in March, Guido pronounced on this PEP in March 2001 (on
python-dev). Here are his reasons for rejection mentioned in
private mail to the author of this PEP:

    ...

    It might be useful, but I really hate the proposed syntax.

    ::

        a = 1
        "foo bar"
        b = 1

    I really have no way to know whether "foo bar" is a docstring
    for a or for b.

    ...

    You can use this convention::

        a = 1
        __doc_a__ = "doc string for a"

    This makes it available at runtime.

    ::

       > Are you completely opposed to adding attribute documentation
       > to Python or is it just the way the implementation works ? I
       > find the syntax proposed in the PEP very intuitive and many
 


id: pep-0218.rst
content:
PEP: 218
Title: Adding a Built-In Set Object Type
Author: Greg Wilson <gvwilson@ddj.com>, Raymond Hettinger <python@rcn.com>
Status: Final
Type: Standards Track
Created: 31-Jul-2000
Python-Version: 2.2
Post-History:


Introduction
============

This PEP proposes adding a Set module to the standard Python
library, and to then make sets a built-in Python type if that
module is widely used.  After explaining why sets are desirable,
and why the common idiom of using dictionaries in their place is
inadequate, we describe how we intend built-in sets to work, and
then how the preliminary Set module will behave.  The last
section discusses the mutability (or otherwise) of sets and set
elements, and the solution which the Set module will implement.


Rationale
=========

Sets are a fundamental mathematical structure, and are very
commonly used in algorithm specifications.  They are much less
frequently used in implementations, even when they are the "right"
structure.  Programmers frequently use lists instead, even when
the ordering information in lists is irrelevant, and by-value
lookups are frequent.  (Most medium-sized C programs contain a
depressing number of start-to-end searches through malloc'd
vectors to determine whether particular items are present or
not...)

Programmers are often told that they can implement sets as
dictionaries with "don't care" values.  Items can be added to
these "sets" by assigning the "don't care" value to them;
membership can be tested using ``dict.has_key``; and items can be
deleted using ``del``.  However, the other main operations on sets
(union, intersection, and difference) are not directly supported
by this representation, since their meaning is ambiguous for
dictionaries containing key/value pairs.


Proposal
========

The long-term goal of this PEP is to add a built-in set type to
Python.  This type will be an unordered collection of unique
values, just as a dictionary is an unordered collection of
key/value pairs.

Iteration and comprehension will be implemented in the obvious
ways, so that::

    for x in S:

will step through the elements of S in arbitrary order, while::

    set(x**2 for x in S)

will produce a set containing the squares of all elements in S,
Membership will be tested using ``in`` and ``not in``, and basic set
operations will be implemented by a mixture of overloaded
operators:

=============  =============================
``|``          union
``&``          intersection
``^``          symmetric difference
``-``          asymmetric difference
``== !=``      equality and inequality tests
``< <= >= >``  subset and superset tests
=============  =============================

and methods:

==================  =============================================
``S.add(x)``        Add "x" to the set.

``S.update(s)``     Add all elements of sequence "s" to the set.

``S.remove(x)``     Remove "x" from the set.  If "x" is not
                    present, this method raises a ``LookupError``
                    exception.

``S.discard(x)``    Remove "x" from the set if it is present, or
                    do nothing if it is not.

``S.pop()``         Remove and return an arbitrary element,
                    raising a ``LookupError`` if the element is
                    not present.

``S.clear()``       Remove all elements from this set.

``S.copy()``        Make a new set.

``s.issuperset()``  Check for a superset relationship.

``s.issubset()``    Check for a subset relationship.
==================  =============================================

and two new built-in conversion functions:

================  ===============================================
``set(x)``        Create a set containing the elements of the
                  collection "x".

``frozenset(x)``  Create an immutable set containing the elements
                  of the collection "x".
================  ===============================================

Notes:

1. We propose using the bitwise operators "``|&``" for intersection
   and union.  While "``+``" for union would be intuitive, "``*``" for
   intersection is not (very few of the people asked guessed what
   it did correctly).

2. We considered using "``+``" to add elements to a set, rather than
   "add".  However, Guido van Rossum pointed out that "``+``" is
   symmetric for other built-in types (although "``*``" is not).  Use
   of "add" will also avoid confusion between that operation and
   set union.


Set Notation
============

The PEP originally proposed ``{1,2,3}`` as the set notation and ``{-}`` for
the empty set.  Experience with Python 2.3's ``sets.py`` showed that
the notation was not necessary.  Also, there was some risk of making
dictionaries less instantly recognizable.

It was also contemplated that the braced notation would support set
comprehensions; however, Python 2.4 provided generator expressions
which fully met that need and did so it a more general way.
(See :pep:`289` for details on generator expressions).

So, Guido ruled that there would not be a set syntax; however, the
issue could be revisited for Python 3000 (see :pep:`3000`).


History
=======

To gain experience with sets, a pure python module was introduced
in Python 2.3.  Based on that implementation, the set and frozenset
types were introduced in Python 2.4.  The improvements are:

* Better hash algorithm for frozensets
* More compact pickle format (storing only an element list
  instead of a dictionary of key:value pairs where the value
  is always ``True``).
* Use a ``__reduce__`` function so that deep copying is automatic.
* The BaseSet concept was eliminated.
* The ``union_update()`` method became just ``update()``.
* Auto-conversion between mutable and immutable sets was dropped.
* The ``_repr`` method was dropped (the need is met by the new
  ``sorted()`` built-in function).

Tim Peters believes that the class's constructor should take a
single sequence as an argument, and populate the set with that
sequence's elements.  His argument is that in most cases,
programmers will be creating sets from pre-existing sequences, so
that this case should be the common one.  However, this would
require users to remember an extra set of parentheses when
initializing a set with known values::

    >>> Set((1, 2, 3, 4))       # case 1

On the other hand, feedback from a small number of novice Python
users (all of whom were very experienced with other languages)
indicates that people will find a "parenthesis-free" syntax more
natural::

    >>> Set(1, 2, 3, 4)         # case 2

Ultimately, we adopted the first strategy in which the initializer
takes a single iterable argument.


Mutability
==========

The most difficult question to resolve in this proposal was
whether sets ought to be able to contain mutable elements.  A
dictionary's keys must be immutable in order to support fast,
reliable lookup.  While it would be easy to require set elements
to be immutable, this would preclude sets of sets (which are
widely used in graph algorithms and other applications).

Earlier drafts of :pep:`218` had only a single set type, but the
``sets.py`` implementation in Python 2.3 has two, Set and
ImmutableSet.  For Python 2.4, the new built-in types were named
``set`` and ``frozenset`` which are slightly less cumbersome.

There are two classes implemented in the "sets" module.  Instances
of the Set class can be modified by the addition or removal of
elements, and the ImmutableSet class is "frozen", with an
unchangeable collection of elements.  Therefore, an ImmutableSet
may be used as a dictionary key or as a set element, but cannot be
updated.  Both types of set require that their elements are
immutable, hashable objects.  Parallel comments apply to the "set"
and "frozenset" built-in types.


Copyright
=========

This document has been placed in the Public Domain.


id: pep-0595.rst
content:
PEP: 595
Title: Improving bugs.python.org
Author: Ezio Melotti <ezio.melotti@gmail.com>, Berker Peksag <berker.peksag@gmail.com>
BDFL-Delegate: Barry Warsaw <barry@python.org>
Status: Withdrawn
Type: Informational
Created: 12-May-2019


Abstract
========

This PEP proposes a list of improvements to make bugs.python.org
more usable for contributors and core developers.  This PEP also
discusses why remaining on Roundup should be preferred over
switching to GitHub Issues, as proposed by :pep:`581`.


Resolution
==========

2020-06-25: With the acceptance of :pep:`581`, the move to GitHub for
issues is proceeding, this PEP is being marked as a withdrawn
informational PEP.


Motivation
==========

On May 14th, 2019 :pep:`581` has been `accepted
<https://mail.python.org/pipermail/python-dev/2019-May/157399.html>`_
without much public discussion and `without a clear consensus
<https://mail.python.org/pipermail/python-committers/2019-May/006755.html>`_.
The PEP contains factual errors and doesn't address some of the
issues that the migration to GitHub Issues might present.

Given the scope of the migration, the amount of work required,
and how it will negatively affect the workflow during the
transition phase, this decision should be re-evaluated.

.. TODO: add a section with background and terminology?
   (e.g. roundup, bpo, instances, github issues, pep581/588)

Roundup advantages over GitHub Issues
=====================================

This section discusses reasons why Roundup should be preferred
over GitHub Issues and Roundup features that are not available
on GitHub Issues.

* **Roundup is the status quo.**  Roundup has been an integral
  part of the CPython workflow for years.  It is a stable product
  that has been tested and customized to adapt to our needs as the
  workflow evolved.

  It is possible to gradually improve it and avoid the disruption
  that a switch to a different system would inevitably bring to
  the workflow.

* **Open-source and Python powered.**  Roundup is an open-source
  project and is written in Python.  By using it and supporting
  it, we also support the Python ecosystem.  Several features
  developed for bpo have also been ported to upstream Roundup
  over the years.

* **Fully customizable.**  Roundup can be (and has been) fully
  customized to fit our needs.

* **Finer-grained access control.**  Roundup allows the creation
  of different roles with different permissions (e.g. create,
  view, edit, etc.) for each individual property, and users can
  have multiple roles.

* **Flexible UI.**  While Roundup UI might look dated, it is
  convenient and flexible.

  For example, on the issue page, each field (e.g. title, type,
  versions, status, linked files and PRs, etc.) have appropriate
  UI elements (input boxes, dropdowns, tables, etc.) that are
  easy to set and also provide a convenient way to get info about
  the issue at a glance.  The number of fields, their values, and
  the UI element they use is also fully customizable.
  GitHub only provides labels.

  The issue list page presents the issues in a compact and easy
  to read table with separate columns for different fields.  For
  comparison, Roundup lists 50 issues in a screen, whereas GitHub
  takes two screens to shows 25 issues.

* **Advanced search.**  Roundup provides an accurate way to search
  and filter by using any combination of issue fields.
  It is also possible to customize the number of results and the
  fields displayed in the table, and the sorting and grouping
  (up to two levels).

  bpo also provides predefined summaries (e.g. "Created by you",
  "Assigned to you", etc.) and allows the creation of custom
  search queries that can be conveniently accessed from the sidebar.

* **Nosy list autocomplete.**  The nosy list has an autocomplete
  feature that suggests maintainers and experts.  The suggestions
  are automatically updated when the `experts index
  <https://devguide.python.org/experts/>`_ changes.

* **Dependencies and Superseders.** Roundup allows to specify
  dependencies that must be addressed before the current issues
  can be closed and a superseder issue to easily mark duplicates
  (for example, `bpo-12078 <https://bugs.python.org/issue12078>`_).
  The list of dependencies can also be used to create
  meta-issues that references several other sub-issues
  (for example, `bpo-26865 <https://bugs.python.org/issue26865>`_).


Improving Roundup
=================

This section lists some of the issues mentioned by :pep:`581`
and other desired features and discusses how they can be implemented
by improving Roundup and/or our instance.

* **REST API support.**  A REST API will make integration with other
  services and the development of new tools and applications easier.

  Upstream Roundup now supports a REST API. Updating the tracker will
  make the REST API available.

* **GitHub login support.**  This will allow users to login
  to bugs.python.org (bpo) without having to create a new account.
  It will also solve issues with confirmation emails being marked
  as spam, and provide two-factor authentication.

  A patch to add this functionality is `already available
  <https://github.com/python/bugs.python.org/issues/7>`_
  and is being integrated at the time of writing.

* **Markdown support and message preview and editing.**  This feature
  will allow the use of Markdown in messages and the ability to
  preview the message before the submission and edit it afterward.

  This can be done, but it will take some work.  Possible solutions
  have been proposed on the `roundup-devel mailing list
  <https://sourceforge.net/p/roundup/mailman/message/36667828/>`_.

* **"Remove me from nosy list" button.**  Add a button on issue pages
  to remove self from the nosy list.

  This feature will be added during GSoC 2019.

* **Mobile friendly theme.**  Current theme of bugs.python.org looks
  dated and it doesn't work well with mobile browsers.

  A mobile-friendly theme that is more modern but still familiar
  will be added.

* **Move reply box close to the last message.**  The reply box is
  located at the top of the page, whereas the last message is at the
  bottom.

  The reply box can be moved or duplicated after the last message.

* **Real-time updates.**  When another users submits changes to an
  issue, they should show up in real time.

  This can be accomplished by using the REST API.

* **Add PR link to BPO emails.**  Currently bpo emails don't include
  links to the corresponding PRs.

  A `patch <https://mail.python.org/pipermail/tracker-discuss/2018-June/004547.html>`_
  is available to change the content of the bpo emails from:

  .. code-block:: text

     components: +Tkinter
     versions: +Python 3.4
     pull_requests: +42

  to:

  .. code-block:: text

     components: +Tkinter
     versions: +Python 3.4
     pull_request: https://github.com/python/cpython/pull/341

* **Python 3 support.**  Using Python 3 will make maintenance easier.

  Upstream Roundup now supports Python 3. Updating the tracker will
  allow us to switch to Python 3.  The instances will need to be
  updated as well.

* **Use upstream Roundup.**  We currently use a fork of Roundup with
  a few modifications, most notably the GitHub integration.  If this
  is ported upstream, we can start using upstream Roundup without
  having to maintain our fork.


PEP 581 issues
==============

This section addresses some errors and inaccuracies found in :pep:`581`.

The "Why GitHub?" section of :pep:`581` lists features currently
available on GitHub Issues but not on Roundup.  Some of this features
are currently supported:

* "Ability to reply to issue and pull request conversations via email."

  * Being able to reply by email has been one of the core features of
    Roundup since the beginning.  It is also possible to create new
    issues or close existing ones, set or modify fields, and add
    attachments.

* "Email notifications containing metadata, integrated wit


id: pep-0581.rst
content:
PEP: 581
Title: Using GitHub Issues for CPython
Author: Mariatta <mariatta@python.org>
BDFL-Delegate: Barry Warsaw <barry@python.org>
Discussions-To: https://discuss.python.org/t/535
Status: Final
Type: Process
Created: 20-Jun-2018
Post-History: 07-Mar-2019
Resolution: https://mail.python.org/pipermail/python-dev/2019-May/157399.html


Abstract
========

This PEP outlines the rationale for migration from Python's issue
tracker on Roundup to GitHub issues.  See :pep:`588` for the detailed
migration plan.


Rationale
=========

CPython's development moved to GitHub in February 2017. All other
projects within the PSF's organization are hosted on GitHub and are
using GitHub issues.  CPython is still using Roundup as the issue
tracker on bugs.python.org (bpo) [#]_.


Why GitHub?
-----------

GitHub has a lot of nice features, readily available out of the box. Some of
these aren't readily available on Roundup / bpo.

- APIs that can be used to build integrations and automations. There are various
  existing integrations and applications available from GitHub Marketplace to
  help with the workflow. New applications are easily installed and enabled.
  In addition, we've had great success with building our own GitHub bots, like
  miss-islington [#]_, bedevere [#]_, and the-knights-who-say-ni [#]_.

- Ability to embed/drag and drop screenshots and debug log files into GitHub
  pull requests and issues.

- Administrators and core developers can edit issues, comments, and pull requests.

- Ability to reply to issue and pull request conversations via email.

- Support for two factor authentication.

- Support for markdown and emoji.

- Preview tab, showing how a comment will be rendered, prior to
  actually posting.

- Support for voting via reactions.

- Support for permalinks [#]_, allowing easy quoting and copying & pasting of
  source code. It is possible to automatically embed code snippet on GitHub
  by pasting a permalink.

- Core developers, volunteers, and the PSF don't have to maintain the
  issue infrastructure/site, giving us more time and resources to focus on the
  development of Python.

- Ability to automatically close issues when a PR has been merged [#]_.

  - Note that this feature exists in bpo too.

- Lower barrier to contribution. With more than 28 million users, an open
  source contributor is more likely to already have an account and be familiar
  with GitHub's interface, making it easier to start contributing.

- Email notifications containing metadata [#]_, integrated with Gmail, allowing
  systematic filtering of emails. While Roundup emails contain some metadata,
  they're not as extensive.

- Additional privacy, such as offering the user a choice to hide an
  email address, while still allowing communication with the user
  through @-mentions.


Issues with Roundup / bpo
-------------------------

- Less than five people maintain bpo. Some of them are core developers.

- The upstream Roundup code is in Mercurial. Without any CI available,
  it puts heavy burden on the few existing maintainers in terms of
  reviewing, testing, and applying patches. While there is an unofficial mirror
  of Roundup on GitHub [#]_, Mercurial patches are still the main way to contribute
  to it.

  There is an open discussion about moving the source code of bpo to
  GitHub [#]_. If the source code of bpo does move to GitHub, it will
  become difficult to update patches from upstream. But as long as it
  is in Mercurial, it is difficult to maintain and onboard new
  contributors.

- In its current state, the project is not equipped to accept lots of
  contributions from people who aren't already familiar with the code
  base.

- The user interface needs an update and redesign. It will require UX/UI research
  to keep it up to date with current web standards, including accessibility.

- Users email addresses were exposed.

  - Note: Exposing email address to registered and logged in users was a decision
    taken when bpo instance was setup. This behavior has been recently modified
    after :pep:`581`'s acceptance.

- REST API is not currently available in bpo. There was an open issue in Roundup
  for adding REST API  [#]_. At the time :pep:`581` was proposed, the ticket received
  no activity since 2016. REST API has been integrated in Roundup in February 2019,
  however it is not yet integrated to bpo.

- It sends a number of unnecessary emails and notifications. An example is the nosy email,
  where email notifications are sent whenever someone adds themselves as "nosy".
  An issue has been filed in upstream Roundup about this since 2012 with
  little traction [#]_. While it can be configured, the request to configure it
  was not addressed/ignored.

- Creating an account has been a hassle. There have been reports of people
  having trouble creating accounts or logging in. Some examples of open tickets:
  "Commas in username causes error in nosy list" [#]_, "An error has occurred .." [#]_,
  "It is not sending me the confirmation email ..." [#]_.


Why not GitLab?
===============

Had we migrated to GitLab instead of GitHub in 2017, this PEP would have been
titled "Using GitLab Issues for CPython".


Why not another issue tracker?
==============================

Using another issue tracker will require yet another learning curve, for having
to learn and get used to a different interface. We'll also need to learn and
figure out how to build the integrations with GitHub.

By using GitHub issues, where the CPython source code is currently
hosted and where pull requests are taking place, we'll be providing
consistent experience to contributors and maintainers, while not
having to jump from one interface to another.


Why not focus on improving Roundup / bpo?
=========================================

GitHub has many features we like that are already available. We still need to
build out additional integrations and update our bots, but this is something
we already know how to do.

In order to really improve Roundup / bpo, it needs to first migrate to GitHub
and add CI and bots. As I understand it, there is hesitation because upstream
Roundup is still in Mercurial. Someone more familiar with Roundup / bpo needs
to champion this effort. (I'm not volunteering, I'm sorry).

I believe the effort of creating and maintaining GitHub integrations and bots
is much less than the effort needed to get Roundup up to speed and then to
continue maintaining it.

Downsides of GitHub
===================

GitHub is not the perfect issue tracker. Several issues we need to be aware of:

- Fear of uncertainty and vendor lock in. GitHub is proprietary and there is
  risk of vendor lock-in. However, this is an existing problem since CPython's
  codebase is already on GitHub. This is also not a unique problem to CPython.
  As a precautionary step, CPython's repository on GitHub has
  been backed up on daily basis since June 2018. [#]_

- Bots maintenance costs money, and also takes up volunteer time. We would
  be shifting maintenance burden from Roundup to the bots. At the very least,
  so far we've been able to address any bugs/issues related to the bots/GitHub
  APIs rather quickly, in a matter of days, instead of months or years. GitHub
  APIs are extensive, and used not only by CPython's bots but also by the wider
  Python community. It makes GitHub APIs more approachable compared to
  the maintenance of Roundup/bpo.

- Using GitHub could possibly increase the triaging effort. This was first raised
  as a Zulip topic [#]_, and also brought up during Core Python sprint in
  September 2018 [#]_. A few solutions have been proposed and considered, such as
  creating a special triage team [#]_.  After :pep:`581`'s acceptance, GitHub released a
  new triaging role, currently in beta. The PSF has been in touch with GitHub
  to have this enabled for Python organization. This is pending GitHub's review [#]_.

- Using GitHub could make it easier for people to post disruptive or spammy 


id: pep-0754.rst
content:
PEP: 754
Title: IEEE 754 Floating Point Special Values
Author: Gregory R. Warnes <gregory_r_warnes@groton.pfizer.com>
Status: Rejected
Type: Standards Track
Created: 28-Mar-2003
Python-Version: 2.3
Post-History:


Rejection Notice
================

This PEP has been rejected. After sitting open for four years, it has
failed to generate sufficient community interest.

Several ideas of this PEP were implemented for Python 2.6. ``float('inf')``
and ``repr(float('inf'))`` are now guaranteed to work on every supported
platform with IEEE 754 semantics. However the ``eval(repr(float('inf')))``
roundtrip is still not supported unless you define inf and nan yourself::

    >>> inf = float('inf')
    >>> inf, 1E400
    (inf, inf)
    >>> neginf = float('-inf')
    >>> neginf, -1E400
    (-inf, -inf)
    >>> nan = float('nan')
    >>> nan, inf * 0.
    (nan, nan)

The math and the sys module also have gained additional features,
``sys.float_info``, ``math.isinf``, ``math.isnan``, ``math.copysign``.


Abstract
========

This PEP proposes an API and a provides a reference module that
generates and tests for IEEE 754 double-precision special values:
positive infinity, negative infinity, and not-a-number (NaN).


Rationale
=========

The IEEE 754 standard defines a set of binary representations and
algorithmic rules for floating point arithmetic.  Included in the
standard is a set of constants for representing special values,
including positive infinity, negative infinity, and indeterminate or
non-numeric results (NaN).  Most modern CPUs implement the
IEEE 754 standard, including the (Ultra)SPARC, PowerPC, and x86
processor series.

Currently, the handling of IEEE 754 special values in Python depends
on the underlying C library.  Unfortunately, there is little
consistency between C libraries in how or whether these values are
handled.  For instance, on some systems "float('Inf')" will properly
return the IEEE 754 constant for positive infinity.  On many systems,
however, this expression will instead generate an error message.

The output string representation for an IEEE 754 special value also
varies by platform.  For example, the expression "float(1e3000)",
which is large enough to generate an overflow, should return a string
representation corresponding to IEEE 754 positive infinity.  Python
2.1.3 on x86 Debian Linux returns "inf".  On Sparc Solaris 8 with
Python 2.2.1, this same expression returns "Infinity", and on
MS-Windows 2000 with Active Python 2.2.1, it returns "1.#INF".

Adding to the confusion, some platforms generate one string on
conversion from floating point and accept a different string for
conversion to floating point.  On these systems ::

    float(str(x))

will generate an error when "x" is an IEEE special value.

In the past, some have recommended that programmers use expressions
like::

    PosInf = 1e300**2
    NaN = PosInf/PosInf

to obtain positive infinity and not-a-number constants.  However, the
first expression generates an error on current Python interpreters.  A
possible alternative is to use::

    PosInf = 1e300000
    NaN = PosInf/PosInf

While this does not generate an error with current Python
interpreters, it is still an ugly and potentially non-portable hack.
In addition, defining NaN in this way does solve the problem of
detecting such values.  First, the IEEE 754 standard provides for an
entire set of constant values for Not-a-Number.  Second, the standard
requires that ::

    NaN != X

for all possible values of X, including NaN.  As a consequence ::

    NaN == NaN

should always evaluate to false.  However, this behavior also is not
consistently implemented.  [e.g. Cygwin Python 2.2.2]

Due to the many platform and library inconsistencies in handling IEEE
special values, it is impossible to consistently set or detect IEEE
754 floating point values in normal Python code without resorting to
directly manipulating bit-patterns.

This PEP proposes a standard Python API and provides a reference
module implementation which allows for consistent handling of IEEE 754
special values on all supported platforms.


API Definition
==============

Constants
---------

NaN
    Non-signalling IEEE 754 "Not a Number" value

PosInf
    IEEE 754 Positive Infinity value

NegInf
    IEEE 754 Negative Infinity value


Functions
---------

isNaN(value)
    Determine if the argument is an IEEE 754 NaN (Not a Number) value.

isPosInf(value)
    Determine if the argument is an IEEE 754 positive infinity value.

isNegInf(value)
    Determine if the argument is an IEEE 754 negative infinity value.

isFinite(value)
    Determine if the argument is a finite IEEE 754 value (i.e., is
    not NaN, positive, or negative infinity).

isInf(value)
    Determine if the argument is an infinite IEEE 754 value (positive
    or negative infinity)


Example
-------

(Run under Python 2.2.1 on Solaris 8.)

>>> import fpconst
>>> val = 1e30000 # should be cause overflow and result in "Inf"
>>> val
Infinity
>>> fpconst.isInf(val)
1
>>> fpconst.PosInf
Infinity
>>> nval = val/val # should result in NaN
>>> nval
NaN
>>> fpconst.isNaN(nval)
1
>>> fpconst.isNaN(val)
0


Implementation
--------------

The reference implementation is provided in the module "fpconst" [1]_,
which is written in pure Python by taking advantage of the "struct"
standard module to directly set or test for the bit patterns that
define IEEE 754 special values.  Care has been taken to generate
proper results on both big-endian and little-endian machines.  The
current implementation is pure Python, but some efficiency could be
gained by translating the core routines into C.

Patch 1151323 "New fpconst module" [2]_ on SourceForge adds the
fpconst module to the Python standard library.


References
==========

See http://babbage.cs.qc.edu/courses/cs341/IEEE-754references.html for
reference material on the IEEE 754 floating point standard.

.. [1] Further information on the reference package is available at
   http://research.warnes.net/projects/rzope/fpconst/

.. [2] http://sourceforge.net/tracker/?func=detail&aid=1151323&group_id=5470&atid=305470



Copyright
=========

This document has been placed in the public domain.


id: pep-0740.rst
content:
PEP: 740
Title: Index support for digital attestations
Author: William Woodruff <william@yossarian.net>,
        Facundo Tuesca <facundo.tuesca@trailofbits.com>,
        Dustin Ingram <di@python.org>
Sponsor: Donald Stufft <donald@stufft.io>
PEP-Delegate: Donald Stufft <donald@stufft.io>
Discussions-To: https://discuss.python.org/t/pep-740-index-support-for-digital-attestations/44498
Status: Final
Type: Standards Track
Topic: Packaging
Created: 08-Jan-2024
Post-History: `02-Jan-2024 <https://discuss.python.org/t/pre-pep-exposing-trusted-publisher-provenance-on-pypi/42337>`__,
              `29-Jan-2024 <https://discuss.python.org/t/pep-740-index-support-for-digital-attestations/44498>`__
Resolution: `17-Jul-2024 <https://discuss.python.org/t/pep-740-index-support-for-digital-attestations/44498/26>`__

.. canonical-pypa-spec:: :ref:`packaging:index-hosted-attestations`

.. canonical-doc:: `PyPI - Digital Attestations <https://docs.pypi.org/attestations/>`__

Abstract
========

This PEP proposes a collection of changes related to the upload and distribution
of digitally signed attestations and metadata used to verify them on a Python
package repository, such as PyPI.

These changes have two subcomponents:

* Changes to the currently unstandardized PyPI upload API, allowing clients
  to upload digital attestations as :ref:`attestation objects <attestation-object>`;
* Changes to the :ref:`HTML and JSON "simple" APIs <packaging:simple-repository-api>`,
  allowing clients to retrieve both digital attestations and
  `Trusted Publishing <https://docs.pypi.org/trusted-publishers/>`_ metadata
  for individual release files as :ref:`provenance objects <provenance-object>`.

This PEP does not make a policy recommendation around mandatory digital
attestations on release uploads or their subsequent verification by installing
clients like ``pip``.

Rationale and Motivation
========================

Desire for digital signatures on Python packages has been repeatedly
expressed by both package maintainers and downstream users:

* Maintainers wish to demonstrate the integrity and authenticity of their
  package uploads;
* Individual downstream users wish to verify package integrity and authenticity
  without placing additional trust in their index's honesty;
* "Bulk" downstream users (such as Operating System distributions) wish to
  perform similar verifications and potentially re-expose or countersign
  for their own downstream packaging ecosystems.

This proposal seeks to accommodate each of the above use cases.

Additionally, this proposal identifies the following motivations:

* Verifiable provenance for Python package distributions: many Python
  packages currently contain *unauthenticated* provenance metadata, such
  as URLs for source hosts. A cryptographic attestation format could enable
  strong *authenticated* links between these packages and their source hosts,
  allowing both the index and downstream users to cryptographically verify that
  a package originates from its claimed source repository.
* Raising attacker requirements: an attacker who seeks to take
  over a Python package can be described along *sophistication*
  (unsophisticated to sophisticated) and *targeting* dimensions
  (opportunistic to targeted).

  Digital attestations impose additional sophistication requirements: the
  attacker must be sufficiently sophisticated to access private signing material
  (or signing identities).
* Index verifiability: in the status quo, the only attestation provided by the
  index is an optional PGP signature per release file
  (see :ref:`PGP signatures <pgp-signatures>`). These signatures are not
  (and cannot be) checked by the index either for well-formedness or for
  validity, since the index has no mechanism for identifying the right public
  key for the signature. This PEP overcomes this limitation
  by ensuring that :ref:`provenance objects <provenance-object>` contain all
  of the metadata needed by the index to verify an attestation's validity.

This PEP proposes a generic attestation format, containing an
:ref:`attestation statement for signature generation <payload-and-signature-generation>`,
with the expectation that index providers adopt the
format with a suitable source of identity for signature verification, such as
Trusted Publishing.

Design Considerations
---------------------

This PEP identifies the following design considerations when evaluating
both its own proposed changes and previous work in the same or adjacent
areas of Python packaging:

1. Index accessibility: digital attestations for Python packages
   are ideally retrievable directly from the index itself, as "detached"
   resources.

   This both simplifies some compatibility concerns (by avoiding
   the need to modify the distribution formats themselves) and also simplifies
   the behavior of potential installing clients (by allowing them to
   retrieve each attestation before its corresponding package without needing
   to do streaming decompression).

2. Verification by the index itself: in addition to enabling verification
   by installing clients, each digital attestation is *ideally* verifiable
   in some form by the index itself.

   This both increases the overall quality
   of attestations uploaded to the index (preventing, for example, users
   from accidentally uploading incorrect or invalid attestations) and also
   enables UI and UX refinements on the index itself (such as a "provenance"
   view for each uploaded package).

3. General applicability: digital attestations should be applicable to
   *any and every* package uploaded to the index, regardless of its format
   (sdist or wheel) or interior contents.

4. Metadata support: this PEP refers to "digital attestations" rather than
   just "digital signatures" to emphasize the ideal presence of additional
   metadata within the cryptographic envelope.

   For example, to prevent domain separation between a distribution's name and
   its contents, this PEP uses '`Statements <https://github.com/in-toto/attestation/blob/v1.0/spec/v1.0/statement.md>`__'
   from the `in-toto project <https://in-toto.io/>`__ to bind the distribution's
   contents (via SHA-256 digest) to its filename.


Previous Work
-------------

.. _pgp-signatures:

PGP signatures
^^^^^^^^^^^^^^

PyPI and other indices have historically supported PGP signatures on uploaded
distributions. These could be supplied during upload, and could be retrieved
by installing clients via the ``data-gpg-sig`` attribute in the :pep:`503`
API, the ``gpg-sig`` key on the :pep:`691` API, or via an adjacent
``.asc``-suffixed URL.

PGP signature uploads have been disabled on PyPI since
`May 2023 <https://blog.pypi.org/posts/2023-05-23-removing-pgp/>`_, after
`an investigation <https://blog.yossarian.net/2023/05/21/PGP-signatures-on-PyPI-worse-than-useless>`_
determined that the majority of signatures (which, themselves, constituted a
tiny percentage of overall uploads) could not be associated with a public key or
otherwise meaningfully verified.

In their previously supported form on PyPI, PGP signatures satisfied
considerations (1) and (3) above but not (2) (owing to the need for external
keyservers and key distribution) or (4) (due to PGP signatures typically being
constructed over just an input file, without any associated signed metadata).

Wheel signatures
^^^^^^^^^^^^^^^^

:pep:`427` (and its :ref:`living PyPA counterpart <packaging:binary-distribution-format>`)
specify the :term:`wheel format <packaging:Wheel>`.

This format includes accommodations for digital signatures embedded directly
into the wheel, in either JWS or S/MIME format. These signatures are specified
over a :pep:`376` RECORD, which is modified to include a cryptographic digest
for each recorded file in the wheel.

While wheel signatures are fully specified, they do not appear to be broadly
used; the official `wheel tooling <https://github.com/pypa/wheel>`_ deprecated
signature generation and veri


id: pep-0768.rst
content:
PEP: 768
Title: Safe external debugger interface for CPython
Author: Pablo Galindo Salgado <pablogsal@python.org>, Matt Wozniski <godlygeek@gmail.com>, Ivona Stojanovic <stojanovic.i@hotmail.com>
Discussions-To: https://discuss.python.org/t/pep-768-safe-external-debugger-interface-for-cpython/73969
Status: Accepted
Type: Standards Track
Created: 25-Nov-2024
Python-Version: 3.14
Post-History: `11-Dec-2024 <https://discuss.python.org/t/pep-768-safe-external-debugger-interface-for-cpython/73969>`__
Resolution: `17-Mar-2025 <https://discuss.python.org/t/pep-768-safe-external-debugger-interface-for-cpython/73969/57>`__

Abstract
========

This PEP proposes adding a zero-overhead debugging interface to CPython that
allows debuggers and profilers to safely attach to running Python processes. The
interface provides safe execution points for attaching debugger code without
modifying the interpreter's normal execution path or adding runtime overhead.

A key application of this interface will be enabling pdb to attach to live
processes by process ID, similar to ``gdb -p``, allowing developers to inspect and
debug Python applications interactively in real-time without stopping or
restarting them.

Motivation
==========


Debugging Python processes in production and live environments presents unique
challenges. Developers often need to analyze application behavior without
stopping or restarting services, which is especially crucial for
high-availability systems. Common scenarios include diagnosing deadlocks,
inspecting memory usage, or investigating unexpected behavior in real-time.

Very few Python tools can attach to running processes, primarily because doing
so requires deep expertise in both operating system debugging interfaces and
CPython internals. While C/C++ debuggers like GDB and LLDB can attach to
processes using well-understood techniques, Python tools must implement all of
these low-level mechanisms plus handle additional complexity. For example, when
GDB needs to execute code in a target process, it:

1. Uses ptrace to allocate a small chunk of executable memory (easier said than done)
2. Writes a small sequence of machine code - typically a function prologue, the
   desired instructions, and code to restore registers
3. Saves all the target thread's registers
4. Changes the instruction pointer to the injected code
5. Lets the process run until it hits a breakpoint at the end of the injected code
6. Restores the original registers and continues execution

Python tools face this same challenge of code injection, but with an additional
layer of complexity. Not only do they need to implement the above mechanism,
they must also understand and safely interact with CPython's runtime state,
including the interpreter loop, garbage collector, thread state, and reference
counting system. This combination of low-level system manipulation and
deep domain specific interpreter knowledge makes implementing Python debugging tools
exceptionally difficult.

The few tools (see for example `DebugPy
<https://github.com/microsoft/debugpy/blob/43f41029eabce338becbd1fa1a09727b3cfb1140/src/debugpy/_vendored/pydevd/pydevd_attach_to_process/linux_and_mac/attach.cpp#L4>`__
and `Memray
<https://github.com/bloomberg/memray/blob/main/src/memray/_memray/inject.cpp>`__)
that do attempt this resort to suboptimal and unsafe methods,
using system debuggers like GDB and LLDB to forcefully inject code. This
approach is fundamentally unsafe because the injected code can execute at any
point during the interpreter's execution cycle - even during critical operations
like memory allocation, garbage collection, or thread state management. When
this happens, the results are catastrophic: attempting to allocate memory while
already inside ``malloc()`` causes crashes, modifying objects during garbage
collection corrupts the interpreter's state, and touching thread state at the
wrong time leads to deadlocks.

Various tools attempt to minimize these risks through complex workarounds, such
as spawning separate threads for injected code or carefully timing their
operations or trying to select some good points to stop the process. However,
these mitigations cannot fully solve the underlying problem: without cooperation
from the interpreter, there's no way to know if it's safe to execute code at any
given moment. Even carefully implemented tools can crash the interpreter because
they're fundamentally working against it rather than with it.


Rationale
=========


Rather than forcing tools to work around interpreter limitations with unsafe
code injection, we can extend CPython with a proper debugging interface that
guarantees safe execution. By adding a few thread state fields and integrating
with the interpreter's existing evaluation loop, we can ensure debugging
operations only occur at well-defined safe points. This eliminates the
possibility of crashes and corruption while maintaining zero overhead during
normal execution.

The key insight is that we don't need to inject code at arbitrary points - we
just need to signal to the interpreter that we want code executed at the next
safe opportunity. This approach works with the interpreter's natural execution
flow rather than fighting against it.

After describing this idea to the PyPy development team, this proposal has
already `been implemented in PyPy <https://github.com/pypy/pypy/pull/5135>`__,
proving both its feasibility and effectiveness. Their implementation
demonstrates that we can provide safe debugging capabilities with zero runtime
overhead during normal execution.  The proposed mechanism not only reduces risks
associated with current debugging approaches but also lays the foundation for
future enhancements. For instance, this framework could enable integration with
popular observability tools, providing real-time insights into interpreter
performance or memory usage. One compelling use case for this interface is
enabling pdb to attach to running Python processes, similar to how gdb allows
users to attach to a program by process ID (``gdb -p <pid>``). With this
feature, developers could inspect the state of a running application, evaluate
expressions, and step through code dynamically. This approach would align
Python's debugging capabilities with those of other major programming languages
and debugging tools that support this mode.

Specification
=============


This proposal introduces a safe debugging mechanism that allows external
processes to trigger code execution in a Python interpreter at well-defined safe
points. The key insight is that rather than injecting code directly via system
debuggers, we can leverage the interpreter's existing evaluation loop and thread
state to coordinate debugging operations.

The mechanism works by having debuggers write to specific memory locations in
the target process that the interpreter then checks during its normal execution
cycle. When the interpreter detects that a debugger wants to attach, it executes the
requested operations only when it's safe to do so - that is, when no internal
locks are held and all data structures are in a consistent state.


Runtime State Extensions
------------------------

A new structure is added to PyThreadState to support remote debugging:

.. code-block:: C

    typedef struct _remote_debugger_support {
        int debugger_pending_call;
        char debugger_script_path[MAX_SCRIPT_PATH_SIZE];
    } _PyRemoteDebuggerSupport;

This structure is appended to ``PyThreadState``, adding only a few fields that
are **never accessed during normal execution**. The ``debugger_pending_call`` field
indicates when a debugger has requested execution, while ``debugger_script_path``
provides a filesystem path to a Python source file (.py) that will be executed when
the interpreter reaches a safe point. The path must point to a Python source file,
not compiled Python code (.pyc) or any other format.

The value for ``MAX_SCRIPT_PATH_SIZE`` will be a trade-off between binary size
an


id: pep-0783.rst
content:
PEP: 783
Title: Emscripten Packaging
Author: Hood Chatham <roberthoodchatham at gmail.com>
Sponsor: Åukasz Langa <lukasz at python.org>
Discussions-To: https://discuss.python.org/t/86862
Status: Draft
Type: Standards Track
Topic: Packaging
Created: 28-Mar-2025
Post-History: `02-Apr-2025 <https://discuss.python.org/t/86862>`__,
              `18-Mar-2025 <https://discuss.python.org/t/84996>`__,

Abstract
========

This PEP proposes a new platform tag series ``pyodide`` for binary Python package
distributions for the Pyodide Python runtime.

`Emscripten <https://emscripten.org/>`__ is a complete open-source compiler
toolchain. It compiles C/C++ code into WebAssembly/JavaScript executables, for
use in JavaScript runtimes, including browsers and Node.js. The Rust language
also maintains an Emscripten target. :pep:`776` specifies Python's support for
Emscripten.


Motivation
==========

Pyodide is a CPython distribution for use in the browser. A web browser is a
universal computing platform, available on Windows, macOS, Linux, and every
smartphone. Hundreds of thousands of students have learned Python through
Pyodide via projects like `Capytale
<https://web.archive.org/web/20241211090946/https://cfp.jupytercon.com/2023/talk/TJ9YEV/>`__
and `PyodideU <https://stanford.edu/~cpiech/bio/papers/pyodideU.pdf>`__. Pyodide
is also increasingly being used by Python packages to provide interactive
documentation.

Pyodide currently maintains ports of 255 different packages at the time of this
writing, including major scientific Python packages like NumPy, SciPy, pandas,
Polars, scikit-learn, OpenCV, PyArrow, and Pillow as well as general purpose
packages like aiohttp, Requests, Pydantic, cryptography, and orjson.

About 60 packages are also testing against Pyodide in their CI, including NumPy,
pandas, awkward-cpp, scikit-image, statsmodels, PyArrow, Hypothesis, and PyO3.

Python package projects cannot deploy binary distributions for Pyodide on PyPI.
Instead they must use other options like ``anaconda.org`` or ``jsdelivr.com``.
This creates friction both for package maintainers and for users.


Rationale
=========

Emscripten uses a variant of musl libc. The Emscripten compiler makes no ABI
stability guarantees between versions. Many Emscripten updates are ABI
compatible by chance, and the Rust Emscripten target behaves as if the ABI were
stable with only `occasional negative consequences
<https://github.com/rust-lang/rust/issues/131467>`__.

There are several linker flags that adjust the Emscripten ABI, so Python
packages built to run with Emscripten must make sure to match the ABI-sensitive
linker flags used to compile the interpreter to avoid load-time or run-time
errors. The Emscripten compiler continuously fixes bugs and adds support for new
web platform features. Thus, there is significant benefit to being able to
update the ABI.

In order to balance the ABI stability needs of package maintainers with the ABI
flexibility to allow the platform to move forward, Pyodide plans to adopt a new
ABI for each feature release of Python.

The Pyodide team also coordinates the ABI flags that Pyodide uses with the
Emscripten ABI that Rust supports in order to ensure that we have support for
the many popular Rust packages. Historically, most of the work for this has
been related to unwinding ABIs. See for instance `this Rust Major Change
Proposal <https://github.com/rust-lang/compiler-team/issues/801>`__.

The ``pyodide`` platform tags only apply to Python interpreters compiled and
linked with the same version of Emscripten as Pyodide, with the same
ABI-sensitive flags.


Specification
=============

The platform tags will take the form:

.. code-block:: text

   pyodide_${YEAR}_${PATCH}_wasm32

Each one of these will be used with a specified Python version. For example, the
platform tag ``pyodide_2025_0`` will be used with Python 3.13.

Emscripten Wheel ABI
--------------------

The specification of the ``pyodide_<abi>`` platform includes:

* Which version of the Emscripten compiler is used
* What libraries are statically linked with the interpreter
* What stack unwinding ABI is to be used
* How the loader handles dependency lookup
* That libraries cannot use ``-pthread``
* That libraries should be linked with ``-sWASM_BIGINT``

The ABI is selected by choosing the appropriate version of the Emscripten
compiler and passing appropriate compiler and linker flags. It is possible for
other people to build their own Python interpreter that is compatible with the
Pyodide ABI, it is not necessary to use the Pyodide distribution itself.

The Pyodide ABIs are fully specified in the `Pyodide Platform ABI
<https://pyodide.org/en/stable/development/abi.html>`__ documentation.

The ``pyodide build`` tool knows how to create wheels that match the Pyodide
ABI. Unlike with manylinux wheels, there is no need for a Docker container to
build the ``pyodide_<abi>`` wheels. All that is needed is a Linux machine and
appropriate versions of Python, Node.js, and Emscripten.

It is possible to validate a wheel by installing and importing it into the
Pyodide runtime. Because Pyodide can run in an environment with strong
sandboxing guarantees, doing this produces no security risks.

Determining the ABI version
---------------------------

The Pyodide ABI version is stored in the ``PYODIDE_ABI_VERSION`` config variable
and can be determined via:

.. code-block:: python

   pyodide_abi_version = sysconfig.get_config_var("PYODIDE_ABI_VERSION")


To generate the list of compatible tags, one can use the following code:

.. code-block:: python

    from packaging.tags import cpython_tags, _generic_platforms

    def _emscripten_platforms() -> Iterator[str]:
        pyodide_abi_version = sysconfig.get_config_var("PYODIDE_ABI_VERSION")
        if pyodide_abi_version:
            yield f"pyodide_{pyodide_abi_version}_wasm32"
        yield from _generic_platforms()

    emscripten_tags = cpython_tags(platforms=_emscripten_platforms())

This code will be added to `pypa/packaging
<https://github.com/pypa/packaging/pull/804>`__.


Package Installers
------------------

Installers should use the ``_emscripten_platforms()`` function shown above to
determine which platforms are compatible with an Emscripten build of CPython. In
particular, the Pyodide ABI version is exposed via
``sysconfig.get_config_var("PYODIDE_ABI_VERSION")``.

Package Indexes
---------------

Package indexes SHOULD accept any wheel whose platform tag matches
the regular expression ``pyodide_[0-9]+_[0-9]+_wasm32``.


Dependency Specifier Markers
----------------------------

According to :pep:`776#platform-identification`, in Emscripten Python
``sys.platform`` returns ``"emscripten"``. To check for the Emscripten platform in a
dependency specifier, one can use ``sys_platform == "emscripten"`` (or its
negation).


Trove Classifier
----------------

Packages that build and test Emscripten wheels can declare this by adding the
``Environment :: WebAssembly :: Emscripten`` classifier. PyPI already accepts uploads of
`packages with this classifier
<https://pypi.org/search/?q=&o=&c=Environment+%3A%3A+WebAssembly+%3A%3A+Emscripten>`__.


Backwards Compatibility
=======================

There are no backwards compatibility concerns in this PEP.


Security Implications
=====================

There are no security implications in this PEP.

How to Teach This
=================

For Pyodide users, we recommend the `Pyodide documentation on installing
packages <https://pyodide.org/en/stable/usage/loading-packages.html>`__.

For package maintainers, we recommend the `Pyodide documentation on building and
testing packages
<https://pyodide.org/en/stable/development/building-and-testing-packages.html>`__.

Reference Implementation
========================

For building packages, `pyodide build
<https://github.com/pyodide/pyodide-build>`__ and `cibuildwheel
<https://github.com/pypa/cibuildwheel/>`__.

For installers to decide whether a wheel tag is compatible wit


id: pep-0620.rst
content:
PEP: 620
Title: Hide implementation details from the C API
Author: Victor Stinner <vstinner@python.org>
Status: Withdrawn
Type: Standards Track
Created: 19-Jun-2020
Python-Version: 3.12

Abstract
========

Introduce C API incompatible changes to hide implementation details.

Once most implementation details will be hidden, evolution of CPython
internals would be less limited by C API backward compatibility issues.
It will be way easier to add new features.

It becomes possible to experiment with more advanced optimizations in
CPython than just micro-optimizations, like tagged pointers.

Define a process to reduce the number of broken C extensions.

The implementation of this PEP is expected to be done carefully over
multiple Python versions. It already started in Python 3.7 and most
changes are already completed. The `Process to reduce the number of
broken C extensions`_ dictates the rhythm.


PEP withdrawn
=============

This PEP was withdrawn by its author since the scope is too broad and the work is
distributed over multiple Python versions, which makes it difficult to make
a decision on the overall PEP. It was split into new PEPs with
narrower and better defined scopes, like :pep:`670`.


Motivation
==========

The C API blocks CPython evolutions
-----------------------------------

Adding or removing members of C structures is causing multiple backward
compatibility issues.

Adding a new member breaks the stable ABI (:pep:`384`), especially for
types declared statically (e.g. ``static PyTypeObject MyType =
{...};``). In Python 3.4, the :pep:`442` "Safe object finalization" added
the ``tp_finalize`` member at the end of the ``PyTypeObject`` structure.
For ABI backward compatibility, a new ``Py_TPFLAGS_HAVE_FINALIZE`` type
flag was required to announce if the type structure contains the
``tp_finalize`` member. The flag was removed in Python 3.8 (`bpo-32388
<https://bugs.python.org/issue32388>`_).

The ``PyTypeObject.tp_print`` member, deprecated since Python 3.0
released in 2009, has been removed in the Python 3.8 development cycle.
But the change broke too many C extensions and had to be reverted before
3.8 final release. Finally, the member was removed again in Python 3.9.

C extensions rely on the ability to access structure members,
indirectly through the C API, or even directly. Modifying structures
like ``PyListObject`` cannot be even considered.

The ``PyTypeObject`` structure is the one which evolved the most, simply
because there was no other way to evolve CPython than modifying it.

A C extension can technically dereference a ``PyObject*`` pointer and
access ``PyObject`` members. This prevents experiments like tagged
pointers (storing small values as ``PyObject*`` which does not point to
a valid ``PyObject`` structure).

Replacing Python garbage collector with a tracing garbage collector
would also need to remove ``PyObject.ob_refcnt`` reference counter,
whereas currently ``Py_INCREF()`` and ``Py_DECREF()`` macros access
directly to ``PyObject.ob_refcnt``.

Same CPython design since 1990: structures and reference counting
-----------------------------------------------------------------

When the CPython project was created, it was written with one principle:
keep the implementation simple enough so it can be maintained by a
single developer. CPython complexity grew a lot and many
micro-optimizations have been implemented, but CPython core design has
not changed.

Members of ``PyObject`` and ``PyTupleObject`` structures have not
changed since the "Initial revision" commit (1990)::

    #define OB_HEAD \
        unsigned int ob_refcnt; \
        struct _typeobject *ob_type;

    typedef struct _object {
        OB_HEAD
    } object;

    typedef struct {
        OB_VARHEAD
        object *ob_item[1];
    } tupleobject;

Only names changed: ``object`` was renamed to ``PyObject`` and
``tupleobject`` was renamed to ``PyTupleObject``.

CPython still tracks Python objects lifetime using reference counting
internally and for third party C extensions (through the Python C API).

All Python objects must be allocated on the heap and cannot be moved.

Why is PyPy more efficient than CPython?
----------------------------------------

The PyPy project is a Python implementation which is 4.2x faster than
CPython on average. PyPy developers chose to not fork CPython, but start
from scratch to have more freedom in terms of optimization choices.

PyPy does not use reference counting, but a tracing garbage collector
which moves objects. Objects can be allocated on the stack (or even not
at all), rather than always having to be allocated on the heap.

Objects layouts are designed with performance in mind. For example, a
list strategy stores integers directly as integers, rather than objects.

Moreover, PyPy also has a JIT compiler which emits fast code thanks to
the efficient PyPy design.

PyPy bottleneck: the Python C API
---------------------------------

While PyPy is way more efficient than CPython to run pure Python code,
it is as efficient or slower than CPython to run C extensions.

Since the C API requires ``PyObject*`` and allows to access directly
structure members, PyPy has to associate a CPython object to PyPy
objects and maintain both consistent. Converting a PyPy object to a
CPython object is inefficient. Moreover, reference counting also has to
be implemented on top of PyPy tracing garbage collector.

These conversions are required because the Python C API is too close to
the CPython implementation: there is no high-level abstraction.
For example, structures members are part of the public C API and nothing
prevents a C extension to get or set directly
``PyTupleObject.ob_item[0]`` (the first item of a tuple).

See `Inside cpyext: Why emulating CPython C API is so Hard
<https://morepypy.blogspot.com/2018/09/inside-cpyext-why-emulating-cpython-c.html>`_
(Sept 2018) by Antonio Cuni for more details.


Rationale
=========

Hide implementation details
---------------------------

Hiding implementation details from the C API has multiple advantages:

* It becomes possible to experiment with more advanced optimizations in
  CPython than just micro-optimizations. For example, tagged pointers,
  and replace the garbage collector with a tracing garbage collector
  which can move objects.
* Adding new features in CPython becomes easier.
* PyPy should be able to avoid conversions to CPython objects in more
  cases: keep efficient PyPy objects.
* It becomes easier to implement the C API for a new Python
  implementation.
* More C extensions will be compatible with Python implementations other
  than CPython.

Relationship with the limited C API
-----------------------------------

The :pep:`384` "Defining a Stable ABI" is implemented in Python 3.4. It introduces the
"limited C API": a subset of the C API. When the limited C API is used,
it becomes possible to build a C extension only once and use it on
multiple Python versions: that's the stable ABI.

The main limitation of the :pep:`384` is that C extensions have to opt-in
for the limited C API. Only very few projects made this choice,
usually to ease distribution of binaries, especially on Windows.

This PEP moves the C API towards the limited C API.

Ideally, the C API will become the limited C API and all C extensions
will use the stable ABI, but this is out of this PEP scope.


Specification
=============

Summary
-------

* (**Completed**) Reorganize the C API header files: create ``Include/cpython/`` and
  ``Include/internal/`` subdirectories.
* (**Completed**) Move private functions exposing implementation details to the internal
  C API.
* (**Completed**) Convert macros to static inline functions.
* (**Completed**) Add new functions ``Py_SET_TYPE()``, ``Py_SET_REFCNT()`` and
  ``Py_SET_SIZE()``. The ``Py_TYPE()``, ``Py_REFCNT()`` and
  ``Py_SIZE()`` macros become functions which cannot be used as l-value.
* (**Completed**) New C API functions must not return borrowed
  references.
* (**In Progr


id: pep-0634.rst
content:
PEP: 634
Title: Structural Pattern Matching: Specification
Author: Brandt Bucher <brandt@python.org>,
        Guido van Rossum <guido@python.org>
BDFL-Delegate:
Discussions-To: python-dev@python.org
Status: Final
Type: Standards Track
Created: 12-Sep-2020
Python-Version: 3.10
Post-History: 22-Oct-2020, 08-Feb-2021
Replaces: 622
Resolution: https://mail.python.org/archives/list/python-committers@python.org/message/SQC2FTLFV5A7DV7RCEAR2I2IKJKGK7W3

.. canonical-doc:: :external+python:ref:`match`

Abstract
========

This PEP provides the technical specification for the match
statement.  It replaces :pep:`622`, which is hereby split in three parts:

- :pep:`634`: Specification
- :pep:`635`: Motivation and Rationale
- :pep:`636`: Tutorial

This PEP is intentionally devoid of commentary; the motivation and all
explanations of our design choices are in :pep:`635`.  First-time readers
are encouraged to start with :pep:`636`, which provides a gentler
introduction to the concepts, syntax and semantics of patterns.


Syntax and Semantics
====================

See `Appendix A <634-appendix-a_>`_ for the complete grammar.

Overview and Terminology
------------------------

The pattern matching process takes as input a pattern (following
``case``) and a subject value (following ``match``).  Phrases to
describe the process include "the pattern is matched with (or against)
the subject value" and "we match the pattern against (or with) the
subject value".

The primary outcome of pattern matching is success or failure.  In
case of success we may say "the pattern succeeds", "the match
succeeds", or "the pattern matches the subject value".

In many cases a pattern contains subpatterns, and success or failure
is determined by the success or failure of matching those subpatterns
against the value (e.g., for OR patterns) or against parts of the
value (e.g., for sequence patterns).  This process typically processes
the subpatterns from left to right until the overall outcome is
determined.  E.g., an OR pattern succeeds at the first succeeding
subpattern, while a sequence patterns fails at the first failing
subpattern.

A secondary outcome of pattern matching may be one or more name
bindings.  We may say "the pattern binds a value to a name".  When
subpatterns tried until the first success, only the bindings due to
the successful subpattern are valid; when trying until the first
failure, the bindings are merged.  Several more rules, explained
below, apply to these cases.


The Match Statement
-------------------

Syntax::

  match_stmt: "match" subject_expr ':' NEWLINE INDENT case_block+ DEDENT
  subject_expr:
      | star_named_expression ',' star_named_expressions?
      | named_expression
  case_block: "case" patterns [guard] ':' block
  guard: 'if' named_expression

The rules ``star_named_expression``, ``star_named_expressions``,
``named_expression`` and ``block`` are part of the `standard Python
grammar <https://docs.python.org/3.10/reference/grammar.html>`_.

The rule ``patterns`` is specified below.

For context, ``match_stmt`` is a new alternative for
``compound_statement``::

  compound_statement:
      | if_stmt
      ...
      | match_stmt


The ``match`` and ``case`` keywords are soft keywords, i.e. they are
not reserved words in other grammatical contexts (including at the
start of a line if there is no colon where expected).  This implies
that they are recognized as keywords when part of a match
statement or case block only, and are allowed to be used in all
other contexts as variable or argument names.


Match Semantics
^^^^^^^^^^^^^^^

The match statement first evaluates the subject expression.  If a
comma is present a tuple is constructed using the standard rules.

The resulting subject value is then used to select the first case
block whose patterns succeeds matching it *and* whose guard condition
(if present) is "truthy".  If no case blocks qualify the match
statement is complete; otherwise, the block of the selected case block
is executed.  The usual rules for executing a block nested inside a
compound statement apply (e.g. an ``if`` statement).

Name bindings made during a successful pattern match outlive the
executed block and can be used after the match statement.

During failed pattern matches, some subpatterns may succeed. For
example, while matching the pattern ``(0, x, 1)`` with the value ``[0,
1, 2]``, the subpattern ``x`` may succeed if the list elements are
matched from left to right.  The implementation may choose to either
make persistent bindings for those partial matches or not. User code
including a match statement should not rely on the bindings being
made for a failed match, but also shouldn't assume that variables are
unchanged by a failed match.  This part of the behavior is left
intentionally unspecified so different implementations can add
optimizations, and to prevent introducing semantic restrictions that
could limit the extensibility of this feature.

The precise pattern binding rules vary per pattern type and are
specified below.


.. _634-guards:

Guards
^^^^^^

If a guard is present on a case block, once the pattern or patterns in
the case block succeed, the expression in the guard is evaluated.  If
this raises an exception, the exception bubbles up.  Otherwise, if the
condition is "truthy" the case block is selected; if it is "falsy" the
case block is not selected.

Since guards are expressions they are allowed to have side effects.
Guard evaluation must proceed from the first to the last case block,
one at a time, skipping case blocks whose pattern(s) don't all
succeed.  (I.e., even if determining whether those patterns succeed
may happen out of order, guard evaluation must happen in order.)
Guard evaluation must stop once a case block is selected.


Irrefutable case blocks
^^^^^^^^^^^^^^^^^^^^^^^

A pattern is considered irrefutable if we can prove from its syntax
alone that it will always succeed.  In particular, capture patterns
and wildcard patterns are irrefutable, and so are AS patterns whose
left-hand side is irrefutable, OR patterns containing at least
one irrefutable pattern, and parenthesized irrefutable patterns.

A case block is considered irrefutable if it has no guard and its
pattern is irrefutable.

A match statement may have at most one irrefutable case block, and it
must be last.


.. _634-patterns:

Patterns
--------

The top-level syntax for patterns is as follows::

  patterns: open_sequence_pattern | pattern
  pattern: as_pattern | or_pattern
  as_pattern: or_pattern 'as' capture_pattern
  or_pattern: '|'.closed_pattern+
  closed_pattern:
      | literal_pattern
      | capture_pattern
      | wildcard_pattern
      | value_pattern
      | group_pattern
      | sequence_pattern
      | mapping_pattern
      | class_pattern


AS Patterns
^^^^^^^^^^^

Syntax::

  as_pattern: or_pattern 'as' capture_pattern

(Note: the name on the right may not be ``_``.)

An AS pattern matches the OR pattern on the left of the ``as``
keyword against the subject.  If this fails, the AS pattern fails.
Otherwise, the AS pattern binds the subject to the name on the right
of the ``as`` keyword and succeeds.


OR Patterns
^^^^^^^^^^^

Syntax::

  or_pattern: '|'.closed_pattern+

When two or more patterns are separated by vertical bars (``|``),
this is called an OR pattern.  (A single closed pattern is just that.)

Only the final subpattern may be irrefutable.

Each subpattern must bind the same set of names.

An OR pattern matches each of its subpatterns in turn to the subject,
until one succeeds.  The OR pattern is then deemed to succeed.
If none of the subpatterns succeed the OR pattern fails.


.. _634-literal_pattern:

Literal Patterns
^^^^^^^^^^^^^^^^

Syntax::

  literal_pattern:
      | signed_number
      | signed_number '+' NUMBER
      | signed_number '-' NUMBER
      | strings
      | 'None'
      | 'True'
      | 'False'
  signed_number: NUMBER | '-' NUMBER

The rule ``strings`` and the token ``NUMBER`` are defined i


id: pep-0608.rst
content:
PEP: 608
Title: Coordinated Python release
Author: Miro HronÄok <miro@hroncok.cz>,
        Victor Stinner <vstinner@python.org>
Status: Rejected
Type: Standards Track
Created: 25-Oct-2019
Python-Version: 3.9

Abstract
========

Block a Python release until a compatible version of selected projects
is available.

The Python release manager can decide to release Python even if a
project is not compatible, if they decide that the project is going to
be fixed soon enough, or if the issue severity is low enough.


Rationale
=========

The PEP involves maintainers of the selected projects in the Python
release cycle. There are multiple benefit:

* Detect more bugs before a Python final release
* Discuss and maybe revert incompatible changes before a Python final
  release
* Increase the number of compatible projects when the new Python final
  version is released

Too few projects are involved in the Python beta phase
------------------------------------------------------

Currently, Python beta versions are available four months before the
final 3.x.0 release.

Bugs reported during the beta phase can be easily fixed and can block a
release if they are serious enough.

Incompatible changes are discussed during the beta phase: enhance
documentation explaining how to update code, or consider to revert these
changes.

Even if more and more projects are tested on the master branch of Python
in their CI, too many projects of the top 50 PyPI projects are only
compatible with the new Python a few weeks, or even months, after the
final Python release.

DeprecationWarning is being ignored
-----------------------------------

Python has well defined process to deprecate features. A
DeprecationWarning must be emitted during at least one Python release,
before a feature can be removed.

In practice, DeprecationWarning warnings are ignored for years in major
Python projects. Usually, maintainers explain that there are too many
warnings and so they simply ignore warnings. Moreover, DeprecationWarning
is silent by default (except in the ``__main__`` module: :pep:`565`).

Even if more and more projects are running their test suite with
warnings treated as errors (``-Werror``), Python core developers still
have no idea how many projects are broken when a feature is removed.

Need to coordinate
------------------

When issues and incompatible changes are discovered and discussed after
the final Python release, it becomes way more complicated and expensive
to fix Python.  Once an API is part of an official final release, Python
should provide backward compatibility for the whole 3.x release
lifetime. Some operating systems can be shipped with the buggy final
release and can take several months before being updated.

Too many projects are only updated to the new Python after the final
Python release, which makes this new Python version barely usable to run
large applications when Python is released.

It is proposed to block a Python release until a compatible version of
all selected projects is available.

Shorter Python release schedule
-------------------------------

The :pep:`PEP 602: Annual Release Cycle for Python
<602>` and the :pep:`PEP 605: A
rolling feature release stream for CPython
<605>` would like to release
Python more often to ship new features more quickly.

The problem is that each Python ``3.x`` release breaks many projects.

Coordinated Python releases reduces the number of broken projects and
makes new Python release more usable.


Specification
=============

By default, a Python release is blocked until a compatible version of
all selected projects is available.

Before releasing the final Python version, the Python release manager is
responsible to send a report of the compatibility status of each project
of the selected projects. It is recommended to send such report at
each beta release to see the evolution and detects issues as soon as
possible.

The Python release manager can decide to release Python even if a
project is not compatible, if they decide that the project is going to
be fixed soon enough, or if the issue severity is low enough.

After each Python release, the project list can be updated to remove
projects and add new ones. For example, to remove old unused
dependencies and add new ones. The list can grow if the whole process
doesn't block Python releases for too long.

Limit the delay
---------------

When a build or test issue with the next Python version is reported to a
project, maintainers have one month to answer. With no answer, the
project can be excluded from the list of projects blocking the Python
release.

Multiple projects are already tested on the master branch of Python in a
CI. Problems can be detected very early in a Python release which should
provide enough time to handle them. More CI can be added for projects
which are not tested on the next Python yet.

Once selected projects issues are known, exceptions can be discussed
between the Python release manager and involved project maintainers on a
case-by-case basis. Not all issues deserve to block a Python release.

Selected projects
-----------------

List of projects blocking a Python release (total: 27):

* Projects (13):

  * aiohttp
  * cryptography
  * Cython
  * Django
  * numpy
  * pandas
  * pip
  * requests
  * scipy
  * Sphinx (needed to build Python)
  * sqlalchemy
  * pytest
  * tox

* Direct and indirect dependencies (14):

  * certifi (needed by urllib3)
  * cffi (needed by cryptography)
  * chardet (needed by Sphinx)
  * colorama (needed by pip)
  * docutils (needed by Sphinx)
  * idna (needed by Sphinx and requests)
  * jinja2 (needed by Sphinx)
  * MarkupSafe (needed by Sphinx)
  * psycopg2 (needed by Django)
  * pycparser (needed by cffi)
  * setuptools (needed by pip and tons of Python projects)
  * six (needed by tons of Python projects)
  * urllib3 (needed by requests)
  * wheel (needed by pip)

How projects are selected
-------------------------

Projects used by to build Python should be in the list, like Sphinx.

Most popular projects are picked from the most downloaded PyPI projects.

Most of project dependencies are included in the list as well, since a
single incompatible dependency can block a whole project. Some
dependencies are excluded to reduce the list length.

Test dependencies as pytest and tox should be included as well. If a
project cannot be tested, a new version cannot be shipped neither.

The list should be long enough to have a good idea of the cost of
porting a project to the next Python, but small enough to not block a
Python release for too long.

Obviously, projects which are not part of the list also are encouraged
to report issues with the next Python and to have a CI running on the
next Python version.


Incompatible changes
====================

The definition here is large: any Python change which cause an issue
when building or testing a project.

See also the :pep:`PEP 606: Python Compatibility Version
<606>` for more examples of
incompatible changes.

Examples
--------

There are different kinds of incompatible changes:

* Change in the Python build. For example, Python 3.8 removed ``'m'``
  (which stands for pymalloc) from ``sys.abiflags`` which impacts Python
  vendors like Linux distributions.
* Change in the C extensions build. For example, Python 3.8 no longer
  links C extensions to libpython, and Python 3.7 removed
  ``os.errno`` alias to the ``errno`` module.
* Removed function. For example, collections aliases to ABC classes
  have been removed in Python 3.9.
* Changed function signature:

  * Reject a type which was previously accepted (ex: only accept ``int``,
    reject ``float``).
  * Add a new mandatory parameter.
  * Convert a positional-or-keyword parameter to positional-only.

* Behavior change. For example, Python 3.8 now serializes XML attributes
  in their insertion order, rather than sorting them by name.
* New warning. Since more and more projects are tested with all warnings
  treated as er


id: pep-0344.rst
content:
PEP: 344
Title: Exception Chaining and Embedded Tracebacks
Author: Ka-Ping Yee
Status: Superseded
Type: Standards Track
Created: 12-May-2005
Python-Version: 2.5
Post-History:


Numbering Note
==============

This PEP has been renumbered to :pep:`3134`.  The text below is the last version
submitted under the old number.


Abstract
========

This PEP proposes three standard attributes on exception instances: the
``__context__`` attribute for implicitly chained exceptions, the
``__cause__`` attribute for explicitly chained exceptions, and the
``__traceback__`` attribute for the traceback.  A new ``raise ... from``
statement sets the ``__cause__`` attribute.


Motivation
==========

During the handling of one exception (exception A), it is possible that another
exception (exception B) may occur.  In today's Python (version 2.4), if this
happens, exception B is propagated outward and exception A is lost.  In order
to debug the problem, it is useful to know about both exceptions.  The
``__context__`` attribute retains this information automatically.

Sometimes it can be useful for an exception handler to intentionally re-raise
an exception, either to provide extra information or to translate an exception
to another type.  The ``__cause__`` attribute provides an explicit way to
record the direct cause of an exception.

In today's Python implementation, exceptions are composed of three parts: the
type, the value, and the traceback.  The ``sys`` module, exposes the current
exception in three parallel variables, ``exc_type``, ``exc_value``, and
``exc_traceback``, the ``sys.exc_info()`` function returns a tuple of these
three parts, and the ``raise`` statement has a three-argument form accepting
these three parts.  Manipulating exceptions often requires passing these three
things in parallel, which can be tedious and error-prone.  Additionally, the
``except`` statement can only provide access to the value, not the traceback.
Adding the ``__traceback__`` attribute to exception values makes all the
exception information accessible from a single place.


History
=======

Raymond Hettinger [1]_ raised the issue of masked exceptions on Python-Dev in
January 2003 and proposed a ``PyErr_FormatAppend()`` function that C modules
could use to augment the currently active exception with more information.
Brett Cannon [2]_ brought up chained exceptions again in June 2003, prompting
a long discussion.

Greg Ewing [3]_ identified the case of an exception occurring in a ``finally``
block during unwinding triggered by an original exception, as distinct from
the case of an exception occurring in an ``except`` block that is handling the
original exception.

Greg Ewing [4]_ and Guido van Rossum [5]_, and probably others, have
previously mentioned adding a traceback attribute to ``Exception`` instances.
This is noted in :pep:`3000`.

This PEP was motivated by yet another recent Python-Dev reposting of the same
ideas [6]_ [7]_.


Rationale
=========

The Python-Dev discussions revealed interest in exception chaining for two
quite different purposes.  To handle the unexpected raising of a secondary
exception, the exception must be retained implicitly. To support intentional
translation of an exception, there must be a way to chain exceptions
explicitly.  This PEP addresses both.

Several attribute names for chained exceptions have been suggested on
Python-Dev [2]_, including ``cause``, ``antecedent``, ``reason``, ``original``,
``chain``, ``chainedexc``, ``xc_chain``, ``excprev``, ``previous`` and
``precursor``.  For an explicitly chained exception, this PEP suggests
``__cause__`` because of its specific meaning.  For an implicitly chained
exception, this PEP proposes the name ``__context__`` because the intended
meaning is more specific than temporal precedence but less specific than
causation: an exception occurs in the context of handling another exception.

This PEP suggests names with leading and trailing double-underscores for these
three attributes because they are set by the Python VM. Only in very special
cases should they be set by normal assignment.

This PEP handles exceptions that occur during ``except`` blocks and
``finally`` blocks in the same way.  Reading the traceback makes it clear
where the exceptions occurred, so additional mechanisms for distinguishing
the two cases would only add unnecessary complexity.

This PEP proposes that the outermost exception object (the one exposed for
matching by ``except`` clauses) be the most recently raised exception for
compatibility with current behaviour.

This PEP proposes that tracebacks display the outermost exception last,
because this would be consistent with the chronological order of tracebacks
(from oldest to most recent frame) and because the actual thrown exception is
easier to find on the last line.

To keep things simpler, the C API calls for setting an exception will not
automatically set the exception's ``__context__``.  Guido van Rossum has
expressed concerns with making such changes [8]_.

As for other languages, Java and Ruby both discard the original exception when
another exception occurs in a ``catch/rescue`` or ``finally/ensure`` clause.
Perl 5 lacks built-in structured exception handling.  For Perl 6, RFC number
88 [9]_ proposes an exception mechanism that implicitly retains chained
exceptions in an array named ``@@``.  In that RFC, the most recently raised
exception is exposed for matching, as in this PEP; also, arbitrary expressions
(possibly involving ``@@``) can be evaluated for exception matching.

Exceptions in C# contain a read-only ``InnerException`` property that may
point to another exception.  Its documentation [10]_ says that "When an
exception X is thrown as a direct result of a previous exception Y, the
``InnerException`` property of X should contain a reference to Y."  This
property is not set by the VM automatically; rather, all exception
constructors take an optional ``innerException`` argument to set it
explicitly.  The ``__cause__`` attribute fulfills the same purpose as
``InnerException``, but this PEP proposes a new form of ``raise`` rather than
extending the constructors of all exceptions. C# also provides a
``GetBaseException`` method that jumps directly to the end of the
``InnerException`` chain; this PEP proposes no analog.

The reason all three of these attributes are presented together in one proposal
is that the ``__traceback__`` attribute provides convenient access to the
traceback on chained exceptions.


Implicit Exception Chaining
===========================

Here is an example to illustrate the ``__context__`` attribute::

    def compute(a, b):
        try:
            a/b
        except Exception, exc:
            log(exc)

    def log(exc):
        file = open('logfile.txt')  # oops, forgot the 'w'
        print >>file, exc
        file.close()

Calling ``compute(0, 0)`` causes a ``ZeroDivisionError``.  The ``compute()``
function catches this exception and calls ``log(exc)``, but the ``log()``
function also raises an exception when it tries to write to a file that wasn't
opened for writing.

In today's Python, the caller of ``compute()`` gets thrown an ``IOError``. The
``ZeroDivisionError`` is lost.  With the proposed change, the instance of
``IOError`` has an additional ``__context__`` attribute that retains the
``ZeroDivisionError``.

The following more elaborate example demonstrates the handling of a mixture of
``finally`` and ``except`` clauses::

    def main(filename):
        file = open(filename)       # oops, forgot the 'w'
        try:
            try:
                compute()
            except Exception, exc:
                log(file, exc)
            finally:
                file.clos()         # oops, misspelled 'close'

    def compute():
        1/0

    def log(file, exc):
        try:
            print >>file, exc       # oops, file is not writable
        except:
            display(exc)

    def display(exc):
        print ex                    # oo


id: pep-0422.rst
content:
PEP: 422
Title: Simpler customisation of class creation
Author: Alyssa Coghlan <ncoghlan@gmail.com>,
        Daniel Urban <urban.dani+py@gmail.com>
Status: Withdrawn
Type: Standards Track
Created: 05-Jun-2012
Python-Version: 3.5
Post-History: 05-Jun-2012, 10-Feb-2013


Abstract
========

Currently, customising class creation requires the use of a custom metaclass.
This custom metaclass then persists for the entire lifecycle of the class,
creating the potential for spurious metaclass conflicts.

This PEP proposes to instead support a wide range of customisation
scenarios through a new ``namespace`` parameter in the class header, and
a new ``__autodecorate__`` hook in the class body.

The new mechanism should be easier to understand and use than
implementing a custom metaclass, and thus should provide a gentler
introduction to the full power Python's metaclass machinery.


PEP Withdrawal
==============

This proposal has been withdrawn in favour of Martin Teichmann's proposal
in :pep:`487`, which achieves the same goals through a simpler, easier to use
``__init_subclass__`` hook that simply isn't invoked for the base class
that defines the hook.


Background
==========

For an already created class ``cls``, the term "metaclass" has a clear
meaning: it is the value of ``type(cls)``.

*During* class creation, it has another meaning: it is also used to refer to
the metaclass hint that may be provided as part of the class definition.
While in many cases these two meanings end up referring to one and the same
object, there are two situations where that is not the case:

* If the metaclass hint refers to an instance of ``type``, then it is
  considered as a candidate metaclass along with the metaclasses of all of
  the parents of the class being defined. If a more appropriate metaclass is
  found amongst the candidates, then it will be used instead of the one
  given in the metaclass hint.
* Otherwise, an explicit metaclass hint is assumed to be a factory function
  and is called directly to create the class object. In this case, the final
  metaclass will be determined by the factory function definition. In the
  typical case (where the factory functions just calls ``type``, or, in
  Python 3.3 or later, ``types.new_class``) the actual metaclass is then
  determined based on the parent classes.

It is notable that only the actual metaclass is inherited - a factory
function used as a metaclass hook sees only the class currently being
defined, and is not invoked for any subclasses.

In Python 3, the metaclass hint is provided using the ``metaclass=Meta``
keyword syntax in the class header. This allows the ``__prepare__`` method
on the metaclass to be used to create the ``locals()`` namespace used during
execution of the class body (for example, specifying the use of
``collections.OrderedDict`` instead of a regular ``dict``).

In Python 2, there was no ``__prepare__`` method (that API was added for
Python 3 by :pep:`3115`). Instead, a class body could set the ``__metaclass__``
attribute, and the class creation process would extract that value from the
class namespace to use as the metaclass hint. There is `published code`_ that
makes use of this feature.

Another new feature in Python 3 is the zero-argument form of the ``super()``
builtin, introduced by :pep:`3135`. This feature uses an implicit ``__class__``
reference to the class being defined to replace the "by name" references
required in Python 2. Just as code invoked during execution of a Python 2
metaclass could not call methods that referenced the class by name (as the
name had not yet been bound in the containing scope), similarly, Python 3
metaclasses cannot call methods that rely on the implicit ``__class__``
reference (as it is not populated until after the metaclass has returned
control to the class creation machinery).

Finally, when a class uses a custom metaclass, it can pose additional
challenges to the use of multiple inheritance, as a new class cannot
inherit from parent classes with unrelated metaclasses. This means that
it is impossible to add a metaclass to an already published class: such
an addition is a backwards incompatible change due to the risk of metaclass
conflicts.


Proposal
========

This PEP proposes that a new mechanism to customise class creation be
added to Python 3.4 that meets the following criteria:

1. Integrates nicely with class inheritance structures (including mixins and
   multiple inheritance)
2. Integrates nicely with the implicit ``__class__`` reference and
   zero-argument ``super()`` syntax introduced by :pep:`3135`
3. Can be added to an existing base class without a significant risk of
   introducing backwards compatibility problems
4. Restores the ability for class namespaces to have some influence on the
   class creation process (above and beyond populating the namespace itself),
   but potentially without the full flexibility of the Python 2 style
   ``__metaclass__`` hook

One mechanism that can achieve this goal is to add a new implicit class
decoration hook, modelled directly on the existing explicit class
decorators, but defined in the class body or in a parent class, rather than
being part of the class definition header.

Specifically, it is proposed that class definitions be able to provide a
class initialisation hook as follows::

   class Example:
       def __autodecorate__(cls):
           # This is invoked after the class is created, but before any
           # explicit decorators are called
           # The usual super() mechanisms are used to correctly support
           # multiple inheritance. The class decorator style signature helps
           # ensure that invoking the parent class is as simple as possible.
           cls = super().__autodecorate__()
           return cls

To simplify the cooperative multiple inheritance case, ``object`` will gain
a default implementation of the hook that returns the class unmodified::

   class object:
       def __autodecorate__(cls):
           return cls

If a metaclass wishes to block implicit class decoration for some reason, it
must arrange for ``cls.__autodecorate__`` to trigger ``AttributeError``.

If present on the created object, this new hook will be called by the class
creation machinery *after* the ``__class__`` reference has been initialised.
For ``types.new_class()``, it will be called as the last step before
returning the created class object. ``__autodecorate__`` is implicitly
converted to a class method when the class is created (prior to the hook
being invoked).

Note, that when ``__autodecorate__`` is called, the name of the class is not
yet bound to the new class object. As a consequence, the two argument form
of ``super()`` cannot be used to call methods (e.g., ``super(Example, cls)``
wouldn't work in the example above). However, the zero argument form of
``super()`` works as expected, since the ``__class__`` reference is already
initialised.

This general proposal is not a new idea (it was first suggested for
inclusion in the language definition `more than 10 years ago`_, and a
similar mechanism has long been supported by `Zope's ExtensionClass`_),
but the situation has changed sufficiently in recent years that
the idea is worth reconsidering for inclusion as a native language feature.

In addition, the introduction of the metaclass ``__prepare__`` method in PEP
3115 allows a further enhancement that was not possible in Python 2: this
PEP also proposes that ``type.__prepare__`` be updated to accept a factory
function as a ``namespace`` keyword-only argument. If present, the value
provided as the ``namespace`` argument will be called without arguments
to create the result of ``type.__prepare__`` instead of using a freshly
created dictionary instance. For example, the following will use
an ordered dictionary as the class namespace::

   class OrderedExample(namespace=collections.OrderedDict):
       def __autodecorate__(cls):
           # cls.__dict__ is still a read-only proxy to


id: pep-0436.rst
content:
PEP: 436
Title: The Argument Clinic DSL
Author: Larry Hastings <larry@hastings.org>
Discussions-To: python-dev@python.org
Status: Final
Type: Standards Track
Created: 22-Feb-2013
Python-Version: 3.4


Abstract
========

This document proposes "Argument Clinic", a DSL to facilitate
argument processing for built-in functions in the implementation of
CPython.


Rationale and Goals
===================

The primary implementation of Python, "CPython", is written in a
mixture of Python and C.  One implementation detail of CPython
is what are called "built-in" functions -- functions available to
Python programs but written in C.  When a Python program calls a
built-in function and passes in arguments, those arguments must be
translated from Python values into C values.  This process is called
"parsing arguments".

As of CPython 3.3, builtin functions nearly always parse their arguments
with one of two functions: the original ``PyArg_ParseTuple()``, [1]_ and
the more modern ``PyArg_ParseTupleAndKeywords()``. [2]_ The former
only handles positional parameters; the latter also accommodates keyword
and keyword-only parameters, and is preferred for new code.

With either function, the caller specifies the translation for
parsing arguments in a "format string": [3]_ each parameter corresponds
to a "format unit", a short character sequence telling the parsing
function what Python types to accept and how to translate them into
the appropriate C value for that parameter.


``PyArg_ParseTuple()`` was reasonable when it was first conceived.
There were only a dozen or so of these "format units"; each one
was distinct, and easy to understand and remember.
But over the years the ``PyArg_Parse`` interface has been extended
in numerous ways.  The modern API is complex, to the point that it
is somewhat painful to use.  Consider:

* There are now forty different "format units"; a few are even three
  characters long.  This makes it difficult for the programmer to
  understand what the format string says--or even perhaps to parse
  it--without constantly cross-indexing it with the documentation.
* There are also six meta-format units that may be buried in the
  format string.  (They are: ``"()|$:;"``.)
* The more format units are added, the less likely it is the
  implementer can pick an easy-to-use mnemonic for the format unit,
  because the character of choice is probably already in use.  In
  other words, the more format units we have, the more obtuse the
  format units become.
* Several format units are nearly identical to others, having only
  subtle differences.  This makes understanding the exact semantics
  of the format string even harder, and can make it difficult to
  figure out exactly which format unit you want.
* The docstring is specified as a static C string, making it mildly
  bothersome to read and edit since it must obey C string quoting rules.
* When adding a new parameter to a function using
  ``PyArg_ParseTupleAndKeywords()``, it's necessary to touch six
  different places in the code: [4]_

  * Declaring the variable to store the argument.
  * Passing in a pointer to that variable in the correct spot in
    ``PyArg_ParseTupleAndKeywords()``, also passing in any
    "length" or "converter" arguments in the correct order.
  * Adding the name of the argument in the correct spot of the
    "keywords" array passed in to
    ``PyArg_ParseTupleAndKeywords()``.
  * Adding the format unit to the correct spot in the format
    string.
  * Adding the parameter to the prototype in the docstring.
  * Documenting the parameter in the docstring.

* There is currently no mechanism for builtin functions to provide
  their "signature" information (see ``inspect.getfullargspec`` and
  ``inspect.Signature``).  Adding this information using a mechanism
  similar to the existing ``PyArg_Parse`` functions would require
  repeating ourselves yet again.

The goal of Argument Clinic is to replace this API with a mechanism
inheriting none of these downsides:

* You need specify each parameter only once.
* All information about a parameter is kept together in one place.
* For each parameter, you specify a conversion function; Argument
  Clinic handles the translation from Python value into C value for
  you.
* Argument Clinic also allows for fine-tuning of argument processing
  behavior with parameterized conversion functions.
* Docstrings are written in plain text.  Function docstrings are
  required; per-parameter docstrings are encouraged.
* From this, Argument Clinic generates for you all the mundane,
  repetitious code and data structures CPython needs internally.
  Once you've specified the interface, the next step is simply to
  write your implementation using native C types.  Every detail of
  argument parsing is handled for you.

Argument Clinic is implemented as a preprocessor.  It draws inspiration
for its workflow directly from [Cog]_ by Ned Batchelder.  To use Clinic,
add a block comment to your C source code beginning and ending with
special text strings, then run Clinic on the file.  Clinic will find the
block comment, process the contents, and write the output back into your
C source file directly after the comment.  The intent is that Clinic's
output becomes part of your source code; it's checked in to revision
control, and distributed with source packages.  This means that Python
will still ship ready-to-build.  It does complicate development slightly;
in order to add a new function, or modify the arguments or documentation
of an existing function using Clinic, you'll need a working Python 3
interpreter.

Future goals of Argument Clinic include:

* providing signature information for builtins,
* enabling alternative implementations of Python to create
  automated library compatibility tests, and
* speeding up argument parsing with improvements to the
  generated code.


DSL Syntax Summary
==================

The Argument Clinic DSL is specified as a comment embedded in a C
file, as follows.  The "Example" column on the right shows you sample
input to the Argument Clinic DSL, and the "Section" column on the left
specifies what each line represents in turn.

Argument Clinic's DSL syntax mirrors the Python ``def``
statement, lending it some familiarity to Python core developers.

::

   +-----------------------+-----------------------------------------------------------------+
   | Section               | Example                                                         |
   +-----------------------+-----------------------------------------------------------------+
   | Clinic DSL start      | /*[clinic]                                                      |
   | Module declaration    | module module_name                                              |
   | Class declaration     | class module_name.class_name                                    |
   | Function declaration  | module_name.function_name  -> return_annotation                 |
   | Parameter declaration |       name : converter(param=value)                             |
   | Parameter docstring   |           Lorem ipsum dolor sit amet, consectetur               |
   |                       |           adipisicing elit, sed do eiusmod tempor               |
   | Function docstring    | Lorem ipsum dolor sit amet, consectetur adipisicing             |
   |                       | elit, sed do eiusmod tempor incididunt ut labore et             |
   | Clinic DSL end        | [clinic]*/                                                      |
   | Clinic output         | ...                                                             |
   | Clinic output end     | /*[clinic end output:<checksum>]*/                              |
   +-----------------------+-----------------------------------------------------------------+

To give some flavor of the proposed DSL syntax, here are some sample Clinic
code blocks.  This first block reflects the normally preferred style, including
blank lines between parameters and p


id: pep-0350.rst
content:
PEP: 350
Title: Codetags
Author: Micah Elliott <mde at tracos.org>
Status: Rejected
Type: Informational
Created: 27-Jun-2005
Post-History: 10-Aug-2005, 26-Sep-2005


Rejection Notice
================

This PEP has been rejected. While the community may be interested,
there is no desire to make the standard library conform to this standard.


Abstract
========

This informational PEP aims to provide guidelines for consistent use
of *codetags*, which would enable the construction of standard
utilities to take advantage of the codetag information, as well as
making Python code more uniform across projects.  Codetags also
represent a very lightweight programming micro-paradigm and become
useful for project management, documentation, change tracking, and
project health monitoring.  This is submitted as a PEP because its
ideas are thought to be Pythonic, although the concepts are not unique
to Python programming.  Herein are the definition of codetags, the
philosophy behind them, a motivation for standardized conventions,
some examples, a specification, a toolset description, and possible
objections to the Codetag project/paradigm.

This PEP is also living as a wiki_ for people to add comments.


What Are Codetags?
==================

Programmers widely use ad-hoc code comment markup conventions to serve
as reminders of sections of code that need closer inspection or
review.  Examples of markup include ``FIXME``, ``TODO``, ``XXX``,
``BUG``, but there many more in wide use in existing software.  Such
markup will henceforth be referred to as *codetags*.  These codetags
may show up in application code, unit tests, scripts, general
documentation, or wherever suitable.

Codetags have been under discussion and in use (hundreds of codetags
in the Python 2.4 sources) in many places (e.g., c2_) for many years.
See References_ for further historic and current information.


Philosophy
==========

If you subscribe to most of these values, then codetags will likely be
useful for you.

1. As much information as possible should be contained **inside the
   source code** (application code or unit tests).  This along with
   use of codetags impedes duplication.  Most documentation can be
   generated from that source code; e.g., by using help2man, man2html,
   docutils, epydoc/pydoc, ctdoc, etc.

2. Information should be almost **never duplicated** -- it should be
   recorded in a single original format and all other locations should
   be automatically generated from the original, or simply be
   referenced.  This is famously known as the Single Point Of
   Truth (SPOT) or Don't Repeat Yourself (DRY) rule.

3. Documentation that gets into customers' hands should be
   **auto-generated** from single sources into all other output
   formats.  People want documentation in many forms.  It is thus
   important to have a documentation system that can generate all of
   these.

4. The **developers are the documentation team**.  They write the code
   and should know the code the best.  There should not be a
   dedicated, disjoint documentation team for any non-huge project.

5. **Plain text** (with non-invasive markup) is the best format for
   writing anything.  All other formats are to be generated from the
   plain text.

Codetag design was influenced by the following goals:

A. Comments should be short whenever possible.

B. Codetag fields should be optional and of minimal length.  Default
   values and custom fields can be set by individual code shops.

C. Codetags should be minimalistic.  The quicker it is to jot
   something down, the more likely it is to get jotted.

D. The most common use of codetags will only have zero to two fields
   specified, and these should be the easiest to type and read.


Motivation
==========

* **Various productivity tools can be built around codetags.**

  See Tools_.

* **Encourages consistency.**

  Historically, a subset of these codetags has been used informally in
  the majority of source code in existence, whether in Python or in
  other languages.  Tags have been used in an inconsistent manner with
  different spellings, semantics, format, and placement.  For example,
  some programmers might include datestamps and/or user identifiers,
  limit to a single line or not, spell the codetag differently than
  others, etc.

* **Encourages adherence to SPOT/DRY principle.**

  E.g., generating a roadmap dynamically from codetags instead of
  keeping TODOs in sync with separate roadmap document.

* **Easy to remember.**

  All codetags must be concise, intuitive, and semantically
  non-overlapping with others.  The format must also be simple.

* **Use not required/imposed.**

  If you don't use codetags already, there's no obligation to start,
  and no risk of affecting code (but see Objections_).  A small subset
  can be adopted and the Tools_ will still be useful (a few codetags
  have probably already been adopted on an ad-hoc basis anyway).  Also
  it is very easy to identify and remove (and possibly record) a
  codetag that is no longer deemed useful.

* **Gives a global view of code.**

  Tools can be used to generate documentation and reports.

* **A logical location for capturing CRCs/Stories/Requirements.**

  The XP community often does not electronically capture Stories, but
  codetags seem like a good place to locate them.

* **Extremely lightweight process.**

  Creating tickets in a tracking system for every thought degrades
  development velocity.  Even if a ticketing system is employed,
  codetags are useful for simply containing links to those tickets.


Examples
========

This shows a simple codetag as commonly found in sources everywhere
(with the addition of a trailing ``<>``)::

    # FIXME: Seems like this loop should be finite. <>
    while True: ...

The following contrived example demonstrates a typical use of
codetags.  It uses some of the available fields to specify the
assignees (a pair of programmers with initials *MDE* and *CLE*), the
Date of expected completion (*Week 14*), and the Priority of the item
(*2*)::

    # FIXME: Seems like this loop should be finite. <MDE,CLE d:14w p:2>
    while True: ...

This codetag shows a bug with fields describing author, discovery
(origination) date, due date, and priority::

    # BUG: Crashes if run on Sundays.
    # <MDE 2005-09-04 d:14w p:2>
    if day == 'Sunday': ...

Here is a demonstration of how not to use codetags.  This has many
problems: 1) Codetags cannot share a line with code; 2) Missing colon
after mnemonic; 3) A codetag referring to codetags is usually useless,
and worse, it is not completable; 4) No need to have a bunch of fields
for a trivial codetag; 5) Fields with unknown values (``t:XXX``)
should not be used::

    i = i + 1   # TODO Add some more codetags.
    # <JRNewbie 2005-04-03 d:2005-09-03 t:XXX d:14w p:0 s:inprogress>


Specification
=============

This describes the format: syntax, mnemonic names, fields, and
semantics, and also the separate DONE File.


General Syntax
--------------

Each codetag should be inside a comment, and can be any number of
lines.  It should not share a line with code.  It should match the
indentation of surrounding code.  The end of the codetag is marked by
a pair of angle brackets ``<>`` containing optional fields, which must
not be split onto multiple lines.  It is preferred to have a codetag
in ``#`` comments instead of string comments.  There can be multiple
fields per codetag, all of which are optional.

.. NOTE: It may be reasonable to allow fields to fit on multiple
   lines, but it complicates parsing and defeats minimalism if you
   use this many fields.

In short, a codetag consists of a mnemonic, a colon, commentary text,
an opening angle bracket, an optional list of fields, and a closing
angle bracket.  E.g., ::

    # MNEMONIC: Some (maybe multi-line) commentary. <field field ...>


Mnemonics
---------

The codetags of interest are listed below, using the following format:

| ``recommended mnemonic (& 


id: pep-0378.rst
content:
PEP: 378
Title: Format Specifier for Thousands Separator
Author: Raymond Hettinger <python@rcn.com>
Status: Final
Type: Standards Track
Created: 12-Mar-2009
Python-Version: 2.7, 3.1
Post-History: 12-Mar-2009


Motivation
==========

Provide a simple, non-locale aware way to format a number
with a thousands separator.

Adding thousands separators is one of the simplest ways to
humanize a program's output, improving its professional appearance
and readability.

In the finance world, output with thousands separators is the norm.
Finance users and non-professional programmers find the locale
approach to be frustrating, arcane and non-obvious.

The locale module presents two other challenges.  First, it is
a global setting and not suitable for multi-threaded apps that
need to serve-up requests in multiple locales.  Second, the
name of a relevant locale (such as "de_DE") can vary from
platform to platform or may not be defined at all.  The docs
for the locale module describe these and `many other challenges`_
in detail.

.. _`many other challenges`:  https://docs.python.org/2.6/library/locale.html#background-details-hints-tips-and-caveats

It is not the goal to replace the locale module, to perform
internationalization tasks, or accommodate every possible
convention.  Such tasks are better suited to robust tools like
`Babel`_. Instead, the goal is to make a common, everyday
task easier for many users.

.. _`Babel`: http://babel.edgewall.org/


Main Proposal (from Alyssa Coghlan, originally called Proposal I)
=================================================================

A comma will be added to the format() specifier mini-language::

[[fill]align][sign][#][0][width][,][.precision][type]

The ',' option indicates that commas should be included in the
output as a thousands separator. As with locales which do not
use a period as the decimal point, locales which use a
different convention for digit separation will need to use the
locale module to obtain appropriate formatting.

The proposal works well with floats, ints, and decimals.
It also allows easy substitution for other separators.
For example::

  format(n, "6,d").replace(",", "_")

This technique is completely general but it is awkward in the
one case where the commas and periods need to be swapped::

  format(n, "6,f").replace(",", "X").replace(".", ",").replace("X", ".")

The *width* argument means the total length including the commas
and decimal point::

  format(1234, "08,d")     -->    '0001,234'
  format(1234.5, "08,.1f") -->    '01,234.5'

The ',' option is defined as shown above for types 'd', 'e',
'f', 'g', 'E', 'G', '%', 'F' and ''. To allow future extensions, it is
undefined for other types: binary, octal, hex, character,
etc.

This proposal has the virtue of being simpler than the alternative
proposal but is much less flexible and meets the needs of fewer
users right out of the box.  It is expected that some other
solution will arise for specifying alternative separators.


Current Version of the Mini-Language
====================================

* `Python 2.6 docs`_

  .. _Python 2.6 docs: https://docs.python.org/2.6/library/string.html#formatstrings

* :pep:`3101` Advanced String Formatting


Research into what Other Languages Do
=====================================

Scanning the web, I've found that thousands separators are
usually one of COMMA, DOT, SPACE, APOSTROPHE or UNDERSCORE.

`C-Sharp`_ provides both styles (picture formatting and type specifiers).
The type specifier approach is locale aware.  The picture formatting only
offers a COMMA as a thousands separator::

    String.Format("{0:n}", 12400)     ==>    "12,400"
    String.Format("{0:0,0}", 12400)   ==>    "12,400"

.. _`C-Sharp`: http://blog.stevex.net/index.php/string-formatting-in-csharp/

`Common Lisp`_ uses a COLON before the ``~D`` decimal type specifier to
emit a COMMA as a thousands separator.  The  general form of ``~D`` is
``~mincol,padchar,commachar,commaintervalD``.  The *padchar* defaults
to SPACE.  The *commachar* defaults to COMMA.  The *commainterval*
defaults to three.

::

    (format nil "~:D" 229345007)   =>   "229,345,007"

.. _`Common Lisp`: http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node200.html


* The `ADA language`_ allows UNDERSCORES in its numeric literals.

.. _`ADA language`: http://archive.adaic.com/standards/83lrm/html/lrm-02-04.html

Visual Basic and its brethren (like `MS Excel`_) use a completely
different style and have ultra-flexible custom format
specifiers like::

    "_($* #,##0_)".

.. _`MS Excel`: http://www.brainbell.com/tutorials/ms-office/excel/Create_Custom_Number_Formats.htm

`COBOL`_ uses picture clauses like::

    PICTURE $***,**9.99CR

.. _`COBOL`: http://en.wikipedia.org/wiki/Cobol#Syntactic_features

Java offers a `Decimal.Format Class`_ that uses picture patterns (one
for positive numbers and an optional one for negatives) such as:
``"#,##0.00;(#,##0.00)"``. It allows arbitrary groupings including
hundreds and ten-thousands and uneven groupings.  The special pattern
characters are non-localized (using a DOT for a decimal separator and
a COMMA for a grouping separator).  The user can supply an alternate
set of symbols using the formatter's *DecimalFormatSymbols* object.

.. _`Decimal.Format Class`: http://java.sun.com/javase/6/docs/api/java/text/DecimalFormat.html


Alternative Proposal (from Eric Smith, originally called Proposal II)
=====================================================================

Make both the thousands separator and decimal separator user
specifiable but not locale aware.  For simplicity, limit the
choices to a COMMA, DOT, SPACE, APOSTROPHE or UNDERSCORE.
The SPACE can be either U+0020 or U+00A0.

Whenever a separator is followed by a precision, it is a
decimal separator and an optional separator preceding it is a
thousands separator.  When the precision is absent, a lone
specifier means a thousands separator::

[[fill]align][sign][#][0][width][tsep][dsep precision][type]

Examples::

  format(1234, "8.1f")     -->    '  1234.0'
  format(1234, "8,1f")     -->    '  1234,0'
  format(1234, "8.,1f")    -->    ' 1.234,0'
  format(1234, "8 ,f")     -->    ' 1 234,0'
  format(1234, "8d")       -->    '    1234'
  format(1234, "8,d")      -->    '   1,234'
  format(1234, "8_d")      -->    '   1_234'

This proposal meets mosts needs, but it comes at the expense
of taking a bit more effort to parse.  Not every possible
convention is covered, but at least one of the options (spaces
or underscores) should be readable, understandable, and useful
to folks from many diverse backgrounds.

As shown in the examples, the *width* argument means the total
length including the thousands separators and decimal separators.

No change is proposed for the locale module.

The thousands separator is defined as shown above for types
'd', 'e', 'f', 'g', '%', 'E', 'G' and 'F'. To allow future
extensions, it is undefined for other types: binary, octal,
hex, character, etc.

The drawback to this alternative proposal is the difficulty
of mentally parsing whether a single separator is a thousands
separator or decimal separator.  Perhaps it is too arcane
to link the decimal separator with the precision specifier.


Commentary
==========

* Some commenters do not like the idea of format strings at all
  and find them to be unreadable.  Suggested alternatives include
  the COBOL style PICTURE approach or a convenience function with
  keyword arguments for every possible combination.

* Some newsgroup respondants think there is no place for any
  scripts that are not internationalized and that it is a step
  backwards to provide a simple way to hardwire a particular choice
  (thus reducing incentive to use a locale sensitive approach).

* Another thought is that embedding some particular convention in
  individual format strings makes it hard to change that convention
  later.  No workable alternative was suggested but the general idea
  is to set the convention once and h


id: pep-0387.rst
content:
PEP: 387
Title: Backwards Compatibility Policy
Author: Benjamin Peterson <benjamin@python.org>
PEP-Delegate: Brett Cannon <brett@python.org>
Status: Active
Type: Process
Created: 18-Jun-2009
Post-History: `19-Jun-2009 <https://mail.python.org/archives/list/python-dev@python.org/thread/YAZQHFUCYEYIX5MIG6LFJLCVQ5ORVUM6/>`__,
              `12-Jun-2020 <https://discuss.python.org/t/pep-387-backwards-compatibilty-policy/4421>`__,
              `19-Dec-2022 <https://discuss.python.org/t/22042>`__,
              `16-Jun-2023 <https://discuss.python.org/t/formalize-the-concept-of-soft-deprecation-dont-schedule-removal-in-pep-387-backwards-compatibility-policy/27957>`__
Replaces: 291


Abstract
========

This PEP outlines Python's backwards compatibility policy.


Rationale
=========

As one of the most used programming languages today [#tiobe]_, the
Python core language and its standard library play a critical role in
millions of applications and libraries. This is fantastic. However, it
means the development team must be very careful not to break this
existing 3rd party code with new releases.

This PEP takes the perspective that "backwards incompatibility" means
preexisting code ceases to comparatively function after a change. It is
acknowledged that this is not a concrete definition, but the expectation
is people in general understand what is meant by
"backwards incompatibility", and if they are unsure they may ask the
Python development team and/or steering council for guidance.


Backwards Compatibility Rules
=============================

This policy applies to all public APIs.  These include:

- Syntax and behavior of these constructs as defined by the reference
  manual.

- The C-API.

- Function, class, module, attribute, and method names and types.

- Given a set of arguments, the return value, side effects, and raised
  exceptions of a function.  This does not preclude changes from
  reasonable bug fixes.

- The position and expected types of arguments and returned values.

- Behavior of classes with regards to subclasses: the conditions under
  which overridden methods are called.

- Documented exceptions and the semantics which lead to their raising.

- Exceptions commonly raised in EAFP scenarios.

Others are explicitly not part of the public API.  They can change or
be removed at any time in any way.  These include:

- Function, class, module, attribute, method, and C-API names and
  types that are prefixed by "_" (except special names).

- Anything documented publicly as being private.
  Note that if something is not documented at all, it is *not*
  automatically considered private.

- Imported modules (unless explicitly documented as part of the public
  API; e.g. importing the ``bacon`` module in the ``spam`` does not
  automatically mean ``spam.bacon`` is part of the public API unless
  it is documented as such).

- Inheritance patterns of internal classes.

- Test suites.  (Anything in the ``Lib/test`` directory or test
  subdirectories of packages.)

- Backward compatibility rules do not apply to any module or API that is
  explicitly documented as **Provisional** per :pep:`411`.


Basic policy for backwards compatibility
----------------------------------------

* In general, incompatibilities should have a large benefit to
  breakage ratio, and the incompatibility should be easy to resolve in
  affected code.  For example, adding an stdlib module with the same
  name as a third party package is generally not acceptable.  Adding
  a method or attribute that conflicts with 3rd party code through
  inheritance, however, is likely reasonable.

* Unless it is going through the deprecation process below, the
  behavior of an API *must* not change in an incompatible fashion
  between any two consecutive releases.  Python's yearly release
  process (:pep:`602`) means that the deprecation period must last at
  least two years.

* Similarly a feature cannot be removed without notice between any two
  consecutive releases.

* For changes that are unable to raise a deprecation warning, consult
  with the steering council.

* The steering council may grant exceptions to this policy. In
  particular, they may shorten the required deprecation period for a
  feature. Exceptions are only granted for extreme situations such as
  dangerously broken or insecure features or features no one could
  reasonably be depending on (e.g., support for completely obsolete
  platforms).


Soft Deprecation
================

A soft deprecation can be used when using an API which should no longer
be used to write new code, but it remains safe to continue using it in
existing code. The API remains documented and tested, but will not be
developed further (no enhancement).

The main difference between a "soft" and a (regular) "hard" deprecation
is that the soft deprecation does not imply scheduling the removal of
the deprecated API.

Another difference is that a soft deprecation does not issue a warning:
it's only mentioned in the documentation, whereas usually a "hard"
deprecation issues a ``DeprecationWarning`` warning at runtime. The
documentation of a soft deprecation should explain why the API should be
avoided, and if possible propose a replacement.

If the decision is made to deprecate (in the regular sense) a feature
that is currently soft deprecated, the deprecation must follow the
`Backwards Compatibility Rules`_ (i.e., there is no exception because
the feature is already soft deprecated).


Making Incompatible Changes
===========================

Making an incompatible change is a gradual process performed over
several releases:

1. Discuss the change.  Depending on the degree of incompatibility,
   this could be on the bug tracker, python-dev, python-list, or the
   appropriate SIG.  A PEP or similar document may be written.
   Hopefully users of the affected API will pipe up to comment.

2. Add a warning to the current ``main`` branch.
   If behavior is changing, the API may gain a new
   function or method to perform the new behavior; old usage should
   raise the warning.  If an API is being removed, simply warn
   whenever it is entered.  ``DeprecationWarning`` is the usual
   warning category to use, but ``PendingDeprecationWarning`` may be
   used in special cases where the old and new versions of the API will
   coexist for many releases [#warnings]_.  The warning message should
   include the release the incompatibility is expected to become the
   default and a link to an issue that users can post feedback to.
   When feasible, also change `typeshed <https://github.com/python/typeshed>`_
   to add the ``@deprecated`` decorator (see :pep:`702`) to the deprecated API,
   so that users of static type checkers have another way to learn
   about the deprecation.

   For C API, a compiler warning generated by the ``Py_DEPRECATED`` macro
   is also acceptable.

3. Wait for the warning to appear in at least two minor Python
   versions of the same major version, or one minor version in an older
   major version (e.g., for a warning in Python 3.10.0, you either wait
   until at least Python 3.12 or Python 4.0 to make the change). It is
   preferred, though, to wait 5 years before removal (e.g., warn starting in
   Python 3.10, removal in 3.15; this happens to coincide with the current
   lifetime of a minor release of Python).

   - If the expected maintenance overhead and security risk of the
     deprecated behavior is small (e.g. an old function is reimplemented
     in terms of a new, more general one), it can stay indefinitely
     (or until the situation changes).

   - If the deprecated feature is replaced by a new one, it should
     generally be removed only after the last Python version
     *without* the new feature reaches end of support.

4. See if there's any feedback.  Users not involved in the original
   discussions may comment now after seeing the warning.  Perhaps
   reconsider.

5. The behavior change or feature removal may now be made de


id: pep-0393.rst
content:
PEP: 393
Title: Flexible String Representation
Author: Martin von LÃ¶wis <martin@v.loewis.de>
Status: Final
Type: Standards Track
Created: 24-Jan-2010
Python-Version: 3.3
Post-History:

Abstract
========

The Unicode string type is changed to support multiple internal
representations, depending on the character with the largest Unicode
ordinal (1, 2, or 4 bytes). This will allow a space-efficient
representation in common cases, but give access to full UCS-4 on all
systems. For compatibility with existing APIs, several representations
may exist in parallel; over time, this compatibility should be phased
out. The distinction between narrow and wide Unicode builds is
dropped.  An implementation of this PEP is available at [1]_.

Rationale
=========

There are two classes of complaints about the current implementation
of the unicode type: on systems only supporting UTF-16, users complain
that non-BMP characters are not properly supported. On systems using
UCS-4 internally (and also sometimes on systems using UCS-2), there is
a complaint that Unicode strings take up too much memory - especially
compared to Python 2.x, where the same code would often use ASCII
strings (i.e. ASCII-encoded byte strings). With the proposed approach,
ASCII-only Unicode strings will again use only one byte per character;
while still allowing efficient indexing of strings containing non-BMP
characters (as strings containing them will use 4 bytes per
character).

One problem with the approach is support for existing applications
(e.g. extension modules). For compatibility, redundant representations
may be computed. Applications are encouraged to phase out reliance on
a specific internal representation if possible. As interaction with
other libraries will often require some sort of internal
representation, the specification chooses UTF-8 as the recommended way
of exposing strings to C code.

For many strings (e.g. ASCII), multiple representations may actually
share memory (e.g. the shortest form may be shared with the UTF-8 form
if all characters are ASCII). With such sharing, the overhead of
compatibility representations is reduced. If representations do share
data, it is also possible to omit structure fields, reducing the base
size of string objects.

Specification
=============

Unicode structures are now defined as a hierarchy of structures,
namely::

  typedef struct {
    PyObject_HEAD
    Py_ssize_t length;
    Py_hash_t hash;
    struct {
        unsigned int interned:2;
        unsigned int kind:2;
        unsigned int compact:1;
        unsigned int ascii:1;
        unsigned int ready:1;
    } state;
    wchar_t *wstr;
  } PyASCIIObject;

  typedef struct {
    PyASCIIObject _base;
    Py_ssize_t utf8_length;
    char *utf8;
    Py_ssize_t wstr_length;
  } PyCompactUnicodeObject;

  typedef struct {
    PyCompactUnicodeObject _base;
    union {
        void *any;
        Py_UCS1 *latin1;
        Py_UCS2 *ucs2;
        Py_UCS4 *ucs4;
    } data;
  } PyUnicodeObject;

Objects for which both size and maximum character are known at
creation time are called "compact" unicode objects; character data
immediately follow the base structure. If the maximum character is
less than 128, they use the PyASCIIObject structure, and the UTF-8
data, the UTF-8 length and the wstr length are the same as the length
of the ASCII data. For non-ASCII strings, the PyCompactObject
structure is used. Resizing compact objects is not supported.

Objects for which the maximum character is not given at creation time
are called "legacy" objects, created through
PyUnicode_FromStringAndSize(NULL, length). They use the
PyUnicodeObject structure. Initially, their data is only in the wstr
pointer; when PyUnicode_READY is called, the data pointer (union) is
allocated. Resizing is possible as long PyUnicode_READY has not been
called.

The fields have the following interpretations:

- length: number of code points in the string (result of sq_length)
- interned: interned-state (SSTATE_*) as in 3.2
- kind: form of string

  + 00 => str is not initialized (data are in wstr)
  + 01 => 1 byte (Latin-1)
  + 10 => 2 byte (UCS-2)
  + 11 => 4 byte (UCS-4);

- compact: the object uses one of the compact representations
  (implies ready)
- ascii: the object uses the PyASCIIObject representation
  (implies compact and ready)
- ready: the canonical representation is ready to be accessed through
  PyUnicode_DATA and PyUnicode_GET_LENGTH. This is set either if the
  object is compact, or the data pointer and length have been
  initialized.
- wstr_length, wstr: representation in platform's wchar_t
  (null-terminated). If wchar_t is 16-bit, this form may use surrogate
  pairs (in which cast wstr_length differs form length).
  wstr_length differs from length only if there are surrogate pairs
  in the representation.
- utf8_length, utf8: UTF-8 representation (null-terminated).
- data: shortest-form representation of the unicode string.
  The string is null-terminated (in its respective representation).

All three representations are optional, although the data form is
considered the canonical representation which can be absent only
while the string is being created. If the representation is absent,
the pointer is NULL, and the corresponding length field may contain
arbitrary data.

The Py_UNICODE type is still supported but deprecated. It is always
defined as a typedef for wchar_t, so the wstr representation can double
as Py_UNICODE representation.

The data and utf8 pointers point to the same memory if the string uses
only ASCII characters (using only Latin-1 is not sufficient). The data
and wstr pointers point to the same memory if the string happens to
fit exactly to the wchar_t type of the platform (i.e. uses some
BMP-not-Latin-1 characters if sizeof(wchar_t) is 2, and uses some
non-BMP characters if sizeof(wchar_t) is 4).

String Creation
---------------

The recommended way to create a Unicode object is to use the function
PyUnicode_New::

   PyObject* PyUnicode_New(Py_ssize_t size, Py_UCS4 maxchar);

Both parameters must denote the eventual size/range of the strings.
In particular, codecs using this API must compute both the number of
characters and the maximum character in advance. A string is
allocated according to the specified size and character range and is
null-terminated; the actual characters in it may be uninitialized.

PyUnicode_FromString and PyUnicode_FromStringAndSize remain supported
for processing UTF-8 input; the input is decoded, and the UTF-8
representation is not yet set for the string.

PyUnicode_FromUnicode remains supported but is deprecated. If the
Py_UNICODE pointer is non-null, the data representation is set. If the
pointer is NULL, a properly-sized wstr representation is allocated,
which can be modified until PyUnicode_READY() is called (explicitly
or implicitly). Resizing a Unicode string remains possible until it
is finalized.

PyUnicode_READY() converts a string containing only a wstr
representation into the canonical representation. Unless wstr and data
can share the memory, the wstr representation is discarded after the
conversion. The macro returns 0 on success and -1 on failure, which
happens in particular if the memory allocation fails.

String Access
-------------

The canonical representation can be accessed using two macros
PyUnicode_Kind and PyUnicode_Data. PyUnicode_Kind gives one of the
values PyUnicode_WCHAR_KIND (0), PyUnicode_1BYTE_KIND (1),
PyUnicode_2BYTE_KIND (2), or PyUnicode_4BYTE_KIND (3). PyUnicode_DATA
gives the void pointer to the data. Access to individual characters
should use PyUnicode_{READ|WRITE}[_CHAR]:

- PyUnicode_READ(kind, data, index)
- PyUnicode_WRITE(kind, data, index, value)
- PyUnicode_READ_CHAR(unicode, index)

All these macros assume that the string is in canonical form;
callers need to ensure this by calling PyUnicode_READY.

A new function PyUnicode_AsUTF8 is provided to access the UTF-8
representation. It is thus identical to the existing
_PyUnicode


id: pep-0392.rst
content:
PEP: 392
Title: Python 3.2 Release Schedule
Author: Georg Brandl <georg@python.org>
Status: Final
Type: Informational
Topic: Release
Created: 30-Dec-2009
Python-Version: 3.2


Abstract
========

This document describes the development and release schedule for the
Python 3.2 series.  The schedule primarily concerns itself with PEP-sized
items.

.. Small features may be added up to and including the first beta
   release.  Bugs may be fixed until the final release, which is planned
   for February 2011.


Release Manager and Crew
========================

- 3.2 Release Manager: Georg Brandl
- Windows installers: Martin v. Loewis
- Mac installers: Ronald Oussoren
- Documentation: Georg Brandl


3.2 Lifespan
============

3.2 will receive bugfix updates approximately every 4-6 months for
approximately 18 months.  After the release of 3.3.0 final (see PEP
398), a final 3.2 bugfix update will be released.  After that,
security updates (source only) will be released until 5 years after
the release of 3.2 final, which was planned for February 2016.

As of 2016-02-20, Python 3.2.x reached end-of-life status.  The final
source release was 3.2.6 in October 2014.


Release Schedule
================

3.2 schedule
------------

- 3.2 alpha 1: August 1, 2010
- 3.2 alpha 2: September 6, 2010
- 3.2 alpha 3: October 12, 2010
- 3.2 alpha 4: November 16, 2010
- 3.2 beta 1: December 6, 2010

(No new features beyond this point.)

- 3.2 beta 2: December 20, 2010
- 3.2 candidate 1: January 16, 2011
- 3.2 candidate 2: January 31, 2011
- 3.2 candidate 3: February 14, 2011
- 3.2 final: February 20, 2011

.. don't forget to update final date above as well

3.2.1 schedule
--------------

- 3.2.1 beta 1: May 8, 2011
- 3.2.1 candidate 1: May 17, 2011
- 3.2.1 candidate 2: July 3, 2011
- 3.2.1 final: July 11, 2011

3.2.2 schedule
--------------

- 3.2.2 candidate 1: August 14, 2011
- 3.2.2 final: September 4, 2011

3.2.3 schedule
--------------

- 3.2.3 candidate 1: February 25, 2012
- 3.2.3 candidate 2: March 18, 2012
- 3.2.3 final: April 11, 2012

3.2.4 schedule
--------------

- 3.2.4 candidate 1: March 23, 2013
- 3.2.4 final: April 6, 2013

3.2.5 schedule (regression fix release)
---------------------------------------

- 3.2.5 final: May 13, 2013

-- Only security releases after 3.2.5 --

3.2.6 schedule
--------------

- 3.2.6 candidate 1 (source-only release): October 4, 2014
- 3.2.6 final (source-only release): October 11, 2014


Features for 3.2
================

Note that :pep:`3003` is in effect: no changes to language
syntax and no additions to the builtins may be made.

No large-scale changes have been recorded yet.


Copyright
=========

This document has been placed in the public domain.


id: pep-0386.rst
content:
PEP: 386
Title: Changing the version comparison module in Distutils
Author: Tarek ZiadÃ© <tarek@ziade.org>
Status: Superseded
Type: Standards Track
Topic: Packaging
Created: 04-Jun-2009
Superseded-By: 440


Abstract
========

Note: This PEP has been superseded by the version identification and
dependency specification scheme defined in :pep:`440`.

This PEP proposed a new version comparison schema system in Distutils.

Motivation
==========

In Python there are no real restrictions yet on how a project should manage its
versions, and how they should be incremented.

Distutils provides a ``version`` distribution meta-data field but it is freeform and
current users, such as PyPI usually consider the latest version pushed as the
``latest`` one, regardless of the expected semantics.

Distutils will soon extend its capabilities to allow distributions to express a
dependency on other distributions through the ``Requires-Dist`` metadata field
(see :pep:`345`) and it will optionally allow use of that field to
restrict the dependency to a set of compatible versions. Notice that this field
is replacing ``Requires`` that was expressing dependencies on modules and packages.

The ``Requires-Dist`` field will allow a distribution to define a dependency on
another package and optionally restrict this dependency to a set of
compatible versions, so one may write::

    Requires-Dist: zope.interface (>3.5.0)

This means that the distribution requires ``zope.interface`` with a version
greater than ``3.5.0``.

This also means that Python projects will need to follow the same convention
as the tool that will be used to install them, so they are able to compare
versions.

That is why this PEP proposes, for the sake of interoperability, a standard
schema to express version information and its comparison semantics.

Furthermore, this will make OS packagers' work easier when repackaging standards
compliant distributions, because as of now it can be difficult to decide how two
distribution versions compare.


Requisites and current status
=============================

It is not in the scope of this PEP to provide a universal versioning schema
intended to support all or even most of existing versioning schemas. There
will always be competing grammars, either mandated by distro or project
policies or by historical reasons that we cannot expect to change.

The proposed schema should be able to express the usual versioning semantics,
so it's possible to parse any alternative versioning schema and transform it
into a compliant one. This is how OS packagers usually deal with the existing
version schemas and is a preferable alternative than supporting an arbitrary
set of versioning schemas.

Conformance to usual practice and conventions, as well as a simplicity are a
plus, to ease frictionless adoption and painless transition. Practicality beats
purity, sometimes.

Projects have very different versioning needs, but the following are widely
considered important semantics:

1. it should be possible to express more than one versioning level
   (usually this is expressed as major and minor revision and, sometimes,
   also a micro revision).
2. a significant number of projects need special meaning versions for
   "pre-releases" (such as "alpha", "beta", "rc"), and these have widely
   used aliases ("a" stands for "alpha", "b" for "beta" and "c" for "rc").
   And these pre-release versions make it impossible to use a simple
   alphanumerical ordering of the version string components.
   (Example: 3.1a1 < 3.1)
3. some projects also need "post-releases" of regular versions,
   mainly for installer work which can't be clearly expressed otherwise.
4. development versions allow packagers of unreleased work to avoid version
   clash with later regular releases.

For people that want to go further and use a tool to manage their version
numbers, the two major ones are:

- The current Distutils system [#distutils]_
- Setuptools [#setuptools]_

Distutils
---------

Distutils currently provides a ``StrictVersion`` and a ``LooseVersion`` class
that can be used to manage versions.

The ``LooseVersion`` class is quite lax. From Distutils doc::

    Version numbering for anarchists and software realists.
    Implements the standard interface for version number classes as
    described above.  A version number consists of a series of numbers,
    separated by either periods or strings of letters.  When comparing
    version numbers, the numeric components will be compared
    numerically, and the alphabetic components lexically.  The following
    are all valid version numbers, in no particular order:

        1.5.1
        1.5.2b2
        161
        3.10a
        8.02
        3.4j
        1996.07.12
        3.2.pl0
        3.1.1.6
        2g6
        11g
        0.960923
        2.2beta29
        1.13++
        5.5.kw
        2.0b1pl0

    In fact, there is no such thing as an invalid version number under
    this scheme; the rules for comparison are simple and predictable,
    but may not always give the results you want (for some definition
    of "want").

This class makes any version string valid, and provides an algorithm to sort
them numerically then lexically. It means that anything can be used to version
your project::

    >>> from distutils.version import LooseVersion as V
    >>> v1 = V('FunkyVersion')
    >>> v2 = V('GroovieVersion')
    >>> v1 > v2
    False

The problem with this is that while it allows expressing any
nesting level it doesn't allow giving special meaning to versions
(pre and post-releases as well as development versions), as expressed in
requisites 2, 3 and 4.

The ``StrictVersion`` class is more strict. From the doc::

    Version numbering for meticulous retentive and software idealists.
    Implements the standard interface for version number classes as
    described above.  A version number consists of two or three
    dot-separated numeric components, with an optional "pre-release" tag
    on the end.  The pre-release tag consists of the letter 'a' or 'b'
    followed by a number.  If the numeric components of two version
    numbers are equal, then one with a pre-release tag will always
    be deemed earlier (lesser) than one without.

    The following are valid version numbers (shown in the order that
    would be obtained by sorting according to the supplied cmp function):

        0.4       0.4.0  (these two are equivalent)
        0.4.1
        0.5a1
        0.5b3
        0.5
        0.9.6
        1.0
        1.0.4a3
        1.0.4b1
        1.0.4

    The following are examples of invalid version numbers:

        1
        2.7.2.2
        1.3.a4
        1.3pl1
        1.3c4

This class enforces a few rules, and makes a decent tool to work with version
numbers::

    >>> from distutils.version import StrictVersion as V
    >>> v2 = V('GroovieVersion')
    Traceback (most recent call last):
    ...
    ValueError: invalid version number 'GroovieVersion'
    >>> v2 = V('1.1')
    >>> v3 = V('1.3')
    >>> v2 < v3
    True

It adds pre-release versions, and some structure, but lacks a few semantic
elements to make it usable, such as development releases or post-release tags,
as expressed in requisites 3 and 4.

Also, note that Distutils version classes have been present for years
but are not really used in the community.


Setuptools
----------

Setuptools provides another version comparison tool [#setuptools-version]_
which does not enforce any rules for the version, but tries to provide a better
algorithm to convert the strings to sortable keys, with a ``parse_version``
function.

From the doc::

    Convert a version string to a chronologically-sortable key

    This is a rough cross between Distutils' StrictVersion and LooseVersion;
    if you give it versions that would work with StrictVersion, then it behaves
    the same; otherwise it acts like a slightly-smarter LooseVersion. It is
    *possible* to create pathological version coding schemes that will f


id: pep-0379.rst
content:
PEP: 379
Title: Adding an Assignment Expression
Author: Jervis Whitley <jervisau@gmail.com>
Status: Withdrawn
Type: Standards Track
Created: 14-Mar-2009
Python-Version: 2.7, 3.2
Post-History:


Abstract
========

This PEP adds a new assignment expression to the Python language
to make it possible to assign the result of an expression in
almost any place.  The new expression will allow the assignment of
the result of an expression at first use (in a comparison for
example).


Motivation and Summary
======================

Issue1714448 "if something as x:" [1]_ describes a feature to allow
assignment of the result of an expression in an if statement to a
name.  It supposed that the ``as`` syntax could be borrowed for this
purpose.  Many times it is not the expression itself that is
interesting, rather one of the terms that make up the
expression. To be clear, something like this::

    if (f_result() == [1, 2, 3]) as res:

seems awfully limited, when this::

    if (f_result() as res) == [1, 2, 3]:

is probably the desired result.


Use Cases
=========

See the Examples section near the end.


Specification
=============

A new expression is proposed with the (nominal) syntax::

    EXPR -> VAR

This single expression does the following:

- Evaluate the value of ``EXPR``, an arbitrary expression;
- Assign the result to ``VAR``, a single assignment target; and
- Leave the result of ``EXPR`` on the Top of Stack (TOS)

Here ``->`` or (``RARROW``) has been used to illustrate the concept that
the result of ``EXPR`` is assigned to ``VAR``.

The translation of the proposed syntax is::

    VAR = (EXPR)
    (EXPR)

The assignment target can be either an attribute, a subscript or
name::

    f() -> name[0]      # where 'name' exists previously.

    f() -> name.attr    # again 'name' exists prior to this expression.

    f() -> name

This expression should be available anywhere that an expression is
currently accepted.

All exceptions that are currently raised during invalid
assignments will continue to be raised when using the assignment
expression.  For example, a ``NameError`` will be raised when in
example 1 and 2 above if ``name`` is not previously defined, or an
``IndexError`` if index 0 was out of range.


Examples from the Standard Library
==================================

The following two examples were chosen after a brief search
through the standard library, specifically both are from ast.py
which happened to be open at the time of the search.

Original::

    def walk(node):
        from collections import deque
        todo = deque([node])
        while todo:
            node = todo.popleft()
            todo.extend(iter_child_nodes(node))
            yield node

Using assignment expression::

    def walk(node):
        from collections import deque
        todo = deque([node])
        while todo:
            todo.extend(iter_child_nodes(todo.popleft() -> node))
            yield node

Original::

    def get_docstring(node, clean=True):
        if not isinstance(node, (FunctionDef, ClassDef, Module)):
            raise TypeError("%r can't have docstrings"
                                % node.__class__.__name__)
        if node.body and isinstance(node.body[0], Expr) and \
        isinstance(node.body[0].value, Str):
            if clean:
                import inspect
                return inspect.cleandoc(node.body[0].value.s)
            return node.body[0].value.s

Using assignment expression::

    def get_docstring(node, clean=True):
        if not isinstance(node, (FunctionDef, ClassDef, Module)):
            raise TypeError("%r can't have docstrings"
                                % node.__class__.__name__)
        if node.body -> body and isinstance(body[0] -> elem, Expr) and \
        isinstance(elem.value -> value, Str):
            if clean:
                import inspect
                return inspect.cleandoc(value.s)
            return value.s


Examples
========

The examples shown below highlight some of the desirable features
of the assignment expression, and some of the possible corner
cases.

1. Assignment in an if statement for use later::

    def expensive():
        import time; time.sleep(1)
        return 'spam'

    if expensive() -> res in ('spam', 'eggs'):
        dosomething(res)

2. Assignment in a while loop clause::

    while len(expensive() -> res) == 4:
        dosomething(res)

3. Keep the iterator object from the for loop::

    for ch in expensive() -> res:
        sell_on_internet(res)

4. Corner case::

    for ch -> please_dont in expensive():
        pass
    # who would want to do this? Not I.


References
==========

.. [1] Issue1714448 "if something as x:", k0wax
       http://bugs.python.org/issue1714448


Copyright
=========

This document has been placed in the public domain.


id: pep-0437.rst
content:
PEP: 437
Title: A DSL for specifying signatures, annotations and argument converters
Author: Stefan Krah <skrah@bytereef.org>
Status: Rejected
Type: Standards Track
Created: 11-Mar-2013
Python-Version: 3.4
Post-History:
Resolution: https://mail.python.org/pipermail/python-dev/2013-May/126117.html

Abstract
========

The Python C-API currently has no mechanism for specifying and auto-generating
function signatures, annotations or custom argument converters.

There are several possible approaches to the problem. Cython uses *cdef*
definitions in *.pyx* files to generate the required information. However,
CPython's C-API functions often require additional initialization and
cleanup snippets that would be hard to specify in a *cdef*.

:pep:`436` proposes a domain specific language (DSL) enclosed in C comments
that largely resembles a per-parameter configuration file. A preprocessor
reads the comment and emits an argument parsing function, docstrings and
a header for the function that utilizes the results of the parsing step.

The latter function is subsequently referred to as the *implementation
function*.


Rejection Notice
================

This PEP was rejected by Guido van Rossum at PyCon US 2013. However, several
of the specific issues raised by this PEP were taken into account when
designing the `second iteration of the PEP 436 DSL`_.


Rationale
=========

Opinions differ regarding the suitability of the :pep:`436` DSL in the context
of a C file. This PEP proposes an alternative DSL. The specific issues with
:pep:`436` that spurred the counter proposal will be explained in the final
section of this PEP.


Scope
=====

The PEP focuses exclusively on the DSL. Topics like the output locations of
docstrings or the generated code are outside the scope of this PEP.

It is however vital that the DSL is suitable for generating custom argument
parsers, a feature that is already implemented in Cython.  Therefore, one of
the goals of this PEP is to keep the DSL close to existing solutions, thus
facilitating a possible inclusion of the relevant parts of Cython into the
CPython source tree.


DSL overview
============

Type safety and annotations
---------------------------

A conversion from a Python to a C value is fully defined by the type of
the converter function.  The PyArg_Parse* family of functions accepts
custom converters in addition to the well-known default converters "i",
"f", etc.

This PEP views the default converters as abstract functions, regardless
of how they are actually implemented.


Include/converters.h
--------------------

Converter functions must be forward-declared. All converter functions
shall be entered into the file Include/converters.h. The file is read
by the preprocessor prior to translating .c files. This is an excerpt::

    /*[converter]
    ##### Default converters #####
    "s":  str                                -> const char *res;
    "s*": [str, bytes, bytearray, rw_buffer] -> Py_buffer &res;
    [...]
    "es#": str -> (const char *res_encoding, char **res, Py_ssize_t *res_length);
    [...]
    ##### Custom converters #####
    path_converter:           [str, bytes, int]  -> path_t &res;
    OS_STAT_DIR_FD_CONVERTER: [int, None]        -> int res;
    [converter_end]*/


Converters are specified by their name, Python input type(s) and C output
type(s).  Default converters must have quoted names, custom converters must
have regular names.  A Python type is given by its name. If a function accepts
multiple Python types, the set is written in list form.

Since the default converters may have multiple implicit return values,
the C output type(s) are written according to the following convention:

The main return value must be named *res*. This is a placeholder for
the actual variable name given later in the DSL. Additional implicit
return values must be prefixed by *res_*.

By default the variables are passed by value to the implementation function.
If the address should be passed instead, *res* must be prefixed with an
ampersand.


Additional declarations may be placed into .c files. Duplicate declarations
are allowed as long as the function types are identical.

It is encouraged to declare custom converter types a second time right
above the converter function definition. The preprocessor will then catch
any mismatch between the declarations.


In order to keep the converter complexity manageable, PY_SSIZE_T_CLEAN will
be deprecated and Py_ssize_t will be assumed for all length arguments.


TBD: Make a list of fantasy types like *rw_buffer*.


Function specifications
-----------------------

Keyword arguments
^^^^^^^^^^^^^^^^^

This example contains the definition of os.stat. The individual sections will
be explained in detail. Grammatically, the whole define block consists of a
function specification and an output section. The function specification in
turn consists of a declaration section, an optional C-declaration section and
an optional cleanup code section.  Sections within the function specification
are separated in yacc style by '%%'::

    /*[define posix_stat]
    def os.stat(path: path_converter, *, dir_fd: OS_STAT_DIR_FD_CONVERTER = None,
                follow_symlinks: "p" = True) -> os.stat_result: pass
    %%
    path_t path = PATH_T_INITIALIZE("stat", 0, 1);
    int dir_fd = DEFAULT_DIR_FD;
    int follow_symlinks = 1;
    %%
    path_cleanup(&path);
    [define_end]*/

    <literal C output>

    /*[define_output_end]*/


Define block
~~~~~~~~~~~~

The function specification block starts with a ``/*[define`` token, followed
by an optional C function name, followed by a right bracket. If the C function
name is not given, it is generated from the declaration name. In the example,
omitting the name *posix_stat* would result in a C function name of *os_stat*.


Declaration
~~~~~~~~~~~

The required declaration is (almost) a valid Python function definition. The
'def' keyword and the function body are redundant, but the author of this PEP
finds the definition more readable if they are present.

The function name may be a path instead of a plain identifier. Each argument
is annotated with the name of the converter function that will be applied to it.

Default values are given in the usual Python manner and may be any valid
Python expression.

The return value may be any Python expression. Usually it will be the name
of an object, but alternative return values could be specified in list form.


C-declarations
~~~~~~~~~~~~~~

This optional section contains C variable declarations. Since the converter
functions have been declared beforehand, the preprocessor can type-check
the declarations.


Cleanup
~~~~~~~

The optional cleanup section contains literal C code that will be inserted
unmodified after the implementation function.


Output
~~~~~~

The output section contains the code emitted by the preprocessor.


Positional-only arguments
^^^^^^^^^^^^^^^^^^^^^^^^^

Functions that do not take keyword arguments are indicated by the presence
of the *slash* special parameter::

    /*[define stat_float_times]
    def os.stat_float_times(/, newval: "i") -> os.stat_result: pass
    %%
    int newval = -1;
    [define_end]*/

The preprocessor translates this definition to a PyArg_ParseTuple() call.
All arguments to the right of the slash are optional arguments.


Left and right optional arguments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Some legacy functions contain optional arguments groups both to the left and
right of a central parameter. It is debatable whether a new tool should support
such functions.  For completeness' sake, this is the proposed syntax::

    /*[define]
    def curses.window.addch(y: "i", x: "i", ch: "O", attr: "l") -> None: pass
    where groups = [[ch], [ch, attr], [y, x, ch], [y, x, ch, attr]]
    [define_end]*/

Here *ch* is the central parameter, *attr* can optionally be added on the
right, and the group [y, x] can optionally be added on the left.

Essentially the rule is that all ordered combi


id: pep-0351.rst
content:
PEP: 351
Title: The freeze protocol
Author: Barry Warsaw <barry@python.org>
Status: Rejected
Type: Standards Track
Created: 14-Apr-2005
Python-Version: 2.5
Post-History:


Abstract
========

This PEP describes a simple protocol for requesting a frozen,
immutable copy of a mutable object.  It also defines a new built-in
function which uses this protocol to provide an immutable copy on any
cooperating object.


Rejection Notice
================

This PEP was rejected.  For a rationale, see `this thread on python-dev`_.

.. _this thread on python-dev: https://mail.python.org/pipermail/python-dev/2006-February/060793.html


Rationale
=========

Built-in objects such dictionaries and sets accept only immutable
objects as keys.  This means that mutable objects like lists cannot be
used as keys to a dictionary.  However, a Python programmer can
convert a list to a tuple; the two objects are similar, but the latter
is immutable, and can be used as a dictionary key.

It is conceivable that third party objects also have similar mutable
and immutable counterparts, and it would be useful to have a standard
protocol for conversion of such objects.

sets.Set objects expose a "protocol for automatic conversion to
immutable" so that you can create sets.Sets of sets.Sets.  :pep:`218`
deliberately dropped this feature from built-in sets.  This PEP
advances that the feature is still useful and proposes a standard
mechanism for its support.


Proposal
========

It is proposed that a new built-in function called freeze() is added.

If freeze() is passed an immutable object, as determined by hash() on
that object not raising a TypeError, then the object is returned
directly.

If freeze() is passed a mutable object (i.e. hash() of that object
raises a TypeError), then freeze() will call that object's
__freeze__() method to get an immutable copy.  If the object does not
have a __freeze__() method, then a TypeError is raised.


Sample implementations
======================

Here is a Python implementation of the freeze() built-in::

    def freeze(obj):
        try:
            hash(obj)
            return obj
        except TypeError:
            freezer = getattr(obj, '__freeze__', None)
            if freezer:
                return freezer()
            raise TypeError('object is not freezable')``

Here are some code samples which show the intended semantics::

    class xset(set):
        def __freeze__(self):
            return frozenset(self)

    class xlist(list):
        def __freeze__(self):
            return tuple(self)

    class imdict(dict):
        def __hash__(self):
            return id(self)

        def _immutable(self, *args, **kws):
            raise TypeError('object is immutable')

        __setitem__ = _immutable
        __delitem__ = _immutable
        clear       = _immutable
        update      = _immutable
        setdefault  = _immutable
        pop         = _immutable
        popitem     = _immutable

    class xdict(dict):
        def __freeze__(self):
            return imdict(self)

    >>> s = set([1, 2, 3])
    >>> {s: 4}
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: set objects are unhashable
    >>> t = freeze(s)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
      File "/usr/tmp/python-lWCjBK.py", line 9, in freeze
    TypeError: object is not freezable
    >>> t = xset(s)
    >>> u = freeze(t)
    >>> {u: 4}
    {frozenset([1, 2, 3]): 4}
    >>> x = 'hello'
    >>> freeze(x) is x
    True
    >>> d = xdict(a=7, b=8, c=9)
    >>> hash(d)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: dict objects are unhashable
    >>> hash(freeze(d))
    -1210776116
    >>> {d: 4}
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: dict objects are unhashable
    >>> {freeze(d): 4}
    {{'a': 7, 'c': 9, 'b': 8}: 4}


Reference implementation
========================

Patch 1335812_ provides the C implementation of this feature.  It adds the
freeze() built-in, along with implementations of the __freeze__()
method for lists and sets.  Dictionaries are not easily freezable in
current Python, so an implementation of dict.__freeze__() is not
provided yet.

.. _1335812: http://sourceforge.net/tracker/index.php?func=detail&aid=1335812&group_id=5470&atid=305470

Open issues
===========

- Should we define a similar protocol for thawing frozen objects?

- Should dicts and sets automatically freeze their mutable keys?

- Should we support "temporary freezing" (perhaps with a method called
  __congeal__()) a la __as_temporarily_immutable__() in sets.Set?

- For backward compatibility with sets.Set, should we support
  __as_immutable__()?  Or should __freeze__() just be renamed to
  __as_immutable__()?


Copyright
=========

This document has been placed in the public domain.


id: pep-0345.rst
content:
PEP: 345
Title: Metadata for Python Software Packages 1.2
Author: Richard Jones <richard@python.org>
Discussions-To: distutils-sig@python.org
Status: Superseded
Type: Standards Track
Topic: Packaging
Created: 28-Apr-2005
Python-Version: 2.7
Post-History: `22-Dec-2009 <https://mail.python.org/archives/list/python-dev@python.org/thread/5QWS2FXKZNNPKNZO6RLPGT5NORS5ZQ2V/>`__
Replaces: 314
Superseded-By: 566
Resolution: https://mail.python.org/archives/list/python-dev@python.org/thread/MKHXVV746H7ZDFN62Z72VNAX6KIRXNRO/

.. canonical-pypa-spec:: :ref:`packaging:core-metadata`


Abstract
========

This PEP describes a mechanism for adding metadata to Python distributions.
It includes specifics of the field names, and their semantics and
usage.

This document specifies version 1.2 of the metadata format.
Version 1.0 is specified in :pep:`241`.
Version 1.1 is specified in :pep:`314`.

Version 1.2 of the metadata format adds a number of optional fields
designed to make third-party packaging of Python Software easier.
These fields are "Requires-Python", "Requires-External", "Requires-Dist",
"Provides-Dist", and "Obsoletes-Dist".  This version also changes the
"Platform" field. Three new fields were also added: "Maintainer",
"Maintainer-email" and "Project-URL".

Last, this new version also adds ``environment markers``.

Fields
======

This section specifies the names and semantics of each of the
supported metadata fields.

Fields marked with "(Multiple use)" may be specified multiple
times in a single PKG-INFO file.  Other fields may only occur
once in a PKG-INFO file.  Fields marked with "(optional)" are
not required to appear in a valid PKG-INFO file; all other
fields must be present.

Metadata-Version
::::::::::::::::

Version of the file format; "1.2" is the only legal value.

Example::

    Metadata-Version: 1.2


Name
::::

The name of the distributions.

Example::

    Name: BeagleVote


Version
:::::::

A string containing the distribution's version number.  This
field  must be in the format specified in :pep:`440`.

Example::

    Version: 1.0a2


Platform (multiple use)
:::::::::::::::::::::::

A Platform specification describing an operating system supported by
the distribution which is not listed in the "Operating System" Trove classifiers.
See "Classifier" below.

Examples::

    Platform: ObscureUnix
    Platform: RareDOS


Supported-Platform (multiple use)
:::::::::::::::::::::::::::::::::

Binary distributions containing a PKG-INFO file will use the
Supported-Platform field in their metadata to specify the OS and
CPU for which the binary distribution was compiled.  The semantics of
the Supported-Platform field are not specified in this PEP.

Example::

    Supported-Platform: RedHat 7.2
    Supported-Platform: i386-win32-2791


Summary
:::::::

A one-line summary of what the distribution does.

Example::

    Summary: A module for collecting votes from beagles.


Description (optional)
::::::::::::::::::::::

A longer description of the distribution that can run to several
paragraphs.  Software that deals with metadata should not assume
any maximum size for this field, though people shouldn't include
their instruction manual as the description.

The contents of this field can be written using reStructuredText
markup [1]_.  For programs that work with the metadata, supporting
markup is optional; programs can also display the contents of the
field as-is.  This means that authors should be conservative in
the markup they use.

To support empty lines and lines with indentation with respect to
the :rfc:`822` format, any CRLF character has to be suffixed by 7 spaces
followed by a pipe ("|") char. As a result, the Description field is
encoded into a folded field that can be interpreted by :rfc:`822#section-3.1.1`
parser.

Example::

    Description: This project provides powerful math functions
            |For example, you can use ``sum()`` to sum numbers:
            |
            |Example::
            |
            |    >>> sum(1, 2)
            |    3
            |

This encoding implies that any occurrences of a CRLF followed by 7 spaces
and a pipe char have to be replaced by a single CRLF when the field is unfolded
using a :rfc:`822` reader.


Keywords (optional)
:::::::::::::::::::

A list of additional keywords to be used to assist searching
for the distribution in a larger catalog.

Example::

    Keywords: dog puppy voting election


Home-page (optional)
::::::::::::::::::::

A string containing the URL for the distribution's home page.

Example::

    Home-page: http://www.example.com/~cschultz/bvote/


Download-URL
::::::::::::

A string containing the URL from which this version of the distribution
can be downloaded.  (This means that the URL can't be something like
".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".)


Author (optional)
:::::::::::::::::

A string containing the author's name at a minimum; additional
contact information may be provided.

Example::

    Author: C. Schultz, Universal Features Syndicate,
            Los Angeles, CA <cschultz@peanuts.example.com>


Author-email (optional)
:::::::::::::::::::::::

A string containing the author's e-mail address.  It can contain
a name and e-mail address in the legal forms for a :rfc:`822`
``From:`` header.

Example::

    Author-email: "C. Schultz" <cschultz@example.com>


Maintainer (optional)
:::::::::::::::::::::

A string containing the maintainer's name at a minimum; additional
contact information may be provided.

Note that this field is intended for use when a project is being
maintained by someone other than the original author:  it should be
omitted if it is identical to ``Author``.

Example::

    Maintainer: C. Schultz, Universal Features Syndicate,
            Los Angeles, CA <cschultz@peanuts.example.com>


Maintainer-email (optional)
:::::::::::::::::::::::::::

A string containing the maintainer's e-mail address.  It can contain
a name and e-mail address in the legal forms for a :rfc:`822`
``From:`` header.

Note that this field is intended for use when a project is being
maintained by someone other than the original author:  it should be
omitted if it is identical to ``Author-email``.

Example::

    Maintainer-email: "C. Schultz" <cschultz@example.com>


License (optional)
::::::::::::::::::

Text indicating the license covering the distribution where the license
is not a selection from the "License" Trove classifiers. See
"Classifier" below.  This field may also be used to specify a
particular version of a license which is named via the ``Classifier``
field, or to indicate a variation or exception to such a license.

Examples::

    License: This software may only be obtained by sending the
            author a postcard, and then the user promises not
            to redistribute it.

    License: GPL version 3, excluding DRM provisions


Classifier (multiple use)
:::::::::::::::::::::::::

Each entry is a string giving a single classification value
for the distribution.  Classifiers are described in :pep:`301`.

Examples::

    Classifier: Development Status :: 4 - Beta
    Classifier: Environment :: Console (Text Based)


Requires-Dist (multiple use)
::::::::::::::::::::::::::::

Each entry contains a string naming some other distutils
project required by this distribution.

The format of a requirement string is identical to that of a
distutils project name (e.g., as found in the ``Name:`` field.
optionally followed by a version declaration within parentheses.

The distutils project names should correspond to names as found
on the `Python Package Index`_.

Version declarations must follow the rules described in
`Version Specifiers`_

Examples::

    Requires-Dist: pkginfo
    Requires-Dist: PasteDeploy
    Requires-Dist: zope.interface (>3.5.0)


Provides-Dist (multiple use)
::::::::::::::::::::::::::::

Each entry contains a string naming a Distutils project which
is contained within this distribution.  This field *must* include
t


id: pep-0423.rst
content:
PEP: 423
Title: Naming conventions and recipes related to packaging
Author: Benoit Bryon <benoit@marmelune.net>
Discussions-To: distutils-sig@python.org
Status: Deferred
Type: Informational
Topic: Packaging
Created: 24-May-2012
Post-History:


Abstract
========

This document deals with:

* names of Python projects,
* names of Python packages or modules being distributed,
* namespace packages.

It provides guidelines and recipes for distribution authors:

* new projects should follow the `guidelines <#overview>`_ below.

* existing projects should be aware of these guidelines and can follow
  `specific recipes for existing projects
  <#how-to-apply-naming-guidelines-on-existing-projects>`_.

PEP Deferral
============

Further consideration of this PEP has been deferred at least until after
:pep:`426` (package metadata 2.0) and related updates have been resolved.

Terminology
===========

Reference is `packaging terminology in Python documentation`_.


Relationship with other PEPs
============================

* :pep:`8#package-and-module-names`
  deals with code style guide, including names of Python
  packages and modules. It covers syntax of package/modules names.

* :pep:`345` deals with packaging metadata, and defines name argument
  of the ``packaging.core.setup()`` function.

* :pep:`420` deals with namespace packages. It brings support of
  namespace packages to Python core. Before, namespaces packages were
  implemented by external libraries.

* :pep:`3108` deals with transition between Python 2.x and Python 3.x
  applied to standard library: some modules to be deleted, some to be
  renamed. It points out that naming conventions matter and is an
  example of transition plan.


Overview
========

Here is a summarized list of guidelines you should follow to choose
names:

* `understand and respect namespace ownership <#respect-ownership>`_.

* if your project is related to another project or community:

  * search for conventions in main project's documentation, because
    `projects should organize community contributions
    <#organize-community-contributions>`_.

  * `follow specific project or related community conventions
    <#follow-community-or-related-project-conventions-if-any>`_, if any.

  * if there is no convention, `follow a standard naming pattern
    <#use-standard-pattern-for-community-contributions>`_.

* make sure your project name is unique, i.e. avoid duplicates:

  * `use top-level namespace for ownership
    <#top-level-namespace-relates-to-code-ownership>`_,
  * `check for name availability
    <#how-to-check-for-name-availability>`_,
  * `register names with PyPI`_.

* make sure distributed packages and modules names are unique, unless
  you explicitly want to distribute alternatives to existing packages
  or modules. `Using the same value for package/module name and
  project name <#use-a-single-name>`_ is the recommended way to
  achieve this.

* `distribute only one package or module at a time
  <#multiple-packages-modules-should-be-rare>`_, unless you know what
  you are doing. It makes it possible to apply the "`use a single
  name`_" rule, and thus make names consistent.

* make it easy to discover and remember your project:

  * `use as much memorable names as possible
    <#pick-memorable-names>`_,
  * `use as much meaningful names as possible
    <#pick-meaningful-names>`_,
  * `use other packaging metadata <#use-packaging-metadata>`_.

* `avoid deep nesting`_. Flat things are easier to use and remember
  than nested ones:

  * one or two namespace levels are recommended, because they are
    almost always enough.
  * even if not recommended, three levels are, de facto, a common
    case.
  * in most cases, you should not need more than three levels.

* `follow PEP 8
  <#follow-pep-8-for-syntax-of-package-and-module-names>`_ for syntax
  of package and module names.

* if you followed specific conventions, or if your project is intended
  to receive contributions from the community, `organize community
  contributions`_.

* `if still in doubt, ask <#if-in-doubt-ask>`_.


If in doubt, ask
================

If you feel unsure after reading this document, ask `Python
community`_ on IRC or on a mailing list.


Top-level namespace relates to code ownership
=============================================

This helps avoid clashes between project names.

Ownership could be:

* an individual.
  Example: `gp.fileupload`_ is owned and maintained by Gael
  Pasgrimaud.

* an organization.
  Examples:

  * `zest.releaser`_ is owned and maintained by Zest Software.
  * `Django`_ is owned and maintained by the Django Software
    Foundation.

* a group or community.
  Example: `sphinx`_ is maintained by developers of the Sphinx
  project, not only by its author, Georg Brandl.

* a group or community related to another package.
  Example: `collective.recaptcha`_ is owned by its author: David
  Glick, Groundwire. But the "collective" namespace is owned by Plone
  community.

Respect ownership
-----------------

Understand the purpose of namespace before you use it.

Don't plug into a namespace you don't own, unless explicitly
authorized.

`If in doubt, ask`_.

As an example, don't plug in "django.contrib" namespace because it is
managed by Django's core contributors.

Exceptions can be defined by project authors. See `Organize community
contributions`_ below.

Also, this rule applies to non-Python projects.

As an example, don't use "apache" as top-level namespace: "Apache" is
the name of an existing project (in the case of "Apache", it is also a
trademark).

Private (including closed-source) projects use a namespace
----------------------------------------------------------

... because private projects are owned by somebody. So apply the
`ownership rule <#top-level-namespace-relates-to-code-ownership>`_.

For internal/customer projects, use your company name as the
namespace.

This rule applies to closed-source projects.

As an example, if you are creating a "climbing" project for the
"Python Sport" company: use "pythonsport.climbing" name, even if it is
closed source.

Individual projects use a namespace
-----------------------------------

... because they are owned by individuals. So apply the
`ownership rule <#top-level-namespace-relates-to-code-ownership>`_.

There is no shame in releasing a project as open source even if it has
an "internal" or "individual" name.

If the project comes to a point where the author wants to change
ownership (i.e. the project no longer belongs to an individual), keep
in mind `it is easy to rename the project
<#how-to-rename-a-project>`_.

Community-owned projects can avoid namespace packages
-----------------------------------------------------

If your project is generic enough (i.e. it is not a contrib to another
product or framework), you can avoid namespace packages. The base
condition is generally that your project is owned by a group (i.e. the
development team) which is dedicated to this project.

Only use a "shared" namespace if you really intend the code to be
community owned.

As an example, `sphinx`_ project belongs to the Sphinx development
team. There is no need to have some "sphinx" namespace package with
only one "sphinx.sphinx" project inside.

In doubt, use an individual/organization namespace
--------------------------------------------------

If your project is really experimental, best choice is to use an
individual or organization namespace:

* it allows projects to be released early.

* it won't block a name if the project is abandoned.

* it doesn't block future changes. When a project becomes mature and
  there is no reason to keep individual ownership, `it remains
  possible to rename the project <#how-to-rename-a-project>`_.


Use a single name
=================

Distribute only one package (or only one module) per project, and use
package (or module) name as project name.

* It avoids possible confusion between project name and distributed
  package or module name


id: pep-0609.rst
content:
PEP: 609
Title: Python Packaging Authority (PyPA) Governance
Author: Dustin Ingram <di@python.org>,
        Pradyun Gedam <pradyunsg@gmail.com>,
        Sumana Harihareswara <sh@changeset.nyc>
Sponsor: Paul Ganssle <paul@ganssle.io>
Discussions-To: https://discuss.python.org/t/pep-609-pypa-governance/2619
Status: Active
Type: Process
Topic: Governance, Packaging
Created: 05-Nov-2019
Post-History: 05-Nov-2019


Abstract
========

This document describes a governance model for the Python Packaging
Authority (PyPA). The model is closely based on existing informal
practices, with the intent of providing clarity into the functioning
of the PyPA and formalizing transparent processes for the PyPA.

Rationale
=========

The Python Packaging Authority (PyPA) is a collaborative community
that maintains and advances many of the relevant projects in Python
packaging. The software and standards developed through the PyPA are
used to package, share, and install Python software and to interact
with indexes of downloadable Python software such as `PyPI`_, the
Python Package Index.

Currently, the PyPA is an informal and loosely defined organization
that lacks true authority, and the inclusion of a given project under
the PyPA umbrella or the creation of new projects has been done in an
ad-hoc, one-off manner. Similarly, individual membership in the PyPA
is not well-defined.

While this model has more or less worked for the PyPA in the past, it
results in an organization which is missing certain features of a
stable ecosystem, namely a clear and transparent decision-making
process. This PEP seeks to rectify this by defining a governance model
for the PyPA.

Terminology
===========

Relevant terms for groups of individual contributors used in this PEP:

**PyPA members**:
  Anyone with the triage bit or commit bit, on at least one project in
  the `PyPA organization`_.

**PyPA committers**:
  Anyone with the commit bit on at least one project in the PyPA
  organization, which should correspond to everyone on the
  `PyPA-Committers`_ mailing list.

**PyPA community**:
  Anyone who is interested in PyPA activity and wants to follow along,
  contribute or make proposals.

**Packaging-WG members**:
  As described in the `Packaging-WG Wiki page`_. For clarity: there is no
  formal relationship between the Packaging-WG and PyPA. This group is
  only included in this list to disambiguate it from PyPA.

Goals
=====

The following section formalizes the goals (and non-goals) of the PyPA
and this governance model.

Goals of the PyPA
-----------------

These goals are the primary motivation for the existence of the PyPA.
These goals are largely already being carried out, even though most
have not been explicitly defined.

Provide support for existing projects under the PyPA
''''''''''''''''''''''''''''''''''''''''''''''''''''

In the event that a given project needs additional support, or no
longer has active maintainers, the PyPA will ensure that the given
project will continue to be supported for users to the extent
necessary.

Foster the creation and acceptance of standards for PyPA projects
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

The PyPA should, as much as possible, strive for standardization and
coordination across PyPA projects, primarily though the governance
process outlined below. PyPA projects are expected to abide by
applicable specifications maintained by the PyPA.

Guide decisions which affect multiple PyPA projects
'''''''''''''''''''''''''''''''''''''''''''''''''''

The PyPA community (especially PyPA members) should be expected to
provide opinions, insight and experience when ecosystem-wide changes
are being proposed.

Determine which projects should be under the guidance of the PyPA
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

For example: accepting new projects from the community, organically
creating projects within the PyPA, etc.

Enforce adherence to a Code of Conduct across all projects
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Generally this means leading by example, but occasionally it may mean
more explicit moderation.

Non-goals of the PyPA
---------------------

These are specific items that are explicitly _not_ goals of the PyPA.

Determine who is and isn't a PyPA member
''''''''''''''''''''''''''''''''''''''''

This is for members of individual projects to decide, as they add new
members to their projects. Maintainership of a project that is under
the PyPA organization automatically transfers membership in the PyPA.

Micromanage individual projects
'''''''''''''''''''''''''''''''

As long as the project is adhering to the Code of Conduct and
following specifications supported by the PyPA, the PyPA should only
concerned with large, ecosystem-wide changes.

Develop and maintain standalone Code of Conduct
'''''''''''''''''''''''''''''''''''''''''''''''

PyPA projects follow `the PSF Code of Conduct`_.


Goals of the PyPA's Governance Model
------------------------------------

These are new goals which the governance model seeks to make possible.

Transparency in PyPA membership
'''''''''''''''''''''''''''''''

Provide a transparent process for decisions taken, regarding project
membership in the PyPA.

Document PyPA's use of PEPs
'''''''''''''''''''''''''''

Formally document how the PyPA uses Python Enhancement Proposals
(PEPs), for maintaining interoperability specifications defined by the
PyPA.

Processes
=========

The processes for the PyPA's activities are outlined below:

Specifications
--------------

The PyPA will use PEPs for defining, and making changes to, the
interoperability specifications maintained by the PyPA. Thus, the
Python Steering Council has the final say in the acceptance of these
interoperability specifications.

It is expected (but not required) that the Python Steering Council
would delegate authority to sponsor and/or approve/reject PEPs related
to packaging interoperability specifications, to individuals within the
PyPA community. At the time of writing (June 2020), the Python Steering
Council has `standing delegations`_ for currently active packaging
interoperability specifications.

The details of the process of proposing and updating the
interoperability specifications are described in the `PyPA
Specifications`_ document.

Governance
----------

PyPA Committer Votes
''''''''''''''''''''

A PyPA member can put forward a proposal and call for a vote on a
public PyPA communication channel. A PyPA committer vote is triggered
when a PyPA committer (not the proposer) seconds the proposal.

The proposal will be put to a vote on the `PyPA-Committers`_ mailing
list, over a 7-day period. Each PyPA committer can vote once, and can
choose one of ``+1`` and ``-1``. If at least two thirds of recorded
votes are ``+1``, then the vote succeeds.

PyPA committer votes are required for, and limited to, the following
kinds of proposals:

Addition of a project to the PyPA
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Proposing the acceptance of a project into the PyPA organization. This
proposal must not be opposed by the existing maintainers of the
project.

Creation of a new project in the PyPA
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Proposing the creation of a new tools / project in the PyPA
organization.

Removal of a project from PyPA
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Proposing the removal of a project in the PyPA organization.

Updates to the Governance/Specification Processes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Proposing changes to how the PyPA operates, including but not limited
to changes to its specification and governance processes, and this
PEP.

Leaving PyPA
''''''''''''

A project that is a part of the PyPA organization, can request to
leave PyPA.

Such requests can made by a committer of the project, on the
`PyPA-Committers`_ mailing list and must clearly state the GitHub
user/organization to transfer the repository to.

If the request is not opposed by anoth


id: pep-0635.rst
content:
PEP: 635
Title: Structural Pattern Matching: Motivation and Rationale
Author: Tobias Kohn <kohnt@tobiaskohn.ch>,
        Guido van Rossum <guido@python.org>
BDFL-Delegate:
Discussions-To: python-dev@python.org
Status: Final
Type: Informational
Created: 12-Sep-2020
Python-Version: 3.10
Post-History: 22-Oct-2020, 08-Feb-2021
Resolution: https://mail.python.org/archives/list/python-committers@python.org/message/SQC2FTLFV5A7DV7RCEAR2I2IKJKGK7W3


Abstract
========

This PEP provides the motivation and rationale for :pep:`634`
("Structural Pattern Matching: Specification").  First-time readers
are encouraged to start with :pep:`636`, which provides a gentler
introduction to the concepts, syntax and semantics of patterns.


Motivation
==========

(Structural) pattern matching syntax is found in many languages, from
Haskell, Erlang and Scala to Elixir and Ruby.  (A proposal for
JavaScript is also under consideration.)

Python already supports a limited form of this through sequence
unpacking assignments, which the new proposal leverages.

Several other common Python idioms are also relevant:

- The ``if ... elif ... elif ... else`` idiom is often used to find
  out the type or shape of an object in an ad-hoc fashion, using one
  or more checks like ``isinstance(x, cls)``, ``hasattr(x, "attr")``,
  ``len(x) == n`` or ``"key" in x`` as guards to select an applicable
  block.  The block can then assume ``x`` supports the interface
  checked by the guard.  For example::

    if isinstance(x, tuple) and len(x) == 2:
        host, port = x
        mode = "http"
    elif isinstance(x, tuple) and len(x) == 3:
        host, port, mode = x
    # Etc.

  Code like this is more elegantly rendered using ``match``::

    match x:
        case host, port:
            mode = "http"
        case host, port, mode:
            pass
        # Etc.

- AST traversal code often looks for nodes matching a given pattern,
  for example the code to detect a node of the shape "A + B * C" might
  look like this::

    if (isinstance(node, BinOp) and node.op == "+"
            and isinstance(node.right, BinOp) and node.right.op == "*"):
        a, b, c = node.left, node.right.left, node.right.right
        # Handle a + b*c

  Using ``match`` this becomes more readable::

    match node:
        case BinOp("+", a, BinOp("*", b, c)):
            # Handle a + b*c

We believe that adding pattern matching to Python will enable Python
users to write cleaner, more readable code for examples like those
above, and many others.

For a more academic discussion to this proposal, see [1]_.


Pattern Matching and OO
-----------------------

Pattern matching is complimentary to the object-oriented paradigm.
Using OO and inheritance we can easily define a method on a base class
that defines default behavior for a specific operation on that class,
and we can override this default behavior in subclasses.  We can also
use the Visitor pattern to separate actions from data.

But this is not sufficient for all situations.  For example, a code
generator may consume an AST, and have many operations where the
generated code needs to vary based not just on the class of a node,
but also on the value of some class attributes, like the ``BinOp``
example above.  The Visitor pattern is insufficiently flexible for
this: it can only select based on the class.

See a `complete example
<https://github.com/gvanrossum/patma/blob/be5969442d0584005492134c3b24eea408709db2/examples/expr.py#L231>`_.

Like the Visitor pattern, pattern matching allows for a strict separation
of concerns: specific actions or data processing is independent of the
class hierarchy or manipulated objects.  When dealing with predefined or
even built-in classes, in particular, it is often impossible to add further
methods to the individual classes.  Pattern matching not only relieves the
programmer or class designer from the burden of the boilerplate code needed
for the Visitor pattern, but is also flexible enough to directly work with
built-in types.  It naturally distinguishes between sequences of different
lengths, which might all share the same class despite obviously differing
structures.  Moreover, pattern matching automatically takes inheritance
into account: a class *D* inheriting from *C* will be handled by a pattern
that targets *C* by default.

Object oriented programming is geared towards single-dispatch: it is a
single instance (or the type thereof) that determines which method is to
be called.  This leads to a somewhat artificial situation in case of binary
operators where both objects might play an equal role in deciding which
implementation to use (Python addresses this through the use of reversed
binary methods).  Pattern matching is structurally better suited to handle
such situations of multi-dispatch, where the action to be taken depends on
the types of several objects to equal parts.


Patterns and Functional Style
-----------------------------

Many Python applications and libraries are not written in a consistent
OO style -- unlike Java, Python encourages defining functions at the
top-level of a module, and for simple data structures, tuples (or
named tuples or lists) and dictionaries are often used exclusively or
mixed with classes or data classes.

Pattern matching is particularly suitable for picking apart such data
structures.  As an extreme example, it's easy to write code that picks
a JSON data structure using ``match``::

  match json_pet:
      case {"type": "cat", "name": name, "pattern": pattern}:
          return Cat(name, pattern)
      case {"type": "dog", "name": name, "breed": breed}:
          return Dog(name, breed)
      case _:
          raise ValueError("Not a suitable pet")

Functional programming generally prefers a declarative style with a focus
on relationships in data.  Side effects are avoided whenever possible.
Pattern matching thus naturally fits and highly supports functional
programming style.


Rationale
=========

This section provides the rationale for individual design decisions.
It takes the place of "Rejected ideas" in the standard PEP format.
It is organized in sections corresponding to the specification (:pep:`634`).


Overview and Terminology
------------------------

Much of the power of pattern matching comes from the nesting of subpatterns.
That the success of a pattern match depends directly on the success of
subpattern is thus a cornerstone of the design.  However, although a
pattern like ``P(Q(), R())`` succeeds only if both subpatterns ``Q()``
and ``R()`` succeed (i.e. the success of pattern ``P`` depends on ``Q``
and ``R``), the pattern ``P`` is checked first.  If ``P`` fails, neither
``Q()`` nor ``R()`` will be tried (this is a direct consequence of the
fact that if ``P`` fails, there are no subjects to match against ``Q()``
and ``R()`` in the first place).

Also note that patterns bind names to values rather than performing an
assignment.  This reflects the fact that patterns aim to not have side
effects, which also means that Capture or AS patterns cannot assign a
value to an attribute or subscript.  We thus consistently use the term
'bind' instead of 'assign' to emphasise this subtle difference between
traditional assignments and name binding in patterns.


The Match Statement
-------------------

The match statement evaluates an expression to produce a subject, finds the
first pattern that matches the subject, and executes the associated block
of code.  Syntactically, the match statement thus takes an expression and
a sequence of case clauses, where each case clause comprises a pattern and
a block of code.

Since case clauses comprise a block of code, they adhere to the existing
indentation scheme with the syntactic structure of
``<keyword> ...: <(indented) block>``, which resembles a compound
statement.  The keyword ``case`` reflects its widespread use in
pattern matching languages, ignoring those languages that use other
syntactic means such as a symbol like ``|``, because it


id: pep-0621.rst
content:
PEP: 621
Title: Storing project metadata in pyproject.toml
Author: Brett Cannon <brett@python.org>,
        Dustin Ingram <di@python.org>,
        Paul Ganssle <paul at ganssle.io>,
        Pradyun Gedam <pradyunsg@gmail.com>,
        SÃ©bastien Eustace <sebastien@eustace.io>,
        Thomas Kluyver <thomas@kluyver.me.uk>,
        Tzu-ping Chung <uranusjr@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-621-round-3/5472
Status: Final
Type: Standards Track
Topic: Packaging
Created: 22-Jun-2020
Post-History: 22-Jun-2020,
              18-Oct-2020,
              24-Oct-2020,
              31-Oct-2020
Resolution: https://discuss.python.org/t/pep-621-round-3/5472/109


.. canonical-pypa-spec:: :ref:`packaging:pyproject-toml-spec`


Abstract
========

This PEP specifies how to write a project's `core metadata`_ in a
``pyproject.toml`` file for packaging-related tools to consume.


Motivation
==========

The key motivators of this PEP are:

- Encourage users to specify core metadata statically for speed,
  ease of specification, unambiguity, and deterministic consumption by
  build back-ends
- Provide a tool-agnostic way of specifying metadata for ease of
  learning and transitioning between build back-ends
- Allow for more code sharing between build back-ends for the
  "boring parts" of a project's metadata

To speak specifically to the motivation for static metadata, that has
been an overall goal of the packaging ecosystem for some time. As
such, making it easy to specify metadata statically is important. This
also means that raising the cost of specifying data as dynamic is
acceptable as users should skew towards wanting to provide static
metadata.

Requiring the distinction between static and dynamic metadata also
helps with disambiguation for when metadata isn't specified. When any
metadata *may* be dynamic, it means you never know if the absence of
metadata is on purpose or because it is to be provided later. By
requiring that dynamic metadata be specified, it disambiguates the
intent when metadata goes unspecified.

This PEP does **not** attempt to standardize all possible metadata
required by a build back-end, only the metadata covered by the
`core metadata`_ specification which are very common across projects
and would stand to benefit from being static and consistently
specified. This means build back-ends are still free and able to
innovate around patterns like how to specify the files to include in a
wheel. There is also an included escape hatch for users and build
back-ends to use when they choose to partially opt-out of this PEP
(compared to opting-out of this PEP entirely, which is also possible).

This PEP is also not trying to change the underlying `core metadata`_
in any way. Such considerations should be done in a separate PEP which
may lead to changes or additions to what this PEP specifies.


Rationale
=========

The design guidelines the authors of this PEP followed were:

- Define a representation of as much of the `core metadata`_ in
  ``pyproject.toml`` as is reasonable
- Define the metadata statically with an escape hatch for those who
  want to define it dynamically later via a build back-end
- Use familiar names where it makes sense, but be willing to use more
  modern terminology
- Try to be ergonomic within a TOML file instead of mirroring how
  build back-ends specify metadata at a low-level when it makes sense
- Learn from other build back-ends in the packaging ecosystem which
  have used TOML for their metadata
- Don't try to standardize things which lack a pre-existing standard
  at a lower-level
- *When* metadata is specified using this PEP, it is considered
  canonical


Specification
=============

When specifying project metadata, tools MUST adhere and honour the
metadata as specified in this PEP. If metadata is improperly specified
then tools MUST raise an error to notify the user about their mistake.

Data specified using this PEP is considered canonical. Tools CANNOT
remove, add or change data that has been statically specified. Only
when a field is marked as ``dynamic`` may a tool provide a "new" value.


Details
-------

Table name
''''''''''

Tools MUST specify fields defined by this PEP in a table named
``[project]``. No tools may add fields to this table which are not
defined by this PEP or subsequent PEPs. For tools wishing to store
their own settings in ``pyproject.toml``, they may use the ``[tool]``
table as defined in :pep:`518`. The lack of a ``[project]`` table
implicitly means the build back-end will dynamically provide all
fields.

``name``
''''''''
- Format: string
- `Core metadata`_: ``Name``
  (`link <https://packaging.python.org/specifications/core-metadata/#name>`__)
- Synonyms

  - Flit_: ``module``/``dist-name``
    (`link <https://flit.readthedocs.io/en/latest/pyproject_toml.html#metadata-section>`__)
  - Poetry_: ``name``
    (`link <https://python-poetry.org/docs/pyproject/#name>`__)
  - Setuptools_: ``name``
    (`link <https://setuptools.readthedocs.io/en/latest/setuptools.html#metadata>`__)

The name of the project.

Tools MUST require users to statically define this field.

Tools SHOULD normalize this name, as specified by :pep:`503`, as soon
as it is read for internal consistency.

``version``
'''''''''''
- Format: string
- `Core metadata`_: ``Version``
  (`link <https://packaging.python.org/specifications/core-metadata/#version>`__)
- Synonyms

  - Flit_: N/A (read from a ``__version__`` attribute)
    (`link <https://flit.readthedocs.io/en/latest/index.html#usage>`__)
  - Poetry_: ``version``
    (`link <https://python-poetry.org/docs/pyproject/#version>`__)
  - Setuptools_: ``version``
    (`link <https://setuptools.readthedocs.io/en/latest/setuptools.html#metadata>`__)

The version of the project as supported by :pep:`440`.

Users SHOULD prefer to specify already-normalized versions.

``description``
'''''''''''''''
- Format: string
- `Core metadata`_: ``Summary``
  (`link <https://packaging.python.org/specifications/core-metadata/#summary>`__)
- Synonyms

  - Flit_: N/A
  - Poetry_: ``description``
    (`link <https://python-poetry.org/docs/pyproject/#description>`__)
  - Setuptools_: ``description``
    (`link <https://setuptools.readthedocs.io/en/latest/setuptools.html#metadata>`__)

The summary description of the project.

``readme``
''''''''''
- Format: String or table
- `Core metadata`_: ``Description``
  (`link <https://packaging.python.org/specifications/core-metadata/#description>`__)
- Synonyms

  - Flit_: ``description-file``
    (`link <https://flit.readthedocs.io/en/latest/pyproject_toml.html#metadata-section>`__)
  - Poetry_: ``readme``
    (`link <https://python-poetry.org/docs/pyproject/#readme>`__)
  - Setuptools_: ``long_description``
    (`link <https://setuptools.readthedocs.io/en/latest/setuptools.html#metadata>`__)

The full description of the project (i.e. the README).

The field accepts either a string or a table. If it is a string then
it is the relative path to a text file containing the full
description. Tools MUST assume the file's encoding is UTF-8. If the
file path ends in a case-insensitive ``.md`` suffix, then tools MUST
assume the content-type is ``text/markdown``. If the file path ends in
a case-insensitive ``.rst``, then tools MUST assume the content-type
is ``text/x-rst``. If a tool recognizes more extensions than this PEP,
they MAY infer the content-type for the user without specifying this
field as ``dynamic``. For all unrecognized suffixes when a
content-type is not provided, tools MUST raise an error.

The ``readme`` field may also take a table. The ``file`` key has a
string value representing a relative path to a file containing the
full description. The ``text`` key has a string value which is the
full description. These keys are mutually-exclusive, thus tools MUST
raise an error if the metadata specifies both keys.

A table specified in the ``readme`` field also has a ``content-type``
field which takes a string specifying the cont


id: pep-0782.rst
content:
PEP: 782
Title: Add PyBytesWriter C API
Author: Victor Stinner <vstinner@python.org>
Discussions-To: https://discuss.python.org/t/86617
Status: Draft
Type: Standards Track
Created: 27-Mar-2025
Python-Version: 3.15
Post-History:
    `18-Feb-2025 <https://discuss.python.org/t/81182>`__


.. highlight:: c


Abstract
========

Add a new ``PyBytesWriter`` C API to create ``bytes`` objects.

Soft deprecate ``PyBytes_FromStringAndSize(NULL, size)`` and
``_PyBytes_Resize()`` APIs. These APIs treat an immutable ``bytes``
object as a mutable object. They remain available and maintained, don't
emit deprecation warning, but are no longer recommended when writing new
code.


Rationale
=========

Disallow creation of incomplete/inconsistent objects
----------------------------------------------------

Creating a Python :class:`bytes` object using
``PyBytes_FromStringAndSize(NULL, size)`` and ``_PyBytes_Resize()``
treats an immutable :class:`bytes` object as mutable. It goes against
the principle that :class:`bytes` objects are immutable. It also creates
an incomplete or "invalid" object since bytes are not initialized. In
Python, a :class:`bytes` object should always have its bytes fully
initialized.

* `Avoid creating incomplete/invalid objects api-evolution#36
  <https://github.com/capi-workgroup/api-evolution/issues/36>`_
* `Disallow mutating immutable objects api-evolution#20
  <https://github.com/capi-workgroup/api-evolution/issues/20>`_
* `Disallow creation of incomplete/inconsistent objects problems#56
  <https://github.com/capi-workgroup/problems/issues/56>`_

Inefficient allocation strategy
-------------------------------

When creating a bytes string and the output size is unknown, one
strategy is to allocate a short buffer and extend it (to the exact size)
each time a larger write is needed.

This strategy is inefficient because it requires to enlarge the buffer
multiple timess. It's more efficient to overallocate the buffer the
first time that a larger write is needed. It reduces the number of
expensive ``realloc()`` operations which can imply a memory copy.


Specification
=============

API
---

.. c:type:: PyBytesWriter

   A Python :class:`bytes` writer instance created by
   :c:func:`PyBytesWriter_Create`.

   The instance must be destroyed by :c:func:`PyBytesWriter_Finish` or
   :c:func:`PyBytesWriter_Discard`.

Create, Finish, Discard
^^^^^^^^^^^^^^^^^^^^^^^

.. c:function:: PyBytesWriter* PyBytesWriter_Create(Py_ssize_t size)

   Create a :c:type:`PyBytesWriter` to write *size* bytes.

   If *size* is greater than zero, allocate *size* bytes, and set the
   writer size to *size*. The caller is responsible to write *size*
   bytes using :c:func:`PyBytesWriter_GetData`.

   On error, set an exception and return NULL.

   *size* must be positive or zero.

.. c:function:: PyObject* PyBytesWriter_Finish(PyBytesWriter *writer)

   Finish a :c:type:`PyBytesWriter` created by
   :c:func:`PyBytesWriter_Create`.

   On success, return a Python :class:`bytes` object.
   On error, set an exception and return ``NULL``.

   The writer instance is invalid after the call in any case.

.. c:function:: PyObject* PyBytesWriter_FinishWithSize(PyBytesWriter *writer, Py_ssize_t size)

   Similar to :c:func:`PyBytesWriter_Finish`, but resize the writer
   to *size* bytes before creating the :class:`bytes` object.

.. c:function:: PyObject* PyBytesWriter_FinishWithPointer(PyBytesWriter *writer, void *buf)

   Similar to :c:func:`PyBytesWriter_Finish`, but resize the writer
   using *buf* pointer before creating the :class:`bytes` object.

   Set an exception and return ``NULL`` if *buf* pointer is outside the
   internal buffer bounds.

   Function pseudo-code::

       Py_ssize_t size = (char*)buf - (char*)PyBytesWriter_GetData(writer);
       return PyBytesWriter_FinishWithSize(writer, size);

.. c:function:: void PyBytesWriter_Discard(PyBytesWriter *writer)

   Discard a :c:type:`PyBytesWriter` created by :c:func:`PyBytesWriter_Create`.

   Do nothing if *writer* is ``NULL``.

   The writer instance is invalid after the call.

High-level API
^^^^^^^^^^^^^^

.. c:function:: int PyBytesWriter_WriteBytes(PyBytesWriter *writer, const void *bytes, Py_ssize_t size)

   Grow the *writer* internal buffer by *size* bytes,
   write *size* bytes of *bytes* at the *writer* end,
   and add *size* to the *writer* size.

   If *size* is equal to ``-1``, call ``strlen(bytes)`` to get the
   string length.

   On success, return ``0``.
   On error, set an exception and return ``-1``.

.. c:function:: int PyBytesWriter_Format(PyBytesWriter *writer, const char *format, ...)

   Similar to ``PyBytes_FromFormat()``, but write the output directly at
   the writer end. Grow the writer internal buffer on demand.
   Then add the written size to the writer size.

   On success, return ``0``.
   On error, set an exception and return ``-1``.

Getters
^^^^^^^

.. c:function:: Py_ssize_t PyBytesWriter_GetSize(PyBytesWriter *writer)

   Get the writer size.

.. c:function:: void* PyBytesWriter_GetData(PyBytesWriter *writer)

   Get the writer data: start of the internal buffer.

   The pointer is valid until :c:func:`PyBytesWriter_Finish` or
   :c:func:`PyBytesWriter_Discard` is called on *writer*.

Low-level API
^^^^^^^^^^^^^

.. c:function:: int PyBytesWriter_Resize(PyBytesWriter *writer, Py_ssize_t size)

   Resize the writer to *size* bytes. It can be used to enlarge or to
   shrink the writer.

   Newly allocated bytes are left uninitialized.

   On success, return ``0``.
   On error, set an exception and return ``-1``.

   *size* must be positive or zero.

.. c:function:: int PyBytesWriter_Grow(PyBytesWriter *writer, Py_ssize_t grow)

   Resize the writer by adding *grow* bytes to the current writer size.

   Newly allocated bytes are left uninitialized.

   On success, return ``0``.
   On error, set an exception and return ``-1``.

   *size* can be negative to shrink the writer.

.. c:function:: void* PyBytesWriter_GrowAndUpdatePointer(PyBytesWriter *writer, Py_ssize_t size, void *buf)

   Similar to :c:func:`PyBytesWriter_Grow`, but update also the *buf*
   pointer.

   The *buf* pointer is moved if the internal buffer is moved in memory.
   The *buf* relative position within the internal buffer is left
   unchanged.

   On error, set an exception and return ``NULL``.

   *buf* must not be ``NULL``.

   Function pseudo-code::

       Py_ssize_t pos = (char*)buf - (char*)PyBytesWriter_GetData(writer);
       if (PyBytesWriter_Grow(writer, size) < 0) {
           return NULL;
       }
       return (char*)PyBytesWriter_GetData(writer) + pos;


Overallocation
--------------

:c:func:`PyBytesWriter_Resize` and :c:func:`PyBytesWriter_Grow`
overallocate the internal buffer to reduce the number of ``realloc()``
calls and so reduce memory copies.

:c:func:`PyBytesWriter_Finish` trims overallocations: it shrinks the
internal buffer to the exact size when creating the final :class:`bytes`
object.


Thread safety
-------------

The API is not thread safe: a writer should only be used by a single
thread at the same time.


Soft deprecations
-----------------

Soft deprecate ``PyBytes_FromStringAndSize(NULL, size)`` and
``_PyBytes_Resize()`` APIs. These APIs treat an immutable ``bytes``
object as a mutable object. They remain available and maintained, don't
emit deprecation warning, but are no longer recommended when writing new
code.

``PyBytes_FromStringAndSize(str, size)`` is not soft deprecated. Only
calls with ``NULL`` *str* are soft deprecated.


Examples
========

High-level API
--------------

Create the bytes string ``b"Hello World!"``::

    PyObject* hello_world(void)
    {
        PyBytesWriter *writer = PyBytesWriter_Create(0);
        if (writer == NULL) {
            goto error;
        }
        if (PyBytesWriter_WriteBytes(writer, "Hello", -1) < 0) {
            goto error;
        }
        if (PyBytesWriter_Format(writer, " %s!", "World") < 0) {
            goto error;
        }


id: pep-0769.rst
content:
PEP: 769
Title: Add a 'default' keyword argument to 'attrgetter', 'itemgetter' and 'getitem'
Author: Facundo Batista <facundo@taniquetil.com.ar>
Discussions-To: https://discuss.python.org/t/76419
Status: Draft
Type: Standards Track
Created: 22-Dec-2024
Python-Version: 3.14
Post-History: `07-Jan-2025 <https://discuss.python.org/t/76419>`__

Abstract
========

This proposal aims to enhance the :mod:`operator` module by adding a
``default`` keyword argument to the ``attrgetter``, ``itemgetter`` and
``getitem`` functions. This addition would allow these functions to return a
specified default value when the targeted attribute or item is missing,
thereby preventing exceptions and simplifying code that handles optional
attributes or items.


Motivation
==========

Currently, ``attrgetter`` and ``itemgetter`` raise exceptions if the
specified attribute or item is absent. This limitation requires
developers to implement additional error handling, leading to more
complex and less readable code.

Introducing a ``default`` parameter would streamline operations involving
optional attributes or items, reducing boilerplate code and enhancing
code clarity.

A similar situation occurs with ``getitem``, with the added nuance that
allowing the specification of a default value in this case would resolve
a longstanding asymmetry with the :func:`getattr` built-in function.


Rationale
=========

The primary design decision is to introduce a single ``default`` parameter
applicable to all specified attributes or items.

This approach maintains simplicity and avoids the complexity of assigning
individual default values to multiple attributes or items. While some
discussions considered allowing multiple defaults, the increased
complexity and potential for confusion led to favoring a single default
value for all cases (more about this below in `Rejected Ideas
<PEP 769 Rejected Ideas_>`__).


Specification
=============

Proposed behaviors:

- **attrgetter**: ``f = attrgetter("name", default=XYZ)`` followed by
  ``f(obj)`` would return ``obj.name`` if the attribute exists, else
  ``XYZ``.

- **itemgetter**: ``f = itemgetter(2, default=XYZ)`` followed by
  ``f(obj)`` would return ``obj[2]`` if that is valid, else ``XYZ``.

- **getitem**: ``getitem(obj, k, XYZ)`` or
  ``getitem(obj, k, default=XYZ)`` would return ``obj[k]`` if that is valid,
  else ``XYZ``.

In the first two cases, the enhancement applies to single and multiple
attribute/item retrievals, with the default value returned for any
missing attribute or item.

No functionality change is incorporated in any case if the extra
default (keyword) argument is not used.


Examples for attrgetter
-----------------------

The current behavior is unchanged::

    >>> class C:
    ...   class D:
    ...     class X:
    ...       pass
    ...   class E:
    ...     pass
    ...
    >>> attrgetter("D")(C)
    <class '__main__.C.D'>
    >>> attrgetter("badname")(C)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: type object 'C' has no attribute 'badname'
    >>> attrgetter("D", "E")(C)
    (<class '__main__.C.D'>, <class '__main__.C.E'>)
    >>> attrgetter("D", "badname")(C)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: type object 'C' has no attribute 'badname'
    >>> attrgetter("D.X")(C)
    <class '__main__.C.D.X'>
    >>> attrgetter("D.badname")(C)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: type object 'D' has no attribute 'badname'

With this PEP, using the proposed ``default`` keyword::

    >>> attrgetter("D", default="noclass")(C)
    <class '__main__.C.D'>
    >>> attrgetter("badname", default="noclass")(C)
    'noclass'
    >>> attrgetter("D", "E", default="noclass")(C)
    (<class '__main__.C.D'>, <class '__main__.C.E'>)
    >>> attrgetter("D", "badname", default="noclass")(C)
    (<class '__main__.C.D'>, 'noclass')
    >>> attrgetter("D.X", default="noclass")(C)
    <class '__main__.C.D.X'>
    >>> attrgetter("D.badname", default="noclass")(C)
    'noclass'


Examples for itemgetter
-----------------------

The current behavior is unchanged::

    >>> obj = ["foo", "bar", "baz"]
    >>> itemgetter(1)(obj)
    'bar'
    >>> itemgetter(5)(obj)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IndexError: list index out of range
    >>> itemgetter(1, 0)(obj)
    ('bar', 'foo')
    >>> itemgetter(1, 5)(obj)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IndexError: list index out of range


With this PEP, using the proposed ``default`` keyword::

    >>> itemgetter(1, default="XYZ")(obj)
    'bar'
    >>> itemgetter(5, default="XYZ")(obj)
    'XYZ'
    >>> itemgetter(1, 0, default="XYZ")(obj)
    ('bar', 'foo')
    >>> itemgetter(1, 5, default="XYZ")(obj)
    ('bar', 'XYZ')


Examples for getitem
--------------------

The current behavior is unchanged::

    >>> obj = ["foo", "bar", "baz"]
    >>> getitem(obj, 1)
    'bar'
    >>> getitem(obj, 5)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IndexError: list index out of range


With this PEP, using the proposed extra default, positionally or with
a keyword::

    >>> getitem(obj, 1, "XYZ")
    'bar'
    >>> getitem(obj, 5, "XYZ")
    'XYZ'
    >>> getitem(obj, 1, default="XYZ")
    'bar'
    >>> getitem(obj, 5, default="XYZ")
    'XYZ'


About Possible Implementations
------------------------------

The implementation of ``attrgetter`` is quite direct: it implies using
``getattr`` and catching a possible ``AttributeError``. So
``attrgetter("name", default=XYZ)(obj)`` would be like::

    try:
        value = getattr(obj, "name")
    except AttributeError:
        value = XYZ

Note we cannot rely on using ``getattr`` with a default value, as it would
be impossible to distinguish what it returned on each step when an
attribute chain is specified (e.g.
``attrgetter("foo.bar.baz", default=XYZ)``).

The implementation for ``itemgetter`` and ``getitem`` is not that
easy. The more straightforward way is also simple to define and
understand: attempting ``__getitem__`` and catching a possible
exception (see below). This way, ``itemgetter(123, default=XYZ)(obj)``
or ``getitem(obj, 123, default=XYZ)`` would be equivalent to::

    try:
        value = obj[123]
    except (IndexError, KeyError):
        value = XYZ

However, for performance reasons the implementation may look more
like the following, which has the same exact behavior::

    if type(obj) == dict:
        value = obj.get(123, XYZ)
    else:
        try:
            value = obj[123]
        except (IndexError, KeyError):
            value = XYZ

Note how the verification is about the exact type and not using
``isinstance``; this is to ensure the exact behavior, which would be
impossible if the object is a user defined one that inherits ``dict``
but overwrites ``get`` (similar reason to not check if the object has
a ``get`` method).

This way, performance is better but it's just an implementation detail,
so we can keep the original explanation on how it behaves.

Regarding the exception to be captured, even if ``__getitem__``
can raise ``IndexError``, ``KeyError``, or ``TypeError`` (see its
`reference`_), only the first two can happen if the container does not
contain the indicated key or index, and the latter is likely to signal
a bug in the code, so we're not capturing it to trigger the default
behavior.


Corner Cases
------------

Providing a ``default`` option would only work if accessing the
item/attribute would fail in the normal case. In other words, the
object accessed should not handle defaults itself.

For example, the following would be redundant/confusing because
``defaultdict`` will never error out when accessing the item::

    >>> from collections import defaultdict
    >>> from operator import itemgetter
    >>> dd = defaultdict


id: pep-0741.rst
content:
PEP: 741
Title: Python Configuration C API
Author: Victor Stinner <vstinner@python.org>
Discussions-To: https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403
Status: Final
Type: Standards Track
Created: 18-Jan-2024
Python-Version: 3.14
Post-History: `19-Jan-2024 <https://discuss.python.org/t/pep-741-python-configuration-c-api/43637>`__,
              `08-Feb-2024 <https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403>`__,
Resolution: https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403/88

Abstract
========

Add a C API to configure the Python initialization without relying on C
structures and the ability to make ABI-compatible changes in the future.

Complete :pep:`587` API by adding ``PyInitConfig_AddModule()`` which can be
used to add a built-in extension module; feature previously referred to
as the "inittab".

Add ``PyConfig_Get()`` and ``PyConfig_Set()`` functions to
get and set the current runtime configuration.

:pep:`587` "Python Initialization Configuration" unified all the ways to
configure the Python **initialization**. This PEP unifies also the
configuration of the Python **preinitialization** and the Python
**initialization** in a single API. Moreover, this PEP only provides a
single choice to embed Python, instead of having two "Python" and
"Isolated" choices (:pep:`587`), to simplify the API further.

The lower level :pep:`587` ``PyConfig`` API remains available for use
cases with an intentionally higher level of coupling to CPython
implementation details (such as emulating the full functionality of
CPython's CLI, including its configuration mechanisms).


Rationale
=========

Get the runtime configuration
-----------------------------

:pep:`587` has no API to **get** the **current** runtime configuration,
only to **configure** the Python **initialization**.

For example, the global configuration variable
``Py_UnbufferedStdioFlag`` was deprecated in Python 3.12 and using
``PyConfig.buffered_stdio`` is recommended instead. It only works to
configure Python, there is no public API to get
``PyConfig.buffered_stdio``.

Users of the limited C API are asking for a public API to get the
current runtime configuration.

Cython needs to get the ``optimization_level`` configuration option:
`issue <https://github.com/python/cpython/issues/99872>`_.

When global configuration variables were deprecated in 2022, `Marc-AndrÃ©
Lemburg requested
<https://github.com/python/cpython/issues/93103#issuecomment-1136462708>`__
a C API to access these configuration variables at runtime (not only
during Python initialization).


Security fix
------------

To fix `CVE-2020-10735
<https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735>`_,
a denial-of-service when converting a very large string to an integer (in base
10), it was discussed to add a new ``PyConfig`` member to stable
branches which affects the ABI.

Gregory P. Smith proposed a different API using text based configuration
file to not be limited by ``PyConfig`` members: `FR: Allow private
runtime config to enable extending without breaking the PyConfig ABI
<https://discuss.python.org/t/fr-allow-private-runtime-config-to-enable-extending-without-breaking-the-pyconfig-abi/18004>`__
(August 2022).

In the end, it was decided to not add a new ``PyConfig`` member to
stable branches, but only add a new ``PyConfig.int_max_str_digits``
member to the development branch (which became Python 3.12). A dedicated
private global variable (unrelated to ``PyConfig``) is used in stable
branches.


Redundancy between PyPreConfig and PyConfig
-------------------------------------------

The Python preinitialization uses the ``PyPreConfig`` structure and the
Python initialization uses the ``PyConfig`` structure. Both structures
have four duplicated members: ``dev_mode``, ``parse_argv``, ``isolated``
and ``use_environment``.

The redundancy is caused by the fact that the two structures are
separated, whereas some ``PyConfig`` members are needed by the
preinitialization.


Embedding Python
----------------

Applications embedding Python
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Examples:

* `Blender 3D graphics <https://www.blender.org/>`_.
* `fontforge <https://fontforge.org/>`_ font editor.
* `Gimp <https://www.gimp.org/>`_.
* `LibreOffice <https://www.libreoffice.org/>`_.
* `OBS Studio <https://obsproject.com/>`_.
* `Tiled <https://www.mapeditor.org/>`_.
* `vim <https://www.vim.org/>`_ text editor.

On Linux, FreeBSD and macOS, applications are usually either statically
linked to a ``libpython``, or load dynamically a ``libpython`` . The
``libpython`` shared library is versioned, example:
``libpython3.12.so`` for Python 3.12 on Linux.

The vim project can target the stable ABI. Usually, the "system Python"
version is used. It's not currently possible to select which Python
version to use. Users would like the ability to select a newer Python
on demand.

On Linux, another approach to deploy an application embedding Python,
such as GIMP, is to include Python in Flatpack, AppImage or Snap
"container". In this case, the application brings its own copy of Python
version with the container.

Libraries embedding Python
^^^^^^^^^^^^^^^^^^^^^^^^^^

Examples:

* `Apache mod_wsgi <https://modwsgi.readthedocs.io/>`_
  (`source <https://github.com/GrahamDumpleton/mod_wsgi/blob/f54eadd6da8e3da0faccd497d4165de435b97242/src/server/wsgi_interp.c#L2367-L2404>`__).
* `nimpy <https://github.com/yglukhov/nimpy>`_:
  Nim - Python bridge.
* `PyO3 <https://github.com/PyO3/pyo3>`__:
  Rust bindings for the Python interpreter.

Utilities creating standalone applications
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* `py2app <https://py2app.readthedocs.io/>`_ for macOS.
* `py2exe <http://www.py2exe.org/>`_ for Windows.
* `pyinstaller <https://pyinstaller.org/>`_.
* `PyOxidizer <https://github.com/indygreg/PyOxidizer>`_:
  it uses the PEP 587 PyConfig API.

These utilities create standalone applications, they are not linked to
libpython.

Set the runtime configuration
-----------------------------

`Marc-AndrÃ© Lemburg requested
<https://discuss.python.org/t/fr-allow-private-runtime-config-to-enable-extending-without-breaking-the-pyconfig-abi/18004/34>`__
a C API to **set** the value of some configuration options at runtime:

* ``optimization_level``
* ``verbose``
* ``parser_debug``
* ``inspect``
* ``write_bytecode``

Previously, it was possible to set directly global configuration
variables:

* ``Py_OptimizeFlag``
* ``Py_VerboseFlag``
* ``Py_DebugFlag``
* ``Py_InspectFlag``
* ``Py_DontWriteBytecodeFlag``

But these configuration flags were deprecated in Python 3.12 and are
scheduled for removal in Python 3.14.


Specification
=============

Add C API functions and structure to configure the Python
initialization:

* Create config:

  * ``PyInitConfig`` opaque structure.
  * ``PyInitConfig_Create()``.
  * ``PyInitConfig_Free(config)``.

* Get options:

  * ``PyInitConfig_HasOption(config, name)``.
  * ``PyInitConfig_GetInt(config, name, &value)``.
  * ``PyInitConfig_GetStr(config, name, &value)``.
  * ``PyInitConfig_GetStrList(config, name, &length, &items)``.
  * ``PyInitConfig_FreeStrList()``.

* Set options:

  * ``PyInitConfig_SetInt(config, name, value)``.
  * ``PyInitConfig_SetStr(config, name, value)``.
  * ``PyInitConfig_SetStrList(config, name, length, items)``.
  * ``PyInitConfig_AddModule(config, name, initfunc)``

* Initialize:

  * ``Py_InitializeFromInitConfig(config)``.

* Error handling:

  * ``PyInitConfig_GetError(config, &err_msg)``.
  * ``PyInitConfig_GetExitcode(config, &exitcode)``.

Add C API functions to get and set the current runtime configuration:

* ``PyConfig_Get(name)``.
* ``PyConfig_GetInt(name, &value)``.
* ``PyConfig_Set(name)``.
* ``PyConfig_Names()``.

The C API uses null-terminated UTF-8 encoded strings to refer to a
configuration option name.

These C API functions are excluded from the limited C API.

PyInitConfig structu


id: pep-0755.rst
content:
PEP: 755
Title: Implicit namespace policy for PyPI
Author: Ofek Lev <ofekmeister@gmail.com>
Sponsor: Barry Warsaw <barry@python.org>
PEP-Delegate: Dustin Ingram <di@python.org>
Discussions-To: https://discuss.python.org/t/63191
Status: Draft
Type: Process
Topic: Packaging
Created: 05-Sep-2024
Post-History: `07-Sep-2024 <https://discuss.python.org/t/63191>`__,

Abstract
========

This PEP codifies an implementation of :pep:`752` for PyPI [1]_.

Motivation
==========

Many projects and communities would benefit from the ability to reserve
namespaces. Since PyPI exists to serve the Python community, it is critical
to gather feedback to ensure that everyone's needs are met.

A dedicated PEP is required because the operational and policy nuances are up
to each package repository to decide.

Rationale
=========

PyPI has been understaffed, receiving the first `dedicated specialist`__ in
July 2024. Due to lack of resources, user support has been lacking for
`package name claims <https://discuss.python.org/t/27436/19>`__,
`organization requests <https://discuss.python.org/t/33764/15>`__,
`storage limit increases <https://discuss.python.org/t/54035>`__,
and even `account recovery <https://discuss.python.org/t/43422/122>`__.

__ https://pyfound.blogspot.com/2024/07/announcing-our-new-pypi-support.html

The `default policy <approval-criteria_>`_ of giving paid organizations more
leniency when reserving namespaces provides the following benefits:

* PyPI would have a constant source of funding for support specialists,
  infrastructure maintenance, bug fixes and new features.
* Although each application would require independent review, less human
  feedback would be required because the process to approve a paid organization
  already bestows a certain amount of trust.

Terminology
===========

Paid/Corporate Organization
    `Corporate organizations`__ are :pep:`organizations <752#organizations>`
    that pay for special functionality on PyPI. This PEP refers to them as
    paid in most circumstances for brevity and to ease understanding for
    non-native speakers.
Root Grant
    A grant as defined by :pep:`PEP 752 terminology <752#terminology>`.
Child Grant
    A grant created from a root grant with the associated namespace being a
    child namespace as defined by :pep:`PEP 752 terminology <752#terminology>`.

__ https://docs.pypi.org/organization-accounts/pricing-and-payments/#corporate-organizations

Implementation
==============

Grant Applications
------------------

Submission
''''''''''

Only organization (non-user) accounts have access to the grant application
form.

Applications for paid organizations receive priority in the reviewing queue.
This is both to offer a meaningful benefit to paid organizations and to ensure
that funding is available for PyPI's operational costs, including more
reviewers.

.. _approval-criteria:

Approval Criteria
'''''''''''''''''

1. The namespace must not be something common like ``tool`` or ``apps``.
2. The namespace should be greater than three characters.
3. The namespace should properly and clearly identify the reservation owner.
4. The organization should be actively using the namespace.
5. There should be evidence that *not* reserving the namespace may cause
   ambiguity, confusion, or other harm to the community.

Organizations that are not paid organizations will represent one of the
following:

* Large, popular open-source projects with many packages
* Universities that actively publish packages
* Government organizations that actively publish packages
* NPOs/NGOs that actively publish packages like
  `Our World in Data <https://github.com/owid>`__

Generally speaking, reviewers should be more tolerant of paid organizations
that apply for grants for which they are not yet using.

For example, while it's reasonable to grant a namespace to a startup or an
existing company with a new product line, it's not as reasonable to grant a
namespace to a community project that doesn't have many users.

Rejections
''''''''''

Rejected applications will receive clear rationale for the decision based on
the approval criteria. Applications rejected due to the namespace being too
common will be persisted internally for future reviewers to reference and new
applications attempting to reserve a namespace that was previously rejected
for that reason will display a warning.

Acceptance
''''''''''

When an application is accepted for a namespace that is used by projects
outside of the organization, an email will be sent to the owners of the
projects notifying them of the new grant. The email will contain a link to the
`namespace's page <namespace-page_>`_.

Grant Types
-----------

There are two types of grants.

.. _root-grant:

Root Grant
''''''''''

An organization gets a root grant for every approved application. This grant
may produce any number of `child grants <child-grant_>`_.

.. _child-grant:

Child Grant
'''''''''''

A child grant may be created by the owner of a `root grant <root-grant_>`_ at
any time without approval. The namespace associated with such grants must be a
child namespace of the root grant's namespace.

Child grants cannot have their own child grants.

.. _grant-ownership:

Grant Ownership
---------------

The owner of a grant may allow any number of other organizations to use the
grant. The grants behave as if they were owned by the organization, i.e. even
the owner cannot upload packages to the namespace. The owner may revoke this
permission at any time.

The owner may transfer ownership to another organization at any time without
approval from PyPI admins. If the organization is a paid organization, the
target for transfer must also be a paid organization. Settings for permitted
organizations are transferred as well.

.. _user-interface:

User Interface
--------------

.. _namespace-page:

Namespace Page
''''''''''''''

The namespace of every active grant will have its own page that has information
such as its :pep:`open <752#open-namespaces>` status, the current owners, the
time at which ownership was granted and the total number of projects that match
the namespace.

Project Page
''''''''''''

Every project's page
(`example <https://pypi.org/project/google-cloud-compute/1.19.2/>`__) that
matches an active namespace grant will indicate what the prefix is (NuGet
currently does not do this) and will stand out as a pill or label. This value
will match the ``prefix`` key in the
:pep:`namespace detail API <752#namespace-detail>`.

Clicking on the namespace will take the user to
`its page <namespace-page_>`_.

Visual Indicators
'''''''''''''''''

For projects that match an active namespace grant, users will be able to
quickly ascertain which of the following scenarios apply:

1. Projects that are tied to a `grant owner <grant-ownership_>`_ will not have
   a visual indicator and users should solely rely on the always-present
   prefix.
2. Projects that are not tied to a `grant owner <grant-ownership_>`_ and the
   matching grant is :pep:`open <752#open-namespaces>` will have a unique
   indicator that does not convey mistrust or danger. A good choice might be
   the `users <https://fontawesome.com/icons/users>`_ icon from Font Awesome or
   the `groups`__ icon from Google Fonts.
3. Projects that are not tied to a `grant owner <grant-ownership_>`_ and the
   matching grant is restricted will have a unique visual indicator. This
   situation arises when the project existed before the grant was created.
   The indicator will convey inauthenticity or lack of trust. A good choice
   might be a warning sign (â ).

__ https://fonts.google.com/icons?selected=Material+Symbols+Outlined:groups

Open Namespaces
---------------

When a `child grant <child-grant_>`_ is created, its
:pep:`open <752#open-namespaces>` status will be inherited from the
`root grant <root-grant_>`_. Owners of child grants may make them open at any
time. If a grant is open, it cannot be made restricted unless the owner of the
grant is the owne


id: pep-0580.rst
content:
PEP: 580
Title: The C call protocol
Author: Jeroen Demeyer <J.Demeyer@UGent.be>
BDFL-Delegate: Petr Viktorin
Status: Rejected
Type: Standards Track
Created: 14-Jun-2018
Python-Version: 3.8
Post-History: 20-Jun-2018, 22-Jun-2018, 16-Jul-2018


Rejection Notice
================

This PEP is rejected in favor of :pep:`590`, which proposes a simpler public
C API for callable objects.


Abstract
========

A new "C call" protocol is proposed.
It is meant for classes representing functions or methods
which need to implement fast calling.
The goal is to generalize all existing optimizations for built-in functions
to arbitrary extension types.

In the reference implementation,
this new protocol is used for the existing classes
``builtin_function_or_method`` and ``method_descriptor``.
However, in the future, more classes may implement it.

**NOTE**: This PEP deals only with the Python/C API,
it does not affect the Python language or standard library.


Motivation
==========

The standard function/method classes ``builtin_function_or_method``
and ``method_descriptor`` allow very efficiently calling C code.
However, they are not subclassable, making them unsuitable for many applications:
for example, they offer limited introspection support
(signatures only using ``__text_signature__``, no arbitrary ``__qualname__``,
no ``inspect.getfile()``).
It's also not possible to store additional data to implement something like
``functools.partial`` or ``functools.lru_cache``.
So, there are many reasons why users would want to implement custom
function/method classes (in a duck-typing sense) in C.
Unfortunately, such custom classes are necessarily slower than
the standard CPython function classes:
the bytecode interpreter has various optimizations
which are specific to instances of
``builtin_function_or_method``, ``method_descriptor``, ``method`` and ``function``.

This PEP also allows to simplify existing code:
checks for ``builtin_function_or_method`` and ``method_descriptor``
could be replaced by simply checking for and using the C call protocol.
Future PEPs may implement the C call protocol for more classes,
enabling even further simplifications.

We also design the C call protocol such that it can easily
be extended with new features in the future.

For more background and motivation, see :pep:`579`.


Overview
========

Currently, CPython has multiple optimizations for fast calling
for a few specific function classes.
A good example is the implementation of the opcode ``CALL_FUNCTION``,
which has the following structure
(`see the actual code <https://github.com/python/cpython/blob/7a2368063f25746d4008a74aca0dc0b82f86ff7b/Python/ceval.c#L4592>`_)::

    if (PyCFunction_Check(func)) {
        return _PyCFunction_FastCallKeywords(func, stack, nargs, kwnames);
    }
    else if (Py_TYPE(func) == &PyMethodDescr_Type) {
        return _PyMethodDescr_FastCallKeywords(func, stack, nargs, kwnames);
    }
    else {
        if (PyMethod_Check(func) && PyMethod_GET_SELF(func) != NULL) {
            /* ... */
        }
        if (PyFunction_Check(func)) {
            return _PyFunction_FastCallKeywords(func, stack, nargs, kwnames);
        }
        else {
            return _PyObject_FastCallKeywords(func, stack, nargs, kwnames);
        }
    }

Calling instances of these special-cased classes
using the ``tp_call`` slot is slower than using the optimizations.
The basic idea of this PEP is to enable such optimizations
for user C code, both as caller and as callee.

The existing class ``builtin_function_or_method`` and a few others
use a ``PyMethodDef`` structure for describing the underlying C function and its signature.
The first concrete change is that this is replaced by a new structure ``PyCCallDef``.
This stores some of the same information as a ``PyMethodDef``,
but with one important addition:
the "parent" of the function (the class or module where it is defined).
Note that ``PyMethodDef`` arrays are still used to construct
functions/methods but no longer for calling them.

Second, we want that every class can use such a ``PyCCallDef`` for optimizing calls,
so the ``PyTypeObject`` structure gains a ``tp_ccalloffset`` field
giving an offset to a ``PyCCallDef *`` in the object structure
and a flag ``Py_TPFLAGS_HAVE_CCALL`` indicating that ``tp_ccalloffset`` is valid.

Third, since we want to deal efficiently with unbound and bound methods too
(as opposed to only plain functions), we need to handle ``__self__`` in the protocol:
after the ``PyCCallDef *`` in the object structure,
there is a ``PyObject *self`` field.
These two fields together are referred to as a ``PyCCallRoot`` structure.

The new protocol for efficiently calling objects using these new structures
is called the "C call protocol".

**NOTE**: In this PEP, the phrases "unbound method" and "bound method"
refer to generic behavior, not to specific classes.
For example, an unbound method gets turned into a bound method
after applying ``__get__``.


New data structures
===================

The ``PyTypeObject`` structure gains a new field ``Py_ssize_t tp_ccalloffset``
and a new flag ``Py_TPFLAGS_HAVE_CCALL``.
If this flag is set, then ``tp_ccalloffset`` is assumed to be a valid
offset inside the object structure (similar to ``tp_dictoffset`` and ``tp_weaklistoffset``).
It must be a strictly positive integer.
At that offset, a ``PyCCallRoot`` structure appears::

    typedef struct {
        const PyCCallDef *cr_ccall;
        PyObject         *cr_self;  /* __self__ argument for methods */
    } PyCCallRoot;

The ``PyCCallDef`` structure contains everything needed to describe how
the function can be called::

    typedef struct {
        uint32_t  cc_flags;
        PyCFunc   cc_func;    /* C function to call */
        PyObject *cc_parent;  /* class or module */
    } PyCCallDef;

The reason for putting ``__self__`` outside of ``PyCCallDef``
is that ``PyCCallDef`` is not meant to be changed after creating the function.
A single ``PyCCallDef`` can be shared
by an unbound method and multiple bound methods.
This wouldn't work if we would put ``__self__`` inside that structure.

**NOTE**: unlike ``tp_dictoffset`` we do not allow negative numbers
for ``tp_ccalloffset`` to mean counting from the end.
There does not seem to be a use case for it and it would only complicate
the implementation.

Parent
------

The ``cc_parent`` field (accessed for example by a ``__parent__``
or ``__objclass__`` descriptor from Python code) can be any Python
object, or NULL.
Custom classes are free to set ``cc_parent`` to whatever they want.
It is only used by the C call protocol if the
``CCALL_OBJCLASS`` flag is set.

For methods of extension types, ``cc_parent`` points to the class
that defines the method (which may be a superclass of ``type(self)``).
This is currently non-trivial to retrieve from a method's code.
In the future, this can be used to access the module state via
the defining class. See the rationale of :pep:`573` for details.

When the flag ``CCALL_OBJCLASS`` is set (as it will be for methods of
extension types), ``cc_parent`` is used for type checks like the following::

    >>> list.append({}, "x")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: descriptor 'append' requires a 'list' object but received a 'dict'

For functions of modules, ``cc_parent`` is set to the module.
Currently, this is exactly the same as ``__self__``.
However, using ``__self__`` for the module is a quirk of the current implementation:
in the future, we want to allow functions which use ``__self__``
in the normal way, for implementing methods.
Such functions can still use ``cc_parent`` instead to refer to the module.

The parent would also typically be used to implement ``__qualname__``.
The new C API function ``PyCCall_GenericGetQualname()`` does exactly that.

Using tp_print
--------------

We propose to replace the existing unused field ``tp_print``
by ``tp_ccalloffset``.
Since ``Py_TPFLAGS_HAVE_CCAL


id: pep-0594.rst
content:
PEP: 594
Title: Removing dead batteries from the standard library
Author: Christian Heimes <christian@python.org>,
        Brett Cannon <brett@python.org>
Discussions-To: https://discuss.python.org/t/13508
Status: Final
Type: Standards Track
Created: 20-May-2019
Python-Version: 3.11
Post-History: 21-May-2019, 04-Feb-2022
Resolution: https://discuss.python.org/t/13508/22


Abstract
========

This PEP proposed a list of standard library modules to be removed from the
standard library. The modules are mostly historic data formats (e.g. Commodore
and SUN file formats), APIs and operating systems that have been superseded a
long time ago (e.g. Mac OS 9), or modules that have security implications and
better alternatives (e.g. password and login).

The PEP follows in the footsteps of other PEPS like :pep:`3108`. The
*Standard Library Reorganization* proposal removed a bunch of modules from
Python 3.0. In 2007, the PEP referred to maintenance burden as:

   "Over the years, certain modules have become a heavy burden upon python-dev
   to maintain. In situations like this, it is better for the module to be
   given to the community to maintain to free python-dev to focus more on
   language support and other modules in the standard library that do not take
   up an undue amount of time and effort."

The withdrawn :pep:`206` from 2000 expresses issues with the Python standard
library in an unvarnished and forthright manner:

   "[...] the standard library modules aren't always the best choices for a
   job. Some library modules were quick hacks (e.g. ``calendar``,
   ``commands``), some were designed poorly and are now near-impossible to
   fix (``cgi``), and some have been rendered obsolete by other, more complete
   modules [...]."


Rationale
=========

Back in the early days of Python, the interpreter came with a large set of
useful modules. This was often referred to as "batteries included"
philosophy and was one of the cornerstones to Python's success story.
Users didn't have to figure out how to download and install separate
packages in order to write a simple web server or parse email.

Times have changed. With the introduction of PyPI (nÃ©e Cheeseshop), setuptools,
and later pip, it became simple and straightforward to download and install
packages. Nowadays Python has a rich and vibrant ecosystem of third-party
packages. It's pretty much standard to either install packages from PyPI or
use one of the many Python or Linux distributions.

On the other hand, Python's standard library is piling up with cruft, unnecessary
duplication of functionality, and dispensable features. This is undesirable
for several reasons.

* Any additional module increases the maintenance cost for the Python core
  development team. The team has limited resources, reduced maintenance cost
  frees development time for other improvements.
* Modules in the standard library are generally favored and seen as the
  de facto solution for a problem. A majority of users only pick third-party
  modules to replace a stdlib module, when they have a compelling reason, e.g.
  ``lxml`` instead of ``xml``. The removal of an unmaintained stdlib module
  increases the chances of a community-contributed module to become widely
  used.
* A lean and mean standard library benefits platforms with limited resources
  like devices with just a few hundred kilobyte of storage (e.g. BBC
  Micro:bit). Python on mobile platforms like BeeWare or WebAssembly
  (e.g. pyodide) also benefit from reduced download size.

The modules in this PEP have been selected for deprecation because their
removal is either least controversial or most beneficial. For example,
least controversial are 30-year-old multimedia formats like the ``sunau``
audio format, which was used on SPARC and NeXT workstations in the late
1980s. The ``crypt`` module has fundamental flaws that are better solved
outside the standard library.

This PEP also designates some modules as not scheduled for removal. Some
modules have been deprecated for several releases or seem unnecessary at
first glance. However it is beneficial to keep the modules in the standard
library, mostly for environments where installing a package from PyPI is not
an option. This can be corporate environments or classrooms where external
code is not permitted without legal approval.

* The usage of FTP is declining, but some files are still provided over
  the FTP protocol or hosters offer FTP to upload content. Therefore,
  ``ftplib`` is going to stay.
* The ``optparse`` and ``getopt`` modules are widely used. They are mature
  modules with very low maintenance overhead.
* According to David Beazley [5]_ the ``wave`` module is easy to teach to
  kids and can make crazy sounds. Making a computer generate sounds is a
  powerful and highly motivating exercise for a nine-year-old aspiring developer.
  It's a fun battery to keep.


Deprecation schedule
====================

3.11
----

Starting with Python 3.11, deprecated modules will start issuing
``DeprecationWarning``. The estimated EOL of Python 3.10, the last
version without the warning, is October 2026.

3.12
----

There should be no specific change compared to Python 3.11.
This is the last version of Python with the deprecated modules,
with an estimated EOL of October 2028.

3.13
----

All modules deprecated by this PEP are removed from the ``main`` branch
of the CPython repository and are no longer distributed as part of Python.


Deprecated modules
==================

The modules are grouped as data encoding, multimedia, network, OS interface,
and misc modules. The majority of modules are for old data formats or
old APIs. Some others are rarely useful and have better replacements on
PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with
audio processing.

.. csv-table:: Table 1: Proposed modules deprecations
   :header: "Module", "Deprecated in", "To be removed", "Added in", "Has maintainer?", "Replacement"
   :widths: 2, 1, 1, 1, 1, 2

    aifc,3.11 (3.0\*),3.13,1993,**yes (inactive)**,\-
    asynchat,**3.6** (3.0\*),3.12,1999,**yes**,:mod:`asyncio`
    asyncore,**3.6** (3.0\*),3.12,1999,**yes**,:mod:`asyncio`
    audioop,3.11 (3.0\*),3.13,1992,**yes**,\-
    cgi,3.11 (2.0\*\*),3.13,1995,no,\-
    cgitb,3.11 (2.0\*\*),3.13,1995,no,\-
    chunk,3.11,3.13,1999,no,\-
    crypt,3.11,3.13,1994,**yes (inactive)**,":pypi:`legacycrypt`, :pypi:`bcrypt`, :pypi:`argon2-cffi`, :mod:`hashlib`, :pypi:`passlib`"
    imghdr,3.11,3.13,1992,no,":pypi:`filetype`, :pypi:`puremagic`, :pypi:`python-magic`"
    mailcap,3.11,3.13,1995,no,\-
    msilib,3.11,3.13,2006,no,\-
    nntplib,3.11,3.13,1992,no,\-
    nis,3.11 (3.0\*),3.13,1992,no,\-
    ossaudiodev,3.11,3.13,2002,no,\-
    pipes,3.11,3.13,1992,no,:mod:`subprocess`
    smtpd,"**3.4.7**, **3.5.4**",3.12,2001,**yes**,:pypi:`aiosmtpd`
    sndhdr,3.11,3.13,1994,no,":pypi:`filetype`, :pypi:`puremagic`, :pypi:`python-magic`"
    spwd,3.11,3.13,2005,no,:pypi:`python-pam`
    sunau,3.11 (3.0\*),3.13,1993,no,\-
    telnetlib,3.11 (3.0\*),3.13,1997,no,":pypi:`telnetlib3`, :pypi:`Exscript`"
    uu,3.11,3.13,1994,no,\-
    xdrlib,3.11,3.13,1992/1996,no,\-

Some module deprecations proposed by :pep:`3108` for 3.0 and :pep:`206` for
2.0. The *added in* column illustrates, when a module was originally designed
and added to the standard library. The *has maintainer* column refers to the
`expert index <https://devguide.python.org/experts/>`_, a list of domain
experts and maintainers in the DevGuide.


Data encoding modules
---------------------


uu and the uu encoding
~~~~~~~~~~~~~~~~~~~~~~

The :external+py3.12:mod:`uu` module provides
uuencode format, an old binary encoding format for email from 1980. The uu
format has been replaced by MIME. The uu codec is provided by the ``binascii``
module.  There's also ``encodings/uu_codec.py`` which is a codec for the
same encoding; it should also be deprecated.


xdrlib
~~~~~~

The :external+py3.12:mod:`xdrlib` module supports
the Sun Ext


id: pep-0219.rst
content:
PEP: 219
Title: Stackless Python
Author: Gordon McMillan <gmcm@hypernet.com>
Status: Deferred
Type: Standards Track
Created: 14-Aug-2000
Python-Version: 2.1
Post-History:


Introduction
============

This PEP discusses changes required to core Python in order to
efficiently support generators, microthreads and coroutines. It is
related to :pep:`220`, which describes how Python should be extended
to support these facilities. The focus of this PEP is strictly on
the changes required to allow these extensions to work.

While these PEPs are based on Christian Tismer's Stackless [1]_
implementation, they do not regard Stackless as a reference
implementation.  Stackless (with an extension module) implements
continuations, and from continuations one can implement
coroutines, microthreads (as has been done by Will Ware [2]_) and
generators. But in more than a year, no one has found any other
productive use of continuations, so there seems to be no demand
for their support.

However, Stackless support for continuations is a relatively minor
piece of the implementation, so one might regard it as "a"
reference implementation (rather than "the" reference
implementation).


Background
==========

Generators and coroutines have been implemented in a number of
languages in a number of ways. Indeed, Tim Peters has done pure
Python implementations of generators [3]_ and coroutines [4]_ using
threads (and a thread-based coroutine implementation exists for
Java). However, the horrendous overhead of a thread-based
implementation severely limits the usefulness of this approach.

Microthreads (a.k.a "green" or "user" threads) and coroutines
involve transfers of control that are difficult to accommodate in
a language implementation based on a single stack. (Generators can
be done on a single stack, but they can also be regarded as a very
simple case of coroutines.)

Real threads allocate a full-sized stack for each thread of
control, and this is the major source of overhead. However,
coroutines and microthreads can be implemented in Python in a way
that involves almost no overhead.  This PEP, therefore, offers a
way for making Python able to realistically manage thousands of
separate "threads" of activity (vs. today's limit of perhaps dozens
of separate threads of activity).

Another justification for this PEP (explored in :pep:`220`) is that
coroutines and generators often allow a more direct expression of
an algorithm than is possible in today's Python.


Discussion
==========

The first thing to note is that Python, while it mingles
interpreter data (normal C stack usage) with Python data (the
state of the interpreted program) on the stack, the two are
logically separate. They just happen to use the same stack.

A real thread gets something approaching a process-sized stack
because the implementation has no way of knowing how much stack
space the thread will require. The stack space required for an
individual frame is likely to be reasonable, but stack switching
is an arcane and non-portable process, not supported by C.

Once Python stops putting Python data on the C stack, however,
stack switching becomes easy.

The fundamental approach of the PEP is based on these two
ideas. First, separate C's stack usage from Python's stack
usage. Secondly, associate with each frame enough stack space to
handle that frame's execution.

In the normal usage, Stackless Python has a normal stack
structure, except that it is broken into chunks. But in the
presence of a coroutine / microthread extension, this same
mechanism supports a stack with a tree structure.  That is, an
extension can support transfers of control between frames outside
the normal "call / return" path.


Problems
========

The major difficulty with this approach is C calling Python. The
problem is that the C stack now holds a nested execution of the
byte-code interpreter. In that situation, a coroutine /
microthread extension cannot be permitted to transfer control to a
frame in a different invocation of the byte-code interpreter. If a
frame were to complete and exit back to C from the wrong
interpreter, the C stack could be trashed.

The ideal solution is to create a mechanism where nested
executions of the byte code interpreter are never needed. The easy
solution is for the coroutine / microthread extension(s) to
recognize the situation and refuse to allow transfers outside the
current invocation.

We can categorize code that involves C calling Python into two
camps: Python's implementation, and C extensions. And hopefully we
can offer a compromise: Python's internal usage (and C extension
writers who want to go to the effort) will no longer use a nested
invocation of the interpreter. Extensions which do not go to the
effort will still be safe, but will not play well with coroutines
/ microthreads.

Generally, when a recursive call is transformed into a loop, a bit
of extra bookkeeping is required. The loop will need to keep its
own "stack" of arguments and results since the real stack can now
only hold the most recent. The code will be more verbose, because
it's not quite as obvious when we're done. While Stackless is not
implemented this way, it has to deal with the same issues.

In normal Python, ``PyEval_EvalCode`` is used to build a frame and
execute it. Stackless Python introduces the concept of a
``FrameDispatcher``. Like ``PyEval_EvalCode``, it executes one frame. But
the interpreter may signal the ``FrameDispatcher`` that a new frame
has been swapped in, and the new frame should be executed. When a
frame completes, the ``FrameDispatcher`` follows the back pointer to
resume the "calling" frame.

So Stackless transforms recursions into a loop, but it is not the
``FrameDispatcher`` that manages the frames. This is done by the
interpreter (or an extension that knows what it's doing).

The general idea is that where C code needs to execute Python
code, it creates a frame for the Python code, setting its back
pointer to the current frame. Then it swaps in the frame, signals
the ``FrameDispatcher`` and gets out of the way. The C stack is now
clean - the Python code can transfer control to any other frame
(if an extension gives it the means to do so).

In the vanilla case, this magic can be hidden from the programmer
(even, in most cases, from the Python-internals programmer). Many
situations present another level of difficulty, however.

The map builtin function involves two obstacles to this
approach. It cannot simply construct a frame and get out of the
way, not just because there's a loop involved, but each pass
through the loop requires some "post" processing. In order to play
well with others, Stackless constructs a frame object for map
itself.

Most recursions of the interpreter are not this complex, but
fairly frequently, some "post" operations are required. Stackless
does not fix these situations because of the amount of code changes
required. Instead, Stackless prohibits transfers out of a nested
interpreter. While not ideal (and sometimes puzzling), this
limitation is hardly crippling.


Advantages
==========

For normal Python, the advantage to this approach is that C stack
usage becomes much smaller and more predictable. Unbounded
recursion in Python code becomes a memory error, instead of a
stack error (and thus, in non-Cupertino operating systems,
something that can be recovered from).  The price, of course, is
the added complexity that comes from transforming recursions of
the byte-code interpreter loop into a higher order loop (and the
attendant bookkeeping involved).

The big advantage comes from realizing that the Python stack is
really a tree, and the frame dispatcher can transfer control
freely between leaf nodes of the tree, thus allowing things like
microthreads and coroutines.


References
==========

.. [1] http://www.stackless.com
.. [2] http://web.archive.org/web/20000815070602/http://world.std.com/~wware/uthread.html
.. [3] Demo/threads/Generator.py in the source distribution
.. [4] ht


id: pep-0543.rst
content:
PEP: 543
Title: A Unified TLS API for Python
Author: Cory Benfield <cory@lukasa.co.uk>,
        Christian Heimes <christian@python.org>
Status: Withdrawn
Type: Standards Track
Created: 17-Oct-2016
Python-Version: 3.7
Post-History: 11-Jan-2017, 19-Jan-2017, 02-Feb-2017, 09-Feb-2017
Superseded-By: 748

Abstract
========

This PEP would define a standard TLS interface in the form of a collection of
abstract base classes. This interface would allow Python implementations and
third-party libraries to provide bindings to TLS libraries other than OpenSSL
that can be used by tools that expect the interface provided by the Python
standard library, with the goal of reducing the dependence of the Python
ecosystem on OpenSSL.


Resolution
==========

2020-06-25: With contemporary agreement with one author, and past
agreement with another, this PEP is withdrawn due to changes in the
APIs of the underlying operating systems.


Rationale
=========

In the 21st century it has become increasingly clear that robust and
user-friendly TLS support is an extremely important part of the ecosystem of
any popular programming language. For most of its lifetime, this role in the
Python ecosystem has primarily been served by the `ssl module`_, which provides
a Python API to the `OpenSSL library`_.

Because the ``ssl`` module is distributed with the Python standard library, it
has become the overwhelmingly most-popular method for handling TLS in Python.
An extraordinary majority of Python libraries, both in the standard library and
on the Python Package Index, rely on the ``ssl`` module for their TLS
connectivity.

Unfortunately, the preeminence of the ``ssl`` module has had a number of
unforeseen side-effects that have had the effect of tying the entire Python
ecosystem tightly to OpenSSL. This has forced Python users to use OpenSSL even
in situations where it may provide a worse user experience than alternative TLS
implementations, which imposes a cognitive burden and makes it hard to provide
"platform-native" experiences.


Problems
--------

The fact that the ``ssl`` module is built into the standard library has meant
that all standard-library Python networking libraries are entirely reliant on
the OpenSSL that the Python implementation has been linked against. This
leads to the following issues:

* It is difficult to take advantage of new, higher-security TLS without
  recompiling Python to get a new OpenSSL. While there are third-party bindings
  to OpenSSL (e.g. `pyOpenSSL`_), these need to be shimmed into a format that
  the standard library understands, forcing projects that want to use them to
  maintain substantial compatibility layers.

* For Windows distributions of Python, they need to be shipped with a copy of
  OpenSSL. This puts the CPython development team in the position of being
  OpenSSL redistributors, potentially needing to ship security updates to the
  Windows Python distributions when OpenSSL vulnerabilities are released.

* For macOS distributions of Python, they need either to be shipped with a copy
  of OpenSSL or linked against the system OpenSSL library. Apple has formally
  deprecated linking against the system OpenSSL library, and even if they had
  not, that library version has been unsupported by upstream for nearly one
  year as of the time of writing. The CPython development team has started
  shipping newer OpenSSLs with the Python available from python.org, but this
  has the same problem as with Windows.

* Many systems, including but not limited to Windows and macOS, do not make
  their system certificate stores available to OpenSSL. This forces users to
  either obtain their trust roots from elsewhere (e.g. `certifi`_) or to
  attempt to export their system trust stores in some form.

  Relying on `certifi`_ is less than ideal, as most system administrators do
  not expect to receive security-critical software updates from PyPI.
  Additionally, it is not easy to extend the `certifi`_ trust bundle to include
  custom roots, or to centrally manage trust using the `certifi`_ model.

  Even in situations where the system certificate stores are made available to
  OpenSSL in some form, the experience is still sub-standard, as OpenSSL will
  perform different validation checks than the platform-native TLS
  implementation. This can lead to users experiencing different behaviour on
  their browsers or other platform-native tools than they experience in Python,
  with little or no recourse to resolve the problem.

* Users may wish to integrate with TLS libraries other than OpenSSL for many
  other reasons, such as OpenSSL missing features (e.g. TLS 1.3 support), or
  because OpenSSL is simply too large and unwieldy for the platform (e.g. for
  embedded Python). Those users are left with the requirement to use
  third-party networking libraries that can interact with their preferred TLS
  library or to shim their preferred library into the OpenSSL-specific ``ssl``
  module API.

Additionally, the ``ssl`` module as implemented today limits the ability of
CPython itself to add support for alternative TLS backends, or remove OpenSSL
support entirely, should either of these become necessary or useful. The
``ssl`` module exposes too many OpenSSL-specific function calls and features to
easily map to an alternative TLS backend.


Proposal
========

This PEP proposes to introduce a few new Abstract Base Classes in Python 3.7 to
provide TLS functionality that is not so strongly tied to OpenSSL. It also
proposes to update standard library modules to use only the interface exposed
by these abstract base classes wherever possible. There are three goals here:

1. To provide a common API surface for both core and third-party developers to
   target their TLS implementations to. This allows TLS developers to provide
   interfaces that can be used by most Python code, and allows network
   developers to have an interface that they can target that will work with a
   wide range of TLS implementations.
2. To provide an API that has few or no OpenSSL-specific concepts leak through.
   The ``ssl`` module today has a number of warts caused by leaking OpenSSL
   concepts through to the API: the new ABCs would remove those specific
   concepts.
3. To provide a path for the core development team to make OpenSSL one of many
   possible TLS backends, rather than requiring that it be present on a system
   in order for Python to have TLS support.

The proposed interface is laid out below.


Interfaces
----------

There are several interfaces that require standardisation. Those interfaces
are:

1. Configuring TLS, currently implemented by the `SSLContext`_ class in the
   ``ssl`` module.
2. Providing an in-memory buffer for doing in-memory encryption or decryption
   with no actual I/O (necessary for asynchronous I/O models), currently
   implemented by the `SSLObject`_ class in the ``ssl`` module.
3. Wrapping a socket object, currently implemented by the `SSLSocket`_ class
   in the ``ssl`` module.
4. Applying TLS configuration to the wrapping objects in (2) and (3). Currently
   this is also implemented by the `SSLContext`_ class in the ``ssl`` module.
5. Specifying TLS cipher suites. There is currently no code for doing this in
   the standard library: instead, the standard library uses OpenSSL cipher
   suite strings.
6. Specifying application-layer protocols that can be negotiated during the
   TLS handshake.
7. Specifying TLS versions.
8. Reporting errors to the caller, currently implemented by the `SSLError`_
   class in the ``ssl`` module.
9. Specifying certificates to load, either as client or server certificates.
10. Specifying which trust database should be used to validate certificates
    presented by a remote peer.
11. Finding a way to get hold of these interfaces at run time.

For the sake of simplicity, this PEP proposes to take a unified approach to
(2) and (3) (that is, buffers and sockets). The Python socket API is a
sizeable one, and implementing a wra


id: pep-0225.rst
content:
PEP: 225
Title: Elementwise/Objectwise Operators
Author: Huaiyu Zhu <hzhu@users.sourceforge.net>,
        Gregory Lielens <gregory.lielens@fft.be>
Status: Rejected
Type: Standards Track
Created: 19-Sep-2000
Python-Version: 2.1
Post-History:


.. rejected::

   The approach in the later :pep:`465` was eventually accepted
   in lieu of this PEP. The :pep:`Rejected Ideas
   <465#rejected-alternatives-to-adding-a-new-operator>`
   of that PEP explains the rationale in more detail.


Introduction
============

This PEP describes a proposal to add new operators to Python which are useful
for distinguishing elementwise and objectwise operations, and summarizes
discussions in the news group comp.lang.python on this topic.  See Credits and
Archives section at end.  Issues discussed here include:

- Background.
- Description of proposed operators and implementation issues.
- Analysis of alternatives to new operators.
- Analysis of alternative forms.
- Compatibility issues
- Description of wider extensions and other related ideas.

A substantial portion of this PEP describes ideas that do not go into the
proposed extension.  They are presented because the extension is essentially
syntactic sugar, so its adoption must be weighed against various possible
alternatives.  While many alternatives may be better in some aspects, the
current proposal appears to be overall advantageous.

The issues concerning elementwise-objectwise operations extends to wider areas
than numerical computation.  This document also describes how the current
proposal may be integrated with more general future extensions.


Background
==========

Python provides six binary infix math operators: ``+`` ``-`` ``*`` ``/`` ``%``
``**`` hereafter generically represented by ``op``.  They can be overloaded
with new semantics for user-defined classes.  However, for objects composed of
homogeneous elements, such as arrays, vectors and matrices in numerical
computation, there are two essentially distinct flavors of semantics.  The
objectwise operations treat these objects as points in multidimensional spaces.
The elementwise operations treat them as collections of individual elements.
These two flavors of operations are often intermixed in the same formulas,
thereby requiring syntactical distinction.

Many numerical computation languages provide two sets of math operators.  For
example, in MatLab, the ordinary ``op`` is used for objectwise operation while
``.op`` is used for elementwise operation. In R, ``op`` stands for elementwise
operation while ``%op%`` stands for objectwise operation.

In Python, there are other methods of representation, some of which already
used by available numerical packages, such as:

- function:   mul(a,b)
- method:     a.mul(b)
- casting:    a.E*b

In several aspects these are not as adequate as infix operators. More details
will be shown later, but the key points are:

- Readability: Even for moderately complicated formulas, infix operators are
  much cleaner than alternatives.

- Familiarity: Users are familiar with ordinary math operators.

- Implementation: New infix operators will not unduly clutter Python syntax.
  They will greatly ease the implementation of numerical packages.

While it is possible to assign current math operators to one flavor of
semantics, there is simply not enough infix operators to overload for the other
flavor.  It is also impossible to maintain visual symmetry between these two
flavors if one of them does not contain symbols for ordinary math operators.


Proposed extension
==================

- Six new binary infix operators ``~+`` ``~-`` ``~*`` ``~/`` ``~%`` ``~**`` are
  added to core Python.  They parallel the existing operators ``+`` ``-`` ``*``
  ``/`` ``%`` ``**``.

- Six augmented assignment operators ``~+=`` ``~-=`` ``~*=`` ``~/=`` ``~%=``
  ``~**=`` are added to core Python.  They parallel the operators ``+=`` ``-=``
  ``*=`` ``/=`` ``%=`` ``**=`` available in Python 2.0.

- Operator ``~op`` retains the syntactical properties of operator ``op``,
  including precedence.

- Operator ``~op`` retains the semantical properties of operator ``op`` on
  built-in number types.

- Operator ``~op`` raise syntax error on non-number builtin types. This is
  temporary until the proper behavior can be agreed upon.

- These operators are overloadable in classes with names that prepend *t* (for
  tilde) to names of ordinary math operators. For example, ``__tadd__`` and
  ``__rtadd__`` work for ``~+`` just as ``__add__`` and ``__radd__`` work for
  ``+``.

- As with existing operators, the ``__r*__()`` methods are invoked when the
  left operand does not provide the appropriate method.

It is intended that one set of ``op`` or ``~op`` is used for elementwise
operations, the other for objectwise operations, but it is not specified which
version of operators stands for elementwise or objectwise operations, leaving
the decision to applications.

The proposed implementation is to patch several files relating to the
tokenizer, parser, grammar and compiler to duplicate the functionality of
corresponding existing operators as necessary. All new semantics are to be
implemented in the classes that overload them.

The symbol ``~`` is already used in Python as the unary *bitwise not* operator.
Currently it is not allowed for binary operators.  The new operators are
completely backward compatible.


Prototype Implementation
========================

Greg Lielens implemented the infix ``~op`` as a patch against Python 2.0b1
source [1]_.

To allow ``~`` to be part of binary operators, the tokenizer would treat ``~+``
as one token.  This means that currently valid expression ``~+1`` would be
tokenized as ``~+`` ``1`` instead of ``~ + 1``.  The parser would then treat ``~+``
as composite of ``~ +``.  The effect is invisible to applications.

Notes about current patch:

- It does not include ``~op=`` operators yet.

- The ``~op`` behaves the same as ``op`` on lists, instead of raising
  exceptions.

These should be fixed when the final version of this proposal is ready.

- It reserves ``xor`` as an infix operator with the semantics equivalent to::

    def __xor__(a, b):
        if not b: return a
        elif not a: return b
        else: 0

This preserves true value as much as possible, otherwise preserve left hand
side value if possible.

This is done so that bitwise operators could be regarded as elementwise
logical operators in the future (see below).


Alternatives to adding new operators
====================================

The discussions on comp.lang.python and python-dev mailing list explored many
alternatives.  Some of the leading alternatives are listed here, using the
multiplication operator as an example.

1. Use function ``mul(a,b)``.

   Advantage:

   - No need for new operators.

   Disadvantage:

   - Prefix forms are cumbersome for composite formulas.
   - Unfamiliar to the intended users.
   - Too verbose for the intended users.
   - Unable to use natural precedence rules.

2. Use method call ``a.mul(b)``.

   Advantage:

   - No need for new operators.

   Disadvantage:

   - Asymmetric for both operands.
   - Unfamiliar to the intended users.
   - Too verbose for the intended users.
   - Unable to use natural precedence rules.

3. Use *shadow classes*.  For matrix class define a shadow array class
   accessible through a method ``.E``, so that for matrices *a* and *b*,
   ``a.E*b`` would be a matrix object that is ``elementwise_mul(a,b)``.

   Likewise define a shadow matrix class for arrays accessible through a method
   ``.M`` so that for arrays *a* and *b*, ``a.M*b`` would be an array that is
   ``matrixwise_mul(a,b)``.

   Advantage:

   - No need for new operators.
   - Benefits of infix operators with correct precedence rules.
   - Clean formulas in applications.

   Disadvantage:

   - Hard to maintain in current Python because ordinary numbers cannot have
     user defined class methods; i.e. ``a.E*b`` will fail if a is a pure
     n


id: pep-0231.rst
content:
PEP: 231
Title: __findattr__()
Author: Barry Warsaw <barry@python.org>
Status: Rejected
Type: Standards Track
Created: 30-Nov-2000
Python-Version: 2.1
Post-History:


Introduction
============

This PEP describes an extension to instance attribute lookup and
modification machinery, which allows pure-Python implementations
of many interesting programming models.  This PEP tracks the
status and ownership of this feature.  It contains a description
of the feature and outlines changes necessary to support the
feature.  This PEP summarizes discussions held in mailing list
forums, and provides URLs for further information, where
appropriate.  The CVS revision history of this file contains the
definitive historical record.


Background
==========

The semantics for Python instances allow the programmer to
customize some aspects of attribute lookup and attribute
modification, through the special methods ``__getattr__()`` and
``__setattr__()`` [1]_.

However, because of certain restrictions imposed by these methods,
there are useful programming techniques that can not be written in
Python alone, e.g. strict Java Bean-like [2]_ interfaces and Zope
style acquisitions [3]_.  In the latter case, Zope solves this by
including a C extension called ExtensionClass [5]_ which modifies
the standard class semantics, and uses a metaclass hook in
Python's class model called alternatively the "Don Beaudry Hook"
or "Don Beaudry Hack" [6]_.

While Zope's approach works, it has several disadvantages.  First,
it requires a C extension.  Second it employs a very arcane, but
truck-sized loophole in the Python machinery.  Third, it can be
difficult for other programmers to use and understand (the
metaclass has well-known brain exploding properties).  And fourth,
because ExtensionClass instances aren't "real" Python instances,
some aspects of the Python runtime system don't work with
ExtensionClass instances.

Proposals for fixing this problem have often been lumped under the
rubric of fixing the "class/type dichotomy"; that is, eliminating
the difference between built-in types and classes [7]_.  While a
laudable goal itself, repairing this rift is not necessary in
order to achieve the types of programming constructs described
above.  This proposal provides an 80% solution with a minimum of
modification to Python's class and instance objects.  It does
nothing to address the type/class dichotomy.


Proposal
========

This proposal adds a new special method called ``__findattr__()`` with
the following semantics:

* If defined in a class, it will be called on all instance
  attribute resolutions instead of ``__getattr__()`` and
  ``__setattr__()``.

* ``__findattr__()`` is never called recursively.  That is, when a
  specific instance's ``__findattr__()`` is on the call stack, further
  attribute accesses for that instance will use the standard
  ``__getattr__()`` and ``__setattr__()`` methods.

* ``__findattr__()`` is called for both attribute access ('getting')
  and attribute modification ('setting').  It is not called for
  attribute deletion.

* When called for getting, it is passed a single argument (not
  counting 'self'): the name of the attribute being accessed.

* When called for setting, it is called with third argument, which
  is the value to set the attribute to.

* ``__findattr__()`` methods have the same caching semantics as
  ``__getattr__()`` and ``__setattr__()``; i.e. if they are present in the
  class at class definition time, they are used, but if they are
  subsequently added to a class later they are not.


Key Differences with the Existing Protocol
==========================================

``__findattr__()``'s semantics are different from the existing
protocol in key ways:

First, ``__getattr__()`` is never called if the attribute is found in
the instance's ``__dict__``.  This is done for efficiency reasons, and
because otherwise, ``__setattr__()`` would have no way to get to the
instance's attributes.

Second, ``__setattr__()`` cannot use "normal" syntax for setting
instance attributes, e.g. "self.name = foo" because that would
cause recursive calls to ``__setattr__()``.

``__findattr__()`` is always called regardless of whether the
attribute is in ``__dict__`` or not, and a flag in the instance object
prevents recursive calls to ``__findattr__()``.  This gives the class
a chance to perform some action for every attribute access.  And
because it is called for both gets and sets, it is easy to write
similar policy for all attribute access.  Further, efficiency is
not a problem because it is only paid when the extended mechanism
is used.


Related Work
============

:pep:`213` describes a different approach to hooking into
attribute access and modification.  The semantics proposed in :pep:`213`
can be implemented using the ``__findattr__()`` hook described
here, with one caveat.  The current reference implementation of
``__findattr__()`` does not support hooking on attribute deletion.
This could be added if it's found desirable.  See example below.


Examples
========

One programming style that this proposal allows is a Java
Bean-like interface to objects, where unadorned attribute access
and modification is transparently mapped to a functional
interface.  E.g.

::

    class Bean:
        def __init__(self, x):
            self.__myfoo = x

        def __findattr__(self, name, *args):
            if name.startswith('_'):
                # Private names
                if args: setattr(self, name, args[0])
                else:    return getattr(self, name)
            else:
                # Public names
                if args: name = '_set_' + name
                else:    name = '_get_' + name
                return getattr(self, name)(*args)

        def _set_foo(self, x):
            self.__myfoo = x

        def _get_foo(self):
            return self.__myfoo


    b = Bean(3)
    print b.foo
    b.foo = 9
    print b.foo


A second, more elaborate example is the implementation of both
implicit and explicit acquisition in pure Python::

    import types

    class MethodWrapper:
        def __init__(self, container, method):
            self.__container = container
            self.__method = method

        def __call__(self, *args, **kws):
            return self.__method.im_func(self.__container, *args, **kws)


    class WrapperImplicit:
        def __init__(self, contained, container):
            self.__contained = contained
            self.__container = container

        def __repr__(self):
            return '<Wrapper: [%s | %s]>' % (self.__container,
                                             self.__contained)

        def __findattr__(self, name, *args):
            # Some things are our own
            if name.startswith('_WrapperImplicit__'):
                if args: return setattr(self, name, *args)
                else:    return getattr(self, name)
            # setattr stores the name on the contained object directly
            if args:
                return setattr(self.__contained, name, args[0])
            # Other special names
            if name == 'aq_parent':
                return self.__container
            elif name == 'aq_self':
                return self.__contained
            elif name == 'aq_base':
                base = self.__contained
                try:
                    while 1:
                        base = base.aq_self
                except AttributeError:
                    return base
            # no acquisition for _ names
            if name.startswith('_'):
                return getattr(self.__contained, name)
            # Everything else gets wrapped
            missing = []
            which = self.__contained
            obj = getattr(which, name, missing)
            if obj is missing:
                which = self.__container
                obj = getattr(which, name, missing)
                if obj is missing:
                    raise AttributeError, name
            of = getattr(ob


id: pep-0557.rst
content:
PEP: 557
Title: Data Classes
Author: Eric V. Smith <eric@trueblade.com>
Status: Final
Type: Standards Track
Created: 02-Jun-2017
Python-Version: 3.7
Post-History: 08-Sep-2017, 25-Nov-2017, 30-Nov-2017, 01-Dec-2017, 02-Dec-2017, 06-Jan-2018, 04-Mar-2018
Resolution: https://mail.python.org/pipermail/python-dev/2017-December/151034.html

Notice for Reviewers
====================

This PEP and the initial implementation were drafted in a separate
repo: https://github.com/ericvsmith/dataclasses.  Before commenting in
a public forum please at least read the discussion_ listed at the
end of this PEP.

Abstract
========

This PEP describes an addition to the standard library called Data
Classes.  Although they use a very different mechanism, Data Classes
can be thought of as "mutable namedtuples with defaults".  Because
Data Classes use normal class definition syntax, you are free to use
inheritance, metaclasses, docstrings, user-defined methods, class
factories, and other Python class features.

A class decorator is provided which inspects a class definition for
variables with type annotations as defined in :pep:`526`, "Syntax for
Variable Annotations".  In this document, such variables are called
fields.  Using these fields, the decorator adds generated method
definitions to the class to support instance initialization, a repr,
comparison methods, and optionally other methods as described in the
Specification_ section.  Such a class is called a Data Class, but
there's really nothing special about the class: the decorator adds
generated methods to the class and returns the same class it was
given.

As an example::

  @dataclass
  class InventoryItem:
      '''Class for keeping track of an item in inventory.'''
      name: str
      unit_price: float
      quantity_on_hand: int = 0

      def total_cost(self) -> float:
          return self.unit_price * self.quantity_on_hand

The ``@dataclass`` decorator will add the equivalent of these methods
to the InventoryItem class::

  def __init__(self, name: str, unit_price: float, quantity_on_hand: int = 0) -> None:
      self.name = name
      self.unit_price = unit_price
      self.quantity_on_hand = quantity_on_hand
  def __repr__(self):
      return f'InventoryItem(name={self.name!r}, unit_price={self.unit_price!r}, quantity_on_hand={self.quantity_on_hand!r})'
  def __eq__(self, other):
      if other.__class__ is self.__class__:
          return (self.name, self.unit_price, self.quantity_on_hand) == (other.name, other.unit_price, other.quantity_on_hand)
      return NotImplemented
  def __ne__(self, other):
      if other.__class__ is self.__class__:
          return (self.name, self.unit_price, self.quantity_on_hand) != (other.name, other.unit_price, other.quantity_on_hand)
      return NotImplemented
  def __lt__(self, other):
      if other.__class__ is self.__class__:
          return (self.name, self.unit_price, self.quantity_on_hand) < (other.name, other.unit_price, other.quantity_on_hand)
      return NotImplemented
  def __le__(self, other):
      if other.__class__ is self.__class__:
          return (self.name, self.unit_price, self.quantity_on_hand) <= (other.name, other.unit_price, other.quantity_on_hand)
      return NotImplemented
  def __gt__(self, other):
      if other.__class__ is self.__class__:
          return (self.name, self.unit_price, self.quantity_on_hand) > (other.name, other.unit_price, other.quantity_on_hand)
      return NotImplemented
  def __ge__(self, other):
      if other.__class__ is self.__class__:
          return (self.name, self.unit_price, self.quantity_on_hand) >= (other.name, other.unit_price, other.quantity_on_hand)
      return NotImplemented

Data Classes save you from writing and maintaining these methods.

Rationale
=========

There have been numerous attempts to define classes which exist
primarily to store values which are accessible by attribute lookup.
Some examples include:

- collections.namedtuple in the standard library.

- typing.NamedTuple in the standard library.

- The popular attrs [#]_ project.

- George Sakkis' recordType recipe [#]_, a mutable data type inspired
  by collections.namedtuple.

- Many example online recipes [#]_, packages [#]_, and questions [#]_.
  David Beazley used a form of data classes as the motivating example
  in a PyCon 2013 metaclass talk [#]_.

So, why is this PEP needed?

With the addition of :pep:`526`, Python has a concise way to specify the
type of class members.  This PEP leverages that syntax to provide a
simple, unobtrusive way to describe Data Classes.  With two exceptions,
the specified attribute type annotation is completely ignored by Data
Classes.

No base classes or metaclasses are used by Data Classes.  Users of
these classes are free to use inheritance and metaclasses without any
interference from Data Classes.  The decorated classes are truly
"normal" Python classes.  The Data Class decorator should not
interfere with any usage of the class.

One main design goal of Data Classes is to support static type
checkers.  The use of :pep:`526` syntax is one example of this, but so is
the design of the ``fields()`` function and the ``@dataclass``
decorator.  Due to their very dynamic nature, some of the libraries
mentioned above are difficult to use with static type checkers.

Data Classes are not, and are not intended to be, a replacement
mechanism for all of the above libraries.  But being in the standard
library will allow many of the simpler use cases to instead leverage
Data Classes.  Many of the libraries listed have different feature
sets, and will of course continue to exist and prosper.

Where is it not appropriate to use Data Classes?

- API compatibility with tuples or dicts is required.

- Type validation beyond that provided by PEPs 484 and 526 is
  required, or value validation or conversion is required.

Specification
=============

All of the functions described in this PEP will live in a module named
``dataclasses``.

A function ``dataclass`` which is typically used as a class decorator
is provided to post-process classes and add generated methods,
described below.

The ``dataclass`` decorator examines the class to find ``field``\s.  A
``field`` is defined as any variable identified in
``__annotations__``.  That is, a variable that has a type annotation.
With two exceptions described below, none of the Data Class machinery
examines the type specified in the annotation.

Note that ``__annotations__`` is guaranteed to be an ordered mapping,
in class declaration order.  The order of the fields in all of the
generated methods is the order in which they appear in the class.

The ``dataclass`` decorator will add various "dunder" methods to the
class, described below.  If any of the added methods already exist on the
class, a ``TypeError`` will be raised.  The decorator returns the same
class that is called on: no new class is created.

The ``dataclass`` decorator is typically used with no parameters and
no parentheses.  However, it also supports the following logical
signature::

  def dataclass(*, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False)

If ``dataclass`` is used just as a simple decorator with no
parameters, it acts as if it has the default values documented in this
signature.  That is, these three uses of ``@dataclass`` are equivalent::

  @dataclass
  class C:
      ...

  @dataclass()
  class C:
      ...

  @dataclass(init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False)
  class C:
      ...

The parameters to ``dataclass`` are:

- ``init``: If true (the default), a ``__init__`` method will be
  generated.

- ``repr``: If true (the default), a ``__repr__`` method will be
  generated.  The generated repr string will have the class name and
  the name and repr of each field, in the order they are defined in
  the class.  Fields that are marked as being excluded from the repr
  are not included.  For example:
  ``InventoryItem(name='widget',


id: pep-0227.rst
content:
PEP: 227
Title: Statically Nested Scopes
Author: Jeremy Hylton <jeremy@alum.mit.edu>
Status: Final
Type: Standards Track
Created: 01-Nov-2000
Python-Version: 2.1
Post-History:

Abstract
========

This PEP describes the addition of statically nested scoping
(lexical scoping) for Python 2.2, and as a source level option
for python 2.1.  In addition, Python 2.1 will issue warnings about
constructs whose meaning may change when this feature is enabled.

The old language definition (2.0 and before) defines exactly three
namespaces that are used to resolve names -- the local, global,
and built-in namespaces.  The addition of nested scopes allows
resolution of unbound local names in enclosing functions'
namespaces.

The most visible consequence of this change is that lambdas (and
other nested functions) can reference variables defined in the
surrounding namespace.  Currently, lambdas must often use default
arguments to explicitly creating bindings in the lambda's
namespace.

Introduction
============

This proposal changes the rules for resolving free variables in
Python functions.  The new name resolution semantics will take
effect with Python 2.2.  These semantics will also be available in
Python 2.1 by adding "from __future__ import nested_scopes" to the
top of a module.  (See :pep:`236`.)

The Python 2.0 definition specifies exactly three namespaces to
check for each name -- the local namespace, the global namespace,
and the builtin namespace.  According to this definition, if a
function A is defined within a function B, the names bound in B
are not visible in A.  The proposal changes the rules so that
names bound in B are visible in A (unless A contains a name
binding that hides the binding in B).

This specification introduces rules for lexical scoping that are
common in Algol-like languages.  The combination of lexical
scoping and existing support for first-class functions is
reminiscent of Scheme.

The changed scoping rules address two problems -- the limited
utility of lambda expressions (and nested functions in general),
and the frequent confusion of new users familiar with other
languages that support nested lexical scopes, e.g. the inability
to define recursive functions except at the module level.

The lambda expression yields an unnamed function that evaluates a
single expression.  It is often used for callback functions.  In
the example below (written using the Python 2.0 rules), any name
used in the body of the lambda must be explicitly passed as a
default argument to the lambda.

::

    from Tkinter import *
    root = Tk()
    Button(root, text="Click here",
           command=lambda root=root: root.test.configure(text="..."))

This approach is cumbersome, particularly when there are several
names used in the body of the lambda.  The long list of default
arguments obscures the purpose of the code.  The proposed
solution, in crude terms, implements the default argument approach
automatically.  The "root=root" argument can be omitted.

The new name resolution semantics will cause some programs to
behave differently than they did under Python 2.0.  In some cases,
programs will fail to compile.  In other cases, names that were
previously resolved using the global namespace will be resolved
using the local namespace of an enclosing function.  In Python
2.1, warnings will be issued for all statements that will behave
differently.

Specification
=============

Python is a statically scoped language with block structure, in
the traditional of Algol.  A code block or region, such as a
module, class definition, or function body, is the basic unit of a
program.

Names refer to objects.  Names are introduced by name binding
operations.  Each occurrence of a name in the program text refers
to the binding of that name established in the innermost function
block containing the use.

The name binding operations are argument declaration, assignment,
class and function definition, import statements, for statements,
and except clauses.  Each name binding occurs within a block
defined by a class or function definition or at the module level
(the top-level code block).

If a name is bound anywhere within a code block, all uses of the
name within the block are treated as references to the current
block.  (Note: This can lead to errors when a name is used within
a block before it is bound.)

If the global statement occurs within a block, all uses of the
name specified in the statement refer to the binding of that name
in the top-level namespace.  Names are resolved in the top-level
namespace by searching the global namespace, i.e. the namespace of
the module containing the code block, and in the builtin
namespace, i.e. the namespace of the ``__builtin__`` module.  The
global namespace is searched first.  If the name is not found
there, the builtin namespace is searched.  The global statement
must precede all uses of the name.

If a name is used within a code block, but it is not bound there
and is not declared global, the use is treated as a reference to
the nearest enclosing function region.  (Note: If a region is
contained within a class definition, the name bindings that occur
in the class block are not visible to enclosed functions.)

A class definition is an executable statement that may contain
uses and definitions of names.  These references follow the normal
rules for name resolution.  The namespace of the class definition
becomes the attribute dictionary of the class.

The following operations are name binding operations.  If they
occur within a block, they introduce new local names in the
current block unless there is also a global declaration.

::

    Function definition: def name ...
    Argument declaration: def f(...name...), lambda ...name...
    Class definition: class name ...
    Assignment statement: name = ...
    Import statement: import name, import module as name,
        from module import name
    Implicit assignment: names are bound by for statements and except
        clauses

There are several cases where Python statements are illegal when
used in conjunction with nested scopes that contain free
variables.

If a variable is referenced in an enclosed scope, it is an error
to delete the name.  The compiler will raise a ``SyntaxError`` for
'del name'.

If the wild card form of import (``import *``) is used in a function
and the function contains a nested block with free variables, the
compiler will raise a ``SyntaxError``.

If exec is used in a function and the function contains a nested
block with free variables, the compiler will raise a ``SyntaxError``
unless the exec explicitly specifies the local namespace for the
exec.  (In other words, "exec obj" would be illegal, but
"exec obj in ns" would be legal.)

If a name bound in a function scope is also the name of a module
global name or a standard builtin name, and the function contains
a nested function scope that references the name, the compiler
will issue a warning.  The name resolution rules will result in
different bindings under Python 2.0 than under Python 2.2.  The
warning indicates that the program may not run correctly with all
versions of Python.

Discussion
==========

The specified rules allow names defined in a function to be
referenced in any nested function defined with that function.  The
name resolution rules are typical for statically scoped languages,
with three primary exceptions:

- Names in class scope are not accessible.
- The global statement short-circuits the normal rules.
- Variables are not declared.

Names in class scope are not accessible.  Names are resolved in
the innermost enclosing function scope.  If a class definition
occurs in a chain of nested scopes, the resolution process skips
class definitions.  This rule prevents odd interactions between
class attributes and local variable access.  If a name binding
operation occurs in a class definition, it creates an attribute on
the resulting class object.  To access this variable in a method,
or in a 


id: pep-0541.rst
content:
PEP: 541
Title: Package Index Name Retention
Author: Åukasz Langa <lukasz@python.org>
BDFL-Delegate: Mark Mangoba <mmangoba@python.org>
Discussions-To: distutils-sig@python.org
Status: Final
Type: Process
Topic: Packaging
Created: 12-Jan-2017
Post-History:
Resolution: https://mail.python.org/pipermail/distutils-sig/2018-March/032089.html


Abstract
========

This PEP proposes an extension to the Terms of Use [1]_ of the Package
Index [2]_, clarifying expectations of package owners regarding
ownership of a package name on the Package Index, specifically with
regards to conflict resolution.

Existing package repositories such as CPAN [3]_, NPM [4]_, and
GitHub [5]_ will be investigated as prior art in this field.


Rationale
=========

Given that package names on the Index are sharing a single flat
namespace, a unique name is a finite resource.  The growing age of
the Package Index causes a constant rise of situations of conflict
between the current use of the name and a different suggested use of
the same name.

This document aims to provide general guidelines for solving the
most typical cases of such conflicts.


Approval Process
================

As the application of this policy has potential legal ramifications for the
Python Software Foundation, the approval process used is more formal than that
used for most PEPs.

Rather than accepting the PEP directly, the assigned BDFL-Delegate will instead
recommend its acceptance to the PSF's Packaging Working Group. After
consultation with the PSF's General Counsel, adoption of the policy will then
be subject to a formal vote within the working group.

This formal approval process will be used for both initial adoption of the
policy, and for adoption of any future amendments.


Specification
=============

The main idea behind this document is that the Package Index serves the
community.  Every user is invited to upload content to the Package Index
under the Terms of Use, understanding that it is at the sole risk of
the user.

While the Package Index is not a backup service, the maintainers of the
Package Index do their best to keep that content accessible indefinitely
in its published form.  However, in certain edge cases the greater
community's needs might overweigh the individual's expectation of
ownership of a package name.

The use cases covered by this document are:

* Abandoned projects:

  * continued maintenance by a different set of users; or
  * removal from the Index for use with a different project.

* Active projects:

  * resolving disputes over a name.

* Invalid projects:

  * projects subject to a claim of intellectual property infringement.

The proposed extension to the Terms of Use, as expressed in the
Implementation section, will be published as a separate document on the
Package Index, linked next to existing Terms of Use in the front page
footer.


Implementation
==============

Reachability
------------

The user of the Package Index is solely responsible for being reachable
by the Package Index maintainers for matters concerning projects that
the user owns.  In every case where contacting the user is necessary,
the maintainers will try to do so at least three times, using the
following means of contact:

* the e-mail address on file in the user's profile on the Package Index;
* the e-mail address listed in the Author field for a given project
  uploaded to the Index; and
* any e-mail addresses found in the given project's documentation
  on the Index or on the listed Home Page.

The maintainers stop trying to reach the user after six weeks.


Abandoned projects
------------------

A project is considered *abandoned* when ALL of the following are met:

* owner not reachable (see Reachability above);
* no releases within the past twelve months; and
* no activity from the owner on the project's home page (or no
  home page listed).

All other projects are considered *active*.

.. _continue-maintenance:

Continued maintenance of an abandoned project
---------------------------------------------

If a candidate appears willing to continue maintenance on an *abandoned*
project, ownership of the name is transferred when ALL of the following
are met:

* the project has been determined *abandoned* by the rules described
  above;
* the candidate is able to demonstrate their own failed attempts to contact
  the existing owner;
* the candidate is able to demonstrate improvements made on the
  candidate's own fork of the project;
* the candidate is able to demonstrate why a fork under a different name
  is not an acceptable workaround; and
* the maintainers of the Package Index don't have any additional
  reservations.

Under no circumstances will a name be reassigned against the wishes of
a reachable owner.

.. _reclaim-name:

Removal of an abandoned project
-------------------------------

Projects are never removed from the Package Index solely on the basis
of abandonment.  Artifacts uploaded to the Package Index hold inherent
historical value.

An *abandoned* project can be transferred to a new owner for purposes
of reusing the name when ALL of the following are met:

* the project has been determined *abandoned* by the rules described
  above;
* the candidate is able to demonstrate their own failed attempts to contact
  the existing owner;
* the candidate is able to demonstrate that the project suggested to
  reuse the name already exists and meets notability requirements;
* the candidate is able to demonstrate why a fork under a different name
  is not an acceptable workaround;
* download statistics on the Package Index for the existing package
  indicate project is not being used; and
* the maintainers of the Package Index don't have any additional
  reservations.


Name conflict resolution for active projects
--------------------------------------------

The maintainers of the Package Index are not arbiters in disputes
around *active* projects.  There are many possible scenarios here,
a non-exclusive list describing some real-world examples is presented
below.  None of the following qualify for package name ownership
transfer:

1. User A and User B share project X.  After some time they part ways
   and each of them wants to continue the project under name X.
2. User A owns a project X outside the Package Index.  User B creates
   a package under the name X on the Index.  After some time, User A
   wants to publish project X on the Index but realizes name is taken.
   This is true even if User A's project X gains notability and the
   User B's project X is not notable.
3. User A publishes project X to the Package Index.  After some time
   User B proposes bug fixes to the project but no new release is
   published by User A.  This is true even if User A agrees to publish
   a new version and later doesn't, even if User B's changes are merged
   to the source code repository for project X.

Again, the list above is not exclusive.  The maintainers of the Package
Index recommend users to get in touch with each other and solve the
issue by respectful communication (see the PSF Code of Conduct [6]_).


Invalid projects
----------------

A project published on the Package Index meeting ANY of the following
is considered invalid and will be removed from the Index:

* project does not conform to Terms of Use;
* project is malware (designed to exploit or harm systems or users directly, to
  facilitate command-and-control attacks, or perform data exfiltration);
* project is spam (designed to advertise or solicit goods or services);
* project contains illegal content;
* project violates copyright, trademarks, patents, or licenses;
* project is name squatting (package has no functionality or is
  empty);
* project name, description, or content violates the Code of Conduct;
* project uses obfuscation to hide or mask functionality;
  or
* project is abusing the Package Index for purposes it was not
  intended.

The Package Index maintainers pre-emptively declare certain package
names as unavaila


id: pep-0555.rst
content:
PEP: 555
Title: Context-local variables (contextvars)
Author: Koos Zevenhoven
Status: Withdrawn
Type: Standards Track
Created: 06-Sep-2017
Python-Version: 3.7
Post-History: 06-Sep-2017


Abstract
========

Sometimes, in special cases, it is desired that code can pass information down the function call chain to the callees without having to explicitly pass the information as arguments to each function in the call chain. This proposal describes a construct which allows code to explicitly switch in and out of a context where a certain context variable has a given value assigned to it. This is a modern alternative to some uses of things like global variables in traditional single-threaded (or thread-unsafe) code and of thread-local storage in traditional *concurrency-unsafe* code (single- or multi-threaded). In particular, the proposed mechanism can also be used with more modern concurrent execution mechanisms such as asynchronously executed coroutines, without the concurrently executed call chains interfering with each other's contexts.

The "call chain" can consist of normal functions, awaited coroutines, or generators. The semantics of context variable scope are equivalent in all cases, allowing code to be refactored freely into *subroutines* (which here refers to functions, sub-generators or sub-coroutines) without affecting the semantics of context variables. Regarding implementation, this proposal aims at simplicity and minimum changes to the CPython interpreter and to other Python interpreters.

Rationale
=========

Consider a modern Python *call chain* (or call tree), which in this proposal refers to any chained (nested) execution of *subroutines*, using any possible combinations of normal function calls, or expressions using ``await`` or ``yield from``. In some cases, passing necessary *information* down the call chain as arguments can substantially complicate the required function signatures, or it can even be impossible to achieve in practice. In these cases, one may search for another place to store this information. Let us look at some historical examples.

The most naive option is to assign the value to a global variable or similar, where the code down the call chain can access it. However, this immediately makes the code thread-unsafe, because with multiple threads, all threads assign to the same global variable, and another thread can interfere at any point in the call chain. Sooner or later, someone will probably find a reason to run the same code in parallel threads.

A somewhat less naive option is to store the information as per-thread information in thread-local storage, where each thread has its own "copy" of the variable which other threads cannot interfere with. Although non-ideal, this has been the best solution in many cases. However, thanks to generators and coroutines, the execution of the call chain can be suspended and resumed, allowing code in other contexts to run concurrently. Therefore, using thread-local storage is *concurrency-unsafe*, because other call chains in other contexts may interfere with the thread-local variable.

Note that in the above two historical approaches, the stored information has the *widest* available scope without causing problems. For a third solution along the same path, one would first define an equivalent of a "thread" for asynchronous execution and concurrency. This could be seen as the largest amount of code and nested calls that is guaranteed to be executed sequentially without ambiguity in execution order. This might be referred to as concurrency-local or task-local storage. In this meaning of "task", there is no ambiguity in the order of execution of the code within one task. (This concept of a task is close to equivalent to a ``Task`` in ``asyncio``, but not exactly.) In such concurrency-locals, it is possible to pass information down the call chain to callees without another code path interfering with the value in the background.

Common to the above approaches is that they indeed use variables with a wide but just-narrow-enough scope. Thread-locals could also be called thread-wide globals---in single-threaded code, they are indeed truly global. And task-locals could be called task-wide globals, because tasks can be very big.

The issue here is that neither global variables, thread-locals nor task-locals are really meant to be used for this purpose of passing information of the execution context down the call chain. Instead of the widest possible variable scope, the scope of the variables should be controlled by the programmer, typically of a library, to have the desired scope---not wider. In other words, task-local variables (and globals and thread-locals) have nothing to do with the kind of context-bound information passing that this proposal intends to enable, even if task-locals can be used to emulate the desired semantics. Therefore, in the following, this proposal describes the semantics and the outlines of an implementation for *context-local variables* (or context variables, contextvars). In fact, as a side effect of this PEP, an async framework can use the proposed feature to implement task-local variables.

Proposal
========

Because the proposed semantics are not a direct extension to anything already available in Python, this proposal is first described in terms of semantics and API at a fairly high level. In particular, Python ``with`` statements are heavily used in the description, as they are a good match with the proposed semantics. However, the underlying ``__enter__`` and ``__exit__`` methods correspond to functions in the lower-level speed-optimized (C) API. For clarity of this document, the lower-level functions are not explicitly named in the definition of the semantics. After describing the semantics and high-level API, the implementation is described, going to a lower level.

Semantics and higher-level API
------------------------------

Core concept
''''''''''''

A context-local variable is represented by a single instance of ``contextvars.Var``, say ``cvar``. Any code that has access to the ``cvar`` object can ask for its value with respect to the current context. In the high-level API, this value is given by the ``cvar.value`` property::

    cvar = contextvars.Var(default="the default value",
                           description="example context variable")

    assert cvar.value == "the default value"  # default still applies

    # In code examples, all ``assert`` statements should
    # succeed according to the proposed semantics.


No assignments to ``cvar`` have been applied for this context, so ``cvar.value`` gives the default value. Assigning new values to contextvars is done in a highly scope-aware manner::

    with cvar.assign(new_value):
        assert cvar.value is new_value
        # Any code here, or down the call chain from here, sees:
        #     cvar.value is new_value
        # unless another value has been assigned in a
        # nested context
        assert cvar.value is new_value
    # the assignment of ``cvar`` to ``new_value`` is no longer visible
    assert cvar.value == "the default value"


Here, ``cvar.assign(value)`` returns another object, namely ``contextvars.Assignment(cvar, new_value)``. The essential part here is that applying a context variable assignment (``Assignment.__enter__``) is paired with a de-assignment (``Assignment.__exit__``). These operations set the bounds for the scope of the assigned value.

Assignments to the same context variable can be nested to override the outer assignment in a narrower context::

    assert cvar.value == "the default value"
    with cvar.assign("outer"):
        assert cvar.value == "outer"
        with cvar.assign("inner"):
            assert cvar.value == "inner"
        assert cvar.value == "outer"
    assert cvar.value == "the default value"


Also multiple variables can be assigned to in a nested manner without affecting each other::

    cvar1 = contextvars.Var()
    cvar2 = 


