# embeddings

id: pep-0470.rst
content:
PEP: 470
Title: Removing External Hosting Support on PyPI
Author: Donald Stufft <donald@stufft.io>
BDFL-Delegate: Paul Moore <p.f.moore@gmail.com>
Discussions-To: distutils-sig@python.org
Status: Final
Type: Process
Topic: Packaging
Created: 12-May-2014
Post-History: 14-May-2014, 05-Jun-2014, 03-Oct-2014, 13-Oct-2014, 26-Aug-2015
Replaces: 438
Resolution: https://mail.python.org/pipermail/distutils-sig/2015-September/026789.html


Abstract
========

This PEP proposes the deprecation and removal of support for hosting files
externally to PyPI as well as the deprecation and removal of the functionality
added by :pep:`438`, particularly rel information to classify different types of
links and the meta-tag to indicate API version.


Rationale
=========

Historically PyPI did not have any method of hosting files nor any method of
automatically retrieving installables, it was instead focused on providing a
central registry of names, to prevent naming collisions, and as a means of
discovery for finding projects to use. In the course of time setuptools began
to scrape these human facing pages, as well as pages linked from those pages,
looking for things it could automatically download and install. Eventually this
became the "Simple" API which used a similar URL structure however it
eliminated any of the extraneous links and information to make the API more
efficient. Additionally PyPI grew the ability for a project to upload release
files directly to PyPI enabling PyPI to act as a repository in addition to an
index.

This gives PyPI two equally important roles that it plays in the Python
ecosystem, that of index to enable easy discovery of Python projects and
central repository to enable easy hosting, download, and installation of Python
projects. Due to the history behind PyPI and the very organic growth it has
experienced the lines between these two roles are blurry, and this blurring has
caused confusion for the end users of both of these roles and this has in turn
caused ire between people attempting to use PyPI in different capacities, most
often when end users want to use PyPI as a repository but the author wants to
use PyPI solely as an index.

This confusion comes down to end users of projects not realizing if a project
is hosted on PyPI or if it relies on an external service. This often manifests
itself when the external service is down but PyPI is not. People will see that
PyPI works, and other projects works, but this one specific one does not. They
oftentimes do not realize who they need to contact in order to get this fixed
or what their remediation steps are.

:pep:`438` attempted to solve this issue by allowing projects to explicitly
declare if they were using the repository features or not, and if they were
not, it had the installers classify the links it found as either "internal",
"verifiable external" or "unverifiable external". :pep:`438` was accepted and
implemented in pip 1.4 (released on Jul 23, 2013) with the final transition
implemented in pip 1.5 (released on Jan 2, 2014).

:pep:`438` was successful in bringing about more people to utilize PyPI's
repository features, an altogether good thing given the global CDN powering
PyPI providing speed ups for a lot of people, however it did so by introducing
a new point of confusion and pain for both the end users and the authors.

By moving to using explicit multiple repositories we can make the lines between
these two roles much more explicit and remove the "hidden" surprises caused by
the current implementation of handling people who do not want to use PyPI as a
repository.


Key User Experience Expectations
--------------------------------

#. Easily allow external hosting to "just work" when appropriately configured
   at the system, user or virtual environment level.
#. Eliminate any and all references to the confusing "verifiable external" and
   "unverifiable external" distinction from the user experience (both when
   installing and when releasing packages).
#. The repository aspects of PyPI should become *just* the default package
   hosting location (i.e. the only one that is treated as opt-out rather than
   opt-in by most client tools in their default configuration). Aside from that
   aspect, hosting on PyPI should not otherwise provide an enhanced user
   experience over hosting your own package repository.
#. Do all of the above while providing default behaviour that is secure against
   most attackers below the nation state adversary level.


Why Additional Repositories?
----------------------------

The two common installer tools, pip and easy_install/setuptools, both support
the concept of additional locations to search for files to satisfy the
installation requirements and have done so for many years. This means that
there is no need to "phase" in a new flag or concept and the solution to
installing a project from a repository other than PyPI will function regardless
of how old (within reason) the end user's installer is. Not only has this
concept existed in the Python tooling for some time, but it is a concept that
exists across languages and even extending to the OS level with OS package
tools almost universally using multiple repository support making it extremely
likely that someone is already familiar with the concept.

Additionally, the multiple repository approach is a concept that is useful
outside of the narrow scope of allowing projects that wish to be included on
the index portion of PyPI but do not wish to utilize the repository portion of
PyPI. This includes places where a company may wish to host a repository that
contains their internal packages or where a project may wish to have multiple
"channels" of releases, such as alpha, beta, release candidate, and final
release. This could also be used for projects wishing to host files which
cannot be uploaded to PyPI, such as multi-gigabyte data files or, currently at
least, Linux Wheels.


Why Not PEP 438 or Similar?
---------------------------

While the additional search location support has existed in pip and setuptools
for quite some time support for :pep:`438` has only existed in pip since the 1.4
version, and still has yet to be implemented in setuptools. The design of
:pep:`438` did mean that users still benefited for projects which did not require
external files even with older installers, however for projects which *did*
require external files, users are still silently being given either potentially
unreliable or, even worse, unsafe files to download. This system is also unique
to Python as it arises out of the history of PyPI, this means that it is almost
certain that this concept will be foreign to most, if not all users, until they
encounter it while attempting to use the Python toolchain.

Additionally, the classification system proposed by :pep:`438` has, in practice,
turned out to be extremely confusing to end users, so much so that it is a
position of this PEP that the situation as it stands is completely untenable.
The common pattern for a user with this system is to attempt to install a
project possibly get an error message (or maybe not if the project ever
uploaded something to PyPI but later switched without removing old files), see
that the error message suggests ``--allow-external``, they reissue the command
adding that flag most likely getting another error message, see that this time
the error message suggests also adding ``--allow-unverified``, and again issue
the command a third time, this time finally getting the thing they wish to
install.

This UX failure exists for several reasons.

#. If pip can locate files at all for a project on the Simple API it will
   simply use that instead of attempting to locate more. This is generally the
   right thing to do as attempting to locate more would erase a large part of
   the benefit of :pep:`438`. This means that if a project *ever* uploaded a file
   that matches what the user has requested for install that will be used
   regardless of ho


id: pep-0316.rst
content:
PEP: 316
Title: Programming by Contract for Python
Author: Terence Way <terry@wayforward.net>
Status: Deferred
Type: Standards Track
Created: 02-May-2003
Post-History:


Abstract
========

This submission describes programming by contract for Python.
Eiffel's Design By Contract(tm) is perhaps the most popular use of
programming contracts [#dbc]_.

Programming contracts extends the language to include invariant
expressions for classes and modules, and pre- and post-condition
expressions for functions and methods.

These expressions (contracts) are similar to assertions: they must be
true or the program is stopped, and run-time checking of the contracts
is typically only enabled while debugging.  Contracts are higher-level
than straight assertions and are typically included in documentation.


Motivation
==========

Python already has assertions, why add extra stuff to the language to
support something like contracts?  The two best reasons are 1) better,
more accurate documentation, and 2) easier testing.

Complex modules and classes never seem to be documented quite right.
The documentation provided may be enough to convince a programmer to
use a particular module or class over another, but the programmer
almost always has to read the source code when the real debugging
starts.

Contracts extend the excellent example provided by the ``doctest``
module [#doctest]_.  Documentation is readable by programmers, yet has
executable tests embedded in it.

Testing code with contracts is easier too.  Comprehensive contracts
are equivalent to unit tests [#xp]_.  Tests exercise the full range of
pre-conditions, and fail if the post-conditions are triggered.
Theoretically, a correctly specified function can be tested completely
randomly.

So why add this to the language?  Why not have several different
implementations, or let programmers implement their own assertions?
The answer is the behavior of contracts under inheritance.

Suppose Alice and Bob use different assertions packages.  If Alice
produces a class library protected by assertions, Bob cannot derive
classes from Alice's library and expect proper checking of
post-conditions and invariants.  If they both use the same assertions
package, then Bob can override Alice's methods yet still test against
Alice's contract assertions.  The natural place to find this
assertions system is in the language's run-time library.


Specification
=============

The docstring of any module or class can include invariant contracts
marked off with a line that starts with the keyword ``inv`` followed
by a colon (:).  Whitespace at the start of the line and around the
colon is ignored.  The colon is either immediately followed by a
single expression on the same line, or by a series of expressions on
following lines indented past the ``inv`` keyword.  The normal Python
rules about implicit and explicit line continuations are followed
here.  Any number of invariant contracts can be in a docstring.

Some examples::

    # state enumeration
    START, CONNECTING, CONNECTED, CLOSING, CLOSED = range(5)

    class conn:

        """A network connection

        inv: self.state in [START, CLOSED,       # closed states
                            CONNECTING, CLOSING, # transition states
                            CONNECTED]

        inv: 0 <= self.seqno < 256
        """

    class circbuf:

        """A circular buffer.

        inv:
            # there can be from 0 to max items on the buffer
            0 <= self.len <= len(self.buf)

            # g is a valid index into buf
            0 <= self.g < len(self.buf)

            # p is also a valid index into buf
            0 <= self.p < len(self.buf)

            # there are len items between get and put
            (self.p - self.g) % len(self.buf) == \
                  self.len % len(self.buf)
        """

Module invariants must be true after the module is loaded, and at the
entry and exit of every public function within the module.

Class invariants must be true after the ``__init__`` function returns,
at the entry of the ``__del__`` function, and at the entry and exit of
every other public method of the class.  Class invariants must use the
self variable to access instance variables.

A method or function is public if its name doesn't start with an
underscore (_), unless it starts and ends with '__' (two underscores).

The docstring of any function or method can have pre-conditions
documented with the keyword ``pre`` following the same rules above.
Post-conditions are documented with the keyword ``post`` optionally
followed by a list of variables.  The variables are in the same scope
as the body of the function or method.  This list declares the
variables that the function/method is allowed to modify.

An example::

    class circbuf:

        def __init__(self, leng):
            """Construct an empty circular buffer.

            pre: leng > 0
            post[self]:
                self.is_empty()
                len(self.buf) == leng
            """

A double-colon (::) can be used instead of a single colon (:) to
support docstrings written using reStructuredText [#rst]_.  For
example, the following two docstrings describe the same contract::

    """pre: leng > 0"""
    """pre:: leng > 0"""

Expressions in pre- and post-conditions are defined in the module
namespace -- they have access to nearly all the variables that the
function can access, except closure variables.

The contract expressions in post-conditions have access to two
additional variables: ``__old__`` which is filled with shallow copies
of values declared in the variable list immediately following the post
keyword, and ``__return__`` which is bound to the return value of the
function or method.

An example::

    class circbuf:

        def get(self):
            """Pull an entry from a non-empty circular buffer.

            pre: not self.is_empty()
            post[self.g, self.len]:
                __return__ == self.buf[__old__.self.g]
                self.len == __old__.self.len - 1
            """

All contract expressions have access to some additional convenience
functions.  To make evaluating the truth of sequences easier, two
functions ``forall`` and ``exists`` are defined as::

    def forall(a, fn = bool):
        """Return True only if all elements in a are true.

        >>> forall([])
        1
        >>> even = lambda x: x % 2 == 0
        >>> forall([2, 4, 6, 8], even)
        1
        >>> forall('this is a test'.split(), lambda x: len(x) == 4)
        0
        """

    def exists(a, fn = bool):
        """Returns True if there is at least one true value in a.

        >>> exists([])
        0
        >>> exists('this is a test'.split(), lambda x: len(x) == 4)
        1
        """

An example::

    def sort(a):
        """Sort a list.

        pre: isinstance(a, type(list))
        post[a]:
            # array size is unchanged
            len(a) == len(__old__.a)

            # array is ordered
            forall([a[i] >= a[i-1] for i in range(1, len(a))])

            # all the old elements are still in the array
            forall(__old__.a, lambda e: __old__.a.count(e) == a.count(e))
        """

To make evaluating conditions easier, the function ``implies`` is
defined.  With two arguments, this is similar to the logical implies
(=>) operator.  With three arguments, this is similar to C's
conditional expression (x?a:b).  This is defined as::

    implies(False, a) => True
    implies(True, a) => a
    implies(False, a, b) => b
    implies(True, a, b) => a

On entry to a function, the function's pre-conditions are checked.  An
assertion error is raised if any pre-condition is false.  If the
function is public, then the class or module's invariants are also
checked.  Copies of variables declared in the post are saved, the
function is called, and if the function exits without raising an
exception, the post-conditions are checked.


Exceptions
----------

Class/module invariants are che


id: pep-0302.rst
content:
PEP: 302
Title: New Import Hooks
Author: Just van Rossum <just@letterror.com>,
        Paul Moore <p.f.moore@gmail.com>
Status: Final
Type: Standards Track
Created: 19-Dec-2002
Python-Version: 2.3
Post-History: 19-Dec-2002

.. warning::
   The language reference for import [10]_ and importlib documentation
   [11]_ now supersede this PEP. This document is no longer updated
   and provided for historical purposes only.


Abstract
========

This PEP proposes to add a new set of import hooks that offer better
customization of the Python import mechanism.  Contrary to the current
``__import__`` hook, a new-style hook can be injected into the existing
scheme, allowing for a finer grained control of how modules are found and how
they are loaded.


Motivation
==========

The only way to customize the import mechanism is currently to override the
built-in ``__import__`` function.  However, overriding ``__import__`` has many
problems.  To begin with:

* An ``__import__`` replacement needs to *fully* reimplement the entire
  import mechanism, or call the original ``__import__`` before or after the
  custom code.

* It has very complex semantics and responsibilities.

* ``__import__`` gets called even for modules that are already in
  ``sys.modules``, which is almost never what you want, unless you're writing
  some sort of monitoring tool.

The situation gets worse when you need to extend the import mechanism from C:
it's currently impossible, apart from hacking Python's ``import.c`` or
reimplementing much of ``import.c`` from scratch.

There is a fairly long history of tools written in Python that allow extending
the import mechanism in various way, based on the ``__import__`` hook.  The
Standard Library includes two such tools: ``ihooks.py`` (by GvR) and
``imputil.py`` [1]_ (Greg Stein), but perhaps the most famous is ``iu.py`` by
Gordon McMillan, available as part of his Installer package.  Their usefulness
is somewhat limited because they are written in Python; bootstrapping issues
need to worked around as you can't load the module containing the hook with
the hook itself.  So if you want the entire Standard Library to be loadable
from an import hook, the hook must be written in C.


Use cases
=========

This section lists several existing applications that depend on import hooks.
Among these, a lot of duplicate work was done that could have been saved if
there had been a more flexible import hook at the time.  This PEP should make
life a lot easier for similar projects in the future.

Extending the import mechanism is needed when you want to load modules that
are stored in a non-standard way.  Examples include modules that are bundled
together in an archive; byte code that is not stored in a ``pyc`` formatted
file; modules that are loaded from a database over a network.

The work on this PEP was partly triggered by the implementation of :pep:`273`,
which adds imports from Zip archives as a built-in feature to Python.  While
the PEP itself was widely accepted as a must-have feature, the implementation
left a few things to desire.  For one thing it went through great lengths to
integrate itself with ``import.c``, adding lots of code that was either
specific for Zip file imports or *not* specific to Zip imports, yet was not
generally useful (or even desirable) either.  Yet the :pep:`273` implementation
can hardly be blamed for this: it is simply extremely hard to do, given the
current state of ``import.c``.

Packaging applications for end users is a typical use case for import hooks,
if not *the* typical use case.  Distributing lots of source or ``pyc`` files
around is not always appropriate (let alone a separate Python installation),
so there is a frequent desire to package all needed modules in a single file.
So frequent in fact that multiple solutions have been implemented over the
years.

The oldest one is included with the Python source code: Freeze [2]_.  It puts
marshalled byte code into static objects in C source code.  Freeze's "import
hook" is hard wired into ``import.c``, and has a couple of issues.  Later
solutions include Fredrik Lundh's Squeeze, Gordon McMillan's Installer, and
Thomas Heller's py2exe [3]_.  MacPython ships with a tool called
``BuildApplication``.

Squeeze, Installer and py2exe use an ``__import__`` based scheme (py2exe
currently uses Installer's ``iu.py``, Squeeze used ``ihooks.py``), MacPython
has two Mac-specific import hooks hard wired into ``import.c``, that are
similar to the Freeze hook.  The hooks proposed in this PEP enables us (at
least in theory; it's not a short-term goal) to get rid of the hard coded
hooks in ``import.c``, and would allow the ``__import__``-based tools to get
rid of most of their ``import.c`` emulation code.

Before work on the design and implementation of this PEP was started, a new
``BuildApplication``-like tool for Mac OS X prompted one of the authors of
this PEP (JvR) to expose the table of frozen modules to Python, in the ``imp``
module.  The main reason was to be able to use the freeze import hook
(avoiding fancy ``__import__`` support), yet to also be able to supply a set
of modules at runtime.  This resulted in issue #642578 [4]_, which was
mysteriously accepted (mostly because nobody seemed to care either way ;-).
Yet it is completely superfluous when this PEP gets accepted, as it offers a
much nicer and general way to do the same thing.


Rationale
=========

While experimenting with alternative implementation ideas to get built-in Zip
import, it was discovered that achieving this is possible with only a fairly
small amount of changes to ``import.c``.  This allowed to factor out the
Zip-specific stuff into a new source file, while at the same time creating a
*general* new import hook scheme: the one you're reading about now.

An earlier design allowed non-string objects on ``sys.path``.  Such an object
would have the necessary methods to handle an import.  This has two
disadvantages: 1) it breaks code that assumes all items on ``sys.path`` are
strings; 2) it is not compatible with the ``PYTHONPATH`` environment variable.
The latter is directly needed for Zip imports.  A compromise came from Jython:
allow string *subclasses* on ``sys.path``, which would then act as importer
objects.  This avoids some breakage, and seems to work well for Jython (where
it is used to load modules from ``.jar`` files), but it was perceived as an
"ugly hack".

This led to a more elaborate scheme, (mostly copied from McMillan's
``iu.py``) in which each in a list of candidates is asked whether it can
handle the ``sys.path`` item, until one is found that can.  This list of
candidates is a new object in the ``sys`` module: ``sys.path_hooks``.

Traversing ``sys.path_hooks`` for each path item for each new import can be
expensive, so the results are cached in another new object in the ``sys``
module: ``sys.path_importer_cache``.  It maps ``sys.path`` entries to importer
objects.

To minimize the impact on ``import.c`` as well as to avoid adding extra
overhead, it was chosen to not add an explicit hook and importer object for
the existing file system import logic (as ``iu.py`` has), but to simply fall
back to the built-in logic if no hook on ``sys.path_hooks`` could handle the
path item.  If this is the case, a ``None`` value is stored in
``sys.path_importer_cache``, again to avoid repeated lookups.  (Later we can
go further and add a real importer object for the built-in mechanism, for now,
the ``None`` fallback scheme should suffice.)

A question was raised: what about importers that don't need *any* entry on
``sys.path``? (Built-in and frozen modules fall into that category.)  Again,
Gordon McMillan to the rescue: ``iu.py`` contains a thing he calls the
*metapath*.  In this PEP's implementation, it's a list of importer objects
that is traversed *before* ``sys.path``.  This list is yet another new object
in the ``sys`` module: ``sys.meta_path``.  Currently, this list is empty by
default, and frozen and built-in mod


id: pep-0464.rst
content:
PEP: 464
Title: Removal of the PyPI Mirror Authenticity API
Author: Donald Stufft <donald@stufft.io>
BDFL-Delegate: Richard Jones <richard@python.org>
Discussions-To: distutils-sig@python.org
Status: Final
Type: Process
Topic: Packaging
Created: 02-Mar-2014
Post-History: 04-Mar-2014
Replaces: 381
Resolution: https://mail.python.org/pipermail/distutils-sig/2014-March/024027.html


Abstract
========

This PEP proposes the deprecation and removal of the PyPI Mirror Authenticity
API, this includes the /serverkey URL and all of the URLs under /serversig.


Rationale
=========

The PyPI mirroring infrastructure (defined in :pep:`381`) provides a means to
mirror the content of PyPI used by the automatic installers, and as a component
of that, it provides a method for verifying the authenticity of the mirrored
content.

This PEP proposes the removal of this API due to:

* There are no known implementations that utilize this API, this includes
  `pip <http://www.pip-installer.org/en/latest/>`_ and
  `setuptools <http://pythonhosted.org//setuptools/>`_.
* Because this API uses DSA it is vulnerable to leaking the private key if
  there is *any* bias in the random nonce.
* This API solves one small corner of the trust problem, however the problem
  itself is much larger and it would be better to have a fully fledged system,
  such as :pep:`The Update Framework <458>`,
  instead.

Due to the issues it has and the lack of use it is the opinion of this PEP
that it does not provide any practical benefit to justify the additional
complexity.


Plan for Deprecation & Removal
==============================

Immediately upon the acceptance of this PEP the Mirror Authenticity API will
be considered deprecated and mirroring agents and installation tools should
stop accessing it.

Instead of actually removing it from the current code base (PyPI 1.0) the
current work to replace PyPI 1.0 with a new code base (PyPI 2.0) will simply
not implement this API. This would cause the API to be "removed" when the
switch from 1.0 to 2.0 occurs.

If PyPI 2.0 has not been deployed in place of PyPI 1.0 by Sept 01 2014 then
this PEP will be implemented in the PyPI 1.0 code base instead (by removing
the associated code).

No changes will be required in the installers, however :pep:`381` compliant
mirroring clients, such as
`bandersnatch <https://pypi.python.org/pypi/bandersnatch/>`_ and
`pep381client <https://pypi.python.org/pypi/pep381client/>`_ will need to be
updated to no longer attempt to mirror the /serversig URLs.


Copyright
=========

This document has been placed in the public domain.


id: pep-8010.rst
content:
PEP: 8010
Title: The Technical Leader Governance Model
Author: Barry Warsaw <barry@python.org>
Status: Rejected
Type: Informational
Topic: Governance
Created: 24-Aug-2018


Abstract
========

This PEP proposes a continuation of the singular technical project
leader model, euphemistically called the `Benevolent Dictator For Life
<https://en.wikipedia.org/wiki/Benevolent_dictator_for_life>`_ (BDFL)
model of Python governance, to be henceforth called in this PEP the
Gracious Umpire Influencing Decisions Officer (GUIDO).  This change in
name reflects both the expanded view of the GUIDO as final arbiter for
the Python language decision making process in consultation with the
wider development community, and the recognition that "for life" while
perhaps aspirational, is not necessarily in the best interest of the
well-being of either the language or the GUIDO themselves.

This PEP describes:

* The rationale for maintaining the singular technical leader model
* The process for how the GUIDO will be selected, elected, retained,
  recalled, and succeeded;
* The roles of the GUIDO in the Python language evolution process;
* The term length of service;
* The relationship of the GUIDO with a Council of Pythonistas (CoP)
  that advise the GUIDO on technical matters;
* The size, election, and roles of the CoP;
* The decision delegation process;
* Any changes to the PEP process to fit the new governance model;

This PEP does *not* name a new BDFL.  Should this model be adopted, it
will be codified in :pep:`13` along with the names of all officeholders
described in this PEP.


PEP Rejection
=============

:pep:`8010` was rejected `by a core developer vote
<https://discuss.python.org/t/python-governance-vote-december-2018-results/546/>`__
described in :pep:`8001` on Monday, December 17, 2018.

:pep:`8016` and the governance model it describes were chosen instead.


Open discussion points
======================

Various tweaks to the parameters of this PEP are allowed during the
governance discussion process, such as the exact size of the CoP, term
lengths of service, and voting procedures.  These will be codified by
the time the PEP is ready to be voted on.

The voting procedures and events described in this PEP will default to
the voting method specified in :pep:`8001`, although as that PEP is still
in discussion at the time of this writing, this is subject to change.

It is allowed, and perhaps even expected, that as experience is gained
with this model, these parameters may be tweaked as future GUIDOs are
named, in order to provide for a smoother governing process.


Why a singular technical leader?
================================

Why this model rather than any other?  It comes down to "vision".
`Design by committee`_ has many known downsides, leading to a language
that accretes new features based on the varied interests of the
contributors at the time.  A famous aphorism is "a camel is a horse
designed by committee".  Can a language that is designed by committee
"hang together"?  Does it feel like a coherent, self-consistent
language where the rules make sense and are easily remembered?

A singular technical leader can promote that vision more than a
committee can, whether that committee is small (e.g. 3 or 5 persons)
or spans the entire Python community.  Every participant will have
their own vision of what "Python" is, and this can lead to indecision
or illogical choices when those individual visions are in conflict.
Should CPython be 3x faster or should we preserve the C API?  That's a
very difficult question to get consensus on, since neither choice is
right or wrong.  But worse than making the wrong decision might be
accepting the status quo because no consensus could be found.


Flexibility
===========

Degrees of flexibility are given to both the GUIDO and CoP by way of
underspecification.  This PEP describes how conflicts will be
resolved, but expects all participants, including core developers,
community members, and office holders, to always have the best
interest of Python and its users at heart.  The PEP assumes that
mutual respect and the best intentions will always lead to consensus,
and that the Code of Conduct governs all interactions and discussions.


The role of the GUIDO
=====================

One of the most important roles of the GUIDO is to provide an
overarching, broad, coherent vision for the evolution of the Python
language, spanning multiple releases.  This is especially important
when decision have lasting impact and competing benefits.  For
example, if backward incompatible changes to the C API leads to a 2x
improvement in Python performance, different community members will
likely advocate convincingly on both sides of the debate, and a clear
consensus may not emerge.  Either choice is equally valid.  In
consultation with the CoP, it will be the GUIDO's vision that guides
the ultimate decision.

The GUIDO is the ultimate authority for decisions on PEPs and other
issues, including whether any particular change is PEP-worthy.  As is
the case today, many --in fact perhaps most-- decisions are handled by
discussion and resolution on the issue tracker, merge requests, and
discussion forums, usually with input or lead by experts in the
particular field.  Where this operating procedure works perfectly
well, it can continue unchanged.  This also helps reduce the workload
on the CoP and GUIDO, leaving only the most important decisions and
broadest view of the landscape to the central authority.

Similarly, should a particular change be deemed to require a PEP, but
the GUIDO, in consultation with the CoP, identifies experts that have
the full confidence to make the final decision, the GUIDO can name a
Delegate for the PEP.  While the GUIDO remains the ultimate authority,
it is expected that the GUIDO will not undermine, and in fact will
support the authority of the Delegate as the final arbiter of the PEP.

The GUIDO has full authority to shut down unproductive discussions,
ideas, and proposals, when it is clear that the proposal runs counter
to the long-term vision for Python.  This is done with compassion for
the advocates of the change, but with the health and well-being of all
community members in mind.  A toxic discussion on a dead-end proposal
does no one any good, and they can be terminated by fiat.

To sum up: the GUIDO has the authority to make a final pronouncement
on any topic, technical or non-technical, except for changing to the
governance PEP itself.


Authority comes from the community
==================================

The GUIDO's authority ultimately resides with the community.  A rogue
GUIDO that loses the confidence of the majority of the community can
be recalled and a new vote conducted.  This is an exceedingly rare and
unlikely event.  This is a sufficient stopgap for the worst-case
scenario, so it should not be undertaken lightly.  The GUIDO should
not fear being deposed because of one decision, even if that decision
isn't favored by the majority of Python developers.  Recall should be
reserved for actions severely detrimental to the Python language or
community.

The Council of Pythonistas (see below) has the responsibility to
initiate a vote of no-confidence.


Length of service and term limits
=================================

The GUIDO shall serve for three Python releases, approximately 4.5
years given the current release cadence.  If Pythonâs release cadence
changes, the length of GUIDOâs term should change to 4.5 years rounded
to whole releases. How the rounding is done is left to the potential
release cadence PEP.  After this time, a new election is held
according to the procedures outlined below.  There are no term limits,
so the GUIDO may run for re-election for as long as they like.

We expect GUIDOs to serve out their entire term of office, but of
course, Life Happens.  Should the GUIDO need to step down before their
term ends, the vacancy will be filled by the process outlined below as
per choosi


id: pep-0458.rst
content:
PEP: 458
Title: Secure PyPI downloads with signed repository metadata
Author: Trishank Karthik Kuppusamy <karthik@trishank.com>,
        Vladimir Diaz <vladimir.diaz@nyu.edu>,
        Marina Moore <mm9693@nyu.edu>,
        Lukas Puehringer <lukas.puehringer@nyu.edu>,
        Joshua Lock <jlock@vmware.com>,
        Lois Anne DeLong <lad278@nyu.edu>,
        Justin Cappos <jcappos@nyu.edu>
Sponsor: Alyssa Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: Donald Stufft <donald@stufft.io>
Discussions-To: https://discuss.python.org/t/pep-458-secure-pypi-downloads-with-package-signing/2648
Status: Accepted
Type: Standards Track
Topic: Packaging
Created: 27-Sep-2013
Post-History: 06-Jan-2019, 13-Nov-2019
Resolution: https://discuss.python.org/t/pep-458-secure-pypi-downloads-with-package-signing/2648/115


Abstract
========

This PEP describes changes to the PyPI infrastructure that are needed to ensure
that users get valid packages from PyPI. These changes should have minimal
impact on other parts of the ecosystem. The PEP focuses on communication between
PyPI and users, and so does not require any action by package developers.
Developers will upload packages using the current process, and PyPI will
automatically generate signed repository metadata for these packages.

In order for the security mechanism to be
effective, additional work will need to be done by PyPI consumers (like pip) to
verify the signatures and metadata provided by PyPI. This verification can be
transparent to users (unless it fails) and provides an automatic security
mechanism. There is documentation for how to consume TUF metadata in the TUF
repository. However, changes to PyPI consumers are not a pre-requisite for
publishing the metadata from PyPI, and can be done
according to the timelines and priorities of individual projects.



Proposed TUF Integration
========================

This PEP proposes how The Update Framework [2]_ (TUF__) should be integrated with the
Python Package Index (PyPI [1]_). TUF was designed to be a flexible
security add-on to a software updater or package manager.  A full implementation
of the framework
integrates best security practices, such as separating role responsibilities,
adopting the many-man rule for signing packages, keeping signing keys offline,
and revocation of expired or compromised signing keys.  As a result, attackers
would need to steal multiple signing keys, which are stored independently,
in order to compromise the role responsible for specifying a repository's available
files.  Or, alternatively, a role
responsible for indicating the latest snapshot of the repository may also have to be
compromised.

__ https://www.linuxfoundation.org/cloud-containers-virtualization/2017/10/cncf-host-two-security-projects-notary-tuf-specification/

The initial integration proposed in this PEP will allow modern package managers,
such as pip [3]_, to be more secure against attacks on PyPI mirrors and PyPI's
own content distribution network, and to better protect users from such attacks.
Specifically, this PEP describes how PyPI processes
should be adapted to generate and incorporate TUF metadata (i.e., the minimum
security model).  This minimum security model supports verification of PyPI
distributions that are signed with keys stored on PyPI. Distributions that are
uploaded by developers are signed by PyPI, requiring no action from developers
(other than uploading the distribution), and are immediately available for
download.  The minimum security model also minimizes PyPI administrative
responsibilities by automating much of the signing process.

There is no discussion in *this* PEP of support for project distributions that
are signed by developers (maximum security model). This possible future extension
is covered in detail in :pep:`480`.  The maximum security model requires more PyPI
administrative work (though no added work for clients), and also proposes
an easy-to-use key management solution for developers/publishers, ideas on how
to interface with a potential future build farm on PyPI infrastructure, and the
feasibility of end-to-end signing.

While it does provide implementation recommendations, this PEP does not
prescribe exactly how package managers, such as pip, should be adapted
to install or update projects from PyPI with TUF metadata.   Package managers
interested in adopting TUF on the client side may consult its `library
documentation`__, which was created for this purpose.

__ https://theupdateframework.readthedocs.io/en/stable/api/tuf.ngclient.html

Non-goals
=========

This PEP does not eliminate any existing features from PyPI. In particular, it
does not replace existing support for OpenPGP signatures. Developers can continue
to upload detached OpenPGP signatures along with distributions. In the future,
:pep:`480` may allow developers to directly sign TUF metadata using their OpenPGP keys.


PEP Status
==========

Due to the amount of work required to implement this PEP, in early
2019 it was deferred until appropriate funding could be secured to
implement the PEP. The Python Software Foundation secured this funding
[22]_ and new PEP coauthors restarted PEP discussion__.

__ https://discuss.python.org/t/pep-458-secure-pypi-downloads-with-package-signing/2648/


Motivation
==========

Attacks on software repositories are common, even in organizations with very
good security practices__. The resulting repository compromise allows an
attacker to edit all files stored on the repository and sign these files using
any keys stored on the repository (online keys). In many signing schemes (like
TLS), this access allows the attacker to replace files on the repository and
make it look like these files are coming from PyPI. Without a way to revoke and
replace the trusted private key, it is very challenging to recover from a
repository compromise. In addition to the dangers of repository compromise,
software repositories are vulnerable to an attacker on the network (MITM)
intercepting and changing files. These and other attacks on software
repositories are detailed here__.

This PEP, together with the follow-up proposal in :pep:`480`, aims to protect users
of PyPI from compromises of the integrity, consistency, and freshness properties
of PyPI packages, and enhances compromise resilience by mitigating key risk and
providing mechanisms to recover from a compromise of PyPI or its signing keys.

__ https://github.com/theupdateframework/pip/wiki/Attacks-on-software-repositories
__ https://theupdateframework.github.io/security.html

On January 5, 2013, the Python Software Foundation (PSF) announced that [4]_ a security
breach had occurred on the
python.org wikis for Python and Jython. As a result, all of the wiki data was destroyed.
Fortunately, the PyPI infrastructure was not affected by this breach.
However, the incident is a reminder that PyPI needed to take defensive steps to
protect users as much as possible in the event of a compromise.  Attacks on
software repositories happen all the time [5]_.  The PSF must accept the
possibility of security breaches and prepare PyPI accordingly because it is a
valuable resource used by thousands, if not millions, of people.

Before the wiki attack, PyPI used MD5 hashes to tell package managers, such as
pip, whether or not a distribution file was corrupted in transit.  However, the absence
of SSL made it hard for package managers to verify transport integrity to PyPI.
It was therefore easy to launch a man-in-the-middle attack between pip and
PyPI, and arbitrarily change the content of distributions.  As a result, users could be tricked into
installing malicious distributions.  After the wiki
attack, several steps were proposed (some of which were implemented) to deliver
a much higher level of security than was previously the case. These steps included
requiring SSL to
communicate with PyPI [6]_, restricting project names [7]_, and migrating from
MD5 to SHA-2 hashes [8]_.

Though necessary, these steps are insufficient


id: pep-0699.rst
content:
PEP: 699
Title: Remove private dict version field added in PEP 509
Author: Ken Jin <kenjin@python.org>
Discussions-To: https://discuss.python.org/t/pep-699-remove-private-dict-version-field-added-in-pep-509/19724
Status: Accepted
Type: Standards Track
Created: 03-Oct-2022
Python-Version: 3.12
Post-History: `05-Oct-2022 <https://discuss.python.org/t/pep-699-remove-private-dict-version-field-added-in-pep-509/19724>`__
Replaces: 509
Resolution: https://discuss.python.org/t/pep-699-remove-private-dict-version-field-added-in-pep-509/19724/13



Abstract
========

:pep:`509` introduced a private ``ma_version_tag`` field for dictionaries to
allow optimizations in CPython and extension libraries. This PEP proposes to
rescind :pep:`509` and declare the field an implementation detail, as it has
already been superseded by alternatives. This will further allow the field to
be reused for future optimization.


Motivation
==========

:pep:`509` introduced the ``ma_version_tag`` field to dictionaries. This 64-bit
field stored a version counter that updates every time a dictionary was
modified. The original proposal was to use this version counter as a
guard for optimizations.

Since CPython 3.11, this field has become unused by internal optimization
efforts. :pep:`659`-specialized instructions use other methods of verifying
that certain optimizations are safe.

To enable further optimizations in CPython, this PEP proposes that the
``ma_version_tag`` field no longer conform to the :pep:`509` specification.
This will allow the CPython developers to store other optimization information,
such as dictionary write watchers.


Rationale
=========
This PEP does not specify what the field may be used for in the future. This is
intentional, as implementation details are subject to change, and the field
should be used only for internal consumption by CPython.


Specification
=============

This specification rescinds that in :pep:`509`. The ``ma_version_tag`` field of
the Python :class:`dict` class is declared to be an internal implementation
detail and may be removed altogether, or may have a different representation.
C extensions should not rely on this field.


Backwards Compatibility
=======================

Certain extensions use ``ma_version_tag`` for fast dictionary or globals
lookups. For example,
`Cython uses the field for fast dynamic module variable lookups <https://github.com/cython/cython/blob/169876872f3cb6198971a1db07e5b8a9d12b3dac/Cython/Utility/ObjectHandling.c#L1556>`_.

This PEP proposes to emit a compiler warning when accessing ``ma_version_tag``.
After two consecutive version releases with warnings, ``ma_version_tag``
will be removed, in line with :pep:`387`.

The biggest user the author could find for this field was Cython.
Discussions with a Cython maintainer indicated that
`removing support for it from Cython is trivial <https://github.com/faster-cpython/ideas/issues/461#issuecomment-1250358596>`_.


Security Implications
=====================

:pep:`509` was concerned with integer overflow. However, this PEP does not
introduce any additional security concerns.


Rejected Ideas
==============

A possible alternative is to preserve the field for backwards compatibility.
This PEP rejects that alternative as future optimizations will consume more
memory, and the field was always considered private and undocumented aside
from the PEP, with no backward compatibility guarantees. Dictionaries in Python
are commonly used, and any increase in their memory consumption will adversely
affect Pythonâs performance.


Reference Implementation
========================

A reference implementation can be found in
`python/cpython#101193 <https://github.com/python/cpython/pull/101193>`_.


Special Thanks
==============

Thanks to C.A.M. Gerlach for edits and wording changes to this document.
Thanks also to Mark Shannon and Kumar Aditya for providing possible
implementations.

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


id: pep-0672.rst
content:
PEP: 672
Title: Unicode-related Security Considerations for Python
Author: Petr Viktorin <encukou@gmail.com>
Status: Active
Type: Informational
Created: 01-Nov-2021
Post-History: 01-Nov-2021

Abstract
========

This document explains possible ways to misuse Unicode to write Python
programs that appear to do something else than they actually do.

This document does not give any recommendations and solutions.


Introduction
============

`Unicode`_ is a system for handling all kinds of written language.
It aims to allow any character from any human language to be
used. Python code may consist of almost all valid Unicode characters.
While this allows programmers from all around the world to express themselves,
it also allows writing code that is potentially confusing to readers.

It is possible to misuse Python's Unicode-related features to write code that
*appears* to do something else than what it does.
Evildoers could take advantage of this to trick code reviewers into
accepting malicious code.

The possible issues generally can't be solved in Python itself without
excessive restrictions of the language.
They should be solved in code editors and review tools
(such as *diff* displays), by enforcing project-specific policies,
and by raising awareness of individual programmers.

This document purposefully does not give any solutions
or recommendations: it is rather a list of things to keep in mind.

This document is specific to Python.
For general security considerations in Unicode text and source code,
see Unicode technical reports [tr36]_, [tr39]_, and [tr55]_.


Acknowledgement
===============

Investigation for this document was prompted by `CVE-2021-42574`_,
*Trojan Source Attacks*, reported by Nicholas Boucher and Ross Anderson,
which focuses on Bidirectional override characters and homoglyphs in a variety
of programming languages.


Confusing Features
==================

This section lists some Unicode-related features that can be surprising
or misusable.


ASCII-only Considerations
-------------------------

ASCII is a subset of Unicode, consisting of the most common symbols, numbers,
Latin letters and control characters.

While issues with the ASCII character set are generally well understood,
the're presented here to help better understanding of the non-ASCII cases.

Confusables and Typos
'''''''''''''''''''''

Some characters look alike.
Before the age of computers, many mechanical typewriters lacked the keys for
the digits ``0`` and ``1``: users typed ``O`` (capital o) and ``l``
(lowercase L) instead. Human readers could tell them apart by context only.
In programming languages, however, distinction between digits and letters is
critical -- and most fonts designed for programmers make it easy to tell them
apart.

Similarly, in fonts designed for human languages, the uppercase âIâ and
lowercase âlâ can look similar. Or the letters ârnâ may be virtually
indistinguishable from the single letter âmâ.
Again, programmers' fonts make these pairs of *confusables*
noticeably different.

However, what is ânoticeablyâ different always depends on the context.
Humans tend to ignore details in longer identifiers: the variable name
``accessibi1ity_options`` can still look indistinguishable from
``accessibility_options``, while they are distinct for the compiler.
The same can be said for plain typos: most humans will not notice the typo in
``responsbility_chain_delegate``.

Control Characters
''''''''''''''''''

Python generally considers all ``CR`` (``\r``), ``LF`` (``\n``), and ``CR-LF``
pairs (``\r\n``) as an end of line characters.
Most code editors do as well, but there are editors that display ânon-nativeâ
line endings as unknown characters (or nothing at all), rather than ending
the line, displaying this example::

    # Don't call this function:
    fire_the_missiles()

as a harmless comment like::

    # Don't call this function:â¬fire_the_missiles()

CPython may treat the control character NUL (``\0``) as end of input,
but many editors simply skip it, possibly showing code that Python will not
run as a regular part of a file.

Some characters can be used to hide/overwrite other characters when source is
listed in common terminals. For example:

* BS (``\b``, Backspace) moves the cursor back, so the character after it
  will overwrite the character before.
* CR (``\r``, carriage return) moves the cursor to the start of line,
  subsequent characters overwrite the start of the line.
* SUB (``\x1A``, Ctrl+Z) means âEnd of textâ on Windows. Some programs
  (such as ``type``) ignore the rest of the file after it.
* ESC (``\x1B``) commonly initiates escape codes which allow arbitrary
  control of the terminal.


Confusable Characters in Identifiers
------------------------------------

Python is not limited to ASCII.
It allows characters of all scripts â Latin letters to ancient Egyptian
hieroglyphs â in identifiers (such as variable names).
See :pep:`3131` for details and rationale.
Only âletters and numbersâ are allowed, so while ``Î³Î¬ÏÎ±`` is a valid Python
identifier, ``ð±`` is not.  (See `Identifiers and keywords`_ for details.)

Non-printing control characters are also not allowed in identifiers.

However, within the allowed set there is a large number of âconfusablesâ.
For example, the uppercase versions of the Latin ``b``, Greek ``Î²`` (Beta), and
Cyrillic ``Ð²`` (Ve) often look identical: ``B``, ``Î`` and ``Ð``, respectively.

This allows identifiers that look the same to humans, but not to Python.
For example, all of the following are distinct identifiers:

* ``scope`` (Latin, ASCII-only)
* ``scÐ¾pe`` (with a Cyrillic ``Ð¾``)
* ``scÎ¿pe`` (with a Greek ``Î¿``)
* ``ÑÑÐ¾ÑÐµ`` (all Cyrillic letters)

Additionally, some letters can look like non-letters:

* The letter for the Hawaiian *Ê»okina* looks like an apostrophe;
  ``Ê»HelloÊ»`` is a Python identifier, not a string.
* The East Asian word for *ten* looks like a plus sign,
  so ``å= 10`` is a complete Python statement. (The âåâ is a word: âtenâ
  rather than â10â.)

.. note::

   The converse also applies â some symbols look like letters â but since
   Python does not allow arbitrary symbols in identifiers, this is not an
   issue.


Confusable  Digits
------------------

Numeric literals in Python only use the ASCII digits 0-9 (and non-digits such
as ``.`` or ``e``).

However, when numbers are converted from strings, such as in the ``int`` and
``float`` constructors or by the ``str.format`` method, any decimal digit
can be used. For example ``ß`` (``NKO DIGIT FIVE``) or ``à¯«``
(``TAMIL DIGIT FIVE``) work as the digit ``5``.

Some scripts include digits that look similar to ASCII ones, but have a
different value. For example::

    >>> int('à§ªà­¨')
    42
    >>> '{Ù¥}'.format('zero', 'one', 'two', 'three', 'four', 'five')
    five


Bidirectional Text
------------------

Some scripts, such as Hebrew or Arabic, are written right-to-left.
Phrases in such scripts interact with nearby text in ways that can be
surprising to people who aren't familiar with these writing systems and their
computer representation.

The exact process is complicated, and explained in Unicode Standard Annex #9,
`Unicode Bidirectional Algorithm`_.

Consider the following code, which assigns a 100-character string to
the variable ``s``::

  s = "X" * 100 #    "X" is assigned

When the ``X`` is replaced by the Hebrew letter ``×``, the line becomes::

  s = "×" * 100 #    "×" is assigned

This command still assigns a 100-character string to ``s``, but
when displayed as general text following the Bidirectional Algorithm
(e.g. in a browser), it appears as ``s = "×"`` followed by a comment.

Other surprising examples include:

* In the statement ``×¢×¨× = 23``, the variable ``×¢×¨×`` is set to the integer 23.

* In the statement ``ÙÙÙØ© = ×¢×¨×``, the variable ``ÙÙÙØ©`` is set
  to the value of ``×¢×¨×``.

* In the statement ``ÙÙÙØ© - (


id: pep-0666.rst
content:
PEP: 666
Title: Reject Foolish Indentation
Author: Laura Creighton <lac@strakt.com>
Status: Rejected
Type: Standards Track
Created: 03-Dec-2001
Python-Version: 2.2
Post-History: 05-Dec-2001


Abstract
========

Everybody agrees that mixing tabs and spaces is a bad idea.  Some
people want more than this.  I propose that we let people define
whatever Python behaviour they want, so it will only run the way
they like it, and will not run the way they don't like it.  We
will do this with a command line switch.  Programs that aren't
formatted the way the programmer wants things will raise
``IndentationError``.

- ``python -TNone`` will refuse to run when there are any tabs.
- ``python -Tn`` will refuse to run when tabs are not exactly ``n`` spaces
- ``python -TOnly`` will refuse to run when blocks are indented by anything
  other than tabs

People who mix tabs and spaces, naturally, will find that their
programs do not run.  Alas, we haven't found a way to give them an
electric shock as from a cattle prod remotely.  (Though if somebody
finds out a way to do this, I will be pleased to add this option to
the PEP.)


Rationale
=========

Python-list@python.org (a.k.a. comp.lang.python) is periodically
awash with discussions about tabs and spaces.  This is inevitable,
given that indentation is syntactically significant in Python.
This has never solved anything, and just makes various people
frustrated and angry.  Eventually they start saying rude things to
each other which is sad for all of us.  And it is also sad that
they are wasting their valuable time which they could spend
creating something with Python.  Moreover, for the Python community
as a whole, from a public relations point of view, this is quite
unfortunate.  The people who aren't posting about tabs and spaces,
are, (unsurprisingly) invisible, while the people who are posting
make the rest of us look somewhat foolish.

The problem is that there is no polite way to say 'Stop wasting
your valuable time and mine.'  People who are already in the middle
of a flame war are not well disposed to believe that you are acting
out of compassion for them, and quite rightly insist that their own
time is their own to do with as they please.  They are stuck like
flies in treacle in this wretched argument, and it is self-evident
that they cannot disengage or they would have already done so.

But today I had to spend time cleaning my keyboard because the 'n'
key is sticking.  So, in addition to feeling compassion for these
people, I am pretty annoyed.  I figure if I make this PEP, we can
then ask Guido to quickly reject it, and then when this argument
next starts up again, we can say 'Guido isn't changing things to
suit the tab-haters or the only-tabbers, so this conversation is a
waste of time.'  Then everybody can quietly believe that a) they
are correct and b) other people are fools and c) they are
undeniably fortunate to not have to share a lab with idiots, (which
is something the arguers could do _now_, but apparently have
forgotten).

And python-list can go back to worrying if it is too smug, rather
than whether it is too hostile for newcomers.  Possibly somebody
could get around to explaining to me what is the difference between
``__getattr__`` and ``__getattribute__`` in non-Classic classes in 2.2, a
question I have foolishly posted in the middle of the current tab
thread.  I would like to know the answer to that question [1]_.

This proposal, if accepted, will probably mean a heck of a lot of
work for somebody.  But since I don't want it accepted, I don't
care.


References
==========

.. [1] Tim Peters already has (private correspondence).  My early 2.2
       didn't have a ``__getattribute__``, and ``__getattr__`` was
       implemented like ``__getattribute__`` now is.  This has been
       fixed.  The important conclusion is that my Decorator Pattern
       is safe and all is right with the world.


Copyright
=========

This document has been placed in the public domain.


id: pep-0100.rst
content:
PEP: 100
Title: Python Unicode Integration
Author: Marc-AndrÃ© Lemburg <mal@lemburg.com>
Status: Final
Type: Standards Track
Created: 10-Mar-2000
Python-Version: 2.0
Post-History:


Historical Note
===============

This document was first written by Marc-Andre in the pre-PEP days,
and was originally distributed as Misc/unicode.txt in Python
distributions up to and included Python 2.1.  The last revision of
the proposal in that location was labeled version 1.7 (CVS
revision 3.10).  Because the document clearly serves the purpose
of an informational PEP in the post-PEP era, it has been moved
here and reformatted to comply with PEP guidelines.  Future
revisions will be made to this document, while Misc/unicode.txt
will contain a pointer to this PEP.

-Barry Warsaw, PEP editor


Introduction
============

The idea of this proposal is to add native Unicode 3.0 support to
Python in a way that makes use of Unicode strings as simple as
possible without introducing too many pitfalls along the way.

Since this goal is not easy to achieve -- strings being one of the
most fundamental objects in Python -- we expect this proposal to
undergo some significant refinements.

Note that the current version of this proposal is still a bit
unsorted due to the many different aspects of the Unicode-Python
integration.

The latest version of this document is always available at:
http://starship.python.net/~lemburg/unicode-proposal.txt

Older versions are available as:
http://starship.python.net/~lemburg/unicode-proposal-X.X.txt

[ed. note: new revisions should be made to this PEP document,
while the historical record previous to version 1.7 should be
retrieved from MAL's url, or Misc/unicode.txt]


Conventions
===========

- In examples we use u = Unicode object and s = Python string

- 'XXX' markings indicate points of discussion (PODs)


General Remarks
===============

- Unicode encoding names should be lower case on output and
  case-insensitive on input (they will be converted to lower case
  by all APIs taking an encoding name as input).

- Encoding names should follow the name conventions as used by the
  Unicode Consortium: spaces are converted to hyphens, e.g. 'utf
  16' is written as 'utf-16'.

- Codec modules should use the same names, but with hyphens
  converted to underscores, e.g. ``utf_8``, ``utf_16``, ``iso_8859_1``.


Unicode Default Encoding
========================

The Unicode implementation has to make some assumption about the
encoding of 8-bit strings passed to it for coercion and about the
encoding to as default for conversion of Unicode to strings when
no specific encoding is given.  This encoding is called <default
encoding> throughout this text.

For this, the implementation maintains a global which can be set
in the site.py Python startup script.  Subsequent changes are not
possible.  The <default encoding> can be set and queried using the
two sys module APIs:

``sys.setdefaultencoding(encoding)``
   Sets the <default encoding> used by the Unicode implementation.
   encoding has to be an encoding which is supported by the
   Python installation, otherwise, a LookupError is raised.

   Note: This API is only available in site.py!  It is
   removed from the sys module by site.py after usage.

``sys.getdefaultencoding()``
   Returns the current <default encoding>.

If not otherwise defined or set, the <default encoding> defaults
to 'ascii'.  This encoding is also the startup default of Python
(and in effect before site.py is executed).

Note that the default site.py startup module contains disabled
optional code which can set the <default encoding> according to
the encoding defined by the current locale.  The locale module is
used to extract the encoding from the locale default settings
defined by the OS environment (see locale.py).  If the encoding
cannot be determined, is unknown or unsupported, the code defaults
to setting the <default encoding> to 'ascii'.  To enable this
code, edit the site.py file or place the appropriate code into the
sitecustomize.py module of your Python installation.


Unicode Constructors
====================

Python should provide a built-in constructor for Unicode strings
which is available through ``__builtins__``::

    u = unicode(encoded_string[,encoding=<default encoding>][,errors="strict"])

    u = u'<unicode-escape encoded Python string>'

    u = ur'<raw-unicode-escape encoded Python string>'

With the 'unicode-escape' encoding being defined as:

- all non-escape characters represent themselves as Unicode
  ordinal (e.g. 'a' -> U+0061).

- all existing defined Python escape sequences are interpreted as
  Unicode ordinals; note that ``\xXXXX`` can represent all Unicode
  ordinals, and ``\OOO`` (octal) can represent Unicode ordinals up to
  U+01FF.

- a new escape sequence, ``\uXXXX``, represents U+XXXX; it is a syntax
  error to have fewer than 4 digits after ``\u``.

For an explanation of possible values for errors see the Codec
section below.

Examples::

    u'abc'          -> U+0061 U+0062 U+0063
    u'\u1234'       -> U+1234
    u'abc\u1234\n'  -> U+0061 U+0062 U+0063 U+1234 U+005c

The 'raw-unicode-escape' encoding is defined as follows:

- ``\uXXXX`` sequence represent the U+XXXX Unicode character if and
  only if the number of leading backslashes is odd

- all other characters represent themselves as Unicode ordinal
  (e.g. 'b' -> U+0062)

Note that you should provide some hint to the encoding you used to
write your programs as pragma line in one the first few comment
lines of the source file (e.g. '# source file encoding: latin-1').
If you only use 7-bit ASCII then everything is fine and no such
notice is needed, but if you include Latin-1 characters not
defined in ASCII, it may well be worthwhile including a hint since
people in other countries will want to be able to read your source
strings too.


Unicode Type Object
===================

Unicode objects should have the type UnicodeType with type name
'unicode', made available through the standard types module.


Unicode Output
==============

Unicode objects have a method .encode([encoding=<default encoding>])
which returns a Python string encoding the Unicode string using the
given scheme (see Codecs).

::

    print u := print u.encode()   # using the <default encoding>

    str(u)  := u.encode()         # using the <default encoding>

    repr(u) := "u%s" % repr(u.encode('unicode-escape'))

Also see Internal Argument Parsing and Buffer Interface for
details on how other APIs written in C will treat Unicode objects.


Unicode Ordinals
================

Since Unicode 3.0 has a 32-bit ordinal character set, the
implementation should provide 32-bit aware ordinal conversion
APIs::

    ord(u[:1]) (this is the standard ord() extended to work with Unicode
                objects)
      --> Unicode ordinal number (32-bit)

    unichr(i)
        --> Unicode object for character i (provided it is 32-bit);
            ValueError otherwise

Both APIs should go into ``__builtins__`` just like their string
counterparts ``ord()`` and ``chr()``.

Note that Unicode provides space for private encodings.  Usage of
these can cause different output representations on different
machines.  This problem is not a Python or Unicode problem, but a
machine setup and maintenance one.


Comparison & Hash Value
=======================

Unicode objects should compare equal to other objects after these
other objects have been coerced to Unicode.  For strings this
means that they are interpreted as Unicode string using the
<default encoding>.

Unicode objects should return the same hash value as their ASCII
equivalent strings.  Unicode strings holding non-ASCII values are
not guaranteed to return the same hash values as the default
encoded equivalent string representation.

When compared using ``cmp()`` (or ``PyObject_Compare()``) the
implementation should mask ``TypeErrors`` raised during the conversion
to remain in synch with the string behavior.  All other errors
such as ``Va


id: pep-0706.rst
content:
PEP: 706
Title: Filter for tarfile.extractall
Author: Petr Viktorin <encukou@gmail.com>
Discussions-To: https://discuss.python.org/t/23903
Status: Final
Type: Standards Track
Created: 09-Feb-2023
Python-Version: 3.12
Post-History: `25-Jan-2023 <https://discuss.python.org/t/23149>`__,
              `15-Feb-2023 <https://discuss.python.org/t/23903>`__,
Resolution: https://discuss.python.org/t/23903/10

.. canonical-doc:: :ref:`tarfile documentation <python:tarfile-extraction-filter>`


Abstract
========

The extraction methods in :external+py3.11:mod:`tarfile` gain a ``filter`` argument,
which allows rejecting files or modifying metadata as the archive is extracted.
Three built-in named filters are provided, aimed at limiting features that
might be surprising or dangerous.
These can be used as-is, or serve as a base for custom filters.

After a deprecation period, a strict (but safer) filter will become the default.


Motivation
==========

The ``tar`` format is used for several use cases, many of which have different
needs. For example:

- A backup of a UNIX workstation should faithfully preserve all kinds of
  details like file permissions, symlinks to system configuration, and various
  kinds of special files.
- When unpacking a data bundle, itâs much more important that the unpacking
  will not have unintended consequences â like exposing a password file by
  symlinking it to a public place.

To support all its use cases, the ``tar`` format has many features.
In many cases, it's best to ignore or disallow some of them when extracting
an archive.

Python allows extracting ``tar`` archives using
:external+py3.11:meth:`tarfile.TarFile.extractall`, whose docs warn to
*never extract archives from untrusted sources without prior inspection*.
However, itâs not clear what kind of inspection should be done.
Indeed, itâs quite tricky to do such an inspection correctly.
As a result, many people donât bother, or do the check incorrectly, resulting in
security issues such as `CVE-2007-4559`_.

Since :external+py3.11:mod:`tarfile` was first written, it's become more
accepted that warnings in documentation are not enough.
Whenever possible, an unsafe operation should be *explicitly requested*;
potentially dangerous operations should *look* dangerous.
However, ``TarFile.extractall`` looks benign in a code review.

Tarfile extraction is also exposed via :external+py3.11:func:`shutil.unpack_archive`,
which allows the user to not care about the kind of archive they're
dealing with.
The API is very inviting for extracting archives without prior inspection,
even though the docs again warn against it.

It has been argued that Python is not wrong -- it behaves exactly as
documented -- but that's beside the point.
Let's improve the situation rather than assign/avoid blame.
Python and its docs are the best place to improve things.


Rationale
=========

How do we improve things?
Unfortunately, we will need to change the defaults, which implies
breaking backwards compatibility. :external+py3.11:meth:`TarFile.extractall <tarfile.TarFile.extractall>`
is what people reach for when they need to extract a tarball.
Its default behaviour needs to change.

What would be the best behaviour? That depends on the use case.
So, we'll add several general âpoliciesâ to control extraction.
They are based on *use cases*, and ideally they should have straightforward
security implications:

- Current behavior: trusting the archive. Suitable e.g. as a building block
  for libraries that do the check themselves, or extracting an archive you just
  made yourself.
- Unpacking a UNIX archive: roughly following GNU ``tar``, e.g. stripping
  leading ``/`` from filenames.
- Unpacking a general data archive: the :external+py3.11:func:`shutil.unpack_archive`
  use case,
  where it's not important to preserve details specific to ``tar`` or
  Unix-like filesystems.

After a deprecation period, the last option -- the most limited
but most secure one -- will become the default.

Even with better general defaults, users should still verify the archives
they extract, and perhaps modify some of the metadata.
Superficially, the following looks like a reasonable way to do this today:

* Call :external+py3.11:meth:`TarFile.getmembers <tarfile.TarFile.getmembers>`
* Verify or modify each member's :external+py3.11:class:`~tarfile.TarInfo`
* Pass the result to ``extractall``'s ``members``

However, there are some issues with this approach:

- It's possible to modify ``TarInfo`` objects, but the changes to them
  affect all subsequent operations on the same ``TarFile`` object.
  This behavior is fine for most uses, but despite that, it would be very
  surprising if ``TarFile.extractall`` did this by default.
- Calling ``getmembers`` can be expensive and it
  `requires a seekable archive <https://github.com/python/cpython/issues/45385#issuecomment-1255615199>`__.
- When verifying members in advance, it may be necessary to track how each
  member would have changed the filesystem, e.g. how symlinks are being set up.
  This is hard. We can't expect users to do it.

To solve these issues we'll:

- Provide a supported way to âcloneâ and modify ``TarInfo`` objects.
  A ``replace`` method, similar to :external+py3.11:func:`dataclasses.replace`
  or :external+py3.11:meth:`namedtuple._replace <collections.somenamedtuple._replace>`
  should do the trick.
- Provide a âfilterâ hook in ``extractall``'s loop that can modify or discard
  members before they are processed.
- Require that this hook is called just before extracting each member,
  so it can scan the *current* state of the disk. This will greatly simplify
  the implementation of policies (both in stdlib and user code),
  at the cost of not being able to do a precise âdry runâ.

The hook API will be very similar to the existing ``filter`` argument
for :external+py3.11:meth:`TarFile.add <tarfile.TarFile.add>`.
We'll also name it ``filter``.
(In some cases âpolicyâ would be a more fitting name,
but the API can be used for more than security policies.)

The built-in policies/filters described above will be implemented using the
public filter API, so they can be used as building blocks or examples.


Setting a precedent
-------------------

If and when other libraries for archive extraction, such as :external+py3.11:mod:`zipfile`,
gain similar functionality, they should mimic this API as much as it's
reasonable.

To enable this for simple cases, the built-in filters will have string names;
e.g. users can pass ``filter='data'`` instead of a specific function that deals
with :external+py3.11:class:`~tarfile.TarInfo` objects.

The :external+py3.11:func:`shutil.unpack_archive` function will get a
``filter`` argument, which it will pass to ``extractall``.

Adding function-based API that would work across archive formats is
out of scope of this PEP.


Full disclosure & redistributor info
------------------------------------

The PEP author works for Red Hat, a redistributor of Python with different
security needs and support periods than CPython in general.
Such redistributors may want to carry vendor patches to:

* Allow configuring the defaults system-wide, and
* Change the default as soon as possible, even in older Python versions.

The proposal makes this easy to do, and it allows users to query
the settings.


Specification
=============

Modifying and forgetting member metadata
----------------------------------------

The :external+py3.11:class:`~tarfile.TarInfo` class will gain a new method,
``replace()``, which will work similarly to ``dataclasses.replace``.
It will return a copy of the ``TarInfo`` object with attributes
replaced as specified by keyword-only arguments:

* ``name``
* ``mtime``
* ``mode``
* ``linkname``
* ``uid``
* ``gid``
* ``uname``
* ``gname``

Any of these, except ``name`` and ``linkname``, will be allowed to be set
to ``None``.
When ``extract`` or ``extractall`` encounters such a ``None``, it will not
set that piece of 


id: pep-0712.rst
content:
PEP: 712
Title: Adding a "converter" parameter to dataclasses.field
Author: Joshua Cannon <joshdcannon@gmail.com>
Sponsor: Eric V. Smith <eric at trueblade.com>
Discussions-To: https://discuss.python.org/t/pep-712-adding-a-converter-parameter-to-dataclasses-field/26126
Status: Rejected
Type: Standards Track
Created: 01-Jan-2023
Python-Version: 3.13
Post-History: `27-Dec-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/NWZQIINJQZDOCZGO6TGCUP2PNW4PEKNY/>`__,
              `19-Jan-2023 <https://discuss.python.org/t/add-converter-to-dataclass-field/22956>`__,
              `23-Apr-2023 <https://discuss.python.org/t/pep-712-adding-a-converter-parameter-to-dataclasses-field/26126>`__,
Resolution: https://discuss.python.org/t/pep-712-adding-a-converter-parameter-to-dataclasses-field/26126/98


Rejection Notice
================

The reasons for the 2024 Steering Council rejection include:

* We did not find evidence of a strong consensus that this feature was needed in the standard library, despite some proponents arguing in favor in order to reduce their dependence on third party packages. For those who need such functionality, we think those existing third party libraries such as attrs and Pydantic (which the PEP references) are acceptable alternatives.
* This feature seems to us like an accumulation of what could be considered more cruft in the standard library, leading us ever farther away from the âsimpleâ use cases that dataclasses are ideal for.
* Reading the âHow to Teach Thisâ section of the PEP gives us pause that the pitfalls and gotchas are significant, with a heightened confusion and complexity outweighing any potential benefits.
* The PEP seems more focused toward helping type checkers than people using the library.


Abstract
========

:pep:`557` added :mod:`dataclasses` to the Python stdlib. :pep:`681` added
:func:`~py3.11:typing.dataclass_transform` to help type checkers understand
several common dataclass-like libraries, such as attrs, Pydantic, and object
relational mapper (ORM) packages such as SQLAlchemy and Django.

A common feature other libraries provide over the standard library
implementation is the ability for the library to convert arguments given at
initialization time into the types expected for each field using a
user-provided conversion function.

Therefore, this PEP adds a ``converter`` parameter to :func:`dataclasses.field`
(along with the requisite changes to :class:`dataclasses.Field` and
:func:`~py3.11:typing.dataclass_transform`) to specify the function to use to
convert the input value for each field to the representation to be stored in
the dataclass.

Motivation
==========

There is no existing, standard way for :mod:`dataclasses` or third-party
dataclass-like libraries to support argument conversion in a type-checkable
way. To work around this limitation, library authors/users are forced to choose
to:

* Opt-in to a custom Mypy plugin. These plugins help Mypy understand the
  conversion semantics, but not other tools.
* Shift conversion responsibility onto the caller of the dataclass
  constructor. This can make constructing certain dataclasses unnecessarily
  verbose and repetitive.
* Provide a custom ``__init__`` which declares "wider" parameter types and
  converts them when setting the appropriate attribute. This not only duplicates
  the typing annotations between the converter and ``__init__``, but also opts
  the user out of many of the features :mod:`dataclasses` provides.
* Provide a custom ``__init__`` but without meaningful type annotations
  for the parameter types requiring conversion.

None of these choices are ideal.

Rationale
=========

Adding argument conversion semantics is useful and beneficial enough that most
dataclass-like libraries provide support. Adding this feature to the standard
library means more users are able to opt-in to these benefits without requiring
third-party libraries. Additionally third-party libraries are able to clue
type-checkers into their own conversion semantics through added support in
:func:`~py3.11:typing.dataclass_transform`, meaning users of those libraries
benefit as well.

Specification
=============

New ``converter`` parameter
---------------------------

This specification introduces a new parameter named ``converter`` to the
:func:`dataclasses.field` function. If provided, it represents a single-argument
callable used to convert all values when assigning to the associated attribute.

For frozen dataclasses, the converter is only used inside a ``dataclass``-synthesized
``__init__`` when setting the attribute. For non-frozen dataclasses, the converter
is used for all attribute assignment (E.g. ``obj.attr = value``), which includes
assignment of default values.

The converter is not used when reading attributes, as the attributes should already
have been converted.

Adding this parameter also implies the following changes:

* A ``converter`` attribute will be added to :class:`dataclasses.Field`.
* ``converter`` will be added to :func:`~py3.11:typing.dataclass_transform`'s
  list of supported field specifier parameters.

Example
'''''''

.. code-block:: python

  def str_or_none(x: Any) -> str | None:
    return str(x) if x is not None else None

  @dataclasses.dataclass
  class InventoryItem:
      # `converter` as a type (including a GenericAlias).
      id: int = dataclasses.field(converter=int)
      skus: tuple[int, ...] = dataclasses.field(converter=tuple[int, ...])
      # `converter` as a callable.
      vendor: str | None = dataclasses.field(converter=str_or_none))
      names: tuple[str, ...] = dataclasses.field(
        converter=lambda names: tuple(map(str.lower, names))
      )  # Note that lambdas are supported, but discouraged as they are untyped.

      # The default value is also converted; therefore the following is not a
      # type error.
      stock_image_path: pathlib.PurePosixPath = dataclasses.field(
        converter=pathlib.PurePosixPath, default="assets/unknown.png"
      )

      # Default value conversion extends to `default_factory`;
      # therefore the following is also not a type error.
      shelves: tuple = dataclasses.field(
        converter=tuple, default_factory=list
      )

  item1 = InventoryItem(
    "1",
    [234, 765],
    None,
    ["PYTHON PLUSHIE", "FLUFFY SNAKE"]
  )
  # item1's repr would be (with added newlines for readability):
  #   InventoryItem(
  #     id=1,
  #     skus=(234, 765),
  #     vendor=None,
  #     names=('PYTHON PLUSHIE', 'FLUFFY SNAKE'),
  #     stock_image_path=PurePosixPath('assets/unknown.png'),
  #     shelves=()
  #   )

  # Attribute assignment also participates in conversion.
  item1.skus = [555]
  # item1's skus attribute is now (555,).


Impact on typing
----------------

A ``converter`` must be a callable that accepts a single positional argument, and
the parameter type corresponding to this positional argument provides the type
of the the synthesized ``__init__`` parameter associated with the field.

In other words, the argument provided for the converter parameter must be
compatible with ``Callable[[T], X]`` where ``T`` is the input type for
the converter and ``X`` is the output type of the converter.

Type-checking ``default`` and ``default_factory``
'''''''''''''''''''''''''''''''''''''''''''''''''

Because default values are unconditionally converted using ``converter``, if
an argument for ``converter`` is provided alongside either ``default`` or
``default_factory``, the type of the default (the ``default`` argument if
provided, otherwise the return value of ``default_factory``) should be checked
using the type of the single argument to the ``converter`` callable.

Converter return type
'''''''''''''''''''''

The return type of the callable must be a type that's compatible with the
field's declared type. This includes the field's type exactly, but can also be
a type that's more specialized (such as a converter returning a ``list[int]``
for a fi


id: pep-0289.rst
content:
PEP: 289
Title: Generator Expressions
Author: Raymond Hettinger <python@rcn.com>
Status: Final
Type: Standards Track
Created: 30-Jan-2002
Python-Version: 2.4
Post-History: 22-Oct-2003


Abstract
========

This PEP introduces generator expressions as a high performance,
memory efficient generalization of list comprehensions :pep:`202` and
generators :pep:`255`.


Rationale
=========

Experience with list comprehensions has shown their widespread
utility throughout Python.  However, many of the use cases do
not need to have a full list created in memory.  Instead, they
only need to iterate over the elements one at a time.

For instance, the following summation code will build a full list of
squares in memory, iterate over those values, and, when the reference
is no longer needed, delete the list::

    sum([x*x for x in range(10)])

Memory is conserved by using a generator expression instead::

    sum(x*x for x in range(10))

Similar benefits are conferred on constructors for container objects::

    s = set(word  for line in page  for word in line.split())
    d = dict( (k, func(k)) for k in keylist)

Generator expressions are especially useful with functions like sum(),
min(), and max() that reduce an iterable input to a single value::

    max(len(line)  for line in file  if line.strip())

Generator expressions also address some examples of functionals coded
with lambda::

    reduce(lambda s, a: s + a.myattr, data, 0)
    reduce(lambda s, a: s + a[3], data, 0)

These simplify to::

    sum(a.myattr for a in data)
    sum(a[3] for a in data)

List comprehensions greatly reduced the need for filter() and map().
Likewise, generator expressions are expected to minimize the need
for itertools.ifilter() and itertools.imap().  In contrast, the
utility of other itertools will be enhanced by generator expressions::

    dotproduct = sum(x*y for x,y in itertools.izip(x_vector, y_vector))

Having a syntax similar to list comprehensions also makes it easy to
convert existing code into a generator expression when scaling up
application.

Early timings showed that generators had a significant performance
advantage over list comprehensions.  However, the latter were highly
optimized for Py2.4 and now the performance is roughly comparable
for small to mid-sized data sets.  As the data volumes grow larger,
generator expressions tend to perform better because they do not
exhaust cache memory and they allow Python to re-use objects between
iterations.

BDFL Pronouncements
===================

This PEP is ACCEPTED for Py2.4.


The Details
===========

(None of this is exact enough in the eye of a reader from Mars, but I
hope the examples convey the intention well enough for a discussion in
c.l.py.  The Python Reference Manual should contain a 100% exact
semantic and syntactic specification.)

1. The semantics of a generator expression are equivalent to creating
   an anonymous generator function and calling it.  For example::

       g = (x**2 for x in range(10))
       print g.next()

   is equivalent to::

       def __gen(exp):
           for x in exp:
               yield x**2
       g = __gen(iter(range(10)))
       print g.next()

   Only the outermost for-expression is evaluated immediately, the other
   expressions are deferred until the generator is run::


       g = (tgtexp  for var1 in exp1 if exp2 for var2 in exp3 if exp4)

   is equivalent to::

    def __gen(bound_exp):
        for var1 in bound_exp:
            if exp2:
                for var2 in exp3:
                    if exp4:
                        yield tgtexp
    g = __gen(iter(exp1))
    del __gen

2. The syntax requires that a generator expression always needs to be
   directly inside a set of parentheses and cannot have a comma on
   either side.  With reference to the file Grammar/Grammar in CVS,
   two rules change:

   a) The rule::

         atom: '(' [testlist] ')'

      changes to::

         atom: '(' [testlist_gexp] ')'

      where testlist_gexp is almost the same as listmaker, but only
      allows a single test after 'for' ... 'in'::

         testlist_gexp: test ( gen_for | (',' test)* [','] )

   b)  The rule for arglist needs similar changes.

   This means that you can write::

       sum(x**2 for x in range(10))

   but you would have to write::

       reduce(operator.add, (x**2 for x in range(10)))

   and also::

       g = (x**2 for x in range(10))

   i.e. if a function call has a single positional argument, it can be
   a generator expression without extra parentheses, but in all other
   cases you have to parenthesize it.

   The exact details were checked in to Grammar/Grammar version 1.49.

3. The loop variable (if it is a simple variable or a tuple of simple
   variables) is not exposed to the surrounding function.  This
   facilitates the implementation and makes typical use cases more
   reliable.  In some future version of Python, list comprehensions
   will also hide the induction variable from the surrounding code
   (and, in Py2.4, warnings will be issued for code accessing the
   induction variable).

   For example::

       x = "hello"
       y = list(x for x in "abc")
       print x    # prints "hello", not "c"

4. List comprehensions will remain unchanged.  For example::

       [x for x in S]    # This is a list comprehension.
       [(x for x in S)]  # This is a list containing one generator
                         # expression.

   Unfortunately, there is currently a slight syntactic difference.
   The expression::

       [x for x in 1, 2, 3]

   is legal, meaning::

       [x for x in (1, 2, 3)]

   But generator expressions will not allow the former version::

       (x for x in 1, 2, 3)

   is illegal.

   The former list comprehension syntax will become illegal in Python
   3.0, and should be deprecated in Python 2.4 and beyond.

   List comprehensions also "leak" their loop variable into the
   surrounding scope.  This will also change in Python 3.0, so that
   the semantic definition of a list comprehension in Python 3.0 will
   be equivalent to list(<generator expression>).  Python 2.4 and
   beyond should issue a deprecation warning if a list comprehension's
   loop variable has the same name as a variable used in the
   immediately surrounding scope.

Early Binding versus Late Binding
=================================

After much discussion, it was decided that the first (outermost)
for-expression should be evaluated immediately and that the remaining
expressions be evaluated when the generator is executed.

Asked to summarize the reasoning for binding the first expression,
Guido offered [1]_::

    Consider sum(x for x in foo()). Now suppose there's a bug in foo()
    that raises an exception, and a bug in sum() that raises an
    exception before it starts iterating over its argument. Which
    exception would you expect to see? I'd be surprised if the one in
    sum() was raised rather the one in foo(), since the call to foo()
    is part of the argument to sum(), and I expect arguments to be
    processed before the function is called.

    OTOH, in sum(bar(x) for x in foo()), where sum() and foo()
    are bugfree, but bar() raises an exception, we have no choice but
    to delay the call to bar() until sum() starts iterating -- that's
    part of the contract of generators. (They do nothing until their
    next() method is first called.)

Various use cases were proposed for binding all free variables when
the generator is defined.  And some proponents felt that the resulting
expressions would be easier to understand and debug if bound immediately.

However, Python takes a late binding approach to lambda expressions and
has no precedent for automatic, early binding.  It was felt that
introducing a new paradigm would unnecessarily introduce complexity.

After exploring many possibilities, a consensus emerged that binding
issues were hard to understand and that users should be strongly
encouraged to use generator expressions inside fu


id: pep-0504.rst
content:
PEP: 504
Title: Using the System RNG by default
Author: Alyssa Coghlan <ncoghlan@gmail.com>
Status: Withdrawn
Type: Standards Track
Created: 15-Sep-2015
Python-Version: 3.6
Post-History: 15-Sep-2015

Abstract
========

Python currently defaults to using the deterministic Mersenne Twister random
number generator for the module level APIs in the ``random`` module, requiring
users to know that when they're performing "security sensitive" work, they
should instead switch to using the cryptographically secure ``os.urandom`` or
``random.SystemRandom`` interfaces or a third party library like
``cryptography``.

Unfortunately, this approach has resulted in a situation where developers that
aren't aware that they're doing security sensitive work use the default module
level APIs, and thus expose their users to unnecessary risks.

This isn't an acute problem, but it is a chronic one, and the often long
delays between the introduction of security flaws and their exploitation means
that it is difficult for developers to naturally learn from experience.

In order to provide an eventually pervasive solution to the problem, this PEP
proposes that Python switch to using the system random number generator by
default in Python 3.6, and require developers to opt-in to using the
deterministic random number generator process wide either by using a new
``random.ensure_repeatable()`` API, or by explicitly creating their own
``random.Random()`` instance.

To minimise the impact on existing code, module level APIs that require
determinism will implicitly switch to the deterministic PRNG.

PEP Withdrawal
==============

During discussion of this PEP, Steven D'Aprano proposed the simpler alternative
of offering a standardised ``secrets`` module that provides "one obvious way"
to handle security sensitive tasks like generating default passwords and other
tokens.

Steven's proposal has the desired effect of aligning the easy way to generate
such tokens and the right way to generate them, without introducing any
compatibility risks for the existing ``random`` module API, so this PEP has
been withdrawn in favour of further work on refining Steven's proposal as
:pep:`506`.


Proposal
========

Currently, it is never correct to use the module level functions in the
``random`` module for security sensitive applications. This PEP proposes to
change that admonition in Python 3.6+ to instead be that it is not correct to
use the module level functions in the ``random`` module for security sensitive
applications if ``random.ensure_repeatable()`` is ever called (directly or
indirectly) in that process.

To achieve this, rather than being bound methods of a ``random.Random``
instance as they are today, the module level callables in ``random`` would
change to be functions that delegate to the corresponding method of the
existing ``random._inst`` module attribute.

By default, this attribute will be bound to a ``random.SystemRandom`` instance.

A new ``random.ensure_repeatable()`` API will then rebind the ``random._inst``
attribute to a ``system.Random`` instance, restoring the same module level
API behaviour as existed in previous Python versions (aside from the
additional level of indirection)::

    def ensure_repeatable():
        """Switch to using random.Random() for the module level APIs

        This switches the default RNG instance from the cryptographically
        secure random.SystemRandom() to the deterministic random.Random(),
        enabling the seed(), getstate() and setstate() operations. This means
        a particular random scenario can be replayed later by providing the
        same seed value or restoring a previously saved state.

        NOTE: Libraries implementing security sensitive operations should
        always explicitly use random.SystemRandom() or os.urandom in order to
        correctly handle applications that call this function.
        """
        if not isinstance(_inst, Random):
            _inst = random.Random()

To minimise the impact on existing code, calling any of the following module
level functions will implicitly call ``random.ensure_repeatable()``:

* ``random.seed``
* ``random.getstate``
* ``random.setstate``

There are no changes proposed to the ``random.Random`` or
``random.SystemRandom`` class APIs - applications that explicitly instantiate
their own random number generators will be entirely unaffected by this
proposal.

Warning on implicit opt-in
--------------------------

In Python 3.6, implicitly opting in to the use of the deterministic PRNG will
emit a deprecation warning using the following check::

    if not isinstance(_inst, Random):
        warnings.warn(DeprecationWarning,
                      "Implicitly ensuring repeatability. "
                      "See help(random.ensure_repeatable) for details")
        ensure_repeatable()

The specific wording of the warning should have a suitable answer added to
Stack Overflow as was done for the custom error message that was added for
missing parentheses in a call to print [#print]_.

In the first Python 3 release after Python 2.7 switches to security fix only
mode, the deprecation warning will be upgraded to a RuntimeWarning so it is
visible by default.

This PEP does *not* propose ever removing the ability to ensure the default RNG
used process wide is a deterministic PRNG that will produce the same series of
outputs given a specific seed. That capability is widely used in modelling
and simulation scenarios, and requiring that ``ensure_repeatable()`` be called
either directly or indirectly is a sufficient enhancement to address the cases
where the module level random API is used for security sensitive tasks in web
applications without due consideration for the potential security implications
of using a deterministic PRNG.

Performance impact
------------------

Due to the large performance difference between ``random.Random`` and
``random.SystemRandom``, applications ported to Python 3.6 will encounter a
significant performance regression in cases where:

* the application is using the module level random API
* cryptographic quality randomness isn't needed
* the application doesn't already implicitly opt back in to the deterministic
  PRNG by calling ``random.seed``,  ``random.getstate``,  or ``random.setstate``
* the application isn't updated to explicitly call ``random.ensure_repeatable``

This would be noted in the Porting section of the Python 3.6 What's New guide,
with the recommendation to include the following code in the ``__main__``
module of affected applications::

    if hasattr(random, "ensure_repeatable"):
        random.ensure_repeatable()

Applications that do need cryptographic quality randomness should be using the
system random number generator regardless of speed considerations, so in those
cases the change proposed in this PEP will fix a previously latent security
defect.

Documentation changes
---------------------

The ``random`` module documentation would be updated to move the documentation
of the ``seed``, ``getstate`` and ``setstate`` interfaces later in the module,
along with the documentation of the new ``ensure_repeatable`` function and the
associated security warning.

That section of the module documentation would also gain a discussion of the
respective use cases for the deterministic PRNG enabled by
``ensure_repeatable`` (games, modelling & simulation, software testing) and the
system RNG that is used by default (cryptography, security token generation).
This discussion will also recommend the use of third party security libraries
for the latter task.

Rationale
=========

Writing secure software under deadline and budget pressures is a hard problem.
This is reflected in regular notifications of data breaches involving personally
identifiable information [#breaches]_, as well as with failures to take
security considerations into account when new systems, like motor vehicles
[#uconnect]_, are connected to the internet. It's also the case that a lo


id: pep-0262.rst
content:
PEP: 262
Title: A Database of Installed Python Packages
Author: A.M. Kuchling <amk@amk.ca>
Status: Rejected
Type: Standards Track
Topic: Packaging
Created: 08-Jul-2001
Post-History: 27-Mar-2002


.. note::
   This PEP was superseded by :pep:`345` and :pep:`376`, which were accepted.
   Therefore, this PEP is (by implication) rejected.


Introduction
============

This PEP describes a format for a database of the Python software
installed on a system.

(In this document, the term "distribution" is used to mean a set
of code that's developed and distributed together.  A "distribution"
is the same as a Red Hat or Debian package, but the term "package"
already has a meaning in Python terminology, meaning "a directory
with an ``__init__.py`` file in it.")


Requirements
============

We need a way to figure out what distributions, and what versions of
those distributions, are installed on a system.  We want to provide
features similar to CPAN, APT, or RPM.  Required use cases that
should be supported are:

* Is distribution X on a system?
* What version of distribution X is installed?
* Where can the new version of distribution X be found?  (This can
  be defined as either "a home page where the user can go and
  find a download link", or "a place where a program can find
  the newest version?"  Both should probably be supported.)
* What files did distribution X put on my system?
* What distribution did the file x/y/z.py come from?
* Has anyone modified x/y/z.py locally?
* What other distributions does this software need?
* What Python modules does this distribution provide?


Database Location
=================

The database lives in a bunch of files under
<prefix>/lib/python<version>/install-db/.  This location will be
called INSTALLDB through the remainder of this PEP.

The structure of the database is deliberately kept simple; each
file in this directory or its subdirectories (if any) describes a
single distribution.  Binary packagings of Python software such as
RPMs can then update Python's database by just installing the
corresponding file into the INSTALLDB directory.

The rationale for scanning subdirectories is that we can move to a
directory-based indexing scheme if the database directory contains
too many entries.  For example, this would let us transparently
switch from INSTALLDB/Numeric to INSTALLDB/N/Nu/Numeric or some
similar hashing scheme.


Database Contents
=================

Each file in INSTALLDB or its subdirectories describes a single
distribution, and has the following contents:

An initial line listing the sections in this file, separated
by whitespace.  Currently this will always be 'PKG-INFO FILES
REQUIRES PROVIDES'.  This is for future-proofing; if we add a
new section, for example to list documentation files, then
we'd add a DOCS section and list it in the contents.  Sections
are always separated by blank lines.

A distribution that uses the Distutils for installation should
automatically update the database.  Distributions that roll their
own installation will have to use the database's API to
manually add or update their own entry.  System package managers
such as RPM or pkgadd can just create the new file in the
INSTALLDB directory.

Each section of the file is used for a different purpose.

PKG-INFO section
----------------

An initial set of :rfc:`822` headers containing the distribution
information for a file, as described in :pep:`241`, "Metadata for
Python Software Packages".

FILES section
-------------

An entry for each file installed by the
distribution. Generated files such as .pyc and .pyo files are
on this list as well as the original .py files installed by a
distribution; their checksums won't be stored or checked,
though.

Each file's entry is a single tab-delimited line that contains
the following fields:

* The file's full path, as installed on the system.

* The file's size

* The file's permissions.  On Windows, this field will always be
  'unknown'

* The owner and group of the file, separated by a tab.
  On Windows, these fields will both be 'unknown'.

* A SHA1 digest of the file, encoded in hex.  For generated files
  such as \*.pyc files, this field must contain the string "-",
  which indicates that the file's checksum should not be verified.


REQUIRES section
----------------

This section is a list of strings giving the services required for
this module distribution to run properly.  This list includes the
distribution name ("python-stdlib") and module names ("rfc822",
"htmllib", "email", "email.Charset").  It will be specified
by an extra 'requires' argument to the ``distutils.core.setup()``
function.  For example::

    setup(..., requires=['xml.utils.iso8601',

Eventually there may be automated tools that look through all of
the code and produce a list of requirements, but it's unlikely
that these tools can handle all possible cases; a manual
way to specify requirements will always be necessary.


PROVIDES section
----------------

This section is a list of strings giving the services provided by
an installed distribution.  This list includes the distribution name
("python-stdlib") and module names ("rfc822", "htmllib", "email",
"email.Charset").

XXX should files be listed?  e.g. $PREFIX/lib/color-table.txt,
to pick up data files, required scripts, etc.

Eventually there may be an option to let module developers add
their own strings to this section.  For example, you might add
"XML parser" to this section, and other module distributions could
then list "XML parser" as one of their dependencies to indicate
that multiple different XML parsers can be used.  For now this
ability isn't supported because it raises too many issues: do we
need a central registry of legal strings, or just let people put
whatever they like?  Etc., etc...


API Description
===============

There's a single fundamental class, InstallationDatabase.  The
code for it lives in distutils/install_db.py.  (XXX any
suggestions for alternate locations in the standard library, or an
alternate module name?)

The InstallationDatabase returns instances of Distribution that contain
all the information about an installed distribution.

XXX Several of the fields in Distribution are duplicates of ones in
distutils.dist.Distribution.  Probably they should be factored out
into the Distribution class proposed here, but can this be done in a
backward-compatible way?

InstallationDatabase has the following interface::

    class InstallationDatabase:
        def __init__ (self, path=None):
            """InstallationDatabase(path:string)
            Read the installation database rooted at the specified path.
            If path is None, INSTALLDB is used as the default.
            """

        def get_distribution (self, distribution_name):
            """get_distribution(distribution_name:string) : Distribution
            Get the object corresponding to a single distribution.
            """

        def list_distributions (self):
            """list_distributions() : [Distribution]
            Return a list of all distributions installed on the system,
            enumerated in no particular order.
            """

        def find_distribution (self, path):
            """find_file(path:string) : Distribution
            Search and return the distribution containing the file 'path'.
            Returns None if the file doesn't belong to any distribution
            that the InstallationDatabase knows about.
            XXX should this work for directories?
            """

    class Distribution:
        """Instance attributes:
        name : string
          Distribution name
        files : {string : (size:int, perms:int, owner:string, group:string,
                           digest:string)}
           Dictionary mapping the path of a file installed by this distribution
           to information about the file.

        The following fields all come from PEP 241.

        version : distutils.version.Version
          Version of this


id: pep-0276.rst
content:
PEP: 276
Title: Simple Iterator for ints
Author: Jim Althoff <james_althoff@i2.com>
Status: Rejected
Type: Standards Track
Created: 12-Nov-2001
Python-Version: 2.3
Post-History:


Abstract
========

Python 2.1 added new functionality to support iterators (:pep:`234`).
Iterators have proven to be useful and convenient in many coding
situations.  It is noted that the implementation of Python's
for-loop control structure uses the iterator protocol as of
release 2.1.  It is also noted that Python provides iterators for
the following builtin types: lists, tuples, dictionaries, strings,
and files.  This PEP proposes the addition of an iterator for the
builtin type int (``types.IntType``).  Such an iterator would simplify
the coding of certain for-loops in Python.

BDFL Pronouncement
==================

This PEP was rejected on 17 June 2005 with a note to python-dev.

Much of the original need was met by the ``enumerate()`` function which
was accepted for Python 2.3.

Also, the proposal both allowed and encouraged misuses such as::

    >>> for i in 3: print i
    0
    1
    2

Likewise, it was not helpful that the proposal would disable the
syntax error in statements like::

    x, = 1

Specification
=============

Define an iterator for types.intType (i.e., the builtin type
"int") that is returned from the builtin function "iter" when
called with an instance of types.intType as the argument.

The returned iterator has the following behavior:

- Assume that object i is an instance of ``types.intType`` (the
  builtin type int) and that i > 0

- ``iter(i)`` returns an iterator object

- said iterator object iterates through the sequence of ints
  0,1,2,...,i-1

  Example:

    ``iter(5)`` returns an iterator object that iterates through the
    sequence of ints 0,1,2,3,4

- if i <= 0, ``iter(i)`` returns an "empty" iterator, i.e., one that
  throws StopIteration upon the first call of its "next" method

In other words, the conditions and semantics of said iterator is
consistent with the conditions and semantics of the ``range()`` and
``xrange()`` functions.

Note that the sequence 0,1,2,...,i-1 associated with the int i is
considered "natural" in the context of Python programming because
it is consistent with the builtin indexing protocol of sequences
in Python.  Python lists and tuples, for example, are indexed
starting at 0 and ending at len(object)-1 (when using positive
indices).  In other words, such objects are indexed with the
sequence 0,1,2,...,len(object)-1


Rationale
=========

A common programming idiom is to take a collection of objects and
apply some operation to each item in the collection in some
established sequential order.  Python provides the "for in"
looping control structure for handling this common idiom.  Cases
arise, however, where it is necessary (or more convenient) to
access each item in an "indexed" collection by iterating through
each index and accessing each item in the collection using the
corresponding index.

For example, one might have a two-dimensional "table" object where one
requires the application of some operation to the first column of
each row in the table.  Depending on the implementation of the table
it might not be possible to access first each row and then each
column as individual objects.  It might, rather, be possible to
access a cell in the table using a row index and a column index.
In such a case it is necessary to use an idiom where one iterates
through a sequence of indices (indexes) in order to access the
desired items in the table.  (Note that the commonly used
DefaultTableModel class in Java-Swing-Jython has this very protocol).

Another common example is where one needs to process two or more
collections in parallel.  Another example is where one needs to
access, say, every second item in a collection.

There are many other examples where access to items in a
collection is facilitated by a computation on an index thus
necessitating access to the indices rather than direct access to
the items themselves.

Let's call this idiom the "indexed for-loop" idiom.  Some
programming languages provide builtin syntax for handling this
idiom.  In Python the common convention for implementing the
indexed for-loop idiom is to use the builtin ``range()`` or ``xrange()``
function to generate a sequence of indices as in, for example::

    for rowcount in range(table.getRowCount()):
        print table.getValueAt(rowcount, 0)

or

::

    for rowcount in xrange(table.getRowCount()):
        print table.getValueAt(rowcount, 0)

From time to time there are discussions in the Python community
about the indexed for-loop idiom.  It is sometimes argued that the
need for using the ``range()`` or ``xrange()`` function for this design
idiom is:

- Not obvious (to new-to-Python programmers),

- Error prone (easy to forget, even for experienced Python
  programmers)

- Confusing and distracting for those who feel compelled to understand
  the differences and recommended usage of ``xrange()`` vis-a-vis ``range()``

- Unwieldy, especially when combined with the ``len()`` function,
  i.e., ``xrange(len(sequence))``

- Not as convenient as equivalent mechanisms in other languages,

- Annoying, a "wart", etc.

And from time to time proposals are put forth for ways in which
Python could provide a better mechanism for this idiom.  Recent
examples include :pep:`204`, "Range Literals", and :pep:`212`, "Loop
Counter Iteration".

Most often, such proposal include changes to Python's syntax and
other "heavyweight" changes.

Part of the difficulty here is that advocating new syntax implies
a comprehensive solution for "general indexing" that has to
include aspects like:

- starting index value

- ending index value

- step value

- open intervals versus closed intervals versus half opened intervals

Finding a new syntax that is comprehensive, simple, general,
Pythonic, appealing to many, easy to implement, not in conflict
with existing structures, not excessively overloading of existing
structures, etc. has proven to be more difficult than one might
anticipate.

The proposal outlined in this PEP tries to address the problem by
suggesting a simple "lightweight" solution that helps the most
common case by using a proven mechanism that is already available
(as of Python 2.1): namely, iterators.

Because for-loops already use "iterator" protocol as of Python
2.1, adding an iterator for types.IntType as proposed in this PEP
would enable by default the following shortcut for the indexed
for-loop idiom::

    for rowcount in table.getRowCount():
        print table.getValueAt(rowcount, 0)

The following benefits for this approach vis-a-vis the current
mechanism of using the ``range()`` or ``xrange()`` functions are claimed
to be:

- Simpler,

- Less cluttered,

- Focuses on the problem at hand without the need to resort to
  secondary implementation-oriented functions (``range()`` and
  ``xrange()``)

And compared to other proposals for change:

- Requires no new syntax

- Requires no new keywords

- Takes advantage of the new and well-established iterator mechanism

And generally:

-  Is consistent with iterator-based "convenience" changes already
   included (as of Python 2.1) for other builtin types such as:
   lists, tuples, dictionaries, strings, and files.


Backwards Compatibility
=======================

The proposed mechanism is generally backwards compatible as it
calls for neither new syntax nor new keywords.  All existing,
valid Python programs should continue to work unmodified.

However, this proposal is not perfectly backwards compatible in
the sense that certain statements that are currently invalid
would, under the current proposal, become valid.

Tim Peters has pointed out two such examples:

1) The common case where one forgets to include ``range()`` or
   ``xrange()``, for example::

       for rowcount in table.getRowCount():
           print table.getValueAt(rowcount, 0)

   in Python 2.2 raises a TypeError exception.

   U


id: pep-0510.rst
content:
PEP: 510
Title: Specialize functions with guards
Author: Victor Stinner <vstinner@python.org>
Status: Rejected
Type: Standards Track
Created: 04-Jan-2016
Python-Version: 3.6


Rejection Notice
================

This PEP was rejected by its author since the design didn't show any
significant speedup, but also because of the lack of time to implement
the most advanced and complex optimizations.


Abstract
========

Add functions to the Python C API to specialize pure Python functions:
add specialized codes with guards. It allows to implement static
optimizers respecting the Python semantics.


Rationale
=========

Python semantics
----------------

Python is hard to optimize because almost everything is mutable: builtin
functions, function code, global variables, local variables, ... can be
modified at runtime. Implement optimizations respecting the Python
semantics requires to detect when "something changes", we will call these
checks "guards".

This PEP proposes to add a public API to the Python C API to add
specialized codes with guards to a function. When the function is
called, a specialized code is used if nothing changed, otherwise use the
original bytecode.

Even if guards help to respect most parts of the Python semantics, it's
hard to optimize Python without making subtle changes on the exact
behaviour. CPython has a long history and many applications rely on
implementation details. A compromise must be found between "everything
is mutable" and performance.

Writing an optimizer is out of the scope of this PEP.


Why not a JIT compiler?
-----------------------

There are multiple JIT compilers for Python actively developed:

* `PyPy <http://pypy.org/>`_
* `Pyston <https://github.com/dropbox/pyston>`_
* `Numba <http://numba.pydata.org/>`_
* `Pyjion <https://github.com/microsoft/pyjion>`_

Numba is specific to numerical computation.  Pyston and Pyjion are still
young.  PyPy is the most complete Python interpreter, it is generally
faster than CPython in micro- and many macro-benchmarks and has a very
good compatibility with CPython (it respects the Python semantics).
There are still issues with Python JIT compilers which avoid them to be
widely used instead of CPython.

Many popular libraries like numpy, PyGTK, PyQt, PySide and wxPython are
implemented in C or C++ and use the Python C API. To have a small memory
footprint and better performances, Python JIT compilers do not use
reference counting to use a faster garbage collector, do not use C
structures of CPython objects and manage memory allocations differently.
PyPy has a ``cpyext`` module which emulates the Python C API but it has
worse performances than CPython and does not support the full Python C
API.

New features are first developed in CPython. In January 2016, the
latest CPython stable version is 3.5, whereas PyPy only supports Python
2.7 and 3.2, and Pyston only supports Python 2.7.

Even if PyPy has a very good compatibility with Python, some modules are
still not compatible with PyPy: see `PyPy Compatibility Wiki
<https://bitbucket.org/pypy/compatibility/wiki/Home>`_. The incomplete
support of the Python C API is part of this problem. There are also
subtle differences between PyPy and CPython like reference counting:
object destructors are always called in PyPy, but can be called "later"
than in CPython. Using context managers helps to control when resources
are released.

Even if PyPy is much faster than CPython in a wide range of benchmarks,
some users still report worse performances than CPython on some specific
use cases or unstable performances.

When Python is used as a scripting program for programs running less
than 1 minute, JIT compilers can be slower because their startup time is
higher and the JIT compiler takes time to optimize the code. For
example, most Mercurial commands take a few seconds.

Numba now supports ahead of time compilation, but it requires decorator
to specify arguments types and it only supports numerical types.

CPython 3.5 has almost no optimization: the peephole optimizer only
implements basic optimizations. A static compiler is a compromise
between CPython 3.5 and PyPy.

.. note::
   There was also the Unladen Swallow project, but it was abandoned in
   2011.


Examples
========

Following examples are not written to show powerful optimizations
promising important speedup, but to be short and easy to understand,
just to explain the principle.

Hypothetical myoptimizer module
-------------------------------

Examples in this PEP uses a hypothetical ``myoptimizer`` module which
provides the following functions and types:

* ``specialize(func, code, guards)``: add the specialized code ``code``
  with guards ``guards`` to the function ``func``
* ``get_specialized(func)``: get the list of specialized codes as a list
  of ``(code, guards)`` tuples where ``code`` is a callable or code object
  and ``guards`` is a list of a guards
* ``GuardBuiltins(name)``: guard watching for
  ``builtins.__dict__[name]`` and ``globals()[name]``. The guard fails
  if ``builtins.__dict__[name]`` is replaced, or if ``globals()[name]``
  is set.


Using bytecode
--------------

Add specialized bytecode where the call to the pure builtin function
``chr(65)`` is replaced with its result ``"A"``::

    import myoptimizer

    def func():
        return chr(65)

    def fast_func():
        return "A"

    myoptimizer.specialize(func, fast_func.__code__,
                           [myoptimizer.GuardBuiltins("chr")])
    del fast_func

Example showing the behaviour of the guard::

    print("func(): %s" % func())
    print("#specialized: %s" % len(myoptimizer.get_specialized(func)))
    print()

    import builtins
    builtins.chr = lambda obj: "mock"

    print("func(): %s" % func())
    print("#specialized: %s" % len(myoptimizer.get_specialized(func)))

Output::

    func(): A
    #specialized: 1

    func(): mock
    #specialized: 0

The first call uses the specialized bytecode which returns the string
``"A"``. The second call removes the specialized code because the
builtin ``chr()`` function was replaced, and executes the original
bytecode calling ``chr(65)``.

On a microbenchmark, calling the specialized bytecode takes 88 ns,
whereas the original function takes 145 ns (+57 ns): 1.6 times as fast.


Using builtin function
----------------------

Add the C builtin ``chr()`` function as the specialized code instead of
a bytecode calling ``chr(obj)``::

    import myoptimizer

    def func(arg):
        return chr(arg)

    myoptimizer.specialize(func, chr,
                           [myoptimizer.GuardBuiltins("chr")])

Example showing the behaviour of the guard::

    print("func(65): %s" % func(65))
    print("#specialized: %s" % len(myoptimizer.get_specialized(func)))
    print()

    import builtins
    builtins.chr = lambda obj: "mock"

    print("func(65): %s" % func(65))
    print("#specialized: %s" % len(myoptimizer.get_specialized(func)))

Output::

    func(): A
    #specialized: 1

    func(): mock
    #specialized: 0

The first call calls the C builtin ``chr()`` function (without creating
a Python frame). The second call removes the specialized code because
the builtin ``chr()`` function was replaced, and executes the original
bytecode.

On a microbenchmark, calling the C builtin takes 95 ns, whereas the
original bytecode takes 155 ns (+60 ns): 1.6 times as fast. Calling
directly ``chr(65)`` takes 76 ns.


Choose the specialized code
===========================

Pseudo-code to choose the specialized code to call a pure Python
function::

    def call_func(func, args, kwargs):
        specialized = myoptimizer.get_specialized(func)
        nspecialized = len(specialized)
        index = 0
        while index < nspecialized:
            specialized_code, guards = specialized[index]

            for guard in guards:
                check = guard(args, kwargs)
                if check:
                    break

            if not check:
                # all guards 


id: pep-0538.rst
content:
PEP: 538
Title: Coercing the legacy C locale to a UTF-8 based locale
Author: Alyssa Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: INADA Naoki
Status: Final
Type: Standards Track
Created: 28-Dec-2016
Python-Version: 3.7
Post-History: 03-Jan-2017,
              07-Jan-2017,
              05-Mar-2017,
              09-May-2017
Resolution: https://mail.python.org/pipermail/python-dev/2017-May/148035.html

Abstract
========

An ongoing challenge with Python 3 on \*nix systems is the conflict between
needing to use the configured locale encoding by default for consistency with
other locale-aware components in the same process or subprocesses,
and the fact that the standard C locale (as defined in POSIX:2001) typically
implies a default text encoding of ASCII, which is entirely inadequate for the
development of networked services and client applications in a multilingual
world.

:pep:`540` proposes a change to CPython's handling of the legacy C locale such
that CPython will assume the use of UTF-8 in such environments, rather than
persisting with the demonstrably problematic assumption of ASCII as an
appropriate encoding for communicating with operating system interfaces.
This is a good approach for cases where network encoding interoperability
is a more important concern than local encoding interoperability.

However, it comes at the cost of making CPython's encoding assumptions diverge
from those of other locale-aware components in the same process, as well as
those of components running in subprocesses that share the same environment.

This can cause interoperability problems with some extension modules (such as
GNU readline's command line history editing), as well as with components
running in subprocesses (such as older Python runtimes).

It also requires non-trivial changes to the internals of how CPython itself
works, rather than relying primarily on existing configuration settings that
are supported by Python versions prior to Python 3.7.

Accordingly, this PEP proposes that independently of the UTF-8 mode proposed
in :pep:`540`, the way the CPython implementation handles the default C locale be
changed to be roughly equivalent to the following existing configuration
settings (supported since Python 3.1)::

    LC_CTYPE=C.UTF-8
    PYTHONIOENCODING=utf-8:surrogateescape

The exact target locale for coercion will be chosen from a predefined list at
runtime based on the actually available locales.

The reinterpreted locale settings will be written back to the environment so
they're visible to other components in the same process and in subprocesses,
but the changed ``PYTHONIOENCODING`` default will be made implicit in order to
avoid causing compatibility problems with Python 2 subprocesses that don't
provide the ``surrogateescape`` error handler.

The new legacy locale coercion behavior can be disabled either by setting
``LC_ALL`` (which may still lead to a Unicode compatibility warning) or by
setting the new ``PYTHONCOERCECLOCALE`` environment variable to ``0``.

With this change, any \*nix platform that does *not* offer at least one of the
``C.UTF-8``, ``C.utf8`` or ``UTF-8`` locales as part of its standard
configuration would only be considered a fully supported platform for CPython
3.7+ deployments when a suitable locale other than the default ``C`` locale is
configured explicitly (e.g. ``en_AU.UTF-8``, ``zh_CN.gb18030``). If :pep:`540` is
accepted in addition to this PEP, then pure Python modules would also be
supported when using the proposed ``PYTHONUTF8`` mode, but expectations for
full Unicode compatibility in extension modules would continue to be limited
to the platforms covered by this PEP.

As it only reflects a change in default settings rather than a fundamentally
new capability, redistributors (such as Linux distributions) with a narrower
target audience than the upstream CPython development team may also choose to
opt in to this locale coercion behaviour for the Python 3.6.x series by
applying the necessary changes as a downstream patch.


Implementation Notes
====================

Attempting to implement the PEP as originally accepted showed that the
proposal to emit locale coercion and compatibility warnings by default
simply wasn't practical (there were too many cases where previously working
code failed *because of the warnings*, rather than because of latent locale
handling defects in the affected code).

As a result, the ``PY_WARN_ON_C_LOCALE`` config flag was removed, and replaced
with a runtime ``PYTHONCOERCECLOCALE=warn`` environment variable setting
that allows developers and system integrators to opt-in to receiving locale
coercion and compatibility warnings, without emitting them by default.

The output examples in the PEP itself have also been updated to remove
the warnings and make them easier to read.


Background
==========

While the CPython interpreter is starting up, it may need to convert from
the ``char *`` format to the ``wchar_t *`` format, or from one of those formats
to ``PyUnicodeObject *``, in a way that's consistent with the locale settings
of the overall system. It handles these cases by relying on the operating
system to do the conversion and then ensuring that the text encoding name
reported by ``sys.getfilesystemencoding()`` matches the encoding used during
this early bootstrapping process.

On Windows, the limitations of the ``mbcs`` format used by default in these
conversions proved sufficiently problematic that :pep:`528` and :pep:`529` were
implemented to bypass the operating system supplied interfaces for binary data
handling and force the use of UTF-8 instead.

On Mac OS X, iOS, and Android, many components, including CPython, already
assume the use of UTF-8 as the system encoding, regardless of the locale
setting. However, this isn't the case for all components, and the discrepancy
can cause problems in some situations (for example, when using the GNU readline
module [16_]).

On non-Apple and non-Android \*nix systems, these operations are handled using
the C locale system in glibc, which has the following characteristics [4]_:

* by default, all processes start in the ``C`` locale, which uses ``ASCII``
  for these conversions. This is almost never what anyone doing multilingual
  text processing actually wants (including CPython and C/C++ GUI frameworks).
* calling ``setlocale(LC_ALL, "")`` reconfigures the active locale based on
  the locale categories configured in the current process environment
* if the locale requested by the current environment is unknown, or no specific
  locale is configured, then the default ``C`` locale will remain active

The specific locale category that covers the APIs that CPython depends on is
``LC_CTYPE``, which applies to "classification and conversion of characters,
and to multibyte and wide characters" [5]_. Accordingly, CPython includes the
following key calls to ``setlocale``:

* in the main ``python`` binary, CPython calls ``setlocale(LC_ALL, "")`` to
  configure the entire C locale subsystem according to the process environment.
  It does this prior to making any calls into the shared CPython library
* in ``Py_Initialize``, CPython calls ``setlocale(LC_CTYPE, "")``, such that
  the configured locale settings for that category *always* match those set in
  the environment. It does this unconditionally, and it *doesn't* revert the
  process state change in ``Py_Finalize``

(This summary of the locale handling omits several technical details related
to exactly where and when the text encoding declared as part of the locale
settings is used - see :pep:`540` for further discussion, as these particular
details matter more when decoupling CPython from the declared C locale than
they do when overriding the locale with one based on UTF-8)

These calls are usually sufficient to provide sensible behaviour, but they can
still fail in the following cases:

* SSH environment forwarding means that SSH clients may sometimes forward
  client locale settings to servers that


