# embeddings

id: pep-3000.rst
content:
PEP: 3000
Title: Python 3000
Author: Guido van Rossum <guido@python.org>
Status: Final
Type: Process
Created: 05-Apr-2006
Post-History:


Abstract
========

This PEP sets guidelines for Python 3000 development.  Ideally, we
first agree on the process, and start discussing features only after
the process has been decided and specified.  In practice, we'll be
discussing features and process simultaneously; often the debate about
a particular feature will prompt a process discussion.


Naming
======

Python 3000, Python 3.0 and Py3K are all names for the same thing.
The project is called Python 3000, or abbreviated to Py3k.  The actual
Python release will be referred to as Python 3.0, and that's
what "python3.0 -V" will print; the actual file names will use the
same naming convention we use for Python 2.x.  I don't want to pick a
new name for the executable or change the suffix for Python source
files.


PEP Numbering
=============

Python 3000 PEPs are numbered starting at :pep:`3000`.  PEPs 3000-3099
are meta-PEPs -- these can be either process or informational PEPs.
PEPs 3100-3999 are feature PEPs.  :pep:`3000` itself (this PEP) is
special; it is the meta-PEP for Python 3000 meta-PEPs (IOW it describe
the process to define processes).  :pep:`3100` is also special; it's a
laundry list of features that were selected for (hopeful) inclusion in
Python 3000 before we started the Python 3000 process for real.  PEP
3099, finally, is a list of features that will *not* change.


Timeline
========

See :pep:`361`, which contains the release schedule for Python
2.6 and 3.0.  These versions will be released in lockstep.

Note: standard library development is expected to ramp up after 3.0a1
is released.

I expect that there will be parallel Python 2.x and 3.x releases for
some time; the Python 2.x releases will continue for a longer time
than the traditional 2.x.y bugfix releases.  Typically, we stop
releasing bugfix versions for 2.x once version 2.(x+1) has been
released.  But I expect there to be at least one or two new 2.x
releases even after 3.0 (final) has been released, probably well into
3.1 or 3.2.  This will to some extent depend on community demand for
continued 2.x support, acceptance and stability of 3.0, and volunteer
stamina.

I expect that Python 3.1 and 3.2 will be released much sooner after
3.0 than has been customary for the 2.x series.  The 3.x release
pattern will stabilize once the community is happy with 3.x.


Compatibility and Transition
============================

Python 3.0 will break backwards compatibility with Python 2.x.

**There is no requirement that Python 2.6 code will run unmodified on
Python 3.0.** Not even a subset.  (Of course there will be a *tiny*
subset, but it will be missing major functionality.)

Python 2.6 will support forward compatibility in the following two
ways:

* It will support a "Py3k warnings mode" which will warn dynamically
  (i.e. at runtime) about features that will stop working in Python
  3.0, e.g. assuming that range() returns a list.
* It will contain backported versions of many Py3k features, either
  enabled through __future__ statements or simply by allowing old and
  new syntax to be used side-by-side (if the new syntax would be a
  syntax error in 2.x).

Instead, and complementary to the forward compatibility features in
2.6, there will be a separate source code conversion tool [1]_.  This
tool can do a context-free source-to-source translation.  For example,
it can translate ``apply(f, args)`` into ``f(*args)``.  However, the
tool cannot do data flow analysis or type inferencing, so it simply
assumes that ``apply`` in this example refers to the old built-in
function.

The recommended development model for a project that needs to support
Python 2.6 and 3.0 simultaneously is as follows:

0. You should have excellent unit tests with close to full coverage.
1. Port your project to Python 2.6.
2. Turn on the Py3k warnings mode.
3. Test and edit until no warnings remain.
4. Use the 2to3 tool to convert this source code to 3.0 syntax.
   **Do not manually edit the output!**
5. Test the converted source code under 3.0.
6. If problems are found, make corrections to the **2.6** version
   of the source code and go back to step 3.
7. When it's time to release, release separate 2.6 and 3.0 tarballs
   (or whatever archive form you use for releases).

It is recommended not to edit the 3.0 source code until you are ready
to reduce 2.6 support to pure maintenance (i.e. the moment when you
would normally move the 2.6 code to a maintenance branch anyway).

PS. We need a meta-PEP to describe the transitional issues in detail.


Implementation Language
=======================

Python 3000 will be implemented in C, and the implementation will be
derived as an evolution of the Python 2 code base. This reflects my
views (which I share with Joel Spolsky [2]_) on the dangers of complete
rewrites. Since Python 3000 as a language is a relatively mild
improvement on Python 2, we can gain a lot by not attempting to
reimplement the language from scratch. I am not against parallel
from-scratch implementation efforts, but my own efforts will be
directed at the language and implementation that I know best.


Meta-Contributions
==================

Suggestions for additional text for this PEP are gracefully accepted
by the author.  Draft meta-PEPs for the topics above and additional
topics are even more welcome!


References
==========

.. [1] The 2to3 tool, in the subversion sandbox
   http://svn.python.org/view/sandbox/trunk/2to3/

.. [2] Joel on Software: Things You Should Never Do, Part I
    http://www.joelonsoftware.com/articles/fog0000000069.html


Copyright
=========

This document has been placed in the public domain.


id: pep-3148.rst
content:
PEP:               3148
Title:             futures - execute computations asynchronously
Author:            Brian Quinlan <brian@sweetapp.com>
Status:            Final
Type:              Standards Track
Created:           16-Oct-2009
Python-Version:    3.2
Post-History:

========
Abstract
========

This PEP proposes a design for a package that facilitates the
evaluation of callables using threads and processes.

==========
Motivation
==========

Python currently has powerful primitives to construct multi-threaded
and multi-process applications but parallelizing simple operations
requires a lot of work i.e. explicitly launching processes/threads,
constructing a work/results queue, and waiting for completion or some
other termination condition (e.g.  failure, timeout).  It is also
difficult to design an application with a global process/thread limit
when each component invents its own parallel execution strategy.

=============
Specification
=============

Naming
------

The proposed package would be called "futures" and would live in a new
"concurrent" top-level package. The rationale behind pushing the
futures library into a "concurrent" namespace has multiple components.
The first, most simple one is to prevent any and all confusion with
the existing "from __future__ import x" idiom which has been in use
for a long time within Python. Additionally, it is felt that adding
the "concurrent" precursor to the name fully denotes what the library
is related to - namely concurrency - this should clear up any addition
ambiguity as it has been noted that not everyone in the community is
familiar with Java Futures, or the Futures term except as it relates
to the US stock market.

Finally; we are carving out a new namespace for the standard library -
obviously named "concurrent". We hope to either add, or move existing,
concurrency-related libraries to this in the future.  A prime example
is the multiprocessing.Pool work, as well as other "addons" included
in that module, which work across thread and process boundaries.

Interface
---------

The proposed package provides two core classes: ``Executor`` and
``Future``. An ``Executor`` receives asynchronous work requests (in terms
of a callable and its arguments) and returns a ``Future`` to represent
the execution of that work request.

Executor
''''''''

``Executor`` is an abstract class that provides methods to execute calls
asynchronously.

``submit(fn, *args, **kwargs)``

    Schedules the callable to be executed as ``fn(*args, **kwargs)``
    and returns a ``Future`` instance representing the execution of the
    callable.

    This is an abstract method and must be implemented by Executor
    subclasses.

``map(func, *iterables, timeout=None)``

    Equivalent to ``map(func, *iterables)`` but func is executed
    asynchronously and several calls to func may be made concurrently.
    The returned iterator raises a ``TimeoutError`` if ``__next__()`` is
    called and the result isn't available after *timeout* seconds from
    the original call to ``map()``.  If *timeout* is not specified or
    ``None`` then there is no limit to the wait time.  If a call raises
    an exception then that exception will be raised when its value is
    retrieved from the iterator.

``shutdown(wait=True)``

    Signal the executor that it should free any resources that it is
    using when the currently pending futures are done executing.
    Calls to ``Executor.submit`` and ``Executor.map`` and made after
    shutdown will raise ``RuntimeError``.

    If wait is ``True`` then this method will not return until all the
    pending futures are done executing and the resources associated
    with the executor have been freed. If wait is ``False`` then this
    method will return immediately and the resources associated with
    the executor will be freed when all pending futures are done
    executing. Regardless of the value of wait, the entire Python
    program will not exit until all pending futures are done
    executing.

| ``__enter__()``
| ``__exit__(exc_type, exc_val, exc_tb)``

    When using an executor as a context manager, ``__exit__`` will call
    ``Executor.shutdown(wait=True)``.


ProcessPoolExecutor
'''''''''''''''''''

The ``ProcessPoolExecutor`` class is an ``Executor`` subclass that uses a
pool of processes to execute calls asynchronously.  The callable
objects and arguments passed to ``ProcessPoolExecutor.submit`` must be
pickleable according to the same limitations as the multiprocessing
module.

Calling ``Executor`` or ``Future`` methods from within a callable
submitted to a ``ProcessPoolExecutor`` will result in deadlock.

``__init__(max_workers)``

    Executes calls asynchronously using a pool of a most *max_workers*
    processes.  If *max_workers* is ``None`` or not given then as many
    worker processes will be created as the machine has processors.

ThreadPoolExecutor
''''''''''''''''''

The ``ThreadPoolExecutor`` class is an ``Executor`` subclass that uses a
pool of threads to execute calls asynchronously.

Deadlock can occur when the callable associated with a ``Future`` waits
on the results of another ``Future``.  For example::

    import time
    def wait_on_b():
        time.sleep(5)
        print(b.result())  # b will never complete because it is waiting on a.
        return 5

    def wait_on_a():
        time.sleep(5)
        print(a.result())  # a will never complete because it is waiting on b.
        return 6


    executor = ThreadPoolExecutor(max_workers=2)
    a = executor.submit(wait_on_b)
    b = executor.submit(wait_on_a)

And::

    def wait_on_future():
        f = executor.submit(pow, 5, 2)
        # This will never complete because there is only one worker thread and
        # it is executing this function.
        print(f.result())

    executor = ThreadPoolExecutor(max_workers=1)
    executor.submit(wait_on_future)

``__init__(max_workers)``

    Executes calls asynchronously using a pool of at most
    *max_workers* threads.

Future Objects
''''''''''''''

The ``Future`` class encapsulates the asynchronous execution of a
callable. ``Future`` instances are returned by ``Executor.submit``.

``cancel()``

    Attempt to cancel the call.  If the call is currently being
    executed then it cannot be cancelled and the method will return
    ``False``, otherwise the call will be cancelled and the method will
    return ``True``.

``cancelled()``

    Return ``True`` if the call was successfully cancelled.

``running()``

    Return ``True`` if the call is currently being executed and cannot
    be cancelled.

``done()``

    Return ``True`` if the call was successfully cancelled or finished
    running.

``result(timeout=None)``

    Return the value returned by the call.  If the call hasn't yet
    completed then this method will wait up to *timeout* seconds.  If
    the call hasn't completed in *timeout* seconds then a
    ``TimeoutError`` will be raised.  If *timeout* is not specified or
    ``None`` then there is no limit to the wait time.

    If the future is cancelled before completing then ``CancelledError``
    will be raised.

    If the call raised then this method will raise the same exception.

``exception(timeout=None)``

    Return the exception raised by the call.  If the call hasn't yet
    completed then this method will wait up to *timeout* seconds.  If
    the call hasn't completed in *timeout* seconds then a
    ``TimeoutError`` will be raised.  If *timeout* is not specified or
    ``None`` then there is no limit to the wait time.

    If the future is cancelled before completing then ``CancelledError``
    will be raised.

    If the call completed without raising then ``None`` is returned.

``add_done_callback(fn)``

    Attaches a callable *fn* to the future that will be called when
    the future is cancelled or finishes running.  *fn* will be called
    with the future as its only argument.

    Added callables are called in the order that they were added


id: pep-3149.rst
content:
PEP: 3149
Title: ABI version tagged .so files
Author: Barry Warsaw <barry@python.org>
Status: Final
Type: Standards Track
Created: 09-Jul-2010
Python-Version: 3.2
Post-History: 14-Jul-2010, 22-Jul-2010
Resolution: https://mail.python.org/pipermail/python-dev/2010-September/103408.html


Abstract
========

:pep:`3147` described an extension to Python's import machinery that
improved the sharing of Python source code, by allowing more than one
byte compilation file (.pyc) to be co-located with each source file.

This PEP defines an adjunct feature which allows the co-location of
extension module files (.so) in a similar manner.  This optional,
build-time feature will enable downstream distributions of Python to
more easily provide more than one Python major version at a time.


Background
==========

:pep:`3147` defined the file system layout for a pure-Python package,
where multiple versions of Python are available on the system.  For
example, where the ``alpha`` package containing source modules ``one.py``
and ``two.py`` exist on a system with Python 3.2 and 3.3, the post-byte
compilation file system layout would be::

    alpha/
        __pycache__/
            __init__.cpython-32.pyc
            __init__.cpython-33.pyc
            one.cpython-32.pyc
            one.cpython-33.pyc
            two.cpython-32.pyc
            two.cpython-33.pyc
        __init__.py
        one.py
        two.py

For packages with extension modules, a similar differentiation is
needed for the module's .so files.  Extension modules compiled for
different Python major versions are incompatible with each other due
to changes in the ABI.  Different configuration/compilation options
for the same Python version can result in different ABIs
(e.g. --with-wide-unicode).

While :pep:`384` defines a stable ABI, it will minimize, but not
eliminate extension module incompatibilities between Python builds or
major versions.  Thus a mechanism for discriminating extension module
file names is proposed.


Rationale
=========

Linux distributions such as Ubuntu [3]_ and Debian [4]_ provide more
than one Python version at the same time to their users.  For example,
Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1,
with Python 2.6 being the default.

In order to share as much as possible between the available Python
versions, these distributions install third party package modules
(``.pyc`` and ``.so`` files) into ``/usr/share/pyshared`` and symlink to
them from ``/usr/lib/pythonX.Y/dist-packages``.  The symlinks exist
because in a pre-:pep:`3147` world (i.e < Python 3.2), the ``.pyc`` files
resulting from byte compilation by the various installed Pythons will
name collide with each other.  For Python versions >= 3.2, all
pure-Python packages can be shared, because the ``.pyc`` files will no
longer cause file system naming conflicts.  Eliminating these symlinks
makes for a simpler, more robust Python distribution.

A similar situation arises with shared library extensions.  Because
extension modules are typically named ``foo.so`` for a ``foo`` extension
module, these would also name collide if ``foo`` was provided for more
than one Python version.

In addition, because different configuration/compilation options for
the same Python version can cause different ABIs to be presented to
extension modules.  On POSIX systems for example, the configure
options ``--with-pydebug``, ``--with-pymalloc``, and
``--with-wide-unicode`` all change the ABI.  This PEP proposes to
encode build-time options in the file name of the ``.so`` extension
module files.

PyPy [5]_ can also benefit from this PEP, allowing it to avoid name
collisions in extension modules built for its API, but with a
different ``.so`` tag.


Proposal
========

The configure/compilation options chosen at Python interpreter
build-time will be encoded in the shared library file name for
extension modules.  This "tag" will appear between the module base
name and the operation file system extension for shared libraries.

The following information *MUST* be included in the shared library
file name:

* The Python implementation (e.g. cpython, pypy, jython, etc.)
* The interpreter's major and minor version numbers

These two fields are separated by a hyphen and no dots are to appear
between the major and minor version numbers.  E.g. ``cpython-32``.

Python implementations *MAY* include additional flags in the file name
tag as appropriate.  For example, on POSIX systems these flags will
also contribute to the file name:

* ``--with-pydebug`` (flag: ``d``)
* ``--with-pymalloc`` (flag: ``m``)
* ``--with-wide-unicode`` (flag: ``u``)

By default in Python 3.2, ``configure`` enables ``--with-pymalloc`` so
shared library file names would appear as ``foo.cpython-32m.so``.
When the other two flags are also enabled, the file names would be
``foo.cpython-32dmu.so``.

The shared library file name tag is used unconditionally; it cannot be
changed.  The tag and extension module suffix are available through
the ``sysconfig`` modules via the following variables::

    >>> sysconfig.get_config_var('EXT_SUFFIX')
    '.cpython-32mu.so'
    >>> sysconfig.get_config_var('SOABI')
    'cpython-32mu'

Note that ``$SOABI`` contains just the tag, while ``$EXT_SUFFIX`` includes the
platform extension for shared library files, and is the exact suffix
added to the extension module name.

For an arbitrary package ``foo``, you might see these files when the
distribution package was installed::

    /usr/lib/python/foo.cpython-32m.so
    /usr/lib/python/foo.cpython-33m.so

(These paths are for example purposes only.  Distributions are free to
use whatever filesystem layout they choose, and nothing in this PEP
changes the locations where from-source builds of Python are
installed.)

Python's dynamic module loader will recognize and import shared
library extension modules with a tag that matches its build-time
options.  For backward compatibility, Python will also continue to
import untagged extension modules, e.g. ``foo.so``.

This shared library tag would be used globally for all distutils-based
extension modules, regardless of where on the file system they are
built.  Extension modules built by means other than distutils would
either have to calculate the tag manually, or fallback to the
non-tagged ``.so`` file name.


Proven approach
===============

The approach described here is already proven, in a sense, on Debian
and Ubuntu system where different extensions are used for debug builds
of Python and extension modules.  Debug builds on Windows also already
use a different file extension for dynamic libraries, and in fact
encoded (in a different way than proposed in this PEP) the Python
major and minor version in the ``.dll`` file name.


Windows
=======

This PEP only addresses build issues on POSIX systems that use the
``configure`` script.  While Windows or other platform support is not
explicitly disallowed under this PEP, platform expertise is needed in
order to evaluate, describe, and implement support on such platforms.
It is not currently clear that the facilities in this PEP are even
useful for Windows.


PEP 384
=======

:pep:`384` defines a stable ABI for extension modules.  In theory,
universal adoption of :pep:`384` would eliminate the need for this PEP
because all extension modules could be compatible with any Python
version.  In practice of course, it will be impossible to achieve
universal adoption, and as described above, different build-time flags
still affect the ABI.  Thus even with a stable ABI, this PEP may still
be necessary.  While a complete specification is reserved for :pep:`384`,
here is a discussion of the relevant issues.

:pep:`384` describes a change to ``PyModule_Create()`` where ``3`` is
passed as the API version if the extension was compiled with
``Py_LIMITED_API``.  This should be formalized into an official macro
called ``PYTHON_ABI_VERSION`` to mirror ``PYTHON_API_VERSION``.  If
and when the ABI changes in an incomp


id: pep-3001.rst
content:
PEP: 3001
Title: Procedure for reviewing and improving standard library modules
Author: Georg Brandl <georg@python.org>
Status: Withdrawn
Type: Process
Created: 05-Apr-2006
Post-History:


Abstract
========

This PEP describes a procedure for reviewing and improving standard
library modules, especially those written in Python, making them ready
for Python 3000.  There can be different steps of refurbishing, each
of which is described in a section below.  Of course, not every step
has to be performed for every module.


Removal of obsolete modules
===========================

All modules marked as deprecated in 2.x versions should be removed for
Python 3000.  The same applies to modules which are seen as obsolete today,
but are too widely used to be deprecated or removed.  Python 3000 is the
big occasion to get rid of them.

There will have to be a document listing all removed modules, together
with information on possible substitutes or alternatives.  This
information will also have to be provided by the python3warn.py porting
helper script mentioned in PEP XXX.


Renaming modules
================

There are proposals for a "great stdlib renaming" introducing a hierarchic
library namespace or a top-level package from which to import standard
modules.  That possibility aside, some modules' names are known to have
been chosen unwisely, a mistake which could never be corrected in the 2.x
series.  Examples are names like "StringIO" or "Cookie".  For Python 3000,
there will be the possibility to give those modules less confusing and
more conforming names.

Of course, each rename will have to be stated in the documentation of
the respective module and perhaps in the global document of Step 1.
Additionally, the python3warn.py script will recognize the old module
names and notify the user accordingly.

If the name change is made in time for another release of the Python 2.x
series, it is worth considering to introduce the new name in the 2.x
branch to ease transition.


Code cleanup
============

As most library modules written in Python have not been touched except
for bug fixes, following the policy of never changing a running system,
many of them may contain code that is not up to the newest language
features and could be rewritten in a more concise, modern Python.

PyChecker should run cleanly over the library.  With a carefully tuned
configuration file, PyLint should also emit as few warnings as possible.

As long as these changes don't change the module's interface and behavior,
no documentation updates are necessary.


Enhancement of test and documentation coverage
==============================================

Code coverage by unit tests varies greatly between modules.  Each test
suite should be checked for completeness, and the remaining classic tests
should be converted to PyUnit (or whatever new shiny testing framework
comes with Python 3000, perhaps py.test?).

It should also be verified that each publicly visible function has a
meaningful docstring which ideally contains several doctests.

No documentation changes are necessary for enhancing test coverage.


Unification of module metadata
==============================

This is a small and probably not very important step.  There have been
various attempts at providing author, version and similar metadata in
modules (such as a "__version__" global).  Those could be standardized
and used throughout the library.

No documentation changes are necessary for this step, too.


Backwards incompatible bug fixes
================================

Over the years, many bug reports have been filed which complained about
bugs in standard library modules, but have subsequently been closed as
"Won't fix" since a fix would have introduced a major incompatibility
which was not acceptable in the Python 2.x series.  In Python 3000, the
fix can be applied if the interface per se is still acceptable.

Each slight behavioral change caused by such fixes must be mentioned in
the documentation, perhaps in a "Changed in Version 3.0" paragraph.


Interface changes
=================

The last and most disruptive change is the overhaul of a module's public
interface.  If a module's interface is to be changed, a justification
should be made beforehand, or a PEP should be written.

The change must be fully documented as "New in Version 3.0", and the
python3warn.py script must know about it.


References
==========

None yet.


Copyright
=========

This document has been placed in the public domain.


id: pep-3003.rst
content:
PEP: 3003
Title: Python Language Moratorium
Author: Brett Cannon, Jesse Noller, Guido van Rossum
Status: Final
Type: Process
Created: 21-Oct-2009
Post-History: 03-Nov-2009


Abstract
========

This PEP proposes a temporary moratorium (suspension) of all changes
to the Python language syntax, semantics, and built-ins for a period
of at least two years from the release of Python 3.1.  In particular, the
moratorium would include Python 3.2 (to be released 18-24 months after
3.1) but allow Python 3.3 (assuming it is not released prematurely) to
once again include language changes.

This suspension of features is designed to allow non-CPython implementations
to "catch up" to the core implementation of the language, help ease adoption
of Python 3.x, and provide a more stable base for the community.


Rationale
=========

This idea was proposed by Guido van Rossum on the python-ideas [1]_ mailing
list. The premise of his email was to slow the alteration of the  Python core
syntax, builtins and semantics to allow non-CPython implementations to catch
up to the current state of Python, both 2.x and 3.x.

Python, as a language is more than the core implementation --
CPython -- with a rich, mature and vibrant community of implementations, such
as Jython [2]_, IronPython [3]_ and PyPy [4]_ that are a benefit not only to
the community, but to the language itself.

Still others, such as Unladen Swallow [5]_ (a branch of CPython) seek not to
create an alternative implementation, but rather they seek to enhance the
performance and implementation of CPython itself.

Python 3.x was a large part of the last several years of Python's
development. Its release, as well as a bevy of changes to the language
introduced by it and the previous 2.6.x releases, puts alternative
implementations at a severe disadvantage in "keeping pace" with core python
development.

Additionally, many of the changes put into the recent releases of the language
as implemented by CPython have not yet seen widespread usage by the
general user population. For example, most users are limited to the version
of the interpreter (typically CPython) which comes pre-installed with their
operating system. Most OS vendors are just barely beginning to ship Python 2.6
-- even fewer are shipping Python 3.x.

As it is expected that Python 2.7 be the effective "end of life" of the Python
2.x code line, with Python 3.x being the future, it is in the best interest of
Python core development to temporarily suspend the alteration of the language
itself to allow all of these external entities to catch up and to assist in
the adoption of, and migration to, Python 3.x

Finally, the moratorium is intended to free up cycles within core development
to focus on other issues, such as the CPython interpreter and improvements
therein, the standard library, etc.

This moratorium does not allow for exceptions -- once accepted, any pending
changes to the syntax or semantics of the language will be postponed until the
moratorium is lifted.

This moratorium does not attempt to apply to any other Python implementation
meaning that if desired other implementations may add features which deviate
from the standard implementation.


Details
=======

Cannot Change
-------------

* New built-ins
* Language syntax
    The grammar file essentially becomes immutable apart from ambiguity
    fixes.
* General language semantics
    The language operates as-is with only specific exemptions (see
    below).
* New __future__ imports
    These are explicitly forbidden, as they effectively change the language
    syntax and/or semantics (albeit using a compiler directive).


Case-by-Case Exemptions
-----------------------

* New methods on built-ins
    The case for adding a method to a built-in object can be made.
* Incorrect language semantics
    If the language semantics turn out to be ambiguous or improperly
    implemented based on the intention of the original design then the
    semantics may change.
* Language semantics that are difficult to implement
    Because other VMs have not begun implementing Python 3.x semantics
    there is a possibility that certain semantics are too difficult to
    replicate. In those cases they can be changed to ease adoption of
    Python 3.x by the other VMs.


Allowed to Change
-----------------

* C API
    It is entirely acceptable to change the underlying C code of
    CPython as long as other restrictions of this moratorium are not
    broken. E.g. removing the GIL would be fine assuming certain
    operations that are currently atomic remain atomic.
* The standard library
    As the standard library is not directly tied to the language
    definition it is not covered by this moratorium.
* Backports of 3.x features to 2.x
    The moratorium only affects features that would be new in 3.x.
* Import semantics
    For example, :pep:`382`.  After all, import semantics vary between
    Python implementations anyway.


Retroactive
===========

It is important to note that the moratorium covers all changes since the release
of Python 3.1. This rule is intended to avoid features being rushed or smuggled
into the CPython source tree while the moratorium is being discussed.  A review
of the NEWS file for the py3k development branch showed no commits would need to
be rolled back in order to meet this goal.


Extensions
==========

The time period of the moratorium can only be extended through a new PEP.


Copyright
=========

This document has been placed in the public domain.

References
==========

.. [1] https://mail.python.org/pipermail/python-ideas/2009-October/006305.html
.. [2] http://www.jython.org/
.. [3] http://www.codeplex.com/IronPython
.. [4] http://codespeak.net/pypy/
.. [5] http://code.google.com/p/unladen-swallow/


id: pep-3002.rst
content:
PEP: 3002
Title: Procedure for Backwards-Incompatible Changes
Author: Steven Bethard <steven.bethard@gmail.com>
Status: Final
Type: Process
Created: 27-Mar-2006
Post-History: 27-Mar-2006, 13-Apr-2006


Abstract
========

This PEP describes the procedure for changes to Python that are
backwards-incompatible between the Python 2.X series and Python 3000.
All such changes must be documented by an appropriate Python 3000 PEP
and must be accompanied by code that can identify when pieces of
Python 2.X code may be problematic in Python 3000.


Rationale
=========

Python 3000 will introduce a number of backwards-incompatible changes
to Python, mainly to streamline the language and to remove some
previous design mistakes.  But Python 3000 is not intended to be a new
and completely different language from the Python 2.X series, and it
is expected that much of the Python user community will make the
transition to Python 3000 when it becomes available.

To encourage this transition, it is crucial to provide a clear and
complete guide on how to upgrade Python 2.X code to Python 3000 code.
Thus, for any backwards-incompatible change, two things are required:

* An official Python Enhancement Proposal (PEP)
* Code that can identify pieces of Python 2.X code that may be
  problematic in Python 3000


Python Enhancement Proposals
=============================

Every backwards-incompatible change must be accompanied by a PEP.
This PEP should follow the usual PEP guidelines and explain the
purpose and reasoning behind the backwards incompatible change.  In
addition to the usual PEP sections, all PEPs proposing
backwards-incompatible changes must include an additional section:
Compatibility Issues.  This section should describe what is backwards
incompatible about the proposed change to Python, and the major sorts
of breakage to be expected.

While PEPs must still be evaluated on a case-by-case basis, a PEP may
be inappropriate for Python 3000 if its Compatibility Issues section
implies any of the following:

* Most or all instances of a Python 2.X construct are incorrect in
  Python 3000, and most or all instances of the Python 3000 construct
  are incorrect in Python 2.X.

  So for example, changing the meaning of the for-loop else-clause
  from "executed when the loop was not broken out of" to "executed
  when the loop had zero iterations" would mean that all Python 2.X
  for-loop else-clauses would be broken, and there would be no way to
  use a for-loop else-clause in a Python-3000-appropriate manner.
  Thus a PEP for such an idea would likely be rejected.

* Many instances of a Python 2.X construct are incorrect in Python
  3000 and the PEP fails to demonstrate real-world use-cases for the
  changes.

  Backwards incompatible changes are allowed in Python 3000, but not
  to excess.  A PEP that proposes backwards-incompatible changes
  should provide good examples of code that visibly benefits from the
  changes.

PEP-writing is time-consuming, so when a number of
backwards-incompatible changes are closely related, they should be
proposed in the same PEP.  Such PEPs will likely have longer
Compatibility Issues sections, however, since they must now describe
the sorts of breakage expected from *all* the proposed changes.


Identifying Problematic Code
============================

In addition to the PEP requirement, backwards incompatible changes to
Python must also be accompanied by code to issue warnings for pieces
of Python 2.X code that will behave differently in Python 3000. Such
warnings will be enabled in Python 2.X using a new command-line
switch: -3. All backwards incompatible changes should be
accompanied by a patch for Python 2.X that, when -3 is
specified, issues warnings for each construct that is being changed.

For example, if ``dict.keys()`` returns an iterator in Python 3000,
the patch to the Python 2.X branch should do something like:

    If -3 was specified, change ``dict.keys()`` to return a
    subclass of ``list`` that issues warnings whenever you use any
    methods other than ``__iter__()``.

Such a patch would mean that warnings are only issued when features
that will not be present in Python 3000 are used, and almost all
existing code should continue to work. (Code that relies on
``dict.keys()`` always returning a ``list`` and not a subclass should
be pretty much non-existent.)


References
==========

TBD


Copyright
=========

This document has been placed in the public domain.


id: pep-3117.rst
content:
PEP: 3117
Title: Postfix type declarations
Author: Georg Brandl <georg@python.org>
Status: Rejected
Type: Standards Track
Created: 01-Apr-2007
Python-Version: 3.0
Post-History:


Abstract
========

This PEP proposes the addition of a postfix type declaration syntax to
Python. It also specifies a new ``typedef`` statement which is used to create
new mappings between types and declarators.

Its acceptance will greatly enhance the Python user experience as well as
eliminate one of the warts that deter users of other programming languages from
switching to Python.


Rationale
=========

Python has long suffered from the lack of explicit type declarations.  Being one
of the few aspects in which the language deviates from its Zen, this wart has
sparked many a discussion between Python heretics and members of the PSU (for
a few examples, see [EX1]_, [EX2]_ or [EX3]_), and it also made it a large-scale
enterprise success unlikely.

However, if one wants to put an end to this misery, a decent Pythonic syntax
must be found. In almost all languages that have them, type declarations lack
this quality: they are verbose, often needing *multiple words* for a single
type, or they are hard to comprehend (e.g., a certain language uses completely
unrelated [#]_ adjectives like ``dim`` for type declaration).

Therefore, this PEP combines the move to type declarations with another bold
move that will once again prove that Python is not only future-proof but
future-embracing: the introduction of Unicode characters as an integral
constituent of source code.

Unicode makes it possible to express much more with much less characters, which
is in accordance with the :pep:`Zen <20>` ("Readability counts."). Additionally, it
eliminates the need for a separate type declaration statement, and last but not
least, it makes Python measure up to Perl 6, which already uses Unicode for its
operators. [#]_


Specification
=============

When the type declaration mode is in operation, the grammar is changed so that
each ``NAME`` must consist of two parts: a name and a type declarator, which is
exactly one Unicode character.

The declarator uniquely specifies the type of the name, and if it occurs on the
left hand side of an expression, this type is enforced: an ``InquisitionError``
exception is raised if the returned type doesn't match the declared type. [#]_

Also, function call result types have to be specified. If the result of the call
does not have the declared type, an ``InquisitionError`` is raised.  Caution: the
declarator for the result should not be confused with the declarator for the
function object (see the example below).

Type declarators after names that are only read, not assigned to, are not strictly
necessary but enforced anyway (see the Python Zen: "Explicit is better than
implicit.").

The mapping between types and declarators is not static. It can be completely
customized by the programmer, but for convenience there are some predefined
mappings for some built-in types:

=========================  ===================================================
Type                       Declarator
=========================  ===================================================
``object``                 ï¿½ (REPLACEMENT CHARACTER)
``int``                    â (DOUBLE-STRUCK CAPITAL N)
``float``                  â® (ESTIMATED SYMBOL)
``bool``                   â (CHECK MARK)
``complex``                â (DOUBLE-STRUCK CAPITAL C)
``str``                    â (LOWER RIGHT PENCIL)
``unicode``                â (BLACK NIB)
``tuple``                  â¯ (PARENTHESIZED LATIN SMALL LETTER T)
``list``                   â¨ (HOT SPRINGS)
``dict``                   â§ (DOUBLE-ENDED MULTIMAP)
``set``                    â (EMPTY SET) (*Note:* this is also for full sets)
``frozenset``              â (SNOWMAN)
``datetime``               â (WATCH)
``function``               Æ (LATIN SMALL LETTER LAMBDA WITH STROKE)
``generator``              â (ATOM SYMBOL)
``Exception``              â (ELECTRIC ARROW)
=========================  ===================================================

The declarator for the ``None`` type is a zero-width space.

These characters should be obvious and easy to remember and type for every
programmer.


Unicode replacement units
=========================

Since even in our modern, globalized world there are still some old-fashioned
rebels who can't or don't want to use Unicode in their source code, and since
Python is a forgiving language, a fallback is provided for those:

Instead of the single Unicode character, they can type ``name${UNICODE NAME OF
THE DECLARATOR}$``. For example, these two function definitions are equivalent::

    def fooÆ(xâ):
        return None

and ::

    def foo${LATIN SMALL LETTER LAMBDA WITH STROKE}$(x${DOUBLE-STRUCK CAPITAL C}$):
        return None${ZERO WIDTH NO-BREAK SPACE}$

This is still easy to read and makes the full power of type-annotated Python
available to ASCII believers.


The ``typedef`` statement
=========================

The mapping between types and declarators can be extended with this new statement.

The syntax is as follows::

    typedef_stmt  ::=  "typedef" expr DECLARATOR

where ``expr`` resolves to a type object. For convenience, the ``typedef`` statement
can also be mixed with the ``class`` statement for new classes, like so::

    typedef class Fooâº(objectï¿½):
        pass


Example
=======

This is the standard ``os.path.normpath`` function, converted to type declaration
syntax::

    def normpathÆ(pathâ)â:
        """Normalize path, eliminating double slashes, etc."""
        if pathâ == '':
            return '.'
        initial_slashesâ = pathâ.startswithÆ('/')â
        # POSIX allows one or two initial slashes, but treats three or more
        # as single slash.
        if (initial_slashesâ and
            pathâ.startswithÆ('//')â and not pathâ.startswithÆ('///')â)â:
            initial_slashesâ = 2
        compsâ¨ = pathâ.splitÆ('/')â¨
        new_compsâ¨ = []â¨
        for compâ in compsâ¨:
            if compâ in ('', '.')â¯:
                continue
            if (compâ != '..' or (not initial_slashesâ and not new_compsâ¨)â or
                 (new_compsâ¨ and new_compsâ¨[-1]â == '..')â)â:
                new_compsâ¨.appendÆ(compâ)
            elif new_compsâ¨:
                new_compsâ¨.popÆ()â
        compsâ¨ = new_compsâ¨
        pathâ = '/'.join(compsâ¨)â
        if initial_slashesâ:
            pathâ = '/'*initial_slashesâ + pathâ
        return pathâ or '.'

As you can clearly see, the type declarations add expressiveness, while at the
same time they make the code look much more professional.


Compatibility issues
====================

To enable type declaration mode, one has to write::

    from __future__ import type_declarations

which enables Unicode parsing of the source [#]_, makes ``typedef`` a keyword
and enforces correct types for all assignments and function calls.


Rejection
=========

After careful considering, much soul-searching, gnashing of teeth and rending
of garments, it has been decided to reject this PEP.


References
==========


.. [EX1] https://mail.python.org/pipermail/python-list/2003-June/210588.html

.. [EX2] https://mail.python.org/pipermail/python-list/2000-May/034685.html

.. [EX3] http://groups.google.com/group/comp.lang.python/browse_frm/thread/6ae8c6add913635a/de40d4ffe9bd4304?lnk=gst&q=type+declarations&rnum=6

.. [#] Though, if you know the language in question, it may not be *that* unrelated.

.. [#] Well, it would, if there was a Perl 6.

.. [#] Since the name ``TypeError`` is already in use, this name has been chosen
   for obvious reasons.

.. [#] The encoding in which the code is written is read from a standard coding
   cookie. There will also be an autodetection mechanism, invoked by ``from
   __future__ import encoding_hell``.


Acknowledgeme


id: pep-3103.rst
content:
PEP: 3103
Title: A Switch/Case Statement
Author: Guido van Rossum <guido@python.org>
Status: Rejected
Type: Standards Track
Created: 25-Jun-2006
Python-Version: 3.0
Post-History: 26-Jun-2006


Rejection Notice
================

A quick poll during my keynote presentation at PyCon 2007 shows this
proposal has no popular support.  I therefore reject it.


Abstract
========

Python-dev has recently seen a flurry of discussion on adding a switch
statement.  In this PEP I'm trying to extract my own preferences from
the smorgasbord of proposals, discussing alternatives and explaining
my choices where I can.  I'll also indicate how strongly I feel about
alternatives I discuss.

This PEP should be seen as an alternative to :pep:`275`.  My views are
somewhat different from that PEP's author, but I'm grateful for the
work done in that PEP.

This PEP introduces canonical names for the many variants that have
been discussed for different aspects of the syntax and semantics, such
as "alternative 1", "school II", "option 3" and so on.  Hopefully
these names will help the discussion.


Rationale
=========

A common programming idiom is to consider an expression and do
different things depending on its value.  This is usually done with a
chain of if/elif tests; I'll refer to this form as the "if/elif
chain".  There are two main motivations to want to introduce new
syntax for this idiom:

- It is repetitive: the variable and the test operator, usually '=='
  or 'in', are repeated in each if/elif branch.

- It is inefficient: when an expression matches the last test value
  (or no test value at all) it is compared to each of the preceding
  test values.

Both of these complaints are relatively mild; there isn't a lot of
readability or performance to be gained by writing this differently.
Yet, some kind of switch statement is found in many languages and it
is not unreasonable to expect that its addition to Python will allow
us to write up certain code more cleanly and efficiently than before.

There are forms of dispatch that are not suitable for the proposed
switch statement; for example, when the number of cases is not
statically known, or when it is desirable to place the code for
different cases in different classes or files.


Basic Syntax
============

I'm considering several variants of the syntax first proposed in PEP
275 here.  There are lots of other possibilities, but I don't see that
they add anything.

I've recently been converted to alternative 1.

I should note that all alternatives here have the "implicit break"
property: at the end of the suite for a particular case, the control
flow jumps to the end of the whole switch statement.  There is no way
to pass control from one case to another.  This in contrast to C,
where an explicit 'break' statement is required to prevent falling
through to the next case.

In all alternatives, the else-suite is optional.  It is more Pythonic
to use 'else' here rather than introducing a new reserved word,
'default', as in C.

Semantics are discussed in the next top-level section.

Alternative 1
-------------

This is the preferred form in :pep:`275`::

    switch EXPR:
        case EXPR:
            SUITE
        case EXPR:
            SUITE
        ...
        else:
            SUITE

The main downside is that the suites where all the action is are
indented two levels deep; this can be remedied by indenting the cases
"half a level" (e.g. 2 spaces if the general indentation level is 4).

Alternative 2
-------------

This is Fredrik Lundh's preferred form; it differs by not indenting
the cases::

    switch EXPR:
    case EXPR:
        SUITE
    case EXPR:
        SUITE
    ....
    else:
        SUITE

Some reasons not to choose this include expected difficulties for
auto-indenting editors, folding editors, and the like; and confused
users.  There are no situations currently in Python where a line
ending in a colon is followed by an unindented line.

Alternative 3
-------------

This is the same as alternative 2 but leaves out the colon after the
switch::

    switch EXPR
    case EXPR:
        SUITE
    case EXPR:
        SUITE
    ....
    else:
        SUITE

The hope of this alternative is that it will not upset the auto-indent
logic of the average Python-aware text editor less.  But it looks
strange to me.

Alternative 4
-------------

This leaves out the 'case' keyword on the basis that it is redundant::

    switch EXPR:
        EXPR:
            SUITE
        EXPR:
            SUITE
        ...
        else:
            SUITE

Unfortunately now we are forced to indent the case expressions,
because otherwise (at least in the absence of an 'else' keyword) the
parser would have a hard time distinguishing between an unindented
case expression (which continues the switch statement) or an unrelated
statement that starts like an expression (such as an assignment or a
procedure call).  The parser is not smart enough to backtrack once it
sees the colon.  This is my least favorite alternative.


Extended Syntax
===============

There is one additional concern that needs to be addressed
syntactically.  Often two or more values need to be treated the same.
In C, this done by writing multiple case labels together without any
code between them.  The "fall through" semantics then mean that these
are all handled by the same code.  Since the Python switch will not
have fall-through semantics (which have yet to find a champion) we
need another solution.  Here are some alternatives.

Alternative A
-------------

Use::

    case EXPR:

to match on a single expression; use::

    case EXPR, EXPR, ...:

to match on multiple expressions.  The is interpreted so that if EXPR
is a parenthesized tuple or another expression whose value is a tuple,
the switch expression must equal that tuple, not one of its elements.
This means that we cannot use a variable to indicate multiple cases.
While this is also true in C's switch statement, it is a relatively
common occurrence in Python (see for example sre_compile.py).

Alternative B
-------------

Use::

    case EXPR:

to match on a single expression; use::

    case in EXPR_LIST:

to match on multiple expressions.  If EXPR_LIST is a single
expression, the 'in' forces its interpretation as an iterable (or
something supporting __contains__, in a minority semantics
alternative).  If it is multiple expressions, each of those is
considered for a match.

Alternative C
-------------

Use::

    case EXPR:

to match on a single expression; use::

    case EXPR, EXPR, ...:

to match on multiple expressions (as in alternative A); and use::

    case *EXPR:

to match on the elements of an expression whose value is an iterable.
The latter two cases can be combined, so that the true syntax is more
like this::

    case [*]EXPR, [*]EXPR, ...:

The ``*`` notation is similar to the use of prefix ``*`` already in use for
variable-length parameter lists and for passing computed argument
lists, and often proposed for value-unpacking (e.g.  ``a, b, *c = X`` as
an alternative to ``(a, b), c = X[:2], X[2:]``).

Alternative D
-------------

This is a mixture of alternatives B and C; the syntax is like
alternative B but instead of the 'in' keyword it uses '*'.  This is
more limited, but still allows the same flexibility.  It uses::

    case EXPR:

to match on a single expression and::

    case *EXPR:

to match on the elements of an iterable.  If one wants to specify
multiple matches in one case, one can write this::

    case *(EXPR, EXPR, ...):

or perhaps this (although it's a bit strange because the relative
priority of '*' and ',' is different than elsewhere)::

    case * EXPR, EXPR, ...:

Discussion
----------

Alternatives B, C and D are motivated by the desire to specify
multiple cases with the same treatment using a variable representing a
set (usually a tuple) rather than spelling them out.  The motivation
for this is usually that if one has several switches over the same set
of cases it's a shame to have 


id: pep-3102.rst
content:
PEP: 3102
Title: Keyword-Only Arguments
Author: Talin <viridia@gmail.com>
Status: Final
Type: Standards Track
Created: 22-Apr-2006
Python-Version: 3.0
Post-History: 28-Apr-2006, 19-May-2006


Abstract
========

This PEP proposes a change to the way that function arguments are
assigned to named parameter slots.  In particular, it enables the
declaration of "keyword-only" arguments: arguments that can only
be supplied by keyword and which will never be automatically
filled in by a positional argument.


Rationale
=========

The current Python function-calling paradigm allows arguments to
be specified either by position or by keyword.  An argument can be
filled in either explicitly by name, or implicitly by position.

There are often cases where it is desirable for a function to take
a variable number of arguments.  The Python language supports this
using the 'varargs' syntax (``*name``), which specifies that any
'left over' arguments be passed into the varargs parameter as a
tuple.

One limitation on this is that currently, all of the regular
argument slots must be filled before the vararg slot can be.

This is not always desirable.  One can easily envision a function
which takes a variable number of arguments, but also takes one
or more 'options' in the form of keyword arguments.  Currently,
the only way to do this is to define both a varargs argument,
and a 'keywords' argument (``**kwargs``), and then manually extract
the desired keywords from the dictionary.


Specification
=============

Syntactically, the proposed changes are fairly simple.  The first
change is to allow regular arguments to appear after a varargs
argument::

    def sortwords(*wordlist, case_sensitive=False):
        ...

This function accepts any number of positional arguments, and it
also accepts a keyword option called 'case_sensitive'.  This
option will never be filled in by a positional argument, but
must be explicitly specified by name.

Keyword-only arguments are not required to have a default value.
Since Python requires that all arguments be bound to a value,
and since the only way to bind a value to a keyword-only argument
is via keyword, such arguments are therefore 'required keyword'
arguments.  Such arguments must be supplied by the caller, and
they must be supplied via keyword.

The second syntactical change is to allow the argument name to
be omitted for a varargs argument. The meaning of this is to
allow for keyword-only arguments for functions that would not
otherwise take a varargs argument::

    def compare(a, b, *, key=None):
        ...

The reasoning behind this change is as follows.  Imagine for a
moment a function which takes several positional arguments, as
well as a keyword argument::

    def compare(a, b, key=None):
        ...

Now, suppose you wanted to have 'key' be a keyword-only argument.
Under the above syntax, you could accomplish this by adding a
varargs argument immediately before the keyword argument::

    def compare(a, b, *ignore, key=None):
        ...

Unfortunately, the 'ignore' argument will also suck up any
erroneous positional arguments that may have been supplied by the
caller.  Given that we'd prefer any unwanted arguments to raise an
error, we could do this::

    def compare(a, b, *ignore, key=None):
        if ignore:  # If ignore is not empty
            raise TypeError

As a convenient shortcut, we can simply omit the 'ignore' name,
meaning 'don't allow any positional arguments beyond this point'.

(Note: After much discussion of alternative syntax proposals, the
BDFL has pronounced in favor of this 'single star' syntax for
indicating the end of positional parameters.)


Function Calling Behavior
=========================

The previous section describes the difference between the old
behavior and the new.  However, it is also useful to have a
description of the new behavior that stands by itself, without
reference to the previous model.  So this next section will
attempt to provide such a description.

When a function is called, the input arguments are assigned to
formal parameters as follows:

- For each formal parameter, there is a slot which will be used
  to contain the value of the argument assigned to that
  parameter.

- Slots which have had values assigned to them are marked as
  'filled'.  Slots which have no value assigned to them yet are
  considered 'empty'.

- Initially, all slots are marked as empty.

- Positional arguments are assigned first, followed by keyword
  arguments.

- For each positional argument:

  * Attempt to bind the argument to the first unfilled
    parameter slot.  If the slot is not a vararg slot, then
    mark the slot as 'filled'.

  * If the next unfilled slot is a vararg slot, and it does
    not have a name, then it is an error.

  * Otherwise, if the next unfilled slot is a vararg slot then
    all remaining non-keyword arguments are placed into the
    vararg slot.

- For each keyword argument:

  * If there is a parameter with the same name as the keyword,
    then the argument value is assigned to that parameter slot.
    However, if the parameter slot is already filled, then that
    is an error.

  * Otherwise, if there is a 'keyword dictionary' argument,
    the argument is added to the dictionary using the keyword
    name as the dictionary key, unless there is already an
    entry with that key, in which case it is an error.

  * Otherwise, if there is no keyword dictionary, and no
    matching named parameter, then it is an error.

- Finally:

  * If the vararg slot is not yet filled, assign an empty tuple
    as its value.

  * For each remaining empty slot: if there is a default value
    for that slot, then fill the slot with the default value.
    If there is no default value, then it is an error.

In accordance with the current Python implementation, any errors
encountered will be signaled by raising ``TypeError``.  (If you want
something different, that's a subject for a different PEP.)


Backwards Compatibility
=======================

The function calling behavior specified in this PEP is a superset
of the existing behavior - that is, it is expected that any
existing programs will continue to work.


Copyright
=========

This document has been placed in the public domain.


id: pep-3116.rst
content:
PEP: 3116
Title: New I/O
Author: Daniel Stutzbach <daniel@stutzbachenterprises.com>,
        Guido van Rossum <guido@python.org>,
        Mike Verdone <mike.verdone@gmail.com>
Status: Final
Type: Standards Track
Created: 26-Feb-2007
Python-Version: 3.0
Post-History: 26-Feb-2007

Rationale and Goals
===================

Python allows for a variety of stream-like (a.k.a. file-like) objects
that can be used via ``read()`` and ``write()`` calls.  Anything that
provides ``read()`` and ``write()`` is stream-like.  However, more
exotic and extremely useful functions like ``readline()`` or
``seek()`` may or may not be available on every stream-like object.
Python needs a specification for basic byte-based I/O streams to which
we can add buffering and text-handling features.

Once we have a defined raw byte-based I/O interface, we can add
buffering and text handling layers on top of any byte-based I/O class.
The same buffering and text handling logic can be used for files,
sockets, byte arrays, or custom I/O classes developed by Python
programmers.  Developing a standard definition of a stream lets us
separate stream-based operations like ``read()`` and ``write()`` from
implementation specific operations like ``fileno()`` and ``isatty()``.
It encourages programmers to write code that uses streams as streams
and not require that all streams support file-specific or
socket-specific operations.

The new I/O spec is intended to be similar to the Java I/O libraries,
but generally less confusing.  Programmers who don't want to muck
about in the new I/O world can expect that the ``open()`` factory
method will produce an object backwards-compatible with old-style file
objects.


Specification
=============

The Python I/O Library will consist of three layers: a raw I/O layer,
a buffered I/O layer, and a text I/O layer.  Each layer is defined by
an abstract base class, which may have multiple implementations.  The
raw I/O and buffered I/O layers deal with units of bytes, while the
text I/O layer deals with units of characters.


Raw I/O
=======

The abstract base class for raw I/O is RawIOBase.  It has several
methods which are wrappers around the appropriate operating system
calls.  If one of these functions would not make sense on the object,
the implementation must raise an IOError exception.  For example, if a
file is opened read-only, the ``.write()`` method will raise an
``IOError``.  As another example, if the object represents a socket,
then ``.seek()``, ``.tell()``, and ``.truncate()`` will raise an
``IOError``.  Generally, a call to one of these functions maps to
exactly one operating system call.

    ``.read(n: int) -> bytes``

       Read up to ``n`` bytes from the object and return them.  Fewer
       than ``n`` bytes may be returned if the operating system call
       returns fewer than ``n`` bytes.  If 0 bytes are returned, this
       indicates end of file.  If the object is in non-blocking mode
       and no bytes are available, the call returns ``None``.

    ``.readinto(b: bytes) -> int``

       Read up to ``len(b)`` bytes from the object and stores them in
       ``b``, returning the number of bytes read.  Like .read, fewer
       than ``len(b)`` bytes may be read, and 0 indicates end of file.
       ``None`` is returned if a non-blocking object has no bytes
       available.  The length of ``b`` is never changed.

    ``.write(b: bytes) -> int``

        Returns number of bytes written, which may be ``< len(b)``.

    ``.seek(pos: int, whence: int = 0) -> int``

    ``.tell() -> int``

    ``.truncate(n: int = None) -> int``

    ``.close() -> None``

Additionally, it defines a few other methods:

    ``.readable() -> bool``

       Returns ``True`` if the object was opened for reading,
       ``False`` otherwise.  If ``False``, ``.read()`` will raise an
       ``IOError`` if called.

    ``.writable() -> bool``

       Returns ``True`` if the object was opened for writing,
       ``False`` otherwise.  If ``False``, ``.write()`` and
       ``.truncate()`` will raise an ``IOError`` if called.

    ``.seekable() -> bool``

       Returns ``True`` if the object supports random access (such as
       disk files), or ``False`` if the object only supports
       sequential access (such as sockets, pipes, and ttys).  If
       ``False``, ``.seek()``, ``.tell()``, and ``.truncate()`` will
       raise an IOError if called.

    ``.__enter__() -> ContextManager``

       Context management protocol.  Returns ``self``.

    ``.__exit__(...) -> None``

       Context management protocol.  Same as ``.close()``.

If and only if a ``RawIOBase`` implementation operates on an
underlying file descriptor, it must additionally provide a
``.fileno()`` member function.  This could be defined specifically by
the implementation, or a mix-in class could be used (need to decide
about this).

    ``.fileno() -> int``

       Returns the underlying file descriptor (an integer)

Initially, three implementations will be provided that implement the
``RawIOBase`` interface: ``FileIO``, ``SocketIO`` (in the socket
module), and ``ByteIO``.  Each implementation must determine whether
the object supports random access as the information provided by the
user may not be sufficient (consider ``open("/dev/tty", "rw")`` or
``open("/tmp/named-pipe", "rw")``).  As an example, ``FileIO`` can
determine this by calling the ``seek()`` system call; if it returns an
error, the object does not support random access.  Each implementation
may provided additional methods appropriate to its type.  The
``ByteIO`` object is analogous to Python 2's ``cStringIO`` library,
but operating on the new bytes type instead of strings.


Buffered I/O
============

The next layer is the Buffered I/O layer which provides more efficient
access to file-like objects.  The abstract base class for all Buffered
I/O implementations is ``BufferedIOBase``, which provides similar methods
to RawIOBase:

    ``.read(n: int = -1) -> bytes``

       Returns the next ``n`` bytes from the object.  It may return
       fewer than ``n`` bytes if end-of-file is reached or the object is
       non-blocking.  0 bytes indicates end-of-file.  This method may
       make multiple calls to ``RawIOBase.read()`` to gather the bytes,
       or may make no calls to ``RawIOBase.read()`` if all of the needed
       bytes are already buffered.

    ``.readinto(b: bytes) -> int``

    ``.write(b: bytes) -> int``

       Write ``b`` bytes to the buffer.  The bytes are not guaranteed to
       be written to the Raw I/O object immediately; they may be
       buffered.  Returns ``len(b)``.

    ``.seek(pos: int, whence: int = 0) -> int``

    ``.tell() -> int``

    ``.truncate(pos: int = None) -> int``

    ``.flush() -> None``

    ``.close() -> None``

    ``.readable() -> bool``

    ``.writable() -> bool``

    ``.seekable() -> bool``

    ``.__enter__() -> ContextManager``

    ``.__exit__(...) -> None``

Additionally, the abstract base class provides one member variable:

    ``.raw``

       A reference to the underlying ``RawIOBase`` object.

The ``BufferedIOBase`` methods signatures are mostly identical to that
of ``RawIOBase`` (exceptions: ``write()`` returns ``None``,
``read()``'s argument is optional), but may have different semantics.
In particular, ``BufferedIOBase`` implementations may read more data
than requested or delay writing data using buffers.  For the most
part, this will be transparent to the user (unless, for example, they
open the same file through a different descriptor).  Also, raw reads
may return a short read without any particular reason; buffered reads
will only return a short read if EOF is reached; and raw writes may
return a short count (even when non-blocking I/O is not enabled!),
while buffered writes will raise ``IOError`` when not all bytes could
be written or buffered.

There are four implementations of the ``BufferedIOBase`` abstract base
class, described below.


``BufferedReader``
------------------

T


id: pep-3128.rst
content:
PEP: 3128
Title: BList: A Faster List-like Type
Author: Daniel Stutzbach <daniel@stutzbachenterprises.com>
Discussions-To: python-3000@python.org
Status: Rejected
Type: Standards Track
Created: 30-Apr-2007
Python-Version: 2.6, 3.0
Post-History: 30-Apr-2007


Rejection Notice
================

Rejected based on Raymond Hettinger's sage advice [4]_:

    After looking at the source, I think this has almost zero chance
    for replacing list().  There is too much value in a simple C API,
    low space overhead for small lists, good performance is common use
    cases, and having performance that is easily understood.  The
    BList implementation lacks these virtues and it trades-off a little
    performance in common cases in for much better performance in
    uncommon cases.  As a Py3.0 PEP, I think it can be rejected.

    Depending on its success as a third-party module, it still has a
    chance for inclusion in the collections module.  The essential
    criteria for that is whether it is a superior choice for some
    real-world use cases.  I've scanned my own code and found no instances
    where BList would have been preferable to a regular list.  However,
    that scan has a selection bias because it doesn't reflect what I would
    have written had BList been available.  So, after a few months, I
    intend to poll comp.lang.python for BList success stories.  If they
    exist, then I have no problem with inclusion in the collections
    module.  After all, its learning curve is near zero -- the only cost
    is the clutter factor stemming from indecision about the most
    appropriate data structure for a given task.


Abstract
========

The common case for list operations is on small lists.  The current
array-based list implementation excels at small lists due to the
strong locality of reference and infrequency of memory allocation
operations.  However, an array takes O(n) time to insert and delete
elements, which can become problematic as the list gets large.

This PEP introduces a new data type, the BList, that has array-like
and tree-like aspects.  It enjoys the same good performance on small
lists as the existing array-based implementation, but offers superior
asymptotic performance for most operations.  This PEP proposes
replacing the makes two mutually exclusive proposals for including the
BList type in Python:

1. Add it to the collections module, or
2. Replace the existing list type


Motivation
==========

The BList grew out of the frustration of needing to rewrite intuitive
algorithms that worked fine for small inputs but took O(n**2) time for
large inputs due to the underlying O(n) behavior of array-based lists.
The deque type, introduced in Python 2.4, solved the most common
problem of needing a fast FIFO queue.  However, the deque type doesn't
help if we need to repeatedly insert or delete elements from the
middle of a long list.

A wide variety of data structure provide good asymptotic performance
for insertions and deletions, but they either have O(n) performance
for other operations (e.g., linked lists) or have inferior performance
for small lists (e.g., binary trees and skip lists).

The BList type proposed in this PEP is based on the principles of
B+Trees, which have array-like and tree-like aspects.  The BList
offers array-like performance on small lists, while offering O(log n)
asymptotic performance for all insert and delete operations.
Additionally, the BList implements copy-on-write under-the-hood, so
even operations like getslice take O(log n) time.  The table below
compares the asymptotic performance of the current array-based list
implementation with the asymptotic performance of the BList.

========= ================                     ====================
Operation Array-based list                     BList
========= ================                     ====================
Copy      O(n)                                 **O(1)**
Append    **O(1)**                             O(log n)
Insert    O(n)                                 **O(log n)**
Get Item  **O(1)**                             O(log n)
Set Item  **O(1)**                             **O(log n)**
Del Item  O(n)                                 **O(log n)**
Iteration O(n)                                 O(n)
Get Slice O(k)                                 **O(log n)**
Del Slice O(n)                                 **O(log n)**
Set Slice O(n+k)                               **O(log k + log n)**
Extend    O(k)                                 **O(log k + log n)**
Sort      O(n log n)                           O(n log n)
Multiply  O(nk)                                **O(log k)**
========= ================                     ====================

An extensive empirical comparison of Python's array-based list and the
BList are available at [2]_.

Use Case Trade-offs
===================

The BList offers superior performance for many, but not all,
operations.  Choosing the correct data type for a particular use case
depends on which operations are used.  Choosing the correct data type
as a built-in depends on balancing the importance of different use
cases and the magnitude of the performance differences.

For the common uses cases of small lists, the array-based list and the
BList have similar performance characteristics.

For the slightly less common case of large lists, there are two common
uses cases where the existing array-based list outperforms the
existing BList reference implementation.  These are:

1. A large LIFO stack, where there are many .append() and .pop(-1)
   operations.  Each operation is O(1) for an array-based list, but
   O(log n) for the BList.

2. A large list that does not change size.  The getitem and setitem
   calls are O(1) for an array-based list, but O(log n) for the BList.

In performance tests on a 10,000 element list, BLists exhibited a 50%
and 5% increase in execution time for these two uses cases,
respectively.

The performance for the LIFO use case could be improved to O(n) time,
by caching a pointer to the right-most leaf within the root node.  For
lists that do not change size, the common case of sequential access
could also be improved to O(n) time via caching in the root node.
However, the performance of these approaches has not been empirically
tested.

Many operations exhibit a tremendous speed-up (O(n) to O(log n)) when
switching from the array-based list to BLists.  In performance tests
on a 10,000 element list, operations such as getslice, setslice, and
FIFO-style insert and deletes on a BList take only 1% of the time
needed on array-based lists.

In light of the large performance speed-ups for many operations, the
small performance costs for some operations will be worthwhile for
many (but not all) applications.

Implementation
==============

The BList is based on the B+Tree data structure.  The BList is a wide,
bushy tree where each node contains an array of up to 128 pointers to
its children.  If the node is a leaf, its children are the
user-visible objects that the user has placed in the list.  If node is
not a leaf, its children are other BList nodes that are not
user-visible.  If the list contains only a few elements, they will all
be a children of single node that is both the root and a leaf.  Since
a node is little more than array of pointers, small lists operate in
effectively the same way as an array-based data type and share the
same good performance characteristics.

The BList maintains a few invariants to ensure good (O(log n))
asymptotic performance regardless of the sequence of insert and delete
operations.  The principle invariants are as follows:

1. Each node has at most 128 children.
2. Each non-root node has at least 64 children.
3. The root node has at least 2 children, unless the list contains
   fewer than 2 elements.
4. The tree is of uniform depth.

If an insert would cause a node to exceed 128 children, the node
spawns a sibling and transfers half of its children to the sibling.
The sibl


id: pep-3100.rst
content:
PEP: 3100
Title: Miscellaneous Python 3.0 Plans
Author: Brett Cannon <brett@python.org>
Status: Final
Type: Process
Created: 20-Aug-2004
Post-History:


Abstract
========

This PEP, previously known as :pep:`3000`, describes smaller scale changes
and new features for which no separate PEP is written yet, all targeted
for Python 3000.

The list of features included in this document is subject to change
and isn't binding on the Python development community; features may be
added, removed, and modified at any time.  The purpose of this list is
to focus our language development effort on changes that are steps to
3.0, and to encourage people to invent ways to smooth the transition.

This document is not a wish-list that anyone can extend.  While there
are two authors of this PEP, we're just supplying the text; the
decisions for which changes are listed in this document are made by
Guido van Rossum, who has chosen them as goals for Python 3.0.

Guido's pronouncements on things that will not change in Python 3.0
are recorded in :pep:`3099`.


General goals
=============

A general goal is to reduce feature duplication by removing old ways
of doing things.  A general principle of the design will be that one
obvious way of doing something is enough. [1]_


Influencing PEPs
================

* :pep:`238` (Changing the Division Operator)
* :pep:`328` (Imports: Multi-Line and Absolute/Relative)
* :pep:`343` (The "with" Statement)
* :pep:`352` (Required Superclass for Exceptions)


Style changes
=============

* The C style guide will be updated to use 4-space indents, never tabs.
  This style should be used for all new files; existing files can be
  updated only if there is no hope to ever merge a particular file from
  the Python 2 HEAD.  Within a file, the indentation style should be
  consistent.  No other style guide changes are planned ATM.


Core language
=============

* True division becomes default behavior :pep:`238` [done]
* ``exec`` as a statement is not worth it -- make it a function [done]
* Add optional declarations for static typing :pep:`3107` [10]_ [done]
* Support only new-style classes; classic classes will be gone [1]_ [done]
* Replace ``print`` by a function [14]_  :pep:`3105` [done]
* The ``softspace`` attribute of files goes away. [done]
* Use ``except E1, E2, E3 as err:`` if you want the error variable. [3]_ [done]
* ``None`` becomes a keyword [4]_; also ``True`` and ``False`` [done]
* ``...`` to become a general expression element [16]_ [done]
* ``as`` becomes a keyword [5]_ (starting in 2.6 already) [done]
* Have list comprehensions be syntactic sugar for passing an
  equivalent generator expression to ``list()``; as a consequence the
  loop variable will no longer be exposed :pep:`289` [done]
* Comparisons other than ``==`` and ``!=`` between disparate types
  will raise an exception unless explicitly supported by the type [6]_ [done]
* floats will not be acceptable as arguments in place of ints for operations
  where floats are inadvertently accepted (PyArg_ParseTuple() i & l formats)
* Remove from ... import * at function scope.  [done] This means that functions
  can always be optimized and support for unoptimized functions can go away.
* Imports :pep:`328`
    + Imports will be absolute by default. [done]
    + Relative imports must be explicitly specified. [done]
    + Indirection entries in ``sys.modules`` (i.e., a value of ``None`` for
      ``A.string`` means to use the top-level ``string`` module) will not be
      supported.
* __init__.py might become optional in sub-packages?  __init__.py will still
  be required for top-level packages.
* Cleanup the Py_InitModule() variants {,3,4} (also import and parser APIs)
* Cleanup the APIs exported in pythonrun, etc.
* Some expressions will require parentheses that didn't in 2.x:

  - List comprehensions will require parentheses around the iterables.
    This will make list comprehensions more similar to generator comprehensions.
    [x for x in 1, 2] will need to be:  [x for x in (1, 2)] [done]
  - Lambdas may have to be parenthesized :pep:`308` [NO]

* In order to get rid of the confusion between __builtin__ and __builtins__,
  it was decided to rename __builtin__ (the module) to builtins, and to leave
  __builtins__ (the sandbox hook) alone. [#builtin]_ [#builtins]_ [done]
* Attributes on functions of the form ``func_whatever`` will be renamed
  ``__whatever__`` [17]_ [done]
* Set literals and comprehensions [19]_ [20]_ [done]
  {x} means set([x]); {x, y} means set([x, y]).
  {F(x) for x in S if P(x)} means set(F(x) for x in S if P(x)).
  NB. {range(x)} means set([range(x)]), NOT set(range(x)).
  There's no literal for an empty set; use set() (or {1}&{2} :-).
  There's no frozenset literal; they are too rarely needed.
* The ``__nonzero__`` special method will be renamed to ``__bool__``
  and have to return a bool. The typeobject slot will be called
  ``tp_bool`` [23]_ [done]
* Dict comprehensions, as first proposed in :pep:`274` [done]
  {K(x): V(x) for x in S if P(x)} means dict((K(x), V(x)) for x in S if P(x)).

To be removed:

* String exceptions: use instances of an Exception class [2]_ [done]
* ``raise Exception, "message"``: use ``raise Exception("message")`` [12]_
  [done]
* ``x``: use ``repr(x)`` [2]_ [done]
* The ``<>`` operator: use ``!=`` instead [3]_ [done]
* The __mod__ and __divmod__ special methods on float. [they should stay] [21]_
* Drop unbound methods [7]_ [26]_ [done]
* METH_OLDARGS [done]
* WITH_CYCLE_GC [done]
* __getslice__, __setslice__, __delslice__ [#sequence-types]_;
  remove slice opcodes and use slice objects. [done]
* ``__oct__``, ``__hex__``: use ``__index__`` in ``oct()`` and ``hex()``
  instead. [done]
* ``__methods__`` and ``__members__`` [done]
* C APIs (see code):
  PyFloat_AsString, PyFloat_AsReprString, PyFloat_AsStringEx,
  PySequence_In, PyEval_EvalFrame, PyEval_CallObject,
  _PyObject_Del, _PyObject_GC_Del, _PyObject_GC_Track, _PyObject_GC_UnTrack
  PyString_AsEncodedString, PyString_AsDecodedString
  PyArg_NoArgs, PyArg_GetInt, intargfunc, intintargfunc

  PyImport_ReloadModule ?


Atomic Types
============

* Remove distinction between int and long types; 'long' built-in type and
  literals with 'L' or 'l' suffix disappear [1]_ [done]
* Make all strings be Unicode, and have a separate bytes() type [1]_
  The new string type will be called 'str'. See :pep:`3137`. [done]
* Return iterable views instead of lists where appropriate for atomic
  type methods (e.g. ``dict.keys()``, ``dict.values()``,
  ``dict.items()``, etc.); iter* methods will be removed. [done]
* Make ``string.join()`` stringify its arguments? [18]_ [NO]
* Fix open() so it returns a ValueError if the mode is bad rather than IOError.
  [done]

To be removed:

* ``basestring.find()`` and ``basestring.rfind()``; use ``basestring.index()``
  or ``basestring.[r]partition()`` or
  ``basestring.rindex()`` in a try/except block??? [13]_ [UNLIKELY]
* ``file.xreadlines()`` method [#file-object]_ [done]
* ``dict.setdefault()``? [15]_ [UNLIKELY]
* ``dict.has_key()`` method; use ``in`` operator [done]
* ``list.sort()`` and ``builtin.sorted()`` methods: eliminate ``cmp``
  parameter [27]_ [done]

Built-in Namespace
==================

* Make built-ins return an iterator where appropriate (e.g. ``range()``,
  ``zip()``, ``map()``, ``filter()``, etc.) [done]
* Remove ``input()`` and rename ``raw_input()`` to ``input()``.
  If you need the old input(), use eval(input()). [done]
* Introduce ``trunc()``, which would call the ``__trunc__()`` method on its
  argument; suggested use is for objects like float where calling ``__int__()``
  has data loss, but an integral representation is still desired? [8]_ [done]
* Exception hierarchy changes :pep:`352` [done]
* Add a ``bin()`` function for a binary representation of integers [done]

To be removed:

* ``apply()``: use ``f(*args, **kw)`` instead [2]_ [done]
* ``buffer()``: must die (use a bytes() type instead) (?) [2]_ [done]
* ``callable()``: ju


id: pep-3114.rst
content:
PEP: 3114
Title: Renaming iterator.next() to iterator.__next__()
Author: Ka-Ping Yee <ping@zesty.ca>
Status: Final
Type: Standards Track
Created: 04-Mar-2007
Python-Version: 3.0
Post-History:


Abstract
========

The iterator protocol in Python 2.x consists of two methods:
``__iter__()`` called on an iterable object to yield an iterator, and
``next()`` called on an iterator object to yield the next item in the
sequence.  Using a ``for`` loop to iterate over an iterable object
implicitly calls both of these methods.  This PEP proposes that the
``next`` method be renamed to ``__next__``, consistent with all the
other protocols in Python in which a method is implicitly called as
part of a language-level protocol, and that a built-in function named
``next`` be introduced to invoke ``__next__`` method, consistent with
the manner in which other protocols are explicitly invoked.


Names With Double Underscores
=============================

In Python, double underscores before and after a name are used to
distinguish names that belong to the language itself.  Attributes and
methods that are implicitly used or created by the interpreter employ
this naming convention; some examples are:

*   ``__file__`` - an attribute automatically created by the interpreter

*   ``__dict__`` - an attribute with special meaning to the interpreter

*   ``__init__`` - a method implicitly called by the interpreter

Note that this convention applies to methods such as ``__init__`` that
are explicitly defined by the programmer, as well as attributes such as
``__file__`` that can only be accessed by naming them explicitly, so it
includes names that are used *or* created by the interpreter.

(Not all things that are called "protocols" are made of methods with
double-underscore names.  For example, the ``__contains__`` method has
double underscores because the language construct ``x in y`` implicitly
calls ``__contains__``.  But even though the ``read`` method is part of
the file protocol, it does not have double underscores because there is
no language construct that implicitly invokes ``x.read()``.)

The use of double underscores creates a separate namespace for names
that are part of the Python language definition, so that programmers
are free to create variables, attributes, and methods that start with
letters, without fear of silently colliding with names that have a
language-defined purpose.  (Colliding with reserved keywords is still
a concern, but at least this will immediately yield a syntax error.)

The naming of the ``next`` method on iterators is an exception to
this convention.  Code that nowhere contains an explicit call to a
``next`` method can nonetheless be silently affected by the presence
of such a method.  Therefore, this PEP proposes that iterators should
have a ``__next__`` method instead of a ``next`` method (with no
change in semantics).


Double-Underscore Methods and Built-In Functions
================================================

The Python language defines several protocols that are implemented or
customized by defining methods with double-underscore names.  In each
case, the protocol is provided by an internal method implemented as a
C function in the interpreter.  For objects defined in Python, this
C function supports customization by implicitly invoking a Python method
with a double-underscore name (it often does a little bit of additional
work beyond just calling the Python method.)

Sometimes the protocol is invoked by a syntactic construct:

*   ``x[y]`` --> internal ``tp_getitem`` --> ``x.__getitem__(y)``

*   ``x + y`` --> internal ``nb_add`` --> ``x.__add__(y)``

*   ``-x`` --> internal ``nb_negative`` --> ``x.__neg__()``

Sometimes there is no syntactic construct, but it is still useful to be
able to explicitly invoke the protocol.  For such cases Python offers a
built-in function of the same name but without the double underscores.

*   ``len(x)`` --> internal ``sq_length`` --> ``x.__len__()``

*   ``hash(x)`` --> internal ``tp_hash`` --> ``x.__hash__()``

*   ``iter(x)`` --> internal ``tp_iter`` --> ``x.__iter__()``

Following this pattern, the natural way to handle ``next`` is to add a
``next`` built-in function that behaves in exactly the same fashion.

*   ``next(x)`` --> internal ``tp_iternext`` --> ``x.__next__()``

Further, it is proposed that the ``next`` built-in function accept a
sentinel value as an optional second argument, following the style of
the ``getattr`` and ``iter`` built-in functions.  When called with two
arguments, ``next`` catches the StopIteration exception and returns
the sentinel value instead of propagating the exception.  This creates
a nice duality between ``iter`` and ``next``:

    iter(function, sentinel) <--> next(iterator, sentinel)


Previous Proposals
==================

This proposal is not a new idea.  The idea proposed here was supported
by the BDFL on python-dev [1]_ and is even mentioned in the original
iterator PEP, :pep:`234`::

    (In retrospect, it might have been better to go for __next__()
    and have a new built-in, next(it), which calls it.__next__().
    But alas, it's too late; this has been deployed in Python 2.2
    since December 2001.)


Objections
==========

There have been a few objections to the addition of more built-ins.
In particular, Martin von Loewis writes [2]_::

    I dislike the introduction of more builtins unless they have a true
    generality (i.e. are likely to be needed in many programs). For this
    one, I think the normal usage of __next__ will be with a for loop, so
    I don't think one would often need an explicit next() invocation.

    It is also not true that most protocols are explicitly invoked through
    builtin functions. Instead, most protocols are can be explicitly invoked
    through methods in the operator module. So following tradition, it
    should be operator.next.

    ...

    As an alternative, I propose that object grows a .next() method,
    which calls __next__ by default.


Transition Plan
===============

Two additional transformations will be added to the 2to3 translation
tool [3]_:

*   Method definitions named ``next`` will be renamed to ``__next__``.

*   Explicit calls to the ``next`` method will be replaced with calls
    to the built-in ``next`` function.  For example, ``x.next()`` will
    become ``next(x)``.

Collin Winter looked into the possibility of automatically deciding
whether to perform the second transformation depending on the presence
of a module-level binding to ``next`` [4]_ and found that it would be
"ugly and slow".  Instead, the translation tool will emit warnings
upon detecting such a binding.  Collin has proposed warnings for the
following conditions [5]_:

*   Module-level assignments to ``next``.

*   Module-level definitions of a function named ``next``.

*   Module-level imports of the name ``next``.

*   Assignments to ``__builtin__.next``.


Approval
========

This PEP was accepted by Guido on March 6, 2007 [6]_.


Implementation
==============

A patch with the necessary changes (except the 2to3 tool) was written
by Georg Brandl and committed as revision 54910.


References
==========

.. [1] Single- vs. Multi-pass iterability (Guido van Rossum)
   https://mail.python.org/pipermail/python-dev/2002-July/026814.html

.. [2] PEP: rename it.next() to it.__next__()... (Martin von Loewis)
   https://mail.python.org/pipermail/python-3000/2007-March/005965.html

.. [3] 2to3 refactoring tool
   https://github.com/python/cpython/tree/ef04c44e29a8276a484f58d03a75a2dec516302d/Lib/lib2to3

.. [4] PEP: rename it.next() to it.__next__()... (Collin Winter)
   https://mail.python.org/pipermail/python-3000/2007-March/006020.html

.. [5] PEP 3113 transition plan
   https://mail.python.org/pipermail/python-3000/2007-March/006044.html

.. [6] PEP: rename it.next() to it.__next__()... (Guido van Rossum)
   https://mail.python.org/pipermail/python-3000/2007-March/006027.html


Copyright
=========

This document has been 


id: pep-3115.rst
content:
PEP: 3115
Title: Metaclasses in Python 3000
Author: Talin <viridia@gmail.com>
Status: Final
Type: Standards Track
Created: 07-Mar-2007
Python-Version: 3.0
Post-History: 11-Mar-2007, 14-Mar-2007


Abstract
========

This PEP proposes changing the syntax for declaring metaclasses,
and alters the semantics for how classes with metaclasses are
constructed.


Rationale
=========

There are two rationales for this PEP, both of which are somewhat
subtle.

The primary reason for changing the way metaclasses work, is that
there are a number of interesting use cases that require the
metaclass to get involved earlier in the class construction process
than is currently possible. Currently, the metaclass mechanism is
essentially a post-processing step. With the advent of class
decorators, much of these post-processing chores can be taken over
by the decorator mechanism.

In particular, there is an important body of use cases where it
would be useful to preserve the order in which a class members are
declared. Ordinary Python objects store their members in a
dictionary, in which ordering is unimportant, and members are
accessed strictly by name. However, Python is often used to
interface with external systems in which the members are organized
according to an implicit ordering. Examples include declaration of C
structs; COM objects; Automatic translation of Python classes into
IDL or database schemas, such as used in an ORM; and so on.

In such cases, it would be useful for a Python programmer to specify
such ordering directly using the declaration order of class members.
Currently, such orderings must be specified explicitly, using some
other mechanism (see the ctypes module for an example.)

Unfortunately, the current method for declaring a metaclass does
not allow for this, since the ordering information has already been
lost by the time the metaclass comes into play. By allowing the
metaclass to get involved in the class construction process earlier,
the new system allows the ordering or other early artifacts of
construction to be preserved and examined.

There proposed metaclass mechanism also supports a number of other
interesting use cases beyond preserving the ordering of declarations.
One use case is to insert symbols into the namespace of the class
body which are only valid during class construction. An example of
this might be "field constructors", small functions that are used in
the creation of class members. Another interesting possibility is
supporting forward references, i.e. references to Python
symbols that are declared further down in the class body.

The other, weaker, rationale is purely cosmetic: The current method
for specifying a metaclass is by assignment to the special variable
``__metaclass__``, which is considered by some to be aesthetically less
than ideal. Others disagree strongly with that opinion. This PEP
will not address this issue, other than to note it, since aesthetic
debates cannot be resolved via logical proofs.


Specification
=============

In the new model, the syntax for specifying a metaclass is via a
keyword argument in the list of base classes::

   class Foo(base1, base2, metaclass=mymeta):
       ...

Additional keywords will also be allowed here, and will be passed to
the metaclass, as in the following example::

   class Foo(base1, base2, metaclass=mymeta, private=True):
       ...

Note that this PEP makes no attempt to define what these other
keywords might be - that is up to metaclass implementors to
determine.

More generally, the parameter list passed to a class definition will
now support all of the features of a function call, meaning that you
can now use ``*args`` and ``**kwargs``-style arguments in the class base
list::

   class Foo(*bases, **kwds):
       ...


Invoking the Metaclass
======================

In the current metaclass system, the metaclass object can be any
callable type. This does not change, however in order to fully
exploit all of the new features the metaclass will need to have an
extra attribute which is used during class pre-construction.

This attribute is named ``__prepare__``, which is invoked as a function
before the evaluation of the class body. The ``__prepare__`` function
takes two positional arguments, and an arbitrary number of keyword
arguments. The two positional arguments are:

=======  ====================================
*name*   the name of the class being created.
*bases*  the list of base classes.
=======  ====================================

The interpreter always tests for the existence of ``__prepare__`` before
calling it; If it is not present, then a regular dictionary is used,
as illustrated in the following Python snippet.

::

   def prepare_class(name, *bases, metaclass=None, **kwargs):
       if metaclass is None:
           metaclass = compute_default_metaclass(bases)
       prepare = getattr(metaclass, '__prepare__', None)
       if prepare is not None:
           return prepare(name, bases, **kwargs)
       else:
           return dict()

The example above illustrates how the arguments to 'class' are
interpreted. The class name is the first argument, followed by
an arbitrary length list of base classes. After the base classes,
there may be one or more keyword arguments, one of which can be
*metaclass*. Note that the *metaclass* argument is not included
in *kwargs*, since it is filtered out by the normal parameter
assignment algorithm. (Note also that *metaclass* is a
keyword-only argument as per :pep:`3102`.)

Even though ``__prepare__`` is not required, the default metaclass
('type') implements it, for the convenience of subclasses calling
it via super().

``__prepare__`` returns a dictionary-like object which is used to store
the class member definitions during evaluation of the class body.
In other words, the class body is evaluated as a function block
(just like it is now), except that the local variables dictionary
is replaced by the dictionary returned from ``__prepare__``. This
dictionary object can be a regular dictionary or a custom mapping
type.

This dictionary-like object is not required to support the full
dictionary interface. A dictionary which supports a limited set of
dictionary operations will restrict what kinds of actions can occur
during evaluation of the class body. A minimal implementation might
only support adding and retrieving values from the dictionary - most
class bodies will do no more than that during evaluation. For some
classes, it may be desirable to support deletion as well. Many
metaclasses will need to make a copy of this dictionary afterwards,
so iteration or other means for reading out the dictionary contents
may also be useful.

The ``__prepare__`` method will most often be implemented as a class
method rather than an instance method because it is called before
the metaclass instance (i.e. the class itself) is created.

Once the class body has finished evaluating, the metaclass will be
called (as a callable) with the class dictionary, which is no
different from the current metaclass mechanism.

Typically, a metaclass will create a custom dictionary - either a
subclass of dict, or a wrapper around it - that will contain
additional properties that are set either before or during the
evaluation of the class body. Then in the second phase, the
metaclass can use these additional properties to further customize
the class.

An example would be a metaclass that uses information about the
ordering of member declarations to create a C struct. The metaclass
would provide a custom dictionary that simply keeps a record of the
order of insertions. This does not need to be a full 'ordered dict'
implementation, but rather just a Python list of (key,value) pairs
that is appended to for each insertion.

Note that in such a case, the metaclass would be required to deal
with the possibility of duplicate keys, but in most cases that is
trivial. The metaclass can use the first declaration, the last,
combine them in some fashion, or simp


id: pep-3101.rst
content:
PEP: 3101
Title: Advanced String Formatting
Author: Talin <viridia@gmail.com>
Status: Final
Type: Standards Track
Created: 16-Apr-2006
Python-Version: 3.0
Post-History: 28-Apr-2006, 06-May-2006, 10-Jun-2007, 14-Aug-2007, 14-Sep-2008


Abstract
========

This PEP proposes a new system for built-in string formatting
operations, intended as a replacement for the existing '%' string
formatting operator.


Rationale
=========

Python currently provides two methods of string interpolation:

- The '%' operator for strings. [1]_

- The string.Template module. [2]_

The primary scope of this PEP concerns proposals for built-in
string formatting operations (in other words, methods of the
built-in string type).

The '%' operator is primarily limited by the fact that it is a
binary operator, and therefore can take at most two arguments.
One of those arguments is already dedicated to the format string,
leaving all other variables to be squeezed into the remaining
argument.  The current practice is to use either a dictionary or a
tuple as the second argument, but as many people have commented
[3]_, this lacks flexibility.  The "all or nothing" approach
(meaning that one must choose between only positional arguments,
or only named arguments) is felt to be overly constraining.

While there is some overlap between this proposal and
string.Template, it is felt that each serves a distinct need,
and that one does not obviate the other.  This proposal is for
a mechanism which, like '%', is efficient for small strings
which are only used once, so, for example, compilation of a
string into a template is not contemplated in this proposal,
although the proposal does take care to define format strings
and the API in such a way that an efficient template package
could reuse the syntax and even some of the underlying
formatting code.


Specification
=============

The specification will consist of the following parts:

- Specification of a new formatting method to be added to the
  built-in string class.

- Specification of functions and flag values to be added to
  the string module, so that the underlying formatting engine
  can be used with additional options.

- Specification of a new syntax for format strings.

- Specification of a new set of special methods to control the
  formatting and conversion of objects.

- Specification of an API for user-defined formatting classes.

- Specification of how formatting errors are handled.

Note on string encodings: When discussing this PEP in the context
of Python 3.0, it is assumed that all strings are unicode strings,
and that the use of the word 'string' in the context of this
document will generally refer to a Python 3.0 string, which is
the same as Python 2.x unicode object.

In the context of Python 2.x, the use of the word 'string' in this
document refers to an object which may either be a regular string
or a unicode object.  All of the function call interfaces
described in this PEP can be used for both strings and unicode
objects, and in all cases there is sufficient information
to be able to properly deduce the output string type (in
other words, there is no need for two separate APIs).
In all cases, the type of the format string dominates - that
is, the result of the conversion will always result in an object
that contains the same representation of characters as the
input format string.


String Methods
--------------

The built-in string class (and also the unicode class in 2.6) will
gain a new method, 'format', which takes an arbitrary number of
positional and keyword arguments::

    "The story of {0}, {1}, and {c}".format(a, b, c=d)

Within a format string, each positional argument is identified
with a number, starting from zero, so in the above example, 'a' is
argument 0 and 'b' is argument 1.  Each keyword argument is
identified by its keyword name, so in the above example, 'c' is
used to refer to the third argument.

There is also a global built-in function, 'format' which formats
a single value::

   print(format(10.0, "7.3g"))

This function is described in a later section.


Format Strings
--------------

Format strings consist of intermingled character data and markup.

Character data is data which is transferred unchanged from the
format string to the output string; markup is not transferred from
the format string directly to the output, but instead is used to
define 'replacement fields' that describe to the format engine
what should be placed in the output string in place of the markup.

Brace characters ('curly braces') are used to indicate a
replacement field within the string::

    "My name is {0}".format('Fred')

The result of this is the string::

    "My name is Fred"

Braces can be escaped by doubling::

    "My name is {0} :-{{}}".format('Fred')

Which would produce::

    "My name is Fred :-{}"

The element within the braces is called a 'field'.  Fields consist
of a 'field name', which can either be simple or compound, and an
optional 'format specifier'.


Simple and Compound Field Names
-------------------------------

Simple field names are either names or numbers.  If numbers, they
must be valid base-10 integers; if names, they must be valid
Python identifiers.  A number is used to identify a positional
argument, while a name is used to identify a keyword argument.

A compound field name is a combination of multiple simple field
names in an expression::

    "My name is {0.name}".format(open('out.txt', 'w'))

This example shows the use of the 'getattr' or 'dot' operator
in a field expression.  The dot operator allows an attribute of
an input value to be specified as the field value.

Unlike some other programming languages, you cannot embed arbitrary
expressions in format strings.  This is by design - the types of
expressions that you can use is deliberately limited.  Only two operators
are supported: the '.' (getattr) operator, and the '[]' (getitem)
operator.  The reason for allowing these operators is that they don't
normally have side effects in non-pathological code.

An example of the 'getitem' syntax::

    "My name is {0[name]}".format(dict(name='Fred'))

It should be noted that the use of 'getitem' within a format string
is much more limited than its conventional usage.  In the above example,
the string 'name' really is the literal string 'name', not a variable
named 'name'.  The rules for parsing an item key are very simple.
If it starts with a digit, then it is treated as a number, otherwise
it is used as a string.

Because keys are not quote-delimited, it is not possible to
specify arbitrary dictionary keys (e.g., the strings "10" or
":-]") from within a format string.

Implementation note: The implementation of this proposal is
not required to enforce the rule about a simple or dotted name
being a valid Python identifier.  Instead, it will rely on the
getattr function of the underlying object to throw an exception if
the identifier is not legal.  The ``str.format()`` function will have
a minimalist parser which only attempts to figure out when it is
"done" with an identifier (by finding a '.' or a ']', or '}',
etc.).


Format Specifiers
-----------------

Each field can also specify an optional set of 'format
specifiers' which can be used to adjust the format of that field.
Format specifiers follow the field name, with a colon (':')
character separating the two::

    "My name is {0:8}".format('Fred')

The meaning and syntax of the format specifiers depends on the
type of object that is being formatted, but there is a standard
set of format specifiers used for any object that does not
override them.

Format specifiers can themselves contain replacement fields.
For example, a field whose field width is itself a parameter
could be specified via::

    "{0:{1}}".format(a, b)

These 'internal' replacement fields can only occur in the format
specifier part of the replacement field.  Internal replacement fields
cannot themselves have format specifiers.  This implies also that
replacement


id: pep-3129.rst
content:
PEP: 3129
Title: Class Decorators
Author: Collin Winter <collinwinter@google.com>
Status: Final
Type: Standards Track
Created: 01-May-2007
Python-Version: 3.0
Post-History: 07-May-2007


Abstract
========

This PEP proposes class decorators, an extension to the function
and method decorators introduced in :pep:`318`.


Rationale
=========

When function decorators were originally debated for inclusion in
Python 2.4, class decorators were seen as
:pep:`obscure and unnecessary <318#motivation>`
thanks to metaclasses.  After several years' experience
with the Python 2.4.x series of releases and an increasing
familiarity with function decorators and their uses, the BDFL and
the community re-evaluated class decorators and recommended their
inclusion in Python 3.0 [#approval]_.

The motivating use-case was to make certain constructs more easily
expressed and less reliant on implementation details of the CPython
interpreter.  While it is possible to express class decorator-like
functionality using metaclasses, the results are generally
unpleasant and the implementation highly fragile [#motivation]_.  In
addition, metaclasses are inherited, whereas class decorators are not,
making metaclasses unsuitable for some, single class-specific uses of
class decorators. The fact that large-scale Python projects like Zope
were going through these wild contortions to achieve something like
class decorators won over the BDFL.


Semantics
=========

The semantics and design goals of class decorators are the same as
for function decorators (:pep:`318#current-syntax`, :pep:`318#design-goals`);
the only
difference is that you're decorating a class instead of a function.
The following two snippets are semantically identical::

  class A:
    pass
  A = foo(bar(A))


  @foo
  @bar
  class A:
    pass

For a detailed examination of decorators, please refer to :pep:`318`.


Implementation
==============

Adapting Python's grammar to support class decorators requires
modifying two rules and adding a new rule::

 funcdef: [decorators] 'def' NAME parameters ['->' test] ':' suite

 compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt |
                with_stmt | funcdef | classdef

need to be changed to ::

 decorated: decorators (classdef | funcdef)

 funcdef: 'def' NAME parameters ['->' test] ':' suite

 compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt |
                with_stmt | funcdef | classdef | decorated

Adding ``decorated`` is necessary to avoid an ambiguity in the
grammar.

The Python AST and bytecode must be modified accordingly.

A reference implementation [#implementation]_ has been provided by
Jack Diederich.


Acceptance
==========

There was virtually no discussion following the posting of this PEP,
meaning that everyone agreed it should be accepted.

The patch was committed to Subversion as revision 55430.


References
==========

.. [#approval]
   https://mail.python.org/pipermail/python-dev/2006-March/062942.html

.. [#motivation]
   https://mail.python.org/pipermail/python-dev/2006-March/062888.html

.. [#implementation]
   https://bugs.python.org/issue1671208



Copyright
=========

This document has been placed in the public domain.


id: pep-3105.rst
content:
PEP: 3105
Title: Make print a function
Author: Georg Brandl <georg@python.org>
Status: Final
Type: Standards Track
Created: 19-Nov-2006
Python-Version: 3.0
Post-History:


Abstract
========

The title says it all -- this PEP proposes a new ``print()`` builtin
that replaces the ``print`` statement and suggests a specific signature
for the new function.


Rationale
=========

The ``print`` statement has long appeared on lists of dubious language
features that are to be removed in Python 3000, such as Guido's "Python
Regrets" presentation [1]_. As such, the objective of this PEP is not
new, though it might become much disputed among Python developers.

The following arguments for a ``print()`` function are distilled from a
python-3000 message by Guido himself [2]_:

* ``print`` is the only application-level functionality that has a
  statement dedicated to it. Within Python's world, syntax is generally
  used as a last resort, when something *can't* be done without help from
  the compiler. Print doesn't qualify for such an exception.

* At some point in application development one quite often feels the need
  to replace ``print`` output by something more sophisticated, like
  logging calls or calls into some other I/O library. With a ``print()``
  function, this is a straightforward string replacement, today it is
  a mess adding all those parentheses and possibly converting ``>>stream``
  style syntax.

* Having special syntax for ``print`` puts up a much larger barrier for
  evolution, e.g. a hypothetical new ``printf()`` function is not too
  far fetched when it will coexist with a ``print()`` function.

* There's no easy way to convert ``print`` statements into another call
  if one needs a different separator, not spaces, or none at all.
  Also, there's no easy way *at all* to conveniently print objects with
  some other separator than a space.

* If ``print()`` is a function, it would be much easier to replace it within
  one module (just ``def print(*args):...``) or even throughout a program
  (e.g. by putting a different function in ``__builtin__.print``). As it is,
  one can do this by writing a class with a ``write()`` method and
  assigning that to ``sys.stdout`` -- that's not bad, but definitely a much
  larger conceptual leap, and it works at a different level than print.


Specification
=============

The signature for ``print()``, taken from various mailings and recently
posted on the python-3000 list [3]_ is::

    def print(*args, sep=' ', end='\n', file=None)

A call like::

    print(a, b, c, file=sys.stderr)

will be equivalent to today's::

    print >>sys.stderr, a, b, c

while the optional ``sep`` and ``end`` arguments specify what is printed
between and after the arguments, respectively.

The ``softspace`` feature (a semi-secret attribute on files currently
used to tell print whether to insert a space before the first item)
will be removed. Therefore, there will not be a direct translation for
today's::

    print "a",
    print

which will not print a space between the ``"a"`` and the newline.


Backwards Compatibility
=======================

The changes proposed in this PEP will render most of today's ``print``
statements invalid.  Only those which incidentally feature parentheses
around all of their arguments will continue to be valid Python syntax
in version 3.0, and of those, only the ones printing a single
parenthesized value will continue to do the same thing.  For example,
in 2.x::

    >>> print ("Hello")
    Hello
    >>> print ("Hello", "world")
    ('Hello', 'world')

whereas in 3.0::

    >>> print ("Hello")
    Hello
    >>> print ("Hello", "world")
    Hello world

Luckily, as it is a statement in Python 2, ``print`` can be detected
and replaced reliably and non-ambiguously by an automated tool, so
there should be no major porting problems (provided someone writes the
mentioned tool).


Implementation
==============

The proposed changes were implemented in the Python 3000 branch in the
Subversion revisions 53685 to 53704. Most of the legacy code in the
library has been converted too, but it is an ongoing effort to catch
every print statement that may be left in the distribution.


References
==========

.. [1] http://legacy.python.org/doc/essays/ppt/regrets/PythonRegrets.pdf

.. [2] Replacement for print in Python 3.0 (Guido van Rossum)
       https://mail.python.org/pipermail/python-dev/2005-September/056154.html

.. [3] print() parameters in py3k (Guido van Rossum)
       https://mail.python.org/pipermail/python-3000/2006-November/004485.html


Copyright
=========

This document has been placed in the public domain.


id: pep-3111.rst
content:
PEP: 3111
Title: Simple input built-in in Python 3000
Author: Andre Roberge <andre.roberge at gmail.com>
Status: Final
Type: Standards Track
Created: 13-Sep-2006
Python-Version: 3.0
Post-History: 22-Dec-2006


Abstract
========

Input and output are core features of computer programs.  Currently,
Python provides a simple means of output through the print keyword
and two simple means of interactive input through the input()
and raw_input() built-in functions.

Python 3.0 will introduce various incompatible changes with previous
Python versions (:pep:`3100`).
Among the proposed changes, print will become a built-in
function, print(), while input() and raw_input() would be removed completely
from the built-in namespace, requiring importing some module to provide
even the most basic input capability.

This PEP proposes that Python 3.0 retains some simple interactive user
input capability, equivalent to raw_input(), within the built-in namespace.

It was accepted by the BDFL in December 2006 [5]_.


Motivation
==========

With its easy readability and its support for many programming styles
(e.g. procedural, object-oriented, etc.) among others, Python is perhaps
the best computer language to use in introductory programming classes.
Simple programs often need to provide information to the user (output)
and to obtain information from the user (interactive input).
Any computer language intended to be used in an educational setting should
provide straightforward methods for both output and interactive input.

The :pep:`current proposals for Python 3.0 <3100>`
include a simple output pathway
via a built-in function named print(), but a more complicated method for
input [e.g. via sys.stdin.readline()], one that requires importing an external
module.  Current versions of Python (pre-3.0) include raw_input() as a
built-in function.  With the availability of such a function, programs that
require simple input/output can be written from day one, without requiring
discussions of importing modules, streams, etc.


Rationale
=========

Current built-in functions, like input() and raw_input(), are found to be
extremely useful in traditional teaching settings. (For more details,
see [2]_ and the discussion that followed.)
While the BDFL has clearly stated [3]_ that input() was not to be kept in
Python 3000, he has also stated that he was not against revising the
decision of killing raw_input().

raw_input() provides a simple mean to ask a question and obtain a response
from a user.  The proposed plans for Python 3.0 would require the replacement
of the single statement::

  name = raw_input("What is your name?")

by the more complicated::

  import sys
  print("What is your name?")
  same = sys.stdin.readline()

However, from the point of view of many Python beginners and educators, the
use of sys.stdin.readline() presents the following problems:

1. Compared to the name "raw_input", the name "sys.stdin.readline()"
is clunky and inelegant.

2. The names "sys" and "stdin" have no meaning for most beginners,
who are mainly interested in *what* the function does, and not *where*
in the package structure it is located.  The lack of meaning also makes
it difficult to remember:
is it "sys.stdin.readline()", or " stdin.sys.readline()"?
To a programming novice, there is not any obvious reason to prefer
one over the other. In contrast, functions simple and direct names like
print, input, and raw_input, and open are easier to remember.

3. The use of "." notation is unmotivated and confusing to many beginners.
For example, it may lead some beginners to think "."  is a standard
character that could be used in any identifier.

4. There is an asymmetry with the print function: why is print not called
sys.stdout.print()?


Specification
=============

The existing ``raw_input()`` function will be renamed to ``input()``.

The Python 2 to 3 conversion tool will replace calls to ``input()`` with
``eval(input())`` and ``raw_input()`` with ``input()``.


Naming Discussion
=================

With ``input()`` effectively removed from the language,
the name ``raw_input()`` makes much less sense and alternatives should be
considered.  The various possibilities mentioned in various forums include::

  ask()
  ask_user()
  get_string()
  input()  # initially rejected by BDFL, later accepted
  prompt()
  read()
  user_input()
  get_response()

While it was initially rejected by the BDFL, it has been suggested that the
most direct solution would be to rename "raw_input" to "input" in Python 3000.
The main objection is that Python 2.x already has a function named "input",
and, even though it is not going to be included in Python 3000,
having a built-in function with the same name but different semantics may
confuse programmers migrating from 2.x to 3000.  Certainly, this is no problem
for beginners, and the scope of the problem is unclear for more experienced
programmers, since raw_input(), while popular with many, is not in
universal use.  In this instance, the good it does for beginners could be
seen to outweigh the harm it does to experienced programmers -
although it could cause confusion for people reading older books or tutorials.

The rationale for accepting the renaming can be found here [4]_.


References
==========

.. [2] The fate of raw_input() in Python 3000
   https://mail.python.org/pipermail/edu-sig/2006-September/006967.html

.. [3] Educational aspects of Python 3000
   https://mail.python.org/pipermail/python-3000/2006-September/003589.html

.. [4] Rationale for going with the straight renaming
   https://mail.python.org/pipermail/python-3000/2006-December/005249.html

.. [5] BDFL acceptance of the PEP
   https://mail.python.org/pipermail/python-3000/2006-December/005257.html

Copyright
=========

This document has been placed in the public domain.


id: pep-3139.rst
content:
PEP: 3139
Title: Cleaning out sys and the "interpreter" module
Author: Benjamin Peterson <benjamin@python.org>
Status: Rejected
Type: Standards Track
Created: 04-Apr-2008
Python-Version: 3.0


Rejection Notice
================

Guido's -0.5 put an end to this PEP.  See
https://mail.python.org/pipermail/python-3000/2008-April/012977.html.


Abstract
========

This PEP proposes a new low-level module for CPython-specific interpreter
functions in order to clean out the sys module and separate general Python
functionality from implementation details.


Rationale
=========

The sys module currently contains functions and data that can be put into two
major groups:

1. Data and functions that are available in all Python implementations and deal
   with the general running of a Python virtual machine.

   - argv
   - byteorder
   - path, path_hooks, meta_path, path_importer_cache, and modules
   - copyright, hexversion, version, and version_info
   - displayhook, __displayhook__
   - excepthook, __excepthook__, exc_info, and exc_clear
   - exec_prefix and prefix
   - executable
   - exit
   - flags, py3kwarning, dont_write_bytecode, and warn_options
   - getfilesystemencoding
   - get/setprofile
   - get/settrace, call_tracing
   - getwindowsversion
   - maxint and maxunicode
   - platform
   - ps1 and ps2
   - stdin, stderr, stdout, __stdin__, __stderr__, __stdout__
   - tracebacklimit


2. Data and functions that affect the CPython interpreter.

   - get/setrecursionlimit
   - get/setcheckinterval
   - _getframe and _current_frame
   - getrefcount
   - get/setdlopenflags
   - settscdumps
   - api_version
   - winver
   - dllhandle
   - float_info
   - _compact_freelists
   - _clear_type_cache
   - subversion
   - builtin_module_names
   - callstats
   - intern

The second collections of items has been steadily increasing over the years
causing clutter in sys.  Guido has even said he doesn't recognize some of things
in it [#bug-1522]_!

Moving these items off to another module would send a clear message to
other Python implementations about what functions need and need not be
implemented.

It has also been proposed that the contents of types module be distributed
across the standard library [#types-removal]_; the interpreter module would
provide an excellent resting place for internal types like frames and code
objects.

Specification
=============

A new builtin module named "interpreter" (see `Naming`_) will be added.

The second list of items above will be split into the stdlib as follows:

The interpreter module
    - get/setrecursionlimit
    - get/setcheckinterval
    - _getframe and _current_frame
    - get/setdlopenflags
    - settscdumps
    - api_version
    - winver
    - dllhandle
    - float_info
    - _clear_type_cache
    - subversion
    - builtin_module_names
    - callstats
    - intern

The gc module:
    - getrefcount
    - _compact_freelists


Transition Plan
===============

Once implemented in 3.x, the interpreter module will be back-ported to 2.6.
Py3k warnings will be added to the sys functions it replaces.


Open Issues
===========


What should move?
-----------------


dont_write_bytecode
^^^^^^^^^^^^^^^^^^^^

Some believe that the writing of bytecode is an implementation detail and should
be moved [#bytecode-issue]_.  The counterargument is that all current, complete
Python implementations do write some sort of bytecode, so it is valuable to be
able to disable it.  Also, if it is moved, some wish to put it in the imp
module.


Move to some to imp?
--------------------

It was noted that dont_write_bytecode or maybe builtin_module_names might fit
nicely in the imp module.


Naming
------

The author proposes the name "interpreter" for the new module.  "pyvm" has also
been suggested [#pyvm-name]_.  The name "cpython" was well liked
[#cpython-name]_.


References
==========

.. [#bug-1522]

   http://bugs.python.org/issue1522

.. [#types-removal]

   https://mail.python.org/pipermail/stdlib-sig/2008-April/000172.html

.. [#bytecode-issue]

   https://mail.python.org/pipermail/stdlib-sig/2008-April/000217.html

.. [#pyvm-name]

   https://mail.python.org/pipermail/python-3000/2007-November/011351.html

.. [#cpython-name]

   https://mail.python.org/pipermail/stdlib-sig/2008-April/000223.html

Copyright
=========

This document has been placed in the public domain.


id: pep-3138.rst
content:
PEP: 3138
Title: String representation in Python 3000
Author: Atsuo Ishimoto <ishimoto at gembook.org>
Status: Final
Type: Standards Track
Created: 05-May-2008
Python-Version: 3.0
Post-History: 05-May-2008, 05-Jun-2008


Abstract
========

This PEP proposes a new string representation form for Python 3000.
In Python prior to Python 3000, the repr() built-in function converted
arbitrary objects to printable ASCII strings for debugging and
logging.  For Python 3000, a wider range of characters, based on the
Unicode standard, should be considered 'printable'.


Motivation
==========

The current repr() converts 8-bit strings to ASCII using following
algorithm.

- Convert CR, LF, TAB and '\\' to '\\r', '\\n', '\\t', '\\\\'.

- Convert other non-printable characters(0x00-0x1f, 0x7f) and
  non-ASCII characters (>= 0x80) to '\\xXX'.

- Backslash-escape quote characters (apostrophe, ') and add the quote
  character at the beginning and the end.

For Unicode strings, the following additional conversions are done.

- Convert leading surrogate pair characters without trailing character
  (0xd800-0xdbff, but not followed by 0xdc00-0xdfff) to '\\uXXXX'.

- Convert 16-bit characters (>= 0x100) to '\\uXXXX'.

- Convert 21-bit characters (>= 0x10000) and surrogate pair characters
  to '\\U00xxxxxx'.

This algorithm converts any string to printable ASCII, and repr() is
used as a handy and safe way to print strings for debugging or for
logging.  Although all non-ASCII characters are escaped, this does not
matter when most of the string's characters are ASCII.  But for other
languages, such as Japanese where most characters in a string are not
ASCII, this is very inconvenient.

We can use ``print(aJapaneseString)`` to get a readable string, but we
don't have a similar workaround for printing strings from collections
such as lists or tuples.  ``print(listOfJapaneseStrings)`` uses repr()
to build the string to be printed, so the resulting strings are always
hex-escaped.  Or when ``open(japaneseFilename)`` raises an exception,
the error message is something like ``IOError: [Errno 2] No such file
or directory: '\u65e5\u672c\u8a9e'``, which isn't helpful.

Python 3000 has a lot of nice features for non-Latin users such as
non-ASCII identifiers, so it would be helpful if Python could also
progress in a similar way for printable output.

Some users might be concerned that such output will mess up their
console if they print binary data like images.  But this is unlikely
to happen in practice because bytes and strings are different types in
Python 3000, so printing an image to the console won't mess it up.

This issue was once discussed by Hye-Shik Chang [1]_, but was rejected.


Specification
=============

- Add a new function to the Python C API ``int Py_UNICODE_ISPRINTABLE
  (Py_UNICODE ch)``.  This function returns 0 if repr() should escape
  the Unicode character ``ch``; otherwise it returns 1.  Characters
  that should be escaped are defined in the Unicode character database
  as:

  * Cc (Other, Control)
  * Cf (Other, Format)
  * Cs (Other, Surrogate)
  * Co (Other, Private Use)
  * Cn (Other, Not Assigned)
  * Zl (Separator, Line), refers to LINE SEPARATOR ('\\u2028').
  * Zp (Separator, Paragraph), refers to PARAGRAPH SEPARATOR
    ('\\u2029').
  * Zs (Separator, Space) other than ASCII space ('\\x20').  Characters
    in this category should be escaped to avoid ambiguity.

- The algorithm to build repr() strings should be changed to:

  * Convert CR, LF, TAB and '\\' to '\\r', '\\n', '\\t', '\\\\'.

  * Convert non-printable ASCII characters (0x00-0x1f, 0x7f) to
    '\\xXX'.

  * Convert leading surrogate pair characters without trailing
    character (0xd800-0xdbff, but not followed by 0xdc00-0xdfff) to
    '\\uXXXX'.

  * Convert non-printable characters (Py_UNICODE_ISPRINTABLE() returns
    0) to '\\xXX', '\\uXXXX' or '\\U00xxxxxx'.

  * Backslash-escape quote characters (apostrophe, 0x27) and add a
    quote character at the beginning and the end.

- Set the Unicode error-handler for sys.stderr to 'backslashreplace'
  by default.

- Add a new function to the Python C API ``PyObject *PyObject_ASCII
  (PyObject *o)``.  This function converts any python object to a
  string using PyObject_Repr() and then hex-escapes all non-ASCII
  characters.  ``PyObject_ASCII()`` generates the same string as
  ``PyObject_Repr()`` in Python 2.

- Add a new built-in function, ``ascii()``.  This function converts
  any python object to a string using repr() and then hex-escapes all
  non-ASCII characters.  ``ascii()`` generates the same string as
  ``repr()`` in Python 2.

- Add a ``'%a'`` string format operator.  ``'%a'`` converts any python
  object to a string using repr() and then hex-escapes all non-ASCII
  characters.  The ``'%a'`` format operator generates the same string
  as ``'%r'`` in Python 2.  Also, add ``'!a'`` conversion flags to the
  ``string.format()`` method and add ``'%A'`` operator to the
  PyUnicode_FromFormat().  They convert any object to an ASCII string
  as ``'%a'`` string format operator.

- Add an ``isprintable()`` method to the string type.
  ``str.isprintable()`` returns False if repr() would escape any
  character in the string; otherwise returns True.  The
  ``isprintable()`` method calls the ``Py_UNICODE_ISPRINTABLE()``
  function internally.


Rationale
=========

The repr() in Python 3000 should be Unicode, not ASCII based, just
like Python 3000 strings.  Also, conversion should not be affected by
the locale setting, because the locale is not necessarily the same as
the output device's locale.  For example, it is common for a daemon
process to be invoked in an ASCII setting, but writes UTF-8 to its log
files.  Also, web applications might want to report the error
information in more readable form based on the HTML page's encoding.

Characters not supported by the user's console could be hex-escaped on
printing, by the Unicode encoder's error-handler.  If the
error-handler of the output file is 'backslashreplace', such
characters are hex-escaped without raising UnicodeEncodeError.  For
example, if the default encoding is ASCII, ``print('Hello Â¢')`` will
print 'Hello \\xa2'.  If the encoding is ISO-8859-1, 'Hello Â¢' will be
printed.

The default error-handler for sys.stdout is 'strict'.  Other
applications reading the output might not understand hex-escaped
characters, so unsupported characters should be trapped when writing.
If unsupported characters must be escaped, the error-handler should be
changed explicitly.  Unlike sys.stdout, sys.stderr doesn't raise
UnicodeEncodingError by default, because the default error-handler is
'backslashreplace'.  So printing error messages containing non-ASCII
characters to sys.stderr will not raise an exception.  Also,
information about uncaught exceptions (exception object, traceback) is
printed by the interpreter without raising exceptions.

Alternate Solutions
-------------------

To help debugging in non-Latin languages without changing repr(),
other suggestions were made.

- Supply a tool to print lists or dicts.

  Strings to be printed for debugging are not only contained by lists
  or dicts, but also in many other types of object.  File objects
  contain a file name in Unicode, exception objects contain a message
  in Unicode, etc.  These strings should be printed in readable form
  when repr()ed.  It is unlikely to be possible to implement a tool to
  print all possible object types.

- Use sys.displayhook and sys.excepthook.

  For interactive sessions, we can write hooks to restore hex escaped
  characters to the original characters.  But these hooks are called
  only when printing the result of evaluating an expression entered in
  an interactive Python session, and don't work for the ``print()``
  function, for non-interactive sessions or for ``logging.debug("%r",
  ...)``, etc.

- Subclass sys.stdout and sys.stderr.

  It is difficult to implement a subclass to restore hex-escaped
  characters s


id: pep-3110.rst
content:
PEP: 3110
Title: Catching Exceptions in Python 3000
Author: Collin Winter <collinwinter@google.com>
Status: Final
Type: Standards Track
Created: 16-Jan-2006
Python-Version: 3.0
Post-History:


Abstract
========

This PEP introduces changes intended to help eliminate ambiguities
in Python's grammar, simplify exception classes, simplify garbage
collection for exceptions and reduce the size of the language in
Python 3.0.


Rationale
=========

1. ``except`` clauses in Python 2.x present a syntactic ambiguity
   where the parser cannot differentiate whether ::

       except <expression>, <expression>:

   should be interpreted as ::

       except <type>, <type>:

   or ::

       except <type>, <name>:

   Python 2 opts for the latter semantic, at the cost of requiring the
   former to be parenthesized, like so ::

    except (<type>, <type>):

2. As specified in :pep:`352`, the ability to treat exceptions
   as tuples will be removed, meaning this code will no longer work ::

       except os.error, (errno, errstr):

   Because the automatic unpacking will no longer be possible, it is
   desirable to remove the ability to use tuples as ``except`` targets.

3. As specified in :pep:`344`, exception instances in Python 3
   will possess a ``__traceback__`` attribute. The Open Issues section
   of that PEP includes a paragraph on garbage collection difficulties
   caused by this attribute, namely a "exception -> traceback ->
   stack frame -> exception" reference cycle, whereby all locals are
   kept in scope until the next GC run. This PEP intends to resolve
   this issue by adding a cleanup semantic to ``except`` clauses in
   Python 3 whereby the target name is deleted at the end of the
   ``except`` suite.

4. In the spirit of :pep:`"there should be one -- and preferably only one
   -- obvious way to do it" <20>`, it is desirable to consolidate
   duplicate functionality. To this end, the ``exc_value``,
   ``exc_type`` and ``exc_traceback`` attributes of the ``sys``
   module [#sys-module]_ will be removed in favor of
   ``sys.exc_info()``, which provides the same information. These
   attributes are already listed in :pep:`3100` as targeted
   for removal.


Grammar Changes
===============

In Python 3, the grammar for ``except`` statements will change
from [#grammar]_ ::

    except_clause: 'except' [test [',' test]]

to ::

    except_clause: 'except' [test ['as' NAME]]

The use of ``as`` in place of the comma token means that ::

    except (AttributeError, os.error):

can be clearly understood as a tuple of exception classes. This new
syntax was first proposed by Greg Ewing [#firstproposal]_ and
endorsed ([#firstproposal]_, [#renaming]_) by the BDFL.

Further, the restriction of the token following ``as`` from ``test``
to ``NAME`` means that only valid identifiers can be used as
``except`` targets.

Note that the grammar above always requires parenthesized tuples as
exception classes. That way, the ambiguous ::

    except A, B:

which would mean different things in Python 2.x and 3.x -- leading to
hard-to-catch bugs -- cannot legally occur in 3.x code.


Semantic Changes
================

In order to resolve the garbage collection issue related to :pep:`344`,
``except`` statements in Python 3 will generate additional bytecode to
delete the target, thus eliminating the reference cycle.
The source-to-source translation, as suggested by Phillip J. Eby
[#except-translation]_, is ::

    try:
        try_body
    except E as N:
        except_body
    ...

gets translated to (in Python 2.5 terms) ::

    try:
        try_body
    except E, N:
        try:
            except_body
        finally:
            N = None
            del N
    ...

An implementation has already been checked into the py3k (formerly
"p3yk") branch [#translation-checkin]_.


Compatibility Issues
====================

Nearly all ``except`` clauses will need to be changed. ``except``
clauses with identifier targets will be converted from ::

    except E, N:

to ::

    except E as N:

``except`` clauses with non-tuple, non-identifier targets
(e.g., ``a.b.c[d]``) will need to be converted from ::

    except E, T:

to ::

    except E as t:
        T = t

Both of these cases can be handled by Guido van Rossum's ``2to3``
utility [#2to3]_ using the ``except`` fixer [#exceptfixer]_.

``except`` clauses with tuple targets will need to be converted
manually, on a case-by-case basis. These changes will usually need
to be accompanied by changes to the exception classes themselves.
While these changes generally cannot be automated, the ``2to3``
utility is able to point out cases where the target of an ``except``
clause is a tuple, simplifying conversion.

Situations where it is necessary to keep an exception instance around
past the end of the ``except`` suite can be easily translated like so
::

    try:
        ...
    except E as N:
        ...
    ...

becomes ::

    try:
        ...
    except E as N:
        n = N
        ...
    ...

This way, when ``N`` is deleted at the end of the block, ``n`` will
persist and can be used as normal.

Lastly, all uses of the ``sys`` module's ``exc_type``, ``exc_value``
and ``exc_traceback`` attributes will need to be removed. They can be
replaced with ``sys.exc_info()[0]``, ``sys.exc_info()[1]`` and
``sys.exc_info()[2]`` respectively, a transformation that can be
performed by ``2to3``'s ``sysexcattrs`` fixer.

2.6 - 3.0 Compatibility
-----------------------

In order to facilitate forwards compatibility between Python 2.6 and 3.0,
the ``except ... as ...:`` syntax will be backported to the 2.x series. The
grammar will thus change from::

    except_clause: 'except' [test [',' test]]

to::

    except_clause: 'except' [test [('as' | ',') test]]

The end-of-suite cleanup semantic for ``except`` statements will not be
included in the 2.x series of releases.


Open Issues
===========

Replacing or Dropping "sys.exc_info()"
--------------------------------------

The idea of dropping ``sys.exc_info()`` or replacing it with a
``sys.exception`` attribute or a ``sys.get_exception()`` function
has been raised several times on python-3000 ([#drop-excinfo]_,
[#replace-excinfo]_) and mentioned in :pep:`344`'s "Open Issues" section.

While a ``2to3`` fixer to replace calls to ``sys.exc_info()``
and some attribute accesses would be trivial, it would be far more
difficult for static analysis to find and fix functions that expect
the values from ``sys.exc_info()`` as arguments. Similarly, this does
not address the need to rewrite the documentation for all APIs that
are defined in terms of ``sys.exc_info()``.


Implementation
==============

This PEP was implemented in revisions 53342 [#r53342]_ and 53349
[#r53349]_.  Support for the new ``except`` syntax in 2.6 was
implemented in revision 55446 [#r55446]_.


References
==========

.. [#sys-module]
   http://docs.python.org/library/sys.html

.. [#firstproposal]
   https://mail.python.org/pipermail/python-dev/2006-March/062449.html

.. [#renaming]
   https://mail.python.org/pipermail/python-dev/2006-March/062640.html

.. [#grammar]
   http://docs.python.org/reference/compound_stmts.html#try

.. [#except-translation]
   https://mail.python.org/pipermail/python-3000/2007-January/005395.html

.. [#translation-checkin]
   http://svn.python.org/view?rev=53342&view=rev

.. [#2to3]
   https://hg.python.org/sandbox/guido/file/2.7/Lib/lib2to3/

.. [#exceptfixer]
   https://hg.python.org/sandbox/guido/file/2.7/Lib/lib2to3/fixes/fix_except.py

.. [#drop-excinfo]
   https://mail.python.org/pipermail/python-3000/2007-January/005385.html

.. [#replace-excinfo]
   https://mail.python.org/pipermail/python-3000/2007-January/005604.html

.. [#r53342]
   http://svn.python.org/view?view=revision&revision=53342

.. [#r53349]
   http://svn.python.org/view?view=revision&revision=53349

.. [#r55446]
   http://svn.python.org/view/python/trunk/?view=rev&rev=55446


Copyright
=========

This document has been placed in the public domai


id: pep-3104.rst
content:
PEP: 3104
Title: Access to Names in Outer Scopes
Author: Ka-Ping Yee <ping@zesty.ca>
Status: Final
Type: Standards Track
Created: 12-Oct-2006
Python-Version: 3.0
Post-History:


Abstract
========

In most languages that support nested scopes, code can refer to or
rebind (assign to) any name in the nearest enclosing scope.
Currently, Python code can refer to a name in any enclosing scope,
but it can only rebind names in two scopes: the local scope (by
simple assignment) or the module-global scope (using a ``global``
declaration).

This limitation has been raised many times on the Python-Dev mailing
list and elsewhere, and has led to extended discussion and many
proposals for ways to remove this limitation.  This PEP summarizes
the various alternatives that have been suggested, together with
advantages and disadvantages that have been mentioned for each.


Rationale
=========

Before version 2.1, Python's treatment of scopes resembled that of
standard C: within a file there were only two levels of scope, global
and local.  In C, this is a natural consequence of the fact that
function definitions cannot be nested.  But in Python, though
functions are usually defined at the top level, a function definition
can be executed anywhere.  This gave Python the syntactic appearance
of nested scoping without the semantics, and yielded inconsistencies
that were surprising to some programmers -- for example, a recursive
function that worked at the top level would cease to work when moved
inside another function, because the recursive function's own name
would no longer be visible in its body's scope.  This violates the
intuition that a function should behave consistently when placed in
different contexts.  Here's an example::

    def enclosing_function():
        def factorial(n):
            if n < 2:
                return 1
            return n * factorial(n - 1)  # fails with NameError
        print factorial(5)

Python 2.1 moved closer to static nested scoping by making visible
the names bound in all enclosing scopes (see :pep:`227`).  This change
makes the above code example work as expected.  However, because any
assignment to a name implicitly declares that name to be local, it is
impossible to rebind a name in an outer scope (except when a
``global`` declaration forces the name to be global).  Thus, the
following code, intended to display a number that can be incremented
and decremented by clicking buttons, doesn't work as someone familiar
with lexical scoping might expect::

    def make_scoreboard(frame, score=0):
        label = Label(frame)
        label.pack()
        for i in [-10, -1, 1, 10]:
            def increment(step=i):
                score = score + step  # fails with UnboundLocalError
                label['text'] = score
            button = Button(frame, text='%+d' % i, command=increment)
            button.pack()
        return label

Python syntax doesn't provide a way to indicate that the name
``score`` mentioned in ``increment`` refers to the variable ``score``
bound in ``make_scoreboard``, not a local variable in ``increment``.
Users and developers of Python have expressed an interest in removing
this limitation so that Python can have the full flexibility of the
Algol-style scoping model that is now standard in many programming
languages, including JavaScript, Perl, Ruby, Scheme, Smalltalk,
C with GNU extensions, and C# 2.0.

It has been argued that such a feature isn't necessary, because
a rebindable outer variable can be simulated by wrapping it in a
mutable object::

    class Namespace:
        pass

    def make_scoreboard(frame, score=0):
        ns = Namespace()
        ns.score = 0
        label = Label(frame)
        label.pack()
        for i in [-10, -1, 1, 10]:
            def increment(step=i):
                ns.score = ns.score + step
                label['text'] = ns.score
            button = Button(frame, text='%+d' % i, command=increment)
            button.pack()
        return label

However, this workaround only highlights the shortcomings of existing
scopes: the purpose of a function is to encapsulate code in its own
namespace, so it seems unfortunate that the programmer should have to
create additional namespaces to make up for missing functionality in
the existing local scopes, and then have to decide whether each name
should reside in the real scope or the simulated scope.

Another common objection is that the desired functionality can be
written as a class instead, albeit somewhat more verbosely.  One
rebuttal to this objection is that the existence of a different
implementation style is not a reason to leave a supported programming
construct (nested scopes) functionally incomplete.  Python is
sometimes called a "multi-paradigm language" because it derives so
much strength, practical flexibility, and pedagogical power from its
support and graceful integration of multiple programming paradigms.

A proposal for scoping syntax appeared on Python-Dev as far back as
1994 [1]_, long before :pep:`227`'s support for nested scopes was
adopted.  At the time, Guido's response was:

    This is dangerously close to introducing CSNS [classic static
    nested scopes]. *If* you were to do so, your proposed semantics
    of scoped seem alright. I still think there is not enough need
    for CSNS to warrant this kind of construct ...

After :pep:`227`, the "outer name rebinding discussion" has reappeared
on Python-Dev enough times that it has become a familiar event,
having recurred in its present form since at least 2003 [2]_.
Although none of the language changes proposed in these discussions
have yet been adopted, Guido has acknowledged that a language change
is worth considering [12]_.


Other Languages
===============

To provide some background, this section describes how some other
languages handle nested scopes and rebinding.

JavaScript, Perl, Scheme, Smalltalk, GNU C, C# 2.0
--------------------------------------------------

These languages use variable declarations to indicate scope.  In
JavaScript, a lexically scoped variable is declared with the ``var``
keyword; undeclared variable names are assumed to be global.  In
Perl, a lexically scoped variable is declared with the ``my``
keyword; undeclared variable names are assumed to be global.  In
Scheme, all variables must be declared (with ``define`` or ``let``,
or as formal parameters).  In Smalltalk, any block can begin by
declaring a list of local variable names between vertical bars.
C and C# require type declarations for all variables.  For all these
cases, the variable belongs to the scope containing the declaration.

Ruby (as of 1.8)
----------------

Ruby is an instructive example because it appears to be the only
other currently popular language that, like Python, tries to support
statically nested scopes without requiring variable declarations, and
thus has to come up with an unusual solution.  Functions in Ruby can
contain other function definitions, and they can also contain code
blocks enclosed in curly braces.  Blocks have access to outer
variables, but nested functions do not.  Within a block, an
assignment to a name implies a declaration of a local variable only
if it would not shadow a name already bound in an outer scope;
otherwise assignment is interpreted as rebinding of the outer name.
Ruby's scoping syntax and rules have also been debated at great
length, and changes seem likely in Ruby 2.0 [28]_.


Overview of Proposals
=====================

There have been many different proposals on Python-Dev for ways to
rebind names in outer scopes.  They all fall into two categories:
new syntax in the scope where the name is bound, or new syntax in
the scope where the name is used.

New Syntax in the Binding (Outer) Scope
---------------------------------------

Scope Override Declaration
''''''''''''''''''''''''''

The proposals in this category all suggest a new kind of declaration
statement similar to JavaScript's ``var``.  A few possible key


id: pep-3099.rst
content:
PEP: 3099
Title: Things that will Not Change in Python 3000
Author: Georg Brandl <georg@python.org>
Status: Final
Type: Process
Created: 04-Apr-2006
Post-History:


Abstract
========

Some ideas are just bad.  While some thoughts on Python evolution are
constructive, some go against the basic tenets of Python so
egregiously that it would be like asking someone to run in a circle:
it gets you nowhere, even for Python 3000, where extraordinary
proposals are allowed.  This PEP tries to list all BDFL pronouncements
on Python 3000 that refer to changes that will not happen and new
features that will not be introduced, sorted by topics, along with
a short explanation or a reference to the relevant thread on the
python-3000 mailing list.

If you think you should suggest any of the listed ideas it would be
better to just step away from the computer, go outside, and enjoy
yourself.  Being active outdoors by napping in a nice patch of grass
is more productive than bringing up a beating-a-dead-horse idea and
having people tell you how dead the idea is.  Consider yourself warned.


Core language
=============

* Python 3000 will not be case-insensitive.

* Python 3000 will not be a rewrite from scratch.

   It will also not use C++ or another language different from C
   as implementation language.  Rather, there will be a gradual
   transmogrification of the codebase.  There's an excellent essay
   by Joel Spolsky explaining why:
   http://www.joelonsoftware.com/articles/fog0000000069.html

* ``self`` will not become implicit.

   Having ``self`` be explicit is a *good thing*.  It makes the code
   clear by removing ambiguity about how a variable resolves.  It also
   makes the difference between functions and methods small.

   Thread: "Draft proposal: Implicit self in Python 3.0"
   https://mail.python.org/pipermail/python-dev/2006-January/059468.html

* ``lambda`` will not be renamed.

   At one point lambda was slated for removal in Python 3000.
   Unfortunately no one was able to come up with a better way of
   providing anonymous functions.  And so lambda is here to stay.

   But it is here to stay as-is.  Adding support for statements is a
   non-starter.  It would require allowing multi-line lambda
   expressions which would mean a multi-line expression could suddenly
   exist.  That would allow for multi-line arguments to function
   calls, for instance.  That is just plain ugly.

   Thread: "genexp syntax / lambda",
   https://mail.python.org/pipermail/python-3000/2006-April/001042.html

* Python will not have programmable syntax.

   Thread: "It's a statement! It's a function! It's BOTH!",
   https://mail.python.org/pipermail/python-3000/2006-April/000286.html

* There won't be a syntax for ``zip()``-style parallel iteration.

   Thread: "Parallel iteration syntax",
   https://mail.python.org/pipermail/python-3000/2006-March/000210.html

* Strings will stay iterable.

   Thread: "Making strings non-iterable",
   https://mail.python.org/pipermail/python-3000/2006-April/000759.html

* There will be no syntax to sort the result of a generator expression
  or list comprehension.  ``sorted()`` covers all use cases.

   Thread: "Adding sorting to generator comprehension",
   https://mail.python.org/pipermail/python-3000/2006-April/001295.html

* Slices and extended slices won't go away (even if the __getslice__
  and __setslice__ APIs may be replaced) nor will they return views
  for the standard object types.

   Thread: Future of slices
   https://mail.python.org/pipermail/python-3000/2006-May/001563.html

* It will not be forbidden to reuse a loop variable inside the loop's
  suite.

   Thread: elimination of scope bleeding of iteration variables
   https://mail.python.org/pipermail/python-dev/2006-May/064761.html

* The parser won't be more complex than LL(1).

   Simple is better than complex.  This idea extends to the parser.
   Restricting Python's grammar to an LL(1) parser is a blessing,
   not a curse.  It puts us in handcuffs that prevent us from going
   overboard and ending up with funky grammar rules like some other
   dynamic languages that will go unnamed, such as Perl.

* No braces.

   This is so obvious that it doesn't need a reference to a mailing
   list. Do ``from __future__ import braces`` to get a definitive
   answer on this subject.

* No more backticks.

   Backticks (\`) will no longer be used as shorthand for ``repr`` --
   but that doesn't mean they are available for other uses.  Even
   ignoring the backwards compatibility confusion, the character
   itself causes too many problems (in some fonts, on some keyboards,
   when typesetting a book, etc).

   Thread: "new operators via backquoting",
   https://mail.python.org/pipermail/python-ideas/2007-January/000054.html

* Referencing the global name ``foo`` will not be spelled ``globals.foo``.
  The ``global`` statement will stay.

   Threads: "replace globals() and global statement with global builtin
   object",
   https://mail.python.org/pipermail/python-3000/2006-July/002485.html,
   "Explicit Lexical Scoping (pre-PEP?)",
   https://mail.python.org/pipermail/python-dev/2006-July/067111.html

* There will be no alternative binding operators such as ``:=``.

   Thread: "Explicit Lexical Scoping (pre-PEP?)",
   https://mail.python.org/pipermail/python-dev/2006-July/066995.html

* We won't be removing container literals.
  That is, {expr: expr, ...}, [expr, ...] and (expr, ...) will stay.

   Thread: "No Container Literals",
   https://mail.python.org/pipermail/python-3000/2006-July/002550.html

* The ``else`` clause in ``while`` and ``for`` loops will not change
  semantics, or be removed.

   Thread: "for/except/else syntax"
   https://mail.python.org/pipermail/python-ideas/2009-October/006083.html


Builtins
========

* ``zip()`` won't grow keyword arguments or other mechanisms to prevent
  it from stopping at the end of the shortest sequence.

   Thread: "have zip() raise exception for sequences of different lengths",
   https://mail.python.org/pipermail/python-3000/2006-August/003338.html

* ``hash()`` won't become an attribute since attributes should be cheap
  to compute, which isn't necessarily the case for a hash.

   Thread: "hash as attribute/property",
   https://mail.python.org/pipermail/python-3000/2006-April/000362.html


Standard types
==============

* Iterating over a dictionary will continue to yield the keys.

   Thread: "Iterating over a dict",
   https://mail.python.org/pipermail/python-3000/2006-April/000283.html

   Thread: have iter(mapping) generate (key, value) pairs
   https://mail.python.org/pipermail/python-3000/2006-June/002368.html

* There will be no ``frozenlist`` type.

   Thread: "Immutable lists",
   https://mail.python.org/pipermail/python-3000/2006-May/002219.html

* ``int`` will not support subscripts yielding a range.

   Thread: "xrange vs. int.__getslice__",
   https://mail.python.org/pipermail/python-3000/2006-June/002450.html


Coding style
============

* The (recommended) maximum line width will remain 80 characters,
  for both C and Python code.

   Thread: "C style guide",
   https://mail.python.org/pipermail/python-3000/2006-March/000131.html


Interactive Interpreter
=======================

* The interpreter prompt (``>>>``) will not change. It gives Guido warm
  fuzzy feelings.

   Thread: "Low-hanging fruit: change interpreter prompt?",
   https://mail.python.org/pipermail/python-3000/2006-November/004891.html


Copyright
=========

This document has been placed in the public domain.


id: pep-3112.rst
content:
PEP: 3112
Title: Bytes literals in Python 3000
Author: Jason Orendorff <jason.orendorff@gmail.com>
Status: Final
Type: Standards Track
Requires: 358
Created: 23-Feb-2007
Python-Version: 3.0
Post-History: 23-Feb-2007


Abstract
========

This PEP proposes a literal syntax for the ``bytes`` objects
introduced in :pep:`358`.  The purpose is to provide a convenient way to
spell ASCII strings and arbitrary binary data.


Motivation
==========

Existing spellings of an ASCII string in Python 3000 include::

    bytes('Hello world', 'ascii')
    'Hello world'.encode('ascii')

The proposed syntax is::

    b'Hello world'

Existing spellings of an 8-bit binary sequence in Python 3000 include::

    bytes([0x7f, 0x45, 0x4c, 0x46, 0x01, 0x01, 0x01, 0x00])
    bytes('\x7fELF\x01\x01\x01\0', 'latin-1')
    '7f454c4601010100'.decode('hex')

The proposed syntax is::

    b'\x7f\x45\x4c\x46\x01\x01\x01\x00'
    b'\x7fELF\x01\x01\x01\0'

In both cases, the advantages of the new syntax are brevity, some
small efficiency gain, and the detection of encoding errors at compile
time rather than at runtime.  The brevity benefit is especially felt
when using the string-like methods of bytes objects::

    lines = bdata.split(bytes('\n', 'ascii'))  # existing syntax
    lines = bdata.split(b'\n')  # proposed syntax

And when converting code from Python 2.x to Python 3000::

    sok.send('EXIT\r\n')  # Python 2.x
    sok.send('EXIT\r\n'.encode('ascii'))  # Python 3000 existing
    sok.send(b'EXIT\r\n')  # proposed


Grammar Changes
===============

The proposed syntax is an extension of the existing string
syntax [#stringliterals]_.

The new syntax for strings, including the new bytes literal, is::

  stringliteral: [stringprefix] (shortstring | longstring)
  stringprefix: "b" | "r" | "br" | "B" | "R" | "BR" | "Br" | "bR"
  shortstring: "'" shortstringitem* "'" | '"' shortstringitem* '"'
  longstring: "'''" longstringitem* "'''" | '"""' longstringitem* '"""'
  shortstringitem: shortstringchar | escapeseq
  longstringitem: longstringchar | escapeseq
  shortstringchar:
    <any source character except "\" or newline or the quote>
  longstringchar: <any source character except "\">
  escapeseq: "\" NL
    | "\\" | "\'" | '\"'
    | "\a" | "\b" | "\f" | "\n" | "\r" | "\t" | "\v"
    | "\ooo" | "\xhh"
    | "\uxxxx" | "\Uxxxxxxxx" | "\N{name}"

The following additional restrictions apply only to bytes literals
(``stringliteral`` tokens with ``b`` or ``B`` in the
``stringprefix``):

- Each ``shortstringchar`` or ``longstringchar`` must be a character
  between 1 and 127 inclusive, regardless of any encoding
  declaration [#encodings]_ in the source file.

- The Unicode-specific escape sequences ``\u``\ *xxxx*,
  ``\U``\ *xxxxxxxx*, and ``\N{``\ *name*\ ``}`` are unrecognized in
  Python 2.x and forbidden in Python 3000.

Adjacent bytes literals are subject to the same concatenation rules as
adjacent string literals [#concat]_.  A bytes literal adjacent to a
string literal is an error.


Semantics
=========

Each evaluation of a bytes literal produces a new ``bytes`` object.
The bytes in the new object are the bytes represented by the
``shortstringitem`` or ``longstringitem`` parts of the literal, in the
same order.


Rationale
=========

The proposed syntax provides a cleaner migration path from Python 2.x
to Python 3000 for most code involving 8-bit strings.  Preserving the
old 8-bit meaning of a string literal is usually as simple as adding a
``b`` prefix.  The one exception is Python 2.x strings containing
bytes >127, which must be rewritten using escape sequences.
Transcoding a source file from one encoding to another, and fixing up
the encoding declaration, should preserve the meaning of the program.
Python 2.x non-Unicode strings violate this principle; Python 3000
bytes literals shouldn't.

A string literal with a ``b`` in the prefix is always a syntax error
in Python 2.5, so this syntax can be introduced in Python 2.6, along
with the ``bytes`` type.

A bytes literal produces a new object each time it is evaluated, like
list displays and unlike string literals.  This is necessary because
bytes literals, like lists and unlike strings, are
mutable [#eachnew]_.


Reference Implementation
========================

Thomas Wouters has checked an implementation into the Py3K branch,
r53872.


References
==========

.. [#stringliterals]
   http://docs.python.org/reference/lexical_analysis.html#string-literals

.. [#encodings]
   http://docs.python.org/reference/lexical_analysis.html#encoding-declarations

.. [#concat]
   http://docs.python.org/reference/lexical_analysis.html#string-literal-concatenation

.. [#eachnew]
   https://mail.python.org/pipermail/python-3000/2007-February/005779.html


Copyright
=========

This document has been placed in the public domain.


id: pep-3106.rst
content:
PEP: 3106
Title: Revamping dict.keys(), .values() and .items()
Author: Guido van Rossum
Status: Final
Type: Standards Track
Created: 19-Dec-2006
Python-Version: 3.0
Post-History:


Abstract
========

This PEP proposes to change the .keys(), .values() and .items()
methods of the built-in dict type to return a set-like or unordered
container object whose contents are derived from the underlying
dictionary rather than a list which is a copy of the keys, etc.; and
to remove the .iterkeys(), .itervalues() and .iteritems() methods.

The approach is inspired by that taken in the Java Collections
Framework [1]_.


Introduction
============

It has long been the plan to change the .keys(), .values() and
.items() methods of the built-in dict type to return a more
lightweight object than a list, and to get rid of .iterkeys(),
.itervalues() and .iteritems().  The idea is that code that currently
(in 2.x) reads::

    for k, v in d.iteritems(): ...

should be rewritten as::

    for k, v in d.items(): ...

(and similar for .itervalues() and .iterkeys(), except the latter is
redundant since we can write that loop as ``for k in d``.)

Code that currently reads::

    a = d.keys()    # assume we really want a list here

(etc.) should be rewritten as

    a = list(d.keys())

There are (at least) two ways to accomplish this.  The original plan
was to simply let .keys(), .values() and .items() return an iterator,
i.e. exactly what iterkeys(), itervalues() and iteritems() return in
Python 2.x.  However, the Java Collections Framework [1]_ suggests
that a better solution is possible: the methods return objects with
set behavior (for .keys() and .items()) or multiset (== bag) behavior
(for .values()) that do not contain copies of the keys, values or
items, but rather reference the underlying dict and pull their values
out of the dict as needed.

The advantage of this approach is that one can still write code like
this::

    a = d.items()
    for k, v in a: ...
    # And later, again:
    for k, v in a: ...

Effectively, iter(d.keys()) (etc.) in Python 3.0 will do what
d.iterkeys() (etc.) does in Python 2.x; but in most contexts we don't
have to write the iter() call because it is implied by a for-loop.

The objects returned by the .keys() and .items() methods behave like
sets.  The object returned by the values() method behaves like a much
simpler unordered collection -- it cannot be a set because duplicate
values are possible.

Because of the set behavior, it will be possible to check whether two
dicts have the same keys by simply testing::

    if a.keys() == b.keys(): ...

and similarly for .items().

These operations are thread-safe only to the extent that using them in
a thread-unsafe way may cause an exception but will not cause
corruption of the internal representation.

As in Python 2.x, mutating a dict while iterating over it using an
iterator has an undefined effect and will in most cases raise a
RuntimeError exception.  (This is similar to the guarantees made by
the Java Collections Framework.)

The objects returned by .keys() and .items() are fully interoperable
with instances of the built-in set and frozenset types; for example::

    set(d.keys()) == d.keys()

is guaranteed to be True (except when d is being modified
simultaneously by another thread).


Specification
=============

I'm using pseudo-code to specify the semantics::

    class dict:

        # Omitting all other dict methods for brevity.
        # The .iterkeys(), .itervalues() and .iteritems() methods
        # will be removed.

        def keys(self):
            return d_keys(self)

        def items(self):
            return d_items(self)

        def values(self):
            return d_values(self)

    class d_keys:

        def __init__(self, d):
            self.__d = d

        def __len__(self):
            return len(self.__d)

        def __contains__(self, key):
            return key in self.__d

        def __iter__(self):
            for key in self.__d:
                yield key

        # The following operations should be implemented to be
        # compatible with sets; this can be done by exploiting
        # the above primitive operations:
        #
        #   <, <=, ==, !=, >=, > (returning a bool)
        #   &, |, ^, - (returning a new, real set object)
        #
        # as well as their method counterparts (.union(), etc.).
        #
        # To specify the semantics, we can specify x == y as:
        #
        #   set(x) == set(y)   if both x and y are d_keys instances
        #   set(x) == y        if x is a d_keys instance
        #   x == set(y)        if y is a d_keys instance
        #
        # and so on for all other operations.

    class d_items:

        def __init__(self, d):
            self.__d = d

        def __len__(self):
            return len(self.__d)

        def __contains__(self, (key, value)):
            return key in self.__d and self.__d[key] == value

        def __iter__(self):
            for key in self.__d:
                yield key, self.__d[key]

        # As well as the set operations mentioned for d_keys above.
        # However the specifications suggested there will not work if
        # the values aren't hashable.  Fortunately, the operations can
        # still be implemented efficiently.  For example, this is how
        # intersection can be specified:

        def __and__(self, other):
            if isinstance(other, (set, frozenset, d_keys)):
                result = set()
                for item in other:
                    if item in self:
                        result.add(item)
                return result
            if not isinstance(other, d_items):
                return NotImplemented
            d = {}
            if len(other) < len(self):
                self, other = other, self
            for item in self:
                if item in other:
                    key, value = item
                    d[key] = value
            return d.items()

        # And here is equality:

        def __eq__(self, other):
            if isinstance(other, (set, frozenset, d_keys)):
                if len(self) != len(other):
                    return False
                for item in other:
                    if item not in self:
                        return False
                return True
            if not isinstance(other, d_items):
                return NotImplemented
            # XXX We could also just compare the underlying dicts...
            if len(self) != len(other):
                return False
            for item in self:
                if item not in other:
                    return False
            return True

        def __ne__(self, other):
            # XXX Perhaps object.__ne__() should be defined this way.
            result = self.__eq__(other)
            if result is not NotImplemented:
                result = not result
            return result

    class d_values:

        def __init__(self, d):
            self.__d = d

        def __len__(self):
            return len(self.__d)

        def __contains__(self, value):
            # This is slow, and it's what "x in y" uses as a fallback
            # if __contains__ is not defined; but I'd rather make it
            # explicit that it is supported.
            for v in self:
                 if v == value:
                     return True
            return False

        def __iter__(self):
            for key in self.__d:
                yield self.__d[key]

        def __eq__(self, other):
            if not isinstance(other, d_values):
                return NotImplemented
            if len(self) != len(other):
                return False
            # XXX Sometimes this could be optimized, but these are the
            # semantics: we can't depend on the values to be hashable
            # or comparable.
            olist = list(other)
            for x in self:
                try:
                    olist.remove(x)
  


id: pep-3107.rst
content:
PEP: 3107
Title: Function Annotations
Author: Collin Winter <collinwinter@google.com>,
        Tony Lownds <tony@lownds.com>
Status: Final
Type: Standards Track
Created: 02-Dec-2006
Python-Version: 3.0
Post-History:


Abstract
========

This PEP introduces a syntax for adding arbitrary metadata annotations
to Python functions [#functerm]_.


Rationale
=========

Because Python's 2.x series lacks a standard way of annotating a
function's parameters and return values, a variety of tools
and libraries have appeared to fill this gap.  Some
utilise the decorators introduced in :pep:`318`, while others parse a
function's docstring, looking for annotations there.

This PEP aims to provide a single, standard way of specifying this
information, reducing the confusion caused by the wide variation in
mechanism and syntax that has existed until this point.


Fundamentals of Function Annotations
====================================

Before launching into a discussion of the precise ins and outs of
Python 3.0's function annotations, let's first talk broadly about
what annotations are and are not:

1. Function annotations, both for parameters and return values, are
   completely optional.

2. Function annotations are nothing more than a way of associating
   arbitrary Python expressions with various parts of a function at
   compile-time.

   By itself, Python does not attach any particular meaning or
   significance to annotations.  Left to its own, Python simply makes
   these expressions available as described in `Accessing Function
   Annotations`_ below.

   The only way that annotations take on meaning is when they are
   interpreted by third-party libraries.  These annotation consumers
   can do anything they want with a function's annotations.  For
   example, one library might use string-based annotations to provide
   improved help messages, like so::

        def compile(source: "something compilable",
                    filename: "where the compilable thing comes from",
                    mode: "is this a single statement or a suite?"):
            ...

   Another library might be used to provide typechecking for Python
   functions and methods.  This library could use annotations to
   indicate the function's expected input and return types, possibly
   something like::

        def haul(item: Haulable, *vargs: PackAnimal) -> Distance:
            ...

   However, neither the strings in the first example nor the
   type information in the second example have any meaning on their
   own; meaning comes from third-party libraries alone.

3. Following from point 2, this PEP makes no attempt to introduce
   any kind of standard semantics, even for the built-in types.
   This work will be left to third-party libraries.


Syntax
======

Parameters
----------

Annotations for parameters take the form of optional expressions that
follow the parameter name::

    def foo(a: expression, b: expression = 5):
        ...

In pseudo-grammar, parameters now look like ``identifier [:
expression] [= expression]``.  That is, annotations always precede a
parameter's default value and both annotations and default values are
optional.  Just like how equal signs are used to indicate a default
value, colons are used to mark annotations.  All annotation
expressions are evaluated when the function definition is executed,
just like default values.

Annotations for excess parameters (i.e., ``*args`` and ``**kwargs``)
are indicated similarly::

    def foo(*args: expression, **kwargs: expression):
        ...

Annotations for nested parameters always follow the name of the
parameter, not the last parenthesis.  Annotating all parameters of a
nested parameter is not required::

    def foo((x1, y1: expression),
            (x2: expression, y2: expression)=(None, None)):
        ...


Return Values
-------------

The examples thus far have omitted examples of how to annotate the
type of a function's return value.  This is done like so::

    def sum() -> expression:
        ...

That is, the parameter list can now be followed by a literal ``->``
and a Python expression.  Like the annotations for parameters, this
expression will be evaluated when the function definition is executed.

The grammar for function definitions [#grammar]_ is now::

    decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
    decorators: decorator+
    funcdef: [decorators] 'def' NAME parameters ['->' test] ':' suite
    parameters: '(' [typedargslist] ')'
    typedargslist: ((tfpdef ['=' test] ',')*
                    ('*' [tname] (',' tname ['=' test])* [',' '**' tname]
                     | '**' tname)
                    | tfpdef ['=' test] (',' tfpdef ['=' test])* [','])
    tname: NAME [':' test]
    tfpdef: tname | '(' tfplist ')'
    tfplist: tfpdef (',' tfpdef)* [',']

Lambda
------

``lambda``'s syntax does not support annotations.  The syntax of
``lambda`` could be changed to support annotations, by requiring
parentheses around the parameter list.  However it was decided
[#lambda]_ not to make this change because:

1. It would be an incompatible change.
2. Lambdas are neutered anyway.
3. The lambda can always be changed to a function.


Accessing Function Annotations
==============================

Once compiled, a function's annotations are available via the
function's ``__annotations__`` attribute.  This attribute is
a mutable dictionary, mapping parameter names to an object
representing the evaluated annotation expression

There is a special key in the ``__annotations__`` mapping,
``"return"``.  This key is present only if an annotation was supplied
for the function's return value.

For example, the following annotation::

    def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9):
        ...

would result in an ``__annotations__`` mapping of ::

    {'a': 'x',
     'b': 11,
     'c': list,
     'return': 9}

The ``return`` key was chosen because it cannot conflict with the name
of a parameter; any attempt to use ``return`` as a parameter name
would result in a ``SyntaxError``.

``__annotations__`` is an empty, mutable dictionary if there are no
annotations on the function or if the functions was created from
a ``lambda`` expression.

Use Cases
=========

In the course of discussing annotations, a number of use-cases have
been raised.  Some of these are presented here, grouped by what kind
of information they convey.  Also included are examples of existing
products and packages that could make use of annotations.

* Providing typing information

  + Type checking ([#typecheck]_, [#maxime]_)
  + Let IDEs show what types a function expects and returns ([#idle]_)
  + Function overloading / generic functions ([#scaling]_)
  + Foreign-language bridges ([#jython]_, [#ironpython]_)
  + Adaptation ([#adaptationpost]_, [#pyprotocols]_)
  + Predicate logic functions
  + Database query mapping
  + RPC parameter marshaling ([#rpyc]_)

* Other information

  + Documentation for parameters and return values ([#pydoc]_)


Standard Library
================

pydoc and inspect
-----------------

The ``pydoc`` module should display the function annotations when
displaying help for a function.  The ``inspect`` module should change
to support annotations.


Relation to Other PEPs
======================

Function Signature Objects (PEP 362)
------------------------------------

Function Signature Objects should expose the function's annotations.
The ``Parameter`` object may change or other changes may be warranted.


Implementation
==============

A reference implementation has been checked into the py3k (formerly
"p3yk") branch as revision 53170 [#implementation]_.


Rejected Proposals
==================

+ The BDFL rejected the author's idea for a special syntax for adding
  annotations to generators as being "too ugly" [#rejectgensyn]_.

+ Though discussed early on ([#threadgen]_, [#threadhof]_), including
  special objects in the stdlib for annotating generator functions and
  higher-order functions was ultim


id: pep-3113.rst
content:
PEP: 3113
Title: Removal of Tuple Parameter Unpacking
Author: Brett Cannon <brett@python.org>
Status: Final
Type: Standards Track
Created: 02-Mar-2007
Python-Version: 3.0
Post-History:


Abstract
========

Tuple parameter unpacking is the use of a tuple as a parameter in a
function signature so as to have a sequence argument automatically
unpacked.  An example is::

  def fxn(a, (b, c), d):
      pass

The use of ``(b, c)`` in the signature requires that the second
argument to the function be a sequence of length two (e.g.,
``[42, -13]``).  When such a sequence is passed it is unpacked and
has its values assigned to the parameters, just as if the statement
``b, c = [42, -13]`` had been executed in the parameter.

Unfortunately this feature of Python's rich function signature
abilities, while handy in some situations, causes more issues than
they are worth.  Thus this PEP proposes their removal from the
language in Python 3.0.


Why They Should Go
==================

Introspection Issues
--------------------

Python has very powerful introspection capabilities.  These extend to
function signatures.  There are no hidden details as to what a
function's call signature is.  In general it is fairly easy to figure
out various details about a function's signature by viewing the
function object and various attributes on it (including the function's
``func_code`` attribute).

But there is great difficulty when it comes to tuple parameters.  The
existence of a tuple parameter is denoted by its name being made of a
``.`` and a number in the ``co_varnames`` attribute of the function's
code object.  This allows the tuple argument to be bound to a name
that only the bytecode is aware of and cannot be typed in Python
source.  But this does not specify the format of the tuple: its
length, whether there are nested tuples, etc.

In order to get all of the details about the tuple from the function
one must analyse the bytecode of the function.  This is because the
first bytecode in the function literally translates into the tuple
argument being unpacked.  Assuming the tuple parameter is
named ``.1`` and is expected to unpack to variables ``spam`` and
``monty`` (meaning it is the tuple ``(spam, monty)``), the first
bytecode in the function will be for the statement
``spam, monty = .1``.  This means that to know all of the details of
the tuple parameter one must look at the initial bytecode of the
function to detect tuple unpacking for parameters formatted as
``\.\d+`` and deduce any and all information about the expected
argument.  Bytecode analysis is how the ``inspect.getargspec``
function is able to provide information on tuple parameters.  This is
not easy to do and is burdensome on introspection tools as they must
know how Python bytecode works (an otherwise unneeded burden as all
other types of parameters do not require knowledge of Python
bytecode).

The difficulty of analysing bytecode not withstanding, there is
another issue with the dependency on using Python bytecode.
IronPython [#ironpython]_ does not use Python's bytecode.  Because it
is based on the .NET framework it instead stores MSIL [#MSIL]_ in
``func_code.co_code`` attribute of the function.  This fact prevents
the ``inspect.getargspec`` function from working when run under
IronPython.  It is unknown whether other Python implementations are
affected but is reasonable to assume if the implementation is not just
a re-implementation of the Python virtual machine.


No Loss of Abilities If Removed
-------------------------------

As mentioned in `Introspection Issues`_, to handle tuple parameters
the function's bytecode starts with the bytecode required to unpack
the argument into the proper parameter names.  This means that there
is no special support required to implement tuple parameters and thus
there is no loss of abilities if they were to be removed, only a
possible convenience (which is addressed in
`Why They Should (Supposedly) Stay`_).

The example function at the beginning of this PEP could easily be
rewritten as::

 def fxn(a, b_c, d):
     b, c = b_c
     pass

and in no way lose functionality.


Exception To The Rule
---------------------

When looking at the various types of parameters that a Python function
can have, one will notice that tuple parameters tend to be an
exception rather than the rule.

Consider :pep:`3102` (keyword-only arguments) and :pep:`3107` (function
annotations).  Both PEPs have been accepted and
introduce new functionality within a function's signature.  And yet
for both PEPs the new feature cannot be applied to tuple parameters as
a whole.  :pep:`3102` has no support for tuple parameters at all (which
makes sense as there is no way to reference a tuple parameter by
name).  :pep:`3107` allows annotations for each item within the tuple
(e.g., ``(x:int, y:int)``), but not the whole tuple (e.g.,
``(x, y):int``).

The existence of tuple parameters also places sequence objects
separately from mapping objects in a function signature.  There is no
way to pass in a mapping object (e.g., a dict) as a parameter and have
it unpack in the same fashion as a sequence does into a tuple
parameter.


Uninformative Error Messages
----------------------------

Consider the following function::

 def fxn((a, b), (c, d)):
     pass

If called as ``fxn(1, (2, 3))`` one is given the error message
``TypeError: unpack non-sequence``.  This error message in no way
tells you which tuple was not unpacked properly.  There is also no
indication that this was a result that occurred because of the
arguments.  Other error messages regarding arguments to functions
explicitly state its relation to the signature:
``TypeError: fxn() takes exactly 2 arguments (0 given)``, etc.


Little Usage
------------

While an informal poll of the handful of Python programmers I know
personally and from the PyCon 2007 sprint indicates a huge majority of
people do not know of this feature and the rest just do not use it,
some hard numbers is needed to back up the claim that the feature is
not heavily used.

Iterating over every line in Python's code repository in the ``Lib/``
directory using the regular expression ``^\s*def\s*\w+\s*\(`` to
detect function and method definitions there were 22,252 matches in
the trunk.

Tacking on ``.*,\s*\(`` to find ``def`` statements that contained a
tuple parameter, only 41 matches were found.  This means that for
``def`` statements, only 0.18% of them seem to use a tuple parameter.


Why They Should (Supposedly) Stay
=================================

Practical Use
-------------

In certain instances tuple parameters can be useful.  A common example
is code that expects a two-item tuple that represents a Cartesian
point.  While true it is nice to be able to have the unpacking of the
x and y coordinates for you, the argument is that this small amount of
practical usefulness is heavily outweighed by other issues pertaining
to tuple parameters.  And as shown in
`No Loss Of Abilities If Removed`_, their use is purely practical and
in no way provide a unique ability that cannot be handled in other
ways very easily.


Self-Documentation For Parameters
---------------------------------

It has been argued that tuple parameters provide a way of
self-documentation for parameters that are expected to be of a certain
sequence format.  Using our Cartesian point example from
`Practical Use`_, seeing ``(x, y)`` as a parameter in a function makes
it obvious that a tuple of length two is expected as an argument for
that parameter.

But Python provides several other ways to document what parameters are
for.  Documentation strings are meant to provide enough information
needed to explain what arguments are expected.  Tuple parameters might
tell you the expected length of a sequence argument, it does not tell
you what that data will be used for.  One must also read the docstring
to know what other arguments are expected if not all parameters are
tuple parameters.

Function annotations (which do not w


id: pep-3136.rst
content:
PEP: 3136
Title: Labeled break and continue
Author: Matt Chisholm <matt-python@theory.org>
Status: Rejected
Type: Standards Track
Created: 30-Jun-2007
Python-Version: 3.1
Post-History:


Rejection Notice
================

This PEP is rejected.
See https://mail.python.org/pipermail/python-3000/2007-July/008663.html.



Abstract
========

This PEP proposes support for labels in Python's ``break`` and
``continue`` statements.  It is inspired by labeled ``break`` and
``continue`` in other languages, and the author's own infrequent but
persistent need for such a feature.


Introduction
============

The ``break`` statement allows the programmer to terminate a loop
early, and the ``continue`` statement allows the programmer to move to
the next iteration of a loop early.  In Python currently, ``break``
and ``continue`` can apply only to the innermost enclosing loop.

Adding support for labels to the ``break`` and ``continue`` statements
is a logical extension to the existing behavior of the ``break`` and
``continue`` statements.  Labeled ``break`` and ``continue`` can
improve the readability and flexibility of complex code which uses
nested loops.

For brevity's sake, the examples and discussion in this PEP usually
refers to the ``break`` statement.  However, all of the examples and
motivations apply equally to labeled ``continue``.


Motivation
==========

If the programmer wishes to move to the next iteration of an outer
enclosing loop, or terminate multiple loops at once, he or she has a
few less-than elegant options.

Here's one common way of imitating labeled ``break`` in Python (For
this and future examples, ``...`` denotes an arbitrary number of
intervening lines of code)::

    for a in a_list:
        time_to_break_out_of_a = False
        ...
        for b in b_list:
            ...
            if condition_one(a, b):
                break
            ...
            if condition_two(a, b):
                time_to_break_out_of_a = True
                break
            ...
        if time_to_break_out_of_a:
            break
        ...


This requires five lines and an extra variable,
``time_to_break_out_of_a``, to keep track of when to break out of the
outer (a) loop.  And those five lines are spread across many lines of
code, making the control flow difficult to understand.

This technique is also error-prone.  A programmer modifying this code
might inadvertently put new code after the end of the inner (b) loop
but before the test for ``time_to_break_out_of_a``, instead of after
the test. This means that code which should have been skipped by
breaking out of the outer loop gets executed incorrectly.

This could also be written with an exception.  The programmer would
declare a special exception, wrap the inner loop in a try, and catch
the exception and break when you see it::

    class BreakOutOfALoop(Exception): pass

    for a in a_list:
        ...
        try:
            for b in b_list:
                ...
                if condition_one(a, b):
                    break
                ...
                if condition_two(a, b):
                    raise BreakOutOfALoop
                ...
        except BreakOutOfALoop:
            break
        ...


Again, though; this requires five lines and a new, single-purpose
exception class (instead of a new variable), and spreads basic control
flow out over many lines.  And it breaks out of the inner loop with
``break`` and out of the other loop with an exception, which is
inelegant. [#toowtdi]_

This next strategy might be the most elegant solution, assuming
condition_two() is inexpensive to compute::

    for a in a_list:
        ...
        for b in b_list:
            ...
            if condition_one(a, b):
                break
            ...
            if condition_two(a, b):
                break
            ...
        if condition_two(a, b)
            break
        ...


Breaking twice is still inelegant.  This implementation also relies on
the fact that the inner (b) loop bleeds b into the outer for loop,
which (although explicitly supported) is both surprising to novices,
and in my opinion counter-intuitive and poor practice.

The programmer must also still remember to put in both breaks on
condition two and not insert code before the second break.  A single
conceptual action, breaking out of both loops on condition_two(),
requires four lines of code at two indentation levels, possibly
separated by many intervening lines at the end of the inner (b) loop.


Other languages
---------------

Now, put aside whatever dislike you may have for other programming
languages, and consider the syntax of labeled ``break`` and
``continue``.  In Perl::

    ALOOP: foreach $a (@a_array){
        ...
        BLOOP: foreach $b (@b_array){
            ...
            if (condition_one($a,$b)){
                last BLOOP; # same as plain old last;
            }
            ...
            if (condition_two($a,$b)){
                last ALOOP;
            }
            ...
        }
        ...
    }


(Notes: Perl uses ``last`` instead of ``break``.  The BLOOP labels
could be omitted; ``last`` and ``continue`` apply to the innermost
loop by default.)

PHP uses a number denoting the number of loops to break out of, rather
than a label::

    foreach ($a_array as $a){
        ....
        foreach ($b_array as $b){
            ....
            if (condition_one($a, $b)){
                break 1;  # same as plain old break
            }
            ....
            if (condition_two($a, $b)){
                break 2;
            }
            ....
        }
        ...
    }


C/C++, Java, and Ruby all have similar constructions.

The control flow regarding when to break out of the outer (a) loop is
fully encapsulated in the ``break`` statement which gets executed when
the break condition is satisfied.  The depth of the break statement
does not matter.  Control flow is not spread out.  No extra variables,
exceptions, or re-checking or storing of control conditions is
required.  There is no danger that code will get inadvertently
inserted after the end of the inner (b) loop and before the break
condition is re-checked inside the outer (a) loop.  These are the
benefits that labeled ``break`` and ``continue`` would bring to
Python.


What this PEP is not
====================

This PEP is not a proposal to add GOTO to Python.  GOTO allows a
programmer to jump to an arbitrary block or line of code, and
generally makes control flow more difficult to follow.  Although
``break`` and ``continue`` (with or without support for labels) can be
considered a type of GOTO, it is much more restricted.  Another Python
construct, ``yield``, could also be considered a form of GOTO -- an
even less restrictive one.  The goal of this PEP is to propose an
extension to the existing control flow tools ``break`` and
``continue``, to make control flow easier to understand, not more
difficult.

Labeled ``break`` and ``continue`` cannot transfer control to another
function or method.  They cannot even transfer control to an arbitrary
line of code in the current scope.  Currently, they can only affect
the behavior of a loop, and are quite different and much more
restricted than GOTO.  This extension allows them to affect any
enclosing loop in the current name-space, but it does not change their
behavior to that of GOTO.


Specification
=============

Under all of these proposals, ``break`` and ``continue`` by themselves
will continue to behave as they currently do, applying to the
innermost loop by default.


Proposal A - Explicit labels
----------------------------

The for and while loop syntax will be followed by an optional ``as``
or ``label`` (contextual) keyword [#keyword]_ and then an identifier,
which may be used to identify the loop out of which to break (or which
should be continued).

The ``break`` (and ``continue``) statements will be followed by an
optional identifier that refers to the loop out of which to break (or
w


id: pep-3122.rst
content:
PEP: 3122
Title: Delineation of the main module
Author: Brett Cannon
Status: Rejected
Type: Standards Track
Created: 27-Apr-2007
Post-History:

.. attention::
   This PEP has been rejected.  Guido views running scripts within a
   package as an anti-pattern [#guido-rejection]_.

Abstract
========

Because of how name resolution works for relative imports in a world
where :pep:`328` is implemented, the ability to execute modules within a
package ceases being possible.  This failing stems from the fact that
the module being executed as the "main" module replaces its
``__name__`` attribute with ``"__main__"`` instead of leaving it as
the absolute name of the module.  This breaks import's ability
to resolve relative imports from the main module into absolute names.

In order to resolve this issue, this PEP proposes to change how the
main module is delineated.  By leaving the ``__name__`` attribute in
a module alone and setting ``sys.main`` to the name of the main
module this will allow at least some instances of executing a module
within a package that uses relative imports.

This PEP does not address the idea of introducing a module-level
function that is automatically executed like :pep:`299` proposes.


The Problem
===========

With the introduction of :pep:`328`, relative imports became dependent on
the ``__name__`` attribute of the module performing the import.  This
is because the use of dots in a relative import are used to strip away
parts of the calling module's name to calculate where in the package
hierarchy an import should fall (prior to :pep:`328` relative
imports could fail and would fall back on absolute imports which had a
chance of succeeding).

For instance, consider the import ``from .. import spam`` made from the
``bacon.ham.beans`` module (``bacon.ham.beans`` is not a package
itself, i.e., does not define ``__path__``).  Name resolution of the
relative import takes the caller's name (``bacon.ham.beans``), splits
on dots, and then slices off the last n parts based on the level
(which is 2).  In this example both ``ham`` and ``beans`` are dropped
and ``spam`` is joined with what is left (``bacon``).  This leads to
the proper import of the module ``bacon.spam``.

This reliance on the ``__name__`` attribute of a module when handling
relative imports becomes an issue when executing a script within a
package.  Because the executing script has its name set to
``'__main__'``, import cannot resolve any relative imports, leading to
an ``ImportError``.

For example, assume we have a package named ``bacon`` with an
``__init__.py`` file containing::

 from . import spam

Also create a module named ``spam`` within the ``bacon`` package (it
can be an empty file).  Now if you try to execute the ``bacon``
package (either through ``python bacon/__init__.py`` or
``python -m bacon``) you will get an ``ImportError`` about trying to
do a relative import from within a non-package.  Obviously the import
is valid, but because of the setting of ``__name__`` to ``'__main__'``
import thinks that ``bacon/__init__.py`` is not in a package since no
dots exist in ``__name__``.  To see how the algorithm works in more
detail, see ``importlib.Import._resolve_name()`` in the sandbox
[#importlib]_.

Currently a work-around is to remove all relative imports in the
module being executed and make them absolute.  This is unfortunate,
though, as one should not be required to use a specific type of
resource in order to make a module in a package be able to be
executed.


The Solution
============

The solution to the problem is to not change the value of ``__name__``
in modules.  But there still needs to be a way to let executing code
know it is being executed as a script.  This is handled with a new
attribute in the ``sys`` module named ``main``.

When a module is being executed as a script, ``sys.main`` will be set
to the name of the module.  This changes the current idiom of::

 if __name__ == '__main__':
     ...

to::

 import sys
 if __name__ == sys.main:
     ...

The newly proposed solution does introduce an added line of
boilerplate which is a module import.  But as the solution does not
introduce a new built-in or module attribute (as discussed in
`Rejected Ideas`_) it has been deemed worth the extra line.

Another issue with the proposed solution (which also applies to all
rejected ideas as well) is that it does not directly solve the problem
of discovering the name of a file.  Consider ``python bacon/spam.py``.
By the file name alone it is not obvious whether ``bacon`` is a
package.  In order to properly find this out both the current
direction must exist on ``sys.path`` as well as ``bacon/__init__.py``
existing.

But this is the simple example.  Consider ``python ../spam.py``.  From
the file name alone it is not at all clear if ``spam.py`` is in a
package or not.  One possible solution is to find out what the
absolute name of ``..``, check if a file named ``__init__.py`` exists,
and then look if the directory is on ``sys.path``.  If it is not, then
continue to walk up the directory until no more ``__init__.py`` files
are found or the directory is found on ``sys.path``.

This could potentially be an expensive process.  If the package depth
happens to be deep then it could require a large amount of disk access
to discover where the package is anchored on ``sys.path``, if at all.
The stat calls alone can be expensive if the file system the executed
script is on is something like NFS.

Because of these issues, only when the ``-m`` command-line argument
(introduced by :pep:`338`) is used will ``__name__`` be set.  Otherwise
the fallback semantics of setting ``__name__`` to ``"__main__"`` will
occur.  ``sys.main`` will still be set to the proper value,
regardless of what ``__name__`` is set to.


Implementation
==============

When the ``-m`` option is used, ``sys.main`` will be set to the
argument passed in.  ``sys.argv`` will be adjusted as it is currently.
Then the equivalent of ``__import__(self.main)`` will occur.  This
differs from current semantics as the ``runpy`` module fetches the
code object for the file specified by the module name in order to
explicitly set ``__name__`` and other attributes.  This is no longer
needed as import can perform its normal operation in this situation.

If a file name is specified, then ``sys.main`` will be set to
``"__main__"``.  The specified file will then be read and have a code
object created and then be executed with ``__name__`` set to
``"__main__"``.  This mirrors current semantics.


Transition Plan
===============

In order for Python 2.6 to be able to support both the current
semantics and the proposed semantics, ``sys.main`` will always be set
to ``"__main__"``.  Otherwise no change will occur for Python 2.6.
This unfortunately means that no benefit from this change will occur
in Python 2.6, but it maximizes compatibility for code that is to
work as much as possible with 2.6 and 3.0.

To help transition to the new idiom, 2to3 [#2to3]_ will gain a rule to
transform the current ``if __name__ == '__main__': ...`` idiom to the
new one.  This will not help with code that checks ``__name__``
outside of the idiom, though.


Rejected Ideas
==============

``__main__`` built-in
---------------------

A counter-proposal to introduce a built-in named ``__main__``.
The value of the built-in would be the name of the module being
executed (just like the proposed ``sys.main``).  This would lead to a
new idiom of::

 if __name__ == __main__:
     ...

A drawback is that the syntactic difference is subtle; the dropping
of quotes around "__main__".  Some believe that for existing Python
programmers bugs will be introduced where the quotation marks will be
put on by accident.  But one could argue that the bug would be
discovered quickly through testing as it is a very shallow bug.

While the name of built-in could obviously be different (e.g.,
``main``) the other drawback is that it introduces a new built-in.
With a simple solution su


id: pep-3123.rst
content:
PEP: 3123
Title: Making PyObject_HEAD conform to standard C
Author: Martin von LÃ¶wis <martin@v.loewis.de>
Status: Final
Type: Standards Track
Created: 27-Apr-2007
Python-Version: 3.0
Post-History:

Abstract
========

Python currently relies on undefined C behavior, with its
usage of ``PyObject_HEAD``. This PEP proposes to change that
into standard C.

Rationale
=========

Standard C defines that an object must be accessed only through a
pointer of its type, and that all other accesses are undefined
behavior, with a few exceptions. In particular, the following
code has undefined behavior::

  struct FooObject{
    PyObject_HEAD
    int data;
  };

  PyObject *foo(struct FooObject*f){
   return (PyObject*)f;
  }

  int bar(){
   struct FooObject *f = malloc(sizeof(struct FooObject));
   struct PyObject *o = foo(f);
   f->ob_refcnt = 0;
   o->ob_refcnt = 1;
   return f->ob_refcnt;
  }

The problem here is that the storage is both accessed as
if it where struct ``PyObject``, and as struct ``FooObject``.

Historically, compilers did not have any problems with this
code. However, modern compilers use that clause as an
optimization opportunity, finding that ``f->ob_refcnt`` and
``o->ob_refcnt`` cannot possibly refer to the same memory, and
that therefore the function should return 0, without having
to fetch the value of ob_refcnt at all in the return
statement. For GCC, Python now uses ``-fno-strict-aliasing``
to work around that problem; with other compilers, it
may just see undefined behavior. Even with GCC, using
``-fno-strict-aliasing`` may pessimize the generated code
unnecessarily.

Specification
=============

Standard C has one specific exception to its aliasing rules precisely
designed to support the case of Python: a value of a struct type may
also be accessed through a pointer to the first field. E.g. if a
struct starts with an ``int``, the ``struct *`` may also be cast to
an ``int *``, allowing to write int values into the first field.

For Python, ``PyObject_HEAD`` and ``PyObject_VAR_HEAD`` will be changed
to not list all fields anymore, but list a single field of type
``PyObject``/``PyVarObject``::

  typedef struct _object {
    _PyObject_HEAD_EXTRA
    Py_ssize_t ob_refcnt;
    struct _typeobject *ob_type;
  } PyObject;

  typedef struct {
    PyObject ob_base;
    Py_ssize_t ob_size;
  } PyVarObject;

  #define PyObject_HEAD        PyObject ob_base;
  #define PyObject_VAR_HEAD    PyVarObject ob_base;

Types defined as fixed-size structure will then include PyObject
as its first field, PyVarObject for variable-sized objects. E.g.::

  typedef struct {
    PyObject ob_base;
    PyObject *start, *stop, *step;
  } PySliceObject;

  typedef struct {
    PyVarObject ob_base;
    PyObject **ob_item;
    Py_ssize_t allocated;
  } PyListObject;

The above definitions of ``PyObject_HEAD`` are normative, so extension
authors MAY either use the macro, or put the ``ob_base`` field explicitly
into their structs.

As a convention, the base field SHOULD be called ob_base. However, all
accesses to ob_refcnt and ob_type MUST cast the object pointer to
PyObject* (unless the pointer is already known to have that type), and
SHOULD use the respective accessor macros. To simplify access to
ob_type, ob_refcnt, and ob_size, macros::

  #define Py_TYPE(o)    (((PyObject*)(o))->ob_type)
  #define Py_REFCNT(o)  (((PyObject*)(o))->ob_refcnt)
  #define Py_SIZE(o)    (((PyVarObject*)(o))->ob_size)

are added. E.g. the code blocks ::

  #define PyList_CheckExact(op) ((op)->ob_type == &PyList_Type)

  return func->ob_type->tp_name;

needs to be changed to::

  #define PyList_CheckExact(op) (Py_TYPE(op) == &PyList_Type)

  return Py_TYPE(func)->tp_name;

For initialization of type objects, the current sequence ::

  PyObject_HEAD_INIT(NULL)
  0, /* ob_size */

becomes incorrect, and must be replaced with ::

  PyVarObject_HEAD_INIT(NULL, 0)

Compatibility with Python 2.6
=============================

To support modules that compile with both Python 2.6 and Python 3.0,
the ``Py_*`` macros are added to Python 2.6. The macros ``Py_INCREF``
and ``Py_DECREF`` will be changed to cast their argument to ``PyObject *``,
so that module authors can also explicitly declare the ``ob_base``
field in modules designed for Python 2.6.

Copyright
=========

This document has been placed in the public domain.


id: pep-3137.rst
content:
PEP: 3137
Title: Immutable Bytes and Mutable Buffer
Author: Guido van Rossum <guido@python.org>
Status: Final
Type: Standards Track
Created: 26-Sep-2007
Python-Version: 3.0
Post-History: 26-Sep-2007, 30-Sep-2007

Introduction
============

After releasing Python 3.0a1 with a mutable bytes type, pressure
mounted to add a way to represent immutable bytes.  Gregory P. Smith
proposed a patch that would allow making a bytes object temporarily
immutable by requesting that the data be locked using the new buffer
API from :pep:`3118`.  This did not seem the right approach to me.

Jeffrey Yasskin, with the help of Adam Hupp, then prepared a patch to
make the bytes type immutable (by crudely removing all mutating APIs)
and fix the fall-out in the test suite.  This showed that there aren't
all that many places that depend on the mutability of bytes, with the
exception of code that builds up a return value from small pieces.

Thinking through the consequences, and noticing that using the array
module as an ersatz mutable bytes type is far from ideal, and
recalling a proposal put forward earlier by Talin, I floated the
suggestion to have both a mutable and an immutable bytes type.  (This
had been brought up before, but until seeing the evidence of Jeffrey's
patch I wasn't open to the suggestion.)

Moreover, a possible implementation strategy became clear: use the old
PyString implementation, stripped down to remove locale support and
implicit conversions to/from Unicode, for the immutable bytes type,
and keep the new PyBytes implementation as the mutable bytes type.

The ensuing discussion made it clear that the idea is welcome but
needs to be specified more precisely.  Hence this PEP.

Advantages
==========

One advantage of having an immutable bytes type is that code objects
can use these.  It also makes it possible to efficiently create hash
tables using bytes for keys; this may be useful when parsing protocols
like HTTP or SMTP which are based on bytes representing text.

Porting code that manipulates binary data (or encoded text) in Python
2.x will be easier using the new design than using the original 3.0
design with mutable bytes; simply replace ``str`` with ``bytes`` and
change '...' literals into b'...' literals.

Naming
======

I propose the following type names at the Python level:

- ``bytes`` is an immutable array of bytes (PyString)

- ``bytearray`` is a mutable array of bytes (PyBytes)

- ``memoryview`` is a bytes view on another object (PyMemory)

The old type named ``buffer`` is so similar to the new type
``memoryview``, introduce by :pep:`3118`, that it is redundant.  The rest
of this PEP doesn't discuss the functionality of ``memoryview``; it is
just mentioned here to justify getting rid of the old ``buffer`` type.
(An earlier version of this PEP proposed ``buffer`` as the new name
for PyBytes; in the end this name was deemed to confusing given the
many other uses of the word buffer.)

While eventually it makes sense to change the C API names, this PEP
maintains the old C API names, which should be familiar to all.

Summary
-------

Here's a simple ASCII-art table summarizing the type names in various
Python versions::

    +--------------+-------------+------------+--------------------------+
    | C name       | 2.x    repr | 3.0a1 repr | 3.0a2               repr |
    +--------------+-------------+------------+--------------------------+
    | PyUnicode    | unicode u'' | str     '' | str                   '' |
    | PyString     | str      '' | str8   s'' | bytes                b'' |
    | PyBytes      | N/A         | bytes  b'' | bytearray bytearray(b'') |
    | PyBuffer     | buffer      | buffer     | N/A                      |
    | PyMemoryView | N/A         | memoryview | memoryview         <...> |
    +--------------+-------------+------------+--------------------------+

Literal Notations
=================

The b'...' notation introduced in Python 3.0a1 returns an immutable
bytes object, whatever variation is used.  To create a mutable array
of bytes, use bytearray(b'...') or bytearray([...]).  The latter form
takes a list of integers in range(256).

Functionality
=============

PEP 3118 Buffer API
-------------------

Both bytes and bytearray implement the :pep:`3118` buffer API.  The bytes
type only implements read-only requests; the bytearray type allows
writable and data-locked requests as well.  The element data type is
always 'B' (i.e. unsigned byte).

Constructors
------------

There are four forms of constructors, applicable to both bytes and
bytearray:

- ``bytes(<bytes>)``, ``bytes(<bytearray>)``, ``bytearray(<bytes>)``,
  ``bytearray(<bytearray>)``: simple copying constructors, with the
  note that ``bytes(<bytes>)`` might return its (immutable)
  argument, but ``bytearray(<bytearray>)`` always makes a copy.

- ``bytes(<str>, <encoding>[, <errors>])``, ``bytearray(<str>,
  <encoding>[, <errors>])``: encode a text string.  Note that the
  ``str.encode()`` method returns an *immutable* bytes object.  The
  <encoding> argument is mandatory; <errors> is optional.
  <encoding> and <errors>, if given, must be ``str`` instances.

- ``bytes(<memory view>)``, ``bytearray(<memory view>)``: construct
  a bytes or bytearray object from anything that implements the PEP
  3118 buffer API.

- ``bytes(<iterable of ints>)``, ``bytearray(<iterable of ints>)``:
  construct a bytes or bytearray object from a stream of integers in
  range(256).

- ``bytes(<int>)``, ``bytearray(<int>)``: construct a
  zero-initialized bytes or bytearray object of a given length.

Comparisons
-----------

The bytes and bytearray types are comparable with each other and
orderable, so that e.g. b'abc' == bytearray(b'abc') < b'abd'.

Comparing either type to a str object for equality returns False
regardless of the contents of either operand.  Ordering comparisons
with str raise TypeError.  This is all conformant to the standard
rules for comparison and ordering between objects of incompatible
types.

(**Note:** in Python 3.0a1, comparing a bytes instance with a str
instance would raise TypeError, on the premise that this would catch
the occasional mistake quicker, especially in code ported from Python
2.x.  However, a long discussion on the python-3000 list pointed out
so many problems with this that it is clearly a bad idea, to be rolled
back in 3.0a2 regardless of the fate of the rest of this PEP.)

Slicing
-------

Slicing a bytes object returns a bytes object.  Slicing a bytearray
object returns a bytearray object.

Slice assignment to a bytearray object accepts anything that
implements the :pep:`3118` buffer API, or an iterable of integers in
range(256).

Indexing
--------

Indexing bytes and bytearray returns small ints (like the bytes type in
3.0a1, and like lists or array.array('B')).

Assignment to an item of a bytearray object accepts an int in
range(256).  (To assign from a bytes sequence, use a slice
assignment.)

Str() and Repr()
----------------

The str() and repr() functions return the same thing for these
objects.  The repr() of a bytes object returns a b'...' style literal.
The repr() of a bytearray returns a string of the form "bytearray(b'...')".

Operators
---------

The following operators are implemented by the bytes and bytearray
types, except where mentioned:

- ``b1 + b2``: concatenation.  With mixed bytes/bytearray operands,
  the return type is that of the first argument (this seems arbitrary
  until you consider how ``+=`` works).

- ``b1 += b2``: mutates b1 if it is a bytearray object.

- ``b * n``, ``n * b``: repetition; n must be an integer.

- ``b *= n``: mutates b if it is a bytearray object.

- ``b1 in b2``, ``b1 not in b2``: substring test; b1 can be any
  object implementing the :pep:`3118` buffer API.

- ``i in b``, ``i not in b``: single-byte membership test; i must
  be an integer (if it is a length-1 bytes array, it is considered
  to be a substring test, with the same outcome).

- ``len(b)``: the number of bytes.

- ``h


id: pep-3109.rst
content:
PEP: 3109
Title: Raising Exceptions in Python 3000
Author: Collin Winter <collinwinter@google.com>
Status: Final
Type: Standards Track
Created: 19-Jan-2006
Python-Version: 3.0
Post-History:


Abstract
========

This PEP introduces changes to Python's mechanisms for raising
exceptions intended to reduce both line noise and the size of the
language.


Rationale
=========

One of Python's guiding maxims is :pep:`"there should be one -- and
preferably only one -- obvious way to do it" <20>`. Python 2.x's
``raise`` statement violates this principle, permitting multiple
ways of expressing the same thought. For example, these statements
are equivalent: ::

    raise E, V

    raise E(V)

There is a third form of the ``raise`` statement, allowing arbitrary
tracebacks to be attached to an exception [#grammar]_: ::

    raise E, V, T

where T is a traceback. As specified in :pep:`344`,
exception objects in Python 3.x will possess a ``__traceback__``
attribute, admitting this translation of the three-expression
``raise`` statement: ::

    raise E, V, T

is translated to ::

    e = E(V)
    e.__traceback__ = T
    raise e

Using these translations, we can reduce the ``raise`` statement from
four forms to two:

1. ``raise`` (with no arguments) is used to re-raise the active
   exception in an ``except`` suite.

2. ``raise EXCEPTION`` is used to raise a new exception. This form has
   two sub-variants: ``EXCEPTION`` may be an exception class or an
   instance of an exception class; valid exception classes are
   BaseException and its subclasses (:pep:`352`). If ``EXCEPTION``
   is a subclass, it will be called with no arguments to obtain
   an exception instance.

   To raise anything else is an error.

There is a further, more tangible benefit to be obtained through this
consolidation, as noted by A.M. Kuchling [#amk-line-noise]_. ::

    PEP 8 doesn't express any preference between the
    two forms of raise statements:
    raise ValueError, 'blah'
    raise ValueError("blah")

    I like the second form better, because if the exception arguments
    are long or include string formatting, you don't need to use line
    continuation characters because of the containing parens.

The BDFL has concurred [#guido-declaration]_ and endorsed the
consolidation of the several ``raise`` forms.


Grammar Changes
===============

In Python 3, the grammar for ``raise`` statements will change
from [#grammar]_ ::

    raise_stmt: 'raise' [test [',' test [',' test]]]

to ::

    raise_stmt: 'raise' [test]


Changes to Builtin Types
========================

Because of its relation to exception raising, the signature for the
``throw()`` method on generator objects will change, dropping the
optional second and third parameters. The signature thus changes (:pep:`342`)
from ::

    generator.throw(E, [V, [T]])

to ::

    generator.throw(EXCEPTION)

Where ``EXCEPTION`` is either a subclass of ``BaseException`` or an
instance of a subclass of ``BaseException``.


Semantic Changes
================

In Python 2, the following ``raise`` statement is legal ::

    raise ((E1, (E2, E3)), E4), V

The interpreter will take the tuple's first element as the exception
type (recursively), making the above fully equivalent to ::

    raise E1, V

As of Python 3.0, support for raising tuples like this will be
dropped. This change will bring ``raise`` statements into line with
the ``throw()`` method on generator objects, which already disallows
this.


Compatibility Issues
====================

All two- and three-expression ``raise`` statements will require
modification, as will all two- and three-expression ``throw()`` calls
on generators. Fortunately, the translation from Python 2.x to
Python 3.x in this case is simple and can be handled mechanically
by Guido van Rossum's 2to3 utility [#2to3]_ using the ``raise`` and
``throw`` fixers ([#raise-fixer]_, [#throw-fixer]_).

The following translations will be performed:

1. Zero- and one-expression ``raise`` statements will be left
   intact.

2. Two-expression ``raise`` statements will be converted from ::

        raise E, V

   to ::

        raise E(V)

   Two-expression ``throw()`` calls will be converted from ::

        generator.throw(E, V)

   to ::

        generator.throw(E(V))

   See point #5 for a caveat to this transformation.

3. Three-expression ``raise`` statements will be converted from ::

        raise E, V, T

   to ::

        e = E(V)
        e.__traceback__ = T
        raise e

   Three-expression ``throw()`` calls will be converted from ::

        generator.throw(E, V, T)

   to ::

        e = E(V)
        e.__traceback__ = T
        generator.throw(e)

   See point #5 for a caveat to this transformation.

4. Two- and three-expression ``raise`` statements where ``E`` is a
   tuple literal can be converted automatically using ``2to3``'s
   ``raise`` fixer. ``raise`` statements where ``E`` is a non-literal
   tuple, e.g., the result of a function call, will need to be
   converted manually.

5. Two- and three-expression ``raise`` statements where ``E`` is an
   exception class and ``V`` is an exception instance will need
   special attention. These cases break down into two camps:

   1. ``raise E, V`` as a long-hand version of the zero-argument
      ``raise`` statement. As an example, assuming F is a subclass
      of E ::

          try:
              something()
          except F as V:
              raise F(V)
          except E as V:
              handle(V)

      This would be better expressed as ::

          try:
              something()
          except F:
              raise
          except E as V:
              handle(V)

   2. ``raise E, V`` as a way of "casting" an exception to another
      class. Taking an example from
      distutils.compiler.unixcompiler ::

           try:
               self.spawn(pp_args)
           except DistutilsExecError as msg:
               raise CompileError(msg)

      This would be better expressed as ::

           try:
               self.spawn(pp_args)
           except DistutilsExecError as msg:
               raise CompileError from msg

      Using the ``raise ... from ...`` syntax introduced in
      :pep:`344`.


Implementation
==============

This PEP was implemented in revision 57783 [#r57783]_.


References
==========

.. [#grammar]
   http://docs.python.org/reference/simple_stmts.html#raise

.. [#amk-line-noise]
   https://mail.python.org/pipermail/python-dev/2005-August/055187.html

.. [#guido-declaration]
   https://mail.python.org/pipermail/python-dev/2005-August/055190.html

.. [#2to3]
   http://svn.python.org/view/sandbox/trunk/2to3/

.. [#raise-fixer]
   http://svn.python.org/view/sandbox/trunk/2to3/fixes/fix_raise.py

.. [#throw-fixer]
   http://svn.python.org/view/sandbox/trunk/2to3/fixes/fix_throw.py

.. [#r57783]
   http://svn.python.org/view/python/branches/py3k/Include/?rev=57783&view=rev


Copyright
=========

This document has been placed in the public domain.


id: pep-3121.rst
content:
PEP: 3121
Title: Extension Module Initialization and Finalization
Author: Martin von LÃ¶wis <martin@v.loewis.de>
Status: Final
Type: Standards Track
Created: 27-Apr-2007
Python-Version: 3.0
Post-History:

.. canonical-doc:: :external+python:c:func:`PyInit_modulename` and
                   :external+python:c:type:`PyModuleDef`

Abstract
========

Extension module initialization currently has a few deficiencies.
There is no cleanup for modules, the entry point name might give
naming conflicts, the entry functions don't follow the usual calling
convention, and multiple interpreters are not supported well. This PEP
addresses these issues.

Problems
========

Module Finalization
-------------------

Currently, extension modules are initialized usually once and then
"live" forever. The only exception is when Py_Finalize() is called:
then the initialization routine is invoked a second time. This is bad
from a resource management point of view: memory and other resources
might get allocated each time initialization is called, but there is
no way to reclaim them. As a result, there is currently no way to
completely release all resources Python has allocated.

Entry point name conflicts
--------------------------

The entry point is currently called init<module>. This might conflict
with other symbols also called init<something>. In particular,
initsocket is known to have conflicted in the past (this specific
problem got resolved as a side effect of renaming the module to
_socket).

Entry point signature
---------------------

The entry point is currently a procedure (returning void).  This
deviates from the usual calling conventions; callers can find out
whether there was an error during initialization only by checking
PyErr_Occurred. The entry point should return a PyObject*, which will
be the module created, or NULL in case of an exception.

Multiple Interpreters
---------------------

Currently, extension modules share their state across all
interpreters. This allows for undesirable information leakage across
interpreters: one script could permanently corrupt objects in an
extension module, possibly breaking all scripts in other interpreters.

Specification
=============

The module initialization routines change their signature
to::

  PyObject *PyInit_<modulename>()

The initialization routine will be invoked once per
interpreter, when the module is imported. It should
return a new module object each time.

In order to store per-module state in C variables,
each module object will contain a block of memory
that is interpreted only by the module. The amount
of memory used for the module is specified at
the point of creation of the module.

In addition to the initialization function, a module
may implement a number of additional callback
functions, which are invoked when the module's
tp_traverse, tp_clear, and tp_free functions are
invoked, and when the module is reloaded.

The entire module definition is combined in a struct
PyModuleDef::

  struct PyModuleDef{
    PyModuleDef_Base m_base;  /* To be filled out by the interpreter */
    Py_ssize_t m_size; /* Size of per-module data */
    PyMethodDef *m_methods;
    inquiry m_reload;
    traverseproc m_traverse;
    inquiry m_clear;
    freefunc m_free;
  };

Creation of a module is changed to expect an optional
PyModuleDef*. The module state will be
null-initialized.

Each module method will be passed the module object
as the first parameter. To access the module data,
a function::

  void* PyModule_GetState(PyObject*);

will be provided. In addition, to lookup a module
more efficiently than going through sys.modules,
a function::

  PyObject* PyState_FindModule(struct PyModuleDef*);

will be provided. This lookup function will use an
index located in the m_base field, to find the
module by index, not by name.

As all Python objects should be controlled through
the Python memory management, usage of "static"
type objects is discouraged, unless the type object
itself has no memory-managed state. To simplify
definition of heap types, a new method::

  PyTypeObject* PyType_Copy(PyTypeObject*);

is added.

Example
=======

xxmodule.c would be changed to remove the initxx
function, and add the following code instead::

  struct xxstate{
    PyObject *ErrorObject;
    PyObject *Xxo_Type;
  };

  #define xxstate(o) ((struct xxstate*)PyModule_GetState(o))

  static int xx_traverse(PyObject *m, visitproc v,
                         void *arg)
  {
    Py_VISIT(xxstate(m)->ErrorObject);
    Py_VISIT(xxstate(m)->Xxo_Type);
    return 0;
  }

  static int xx_clear(PyObject *m)
  {
    Py_CLEAR(xxstate(m)->ErrorObject);
    Py_CLEAR(xxstate(m)->Xxo_Type);
    return 0;
  }

  static struct PyModuleDef xxmodule = {
    {}, /* m_base */
    sizeof(struct xxstate),
    &xx_methods,
    0,  /* m_reload */
    xx_traverse,
    xx_clear,
    0,  /* m_free - not needed, since all is done in m_clear */
  }

  PyObject*
  PyInit_xx()
  {
    PyObject *res = PyModule_New("xx", &xxmodule);
    if (!res) return NULL;
    xxstate(res)->ErrorObject = PyErr_NewException("xx.error", NULL, NULL);
    if (!xxstate(res)->ErrorObject) {
      Py_DECREF(res);
      return NULL;
    }
    xxstate(res)->XxoType = PyType_Copy(&Xxo_Type);
    if (!xxstate(res)->Xxo_Type) {
      Py_DECREF(res);
      return NULL;
    }
    return res;
  }


Discussion
==========

Tim Peters reports in [1]_ that PythonLabs considered such a feature
at one point, and lists the following additional hooks which aren't
currently supported in this PEP:

* when the module object is deleted from sys.modules

* when Py_Finalize is called

* when Python exits

* when the Python DLL is unloaded (Windows only)


References
==========

.. [1] Tim Peters, reporting earlier conversation about such a feature
   https://mail.python.org/pipermail/python-3000/2006-April/000726.html


Copyright
=========

This document has been placed in the public domain.


id: pep-3135.rst
content:
PEP: 3135
Title: New Super
Author: Calvin Spealman <ironfroggy@gmail.com>,
        Tim Delaney <timothy.c.delaney@gmail.com>,
        Lie Ryan <lie.1296@gmail.com>
Status: Final
Type: Standards Track
Created: 28-Apr-2007
Python-Version: 3.0
Post-History: `28-Apr-2007 <https://mail.python.org/pipermail/python-dev/2007-April/072807.html>`__,
              `29-Apr-2007 <https://mail.python.org/pipermail/python-dev/2007-April/072835.html>`__,
              `29-Apr-2007 <https://mail.python.org/pipermail/python-dev/2007-April/072858.html>`__,
              `14-May-2007 <https://mail.python.org/pipermail/python-dev/2007-May/073127.html>`__,
              `12-Mar-2009 <https://mail.python.org/pipermail/python-bugs-list/2009-March/072665.html>`__


Numbering Note
==============

This PEP started its life as :pep:`367`.  Since it is now targeted
for Python 3000, it has been moved into the 3xxx space.


Abstract
========

This PEP proposes syntactic sugar for use of the ``super`` type to automatically
construct instances of the super type binding to the class that a method was
defined in, and the instance (or class object for classmethods) that the method
is currently acting upon.

The premise of the new super usage suggested is as follows::

    super().foo(1, 2)

to replace the old::

    super(Foo, self).foo(1, 2)


Rationale
=========

The current usage of super requires an explicit passing of both the class and
instance it must operate from, requiring a breaking of the DRY (Don't Repeat
Yourself) rule. This hinders any change in class name, and is often considered
a wart by many.


Specification
=============

Within the specification section, some special terminology will be used to
distinguish similar and closely related concepts. "super class" will refer to
the actual builtin class named "super". A "super instance" is simply an
instance of the super class, which is associated with another class and
possibly with an instance of that class.

The new ``super`` semantics are only available in Python 3.0.

Replacing the old usage of super, calls to the next class in the MRO (method
resolution order) can be made without explicitly passing the class object
(although doing so will still be supported). Every function
will have a cell named ``__class__`` that contains the class object that the
function is defined in.

The new syntax::

    super()

is equivalent to::

    super(__class__, <firstarg>)

where ``__class__`` is the class that the method was defined in, and
``<firstarg>`` is the first parameter of the method (normally ``self``
for instance methods, and ``cls`` for class methods). For functions
defined outside a class body, ``__class__`` is not defined, and will
result in runtime ``SystemError``.

While ``super`` is not a reserved word, the parser recognizes the use
of ``super`` in a method definition and only passes in the
``__class__`` cell when this is found.  Thus, calling a global alias
of ``super`` without arguments will not necessarily work.


Closed Issues
=============

Determining the class object to use
-----------------------------------

The class object is taken from a cell named ``__class__``.


Should ``super`` actually become a keyword?
-------------------------------------------

No. It is not necessary for super to become a keyword.

super used with __call__ attributes
-----------------------------------

It was considered that it might be a problem that instantiating super instances
the classic way, because calling it would lookup the __call__ attribute and
thus try to perform an automatic super lookup to the next class in the MRO.
However, this was found to be false, because calling an object only looks up
the __call__ method directly on the object's type. The following example shows
this in action.

.. code:: python

    class A(object):
        def __call__(self):
            return '__call__'
        def __getattribute__(self, attr):
            if attr == '__call__':
                return lambda: '__getattribute__'
    a = A()
    assert a() == '__call__'
    assert a.__call__() == '__getattribute__'

In any case, this issue goes away entirely because classic calls to
``super(<class>, <instance>)`` are still supported with the same meaning.


Alternative Proposals
=====================

No Changes
----------

Although its always attractive to just keep things how they are, people have
sought a change in the usage of super calling for some time, and for good
reason, all mentioned previously.

- Decoupling from the class name (which might not even be bound to the
  right class anymore!)
- Simpler looking, cleaner super calls would be better

Dynamic attribute on super type
-------------------------------

The proposal adds a dynamic attribute lookup to the super type, which will
automatically determine the proper class and instance parameters. Each super
attribute lookup identifies these parameters and performs the super lookup on
the instance, as the current super implementation does with the explicit
invocation of a super instance upon a class and instance.

This proposal relies on sys._getframe(), which is not appropriate for anything
except a prototype implementation.

self.__super__.foo(\*args)
--------------------------

The __super__ attribute is mentioned in this PEP in several places, and could
be a candidate for the complete solution, actually using it explicitly instead
of any super usage directly. However, double-underscore names are usually an
internal detail, and attempted to be kept out of everyday code.

super(self, \*args) or __super__(self, \*args)
----------------------------------------------

This solution only solves the problem of the type indication, does not handle
differently named super methods, and is explicit about the name of the
instance. It is less flexible without being able to enacted on other method
names, in cases where that is needed. One use case this fails is where a
base-class has a factory classmethod and a subclass has two factory
classmethods,both of which needing to properly make super calls to the one
in the base-class.

super.foo(self, \*args)
-----------------------

This variation actually eliminates the problems with locating the proper
instance, and if any of the alternatives were pushed into the spotlight, I
would want it to be this one.

super(\*p, \*\*kw)
------------------

There has been the proposal that directly calling ``super(*p, **kw)`` would
be equivalent to calling the method on the ``super`` object with the same name
as the method currently being executed i.e. the following two methods would be
equivalent:

.. code:: python

    def f(self, *p, **kw):
        super.f(*p, **kw)

.. code:: python

    def f(self, *p, **kw):
        super(*p, **kw)

There is strong sentiment for and against this, but implementation and style
concerns are obvious. Guido has suggested that this should be excluded from
this PEP on the principle of KISS (Keep It Simple Stupid).


History
=======

29-Apr-2007
  - Changed title from "Super As A Keyword" to "New Super"
  - Updated much of the language and added a terminology section
    for clarification in confusing places.
  - Added reference implementation and history sections.

06-May-2007
  - Updated by Tim Delaney to reflect discussions on the python-3000
    and python-dev mailing lists.

12-Mar-2009
  - Updated to reflect the current state of implementation.


References
==========

[1] Fixing super anyone?
\   (https://mail.python.org/pipermail/python-3000/2007-April/006667.html)

[2] PEP 3130: Access to Module/Class/Function Currently Being Defined (this)
\   (https://mail.python.org/pipermail/python-ideas/2007-April/000542.html)


Copyright
=========

This document has been placed in the public domain.


id: pep-3134.rst
content:
PEP: 3134
Title: Exception Chaining and Embedded Tracebacks
Author: Ka-Ping Yee
Status: Final
Type: Standards Track
Created: 12-May-2005
Python-Version: 3.0
Post-History:


Numbering Note
==============

This PEP started its life as :pep:`344`.  Since it is now targeted for Python
3000, it has been moved into the 3xxx space.


Abstract
========

This PEP proposes three standard attributes on exception instances: the
``__context__`` attribute for implicitly chained exceptions, the ``__cause__``
attribute for explicitly chained exceptions, and the ``__traceback__``
attribute for the traceback.  A new ``raise ... from`` statement sets the
``__cause__`` attribute.


Motivation
==========

During the handling of one exception (exception A), it is possible that another
exception (exception B) may occur.  In today's Python (version 2.4), if this
happens, exception B is propagated outward and exception A is lost.  In order
to debug the problem, it is useful to know about both exceptions.  The
``__context__`` attribute retains this information automatically.

Sometimes it can be useful for an exception handler to intentionally re-raise
an exception, either to provide extra information or to translate an exception
to another type.  The ``__cause__`` attribute provides an explicit way to
record the direct cause of an exception.

In today's Python implementation, exceptions are composed of three parts: the
type, the value, and the traceback.  The ``sys`` module, exposes the current
exception in three parallel variables, ``exc_type``, ``exc_value``, and
``exc_traceback``, the ``sys.exc_info()`` function returns a tuple of these
three parts, and the ``raise`` statement has a three-argument form accepting
these three parts.  Manipulating exceptions often requires passing these three
things in parallel, which can be tedious and error-prone.  Additionally, the
``except`` statement can only provide access to the value, not the traceback.
Adding the ``__traceback__`` attribute to exception values makes all the
exception information accessible from a single place.


History
=======

Raymond Hettinger [1]_ raised the issue of masked exceptions on Python-Dev in
January 2003 and proposed a ``PyErr_FormatAppend()`` function that C modules
could use to augment the currently active exception with more information.
Brett Cannon [2]_ brought up chained exceptions again in June 2003, prompting
a long discussion.

Greg Ewing [3]_ identified the case of an exception occurring in a ``finally``
block during unwinding triggered by an original exception, as distinct from
the case of an exception occurring in an ``except`` block that is handling the
original exception.

Greg Ewing [4]_ and Guido van Rossum [5]_, and probably others, have
previously mentioned adding a traceback attribute to Exception instances.
This is noted in :pep:`3000`.

This PEP was motivated by yet another recent Python-Dev reposting of the same
ideas [6]_ [7]_.


Rationale
=========

The Python-Dev discussions revealed interest in exception chaining for two
quite different purposes.  To handle the unexpected raising of a secondary
exception, the exception must be retained implicitly. To support intentional
translation of an exception, there must be a way to chain exceptions
explicitly.  This PEP addresses both.

Several attribute names for chained exceptions have been suggested on
Python-Dev [2]_, including ``cause``, ``antecedent``, ``reason``, ``original``,
``chain``, ``chainedexc``, ``exc_chain``, ``excprev``, ``previous``, and
``precursor``.  For an explicitly chained exception, this PEP suggests
``__cause__`` because of its specific meaning.  For an implicitly chained
exception, this PEP proposes the name ``__context__`` because the intended
meaning is more specific than temporal precedence but less specific than
causation: an exception occurs in the context of handling another exception.

This PEP suggests names with leading and trailing double-underscores for these
three attributes because they are set by the Python VM. Only in very special
cases should they be set by normal assignment.

This PEP handles exceptions that occur during ``except`` blocks and ``finally``
blocks in the same way.  Reading the traceback makes it clear where the
exceptions occurred, so additional mechanisms for distinguishing the two cases
would only add unnecessary complexity.

This PEP proposes that the outermost exception object (the one exposed for
matching by ``except`` clauses) be the most recently raised exception for
compatibility with current behaviour.

This PEP proposes that tracebacks display the outermost exception last, because
this would be consistent with the chronological order of tracebacks (from
oldest to most recent frame) and because the actual thrown exception is easier
to find on the last line.

To keep things simpler, the C API calls for setting an exception will not
automatically set the exception's ``__context__``.  Guido van Rossum has
expressed concerns with making such changes [8]_.

As for other languages, Java and Ruby both discard the original exception when
another exception occurs in a ``catch``/``rescue`` or ``finally``/``ensure``
clause.  Perl 5 lacks built-in structured exception handling.  For Perl 6, RFC
number 88 [9]_ proposes an exception mechanism that implicitly retains chained
exceptions in an array named ``@@``.  In that RFC, the most recently raised
exception is exposed for matching, as in this PEP; also, arbitrary expressions
(possibly involving ``@@``) can be evaluated for exception matching.

Exceptions in C# contain a read-only ``InnerException`` property that may point
to another exception.  Its documentation [10]_ says that "When an exception X
is thrown as a direct result of a previous exception Y, the ``InnerException``
property of X should contain a reference to Y."  This property is not set by
the VM automatically; rather, all exception constructors take an optional
``innerException`` argument to set it explicitly.  The ``__cause__`` attribute
fulfills the same purpose as ``InnerException``, but this PEP proposes a new
form of ``raise`` rather than extending the constructors of all exceptions. C#
also provides a ``GetBaseException`` method that jumps directly to the end of
the ``InnerException`` chain; this PEP proposes no analog.

The reason all three of these attributes are presented together in one proposal
is that the ``__traceback__`` attribute provides convenient access to the
traceback on chained exceptions.


Implicit Exception Chaining
===========================

Here is an example to illustrate the ``__context__`` attribute::

    def compute(a, b):
        try:
            a/b
        except Exception, exc:
            log(exc)

    def log(exc):
        file = open('logfile.txt')  # oops, forgot the 'w'
        print >>file, exc
        file.close()

Calling ``compute(0, 0)`` causes a ``ZeroDivisionError``.  The ``compute()``
function catches this exception and calls ``log(exc)``, but the ``log()``
function also raises an exception when it tries to write to a file that wasn't
opened for writing.

In today's Python, the caller of ``compute()`` gets thrown an ``IOError``. The
``ZeroDivisionError`` is lost.  With the proposed change, the instance of
``IOError`` has an additional ``__context__`` attribute that retains the
``ZeroDivisionError``.

The following more elaborate example demonstrates the handling of a mixture of
``finally`` and ``except`` clauses::

    def main(filename):
        file = open(filename)       # oops, forgot the 'w'
        try:
            try:
                compute()
            except Exception, exc:
                log(file, exc)
        finally:
            file.clos()             # oops, misspelled 'close'

    def compute():
        1/0

    def log(file, exc):
        try:
            print >>file, exc       # oops, file is not writable
        except:
            display(exc)

    def display(exc):
        print ex               


id: pep-3120.rst
content:
PEP: 3120
Title: Using UTF-8 as the default source encoding
Author: Martin von LÃ¶wis <martin@v.loewis.de>
Status: Final
Type: Standards Track
Created: 15-Apr-2007
Python-Version: 3.0
Post-History:


Specification
=============

This PEP proposes to change the default source encoding from ASCII to
UTF-8. Support for alternative source encodings (:pep:`263`) continues to
exist; an explicit encoding declaration takes precedence over the
default.


A Bit of History
================

In Python 1, the source encoding was unspecified, except that the
source encoding had to be a superset of the system's basic execution
character set (i.e. an ASCII superset, on most systems).  The source
encoding was only relevant for the lexis itself (bytes representing
letters for keywords, identifiers, punctuation, line breaks, etc).
The contents of a string literal was copied literally from the file
on source.

In Python 2.0, the source encoding changed to Latin-1 as a side effect
of introducing Unicode. For Unicode string literals, the characters
were still copied literally from the source file, but widened on a
character-by-character basis. As Unicode gives a fixed interpretation
to code points, this algorithm effectively fixed a source encoding, at
least for files containing non-ASCII characters in Unicode literals.

:pep:`263` identified the problem that you can use only those Unicode
characters in a Unicode literal which are also in Latin-1, and
introduced a syntax for declaring the source encoding. If no source
encoding was given, the default should be ASCII. For compatibility
with Python 2.0 and 2.1, files were interpreted as Latin-1 for a
transitional period. This transition ended with Python 2.5, which
gives an error if non-ASCII characters are encountered and no source
encoding is declared.

Rationale
=========

With :pep:`263`, using arbitrary non-ASCII characters in a Python file is
possible, but tedious. One has to explicitly add an encoding
declaration. Even though some editors (like IDLE and Emacs) support
the declarations of :pep:`263`, many editors still do not (and never
will); users have to explicitly adjust the encoding which the editor
assumes on a file-by-file basis.

When the default encoding is changed to UTF-8, adding non-ASCII text
to Python files becomes easier and more portable: On some systems,
editors will automatically choose UTF-8 when saving text (e.g. on Unix
systems where the locale uses UTF-8). On other systems, editors will
guess the encoding when reading the file, and UTF-8 is easy to
guess. Yet other editors support associating a default encoding with a
file extension, allowing users to associate .py with UTF-8.

For Python 2, an important reason for using non-UTF-8 encodings was
that byte string literals would be in the source encoding at run-time,
allowing then to output them to a file or render them to the user
as-is. With Python 3, all strings will be Unicode strings, so the
original encoding of the source will have no impact at run-time.

Implementation
==============

The parser needs to be changed to accept bytes > 127 if no source
encoding is specified; instead of giving an error, it needs to check
that the bytes are well-formed UTF-8 (decoding is not necessary,
as the parser converts all source code to UTF-8, anyway).

IDLE needs to be changed to use UTF-8 as the default encoding.


Copyright
=========

This document has been placed in the public domain.


id: pep-3108.rst
content:
PEP: 3108
Title: Standard Library Reorganization
Author: Brett Cannon <brett@python.org>
Status: Final
Type: Standards Track
Created: 01-Jan-2007
Python-Version: 3.0
Post-History: 28-Apr-2008

.. note::
   The merging of profile/cProfile as of Python 3.3 did not
   occur, and thus is considered abandoned (although it would be
   acceptable to do in the future).

Abstract
========

Just like the language itself, Python's standard library (stdlib) has
grown over the years to be very rich.  But over time some modules
have lost their need to be included with Python.  There has also been
an introduction of a naming convention for modules since Python's
inception that not all modules follow.

Python 3.0 presents a chance to remove modules that do not have
long term usefulness.  This chance also allows for the renaming of
modules so that they follow the :pep:`Python style guide <8>`.  This
PEP lists modules that should not be included in Python 3.0 or which
need to be renamed.


Modules to Remove
=================

Guido pronounced that "silly old stuff" is to be deleted from the
stdlib for Py3K [#silly-old-stuff]_.  This is open-ended on purpose.
Each module to be removed needs to have a justification as to why it
should no longer be distributed with Python.  This can range from the
module being deprecated in Python 2.x to being for a platform that is
no longer widely used.

This section of the PEP lists the various modules to be removed. Each
subsection represents a different reason for modules to be
removed.  Each module must have a specific justification on top of
being listed in a specific subsection so as to make sure only modules
that truly deserve to be removed are in fact removed.

When a reason mentions how long it has been since a module has been
"uniquely edited", it is in reference to how long it has been since a
checkin was done specifically for the module and not for a change that
applied universally across the entire stdlib.  If an edit time is not
denoted as "unique" then it is the last time the file was edited,
period.


Previously deprecated [done]
----------------------------

:pep:`4` lists all modules that have been deprecated in the stdlib.
The specified motivations mirror those listed in
:pep:`4`. All modules listed
in the PEP at the time of the first alpha release of Python 3.0 will
be removed.

The entire contents of lib-old will also be removed.  These modules
have already been removed from being imported but are kept in the
distribution for Python for users that rely upon the code.

* cfmfile

  + Documented as deprecated since Python 2.4 without an explicit
    reason.

* cl

  + Documented as obsolete since Python 2.0 or earlier.
  + Interface to SGI hardware.

* md5

  + Supplanted by the ``hashlib`` module.

* mimetools

  + Documented as obsolete in a previous version.
  + Supplanted by the ``email`` package.

* MimeWriter

  + Supplanted by the ``email`` package.

* mimify

  + Supplanted by the ``email`` package.

* multifile

  + Supplanted by the ``email`` package.

* posixfile

  + Locking is better done by ``fcntl.lockf()``.

* rfc822

  + Supplanted by the ``email`` package.

* sha

  + Supplanted by the ``hashlib`` package.

* sv

  + Documented as obsolete since Python 2.0 or earlier.
  + Interface to obsolete SGI Indigo hardware.

* timing

  + Documented as obsolete since Python 2.0 or earlier.
  + ``time.clock()`` gives better time resolution.


Platform-specific with minimal use [done]
------------------------------------------

Python supports many platforms, some of which are not widely used or
maintained.
And on some of these platforms there are modules that have limited use
to people on those platforms.  Because of their limited usefulness it
would be better to no longer burden the Python development team with
their maintenance.

The modules mentioned below are documented. All undocumented modules
for the specified platforms will also be removed.

IRIX
////

The IRIX operating system is no longer produced [#irix-retirement]_.
Removing all modules from the plat-irix[56] directory has been deemed
reasonable because of this fact.

+ AL/al

  - Provides sound support on Indy and Indigo workstations.
  - Both workstations are no longer available.
  - Code has not been uniquely edited in three years.

+ cd/CD

  - CD drive control for SGI systems.
  - SGI no longer sells machines with IRIX on them.
  - Code has not been uniquely edited in 14 years.

+ cddb

  - Undocumented.

+ cdplayer

  - Undocumented.

+ cl/CL/CL_old

  - Compression library for SGI systems.
  - SGI no longer sells machines with IRIX on them.
  - Code has not been uniquely edited in 14 years.

+ DEVICE/GL/gl/cgen/cgensuport

  - GL access, which is the predecessor to OpenGL.
  - Has not been edited in at least eight years.
  - Third-party libraries provide better support (PyOpenGL [#pyopengl]_).

+ ERRNO

  - Undocumented.

+ FILE

  - Undocumented.

+ FL/fl/flp

  - Wrapper for the FORMS library [#irix-forms]_
  - FORMS has not been edited in 12 years.
  - Library is not widely used.
  - First eight hits on Google are for Python docs for fl.

+ fm

  - Wrapper to the IRIS Font Manager library.
  - Only available on SGI machines which no longer come with IRIX.

+ GET

  - Undocumented.

+ GLWS

  - Undocumented.

+ imgfile

  - Wrapper for SGI libimage library for imglib image files
    (``.rgb`` files).
  - Python Imaging Library provides read-only support [#pil]_.
  - Not uniquely edited in 13 years.

+ IN

  - Undocumented.

+ IOCTL

  - Undocumented.

+ jpeg

  - Wrapper for JPEG (de)compressor.
  - Code not uniquely edited in nine years.
  - Third-party libraries provide better support
    (Python Imaging Library [#pil]_).

+ panel

  - Undocumented.

+ panelparser

  - Undocumented.

+ readcd

  - Undocumented.

+ SV

  - Undocumented.

+ torgb

  - Undocumented.

+ WAIT

  - Undocumented.


Mac-specific modules
////////////////////

The Mac-specific modules are not well-maintained (e.g., the bgen
tool used to auto-generate many of the modules has never been
updated to support UCS-4). It is also not Python's place to maintain
such a large amount of OS-specific modules. Thus all modules under
``Lib/plat-mac`` and ``Mac`` are to be removed.

A stub module for proxy access will be provided for use by urllib.

* _builtinSuites

  - Undocumented.
  - Package under lib-scriptpackages.

* Audio_mac

  - Undocumented.

* aepack

  - OSA support is better through third-party modules.

    * Appscript [#appscript]_.

  - Hard-coded endianness which breaks on Intel Macs.
  - Might need to rename if Carbon package dependent.

* aetools

  - See aepack.

* aetypes

  - See aepack.

* applesingle

  - Undocumented.
  - AppleSingle is a binary file format for A/UX.
  - A/UX no longer distributed.

* appletrawmain

  - Undocumented.

* appletrunner

  - Undocumented.

* argvemulator

  - Undocumented.

* autoGIL

  - Very bad model for using Python with the CFRunLoop.

* bgenlocations

  - Undocumented.

* buildtools

  - Documented as deprecated since Python 2.3 without an explicit
    reason.

* bundlebuilder

  - Undocumented.

* Carbon

  - Carbon development has stopped.
  - Does not support 64-bit systems completely.
  - Dependent on bgen which has never been updated to support UCS-4
    Unicode builds of Python.

* CodeWarrior

  - Undocumented.
  - Package under lib-scriptpackages.

* ColorPicker

  - Better to use Cocoa for GUIs.

* EasyDialogs

  - Better to use Cocoa for GUIs.

* Explorer

  - Undocumented.
  - Package under lib-scriptpackages.

* Finder

  - Undocumented.
  - Package under lib-scriptpackages.

* findertools

  - No longer useful.

* FrameWork

  - Poorly documented.
  - Not updated to support Carbon Events.

* gensuitemodule

  - See aepack.

* ic

* icglue

* icopen

  - Not needed on OS X.
  - Meant to replace 'open' which is usually a bad thing to do.

* macerrors

  - Undocumented.

* MacOS

  - Would also mean the rem


id: pep-3124.rst
content:
PEP: 3124
Title: Overloading, Generic Functions, Interfaces, and Adaptation
Author: Phillip J. Eby <pje@telecommunity.com>
Discussions-To: python-3000@python.org
Status: Deferred
Type: Standards Track
Requires: 3107, 3115, 3119
Created: 28-Apr-2007
Post-History: 30-Apr-2007
Replaces: 245, 246


Deferred
========

See https://mail.python.org/pipermail/python-3000/2007-July/008784.html.


Abstract
========

This PEP proposes a new standard library module, ``overloading``, to
provide generic programming features including dynamic overloading
(aka generic functions), interfaces, adaptation, method combining (ala
CLOS and AspectJ), and simple forms of aspect-oriented programming
(AOP).

The proposed API is also open to extension; that is, it will be
possible for library developers to implement their own specialized
interface types, generic function dispatchers, method combination
algorithms, etc., and those extensions will be treated as first-class
citizens by the proposed API.

The API will be implemented in pure Python with no C, but may have
some dependency on CPython-specific features such as ``sys._getframe``
and the ``func_code`` attribute of functions.  It is expected that
e.g. Jython and IronPython will have other ways of implementing
similar functionality (perhaps using Java or C#).


Rationale and Goals
===================

Python has always provided a variety of built-in and standard-library
generic functions, such as ``len()``, ``iter()``, ``pprint.pprint()``,
and most of the functions in the ``operator`` module.  However, it
currently:

1. does not have a simple or straightforward way for developers to
   create new generic functions,

2. does not have a standard way for methods to be added to existing
   generic functions (i.e., some are added using registration
   functions, others require defining ``__special__`` methods,
   possibly by monkeypatching), and

3. does not allow dispatching on multiple argument types (except in
   a limited form for arithmetic operators, where "right-hand"
   (``__r*__``) methods can be used to do two-argument dispatch.

In addition, it is currently a common anti-pattern for Python code
to inspect the types of received arguments, in order to decide what
to do with the objects.  For example, code may wish to accept either
an object of some type, or a sequence of objects of that type.

Currently, the "obvious way" to do this is by type inspection, but
this is brittle and closed to extension.  A developer using an
already-written library may be unable to change how their objects are
treated by such code, especially if the objects they are using were
created by a third party.

Therefore, this PEP proposes a standard library module to address
these, and related issues, using decorators and argument annotations
(:pep:`3107`).  The primary features to be provided are:

* a dynamic overloading facility, similar to the static overloading
  found in languages such as Java and C++, but including optional
  method combination features as found in CLOS and AspectJ.

* a simple "interfaces and adaptation" library inspired by Haskell's
  typeclasses (but more dynamic, and without any static type-checking),
  with an extension API to allow registering user-defined interface
  types such as those found in PyProtocols and Zope.

* a simple "aspect" implementation to make it easy to create stateful
  adapters and to do other stateful AOP.

These features are to be provided in such a way that extended
implementations can be created and used.  For example, it should be
possible for libraries to define new dispatching criteria for
generic functions, and new kinds of interfaces, and use them in
place of the predefined features.  For example, it should be possible
to use a ``zope.interface`` interface object to specify the desired
type of a function argument, as long as the ``zope.interface`` package
registered itself correctly (or a third party did the registration).

In this way, the proposed API simply offers a uniform way of accessing
the functionality within its scope, rather than prescribing a single
implementation to be used for all libraries, frameworks, and
applications.


User API
========

The overloading API will be implemented as a single module, named
``overloading``, providing the following features:


Overloading/Generic Functions
-----------------------------

The ``@overload`` decorator allows you to define alternate
implementations of a function, specialized by argument type(s).  A
function with the same name must already exist in the local namespace.
The existing function is modified in-place by the decorator to add
the new implementation, and the modified function is returned by the
decorator.  Thus, the following code::

    from overloading import overload
    from collections import Iterable

    def flatten(ob):
        """Flatten an object to its component iterables"""
        yield ob

    @overload
    def flatten(ob: Iterable):
        for o in ob:
            for ob in flatten(o):
                yield ob

    @overload
    def flatten(ob: basestring):
        yield ob

creates a single ``flatten()`` function whose implementation roughly
equates to::

    def flatten(ob):
        if isinstance(ob, basestring) or not isinstance(ob, Iterable):
            yield ob
        else:
            for o in ob:
                for ob in flatten(o):
                    yield ob

**except** that the ``flatten()`` function defined by overloading
remains open to extension by adding more overloads, while the
hardcoded version cannot be extended.

For example, if someone wants to use ``flatten()`` with a string-like
type that doesn't subclass ``basestring``, they would be out of luck
with the second implementation.  With the overloaded implementation,
however, they can either write this::

    @overload
    def flatten(ob: MyString):
        yield ob

or this (to avoid copying the implementation)::

    from overloading import RuleSet
    RuleSet(flatten).copy_rules((basestring,), (MyString,))

(Note also that, although :pep:`3119` proposes that it should be possible
for abstract base classes like ``Iterable`` to allow classes like
``MyString`` to claim subclass-hood, such a claim is *global*,
throughout the application.  In contrast, adding a specific overload
or copying a rule is specific to an individual function, and therefore
less likely to have undesired side effects.)


``@overload`` vs. ``@when``
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``@overload`` decorator is a common-case shorthand for the more
general ``@when`` decorator.  It allows you to leave out the name of
the function you are overloading, at the expense of requiring the
target function to be in the local namespace.  It also doesn't support
adding additional criteria besides the ones specified via argument
annotations.  The following function definitions have identical
effects, except for name binding side-effects (which will be described
below)::

    from overloading import when

    @overload
    def flatten(ob: basestring):
        yield ob

    @when(flatten)
    def flatten(ob: basestring):
        yield ob

    @when(flatten)
    def flatten_basestring(ob: basestring):
        yield ob

    @when(flatten, (basestring,))
    def flatten_basestring(ob):
        yield ob

The first definition above will bind ``flatten`` to whatever it was
previously bound to.  The second will do the same, if it was already
bound to the ``when`` decorator's first argument.  If ``flatten`` is
unbound or bound to something else, it will be rebound to the function
definition as given.  The last two definitions above will always bind
``flatten_basestring`` to the function definition as given.

Using this approach allows you to both give a method a descriptive
name (often useful in tracebacks!) and to reuse the method later.

Except as otherwise specified, all ``overloading`` decorators have the
same signature and binding rules as ``@when``.  They accept a function
and an opt


id: pep-3130.rst
content:
PEP: 3130
Title: Access to Current Module/Class/Function
Author: Jim J. Jewett <jimjjewett@gmail.com>
Status: Rejected
Type: Standards Track
Created: 22-Apr-2007
Python-Version: 3.0
Post-History: 22-Apr-2007


Rejection Notice
================

This PEP is rejected.  It is not clear how it should be
implemented or what the precise semantics should be in edge cases,
and there aren't enough important use cases given.  response has
been lukewarm at best.


Abstract
========

It is common to need a reference to the current module, class,
or function, but there is currently no entirely correct way to
do this.  This PEP proposes adding the keywords ``__module__``,
``__class__``, and ``__function__``.


Rationale for ``__module__``
============================

Many modules export various functions, classes, and other objects,
but will perform additional activities (such as running unit
tests) when run as a script.  The current idiom is to test whether
the module's name has been set to magic value.

::

    if __name__ == "__main__": ...

More complicated introspection requires a module to (attempt to)
import itself.  If importing the expected name actually produces
a different module, there is no good workaround.

::

    # __import__ lets you use a variable, but... it gets more
    # complicated if the module is in a package.
    __import__(__name__)

    # So just go to sys modules... and hope that the module wasn't
    # hidden/removed (perhaps for security), that __name__ wasn't
    # changed, and definitely hope that no other module with the
    # same name is now available.
    class X(object):
        pass

    import sys
    mod = sys.modules[__name__]
    mod = sys.modules[X.__class__.__module__]

Proposal:  Add a ``__module__`` keyword which refers to the module
currently being defined (executed).  (But see open issues.)

::

    # XXX sys.main is still changing as draft progresses.  May
    # really need sys.modules[sys.main]
    if __module__ is sys.main:    # assumes PEP (3122), Cannon
        ...


Rationale for ``__class__``
===========================

Class methods are passed the current instance; from this they can
determine ``self.__class__`` (or cls, for class methods).
Unfortunately, this reference is to the object's actual class,
which may be a subclass of the defining class.  The current
workaround is to repeat the name of the class, and assume that the
name will not be rebound.

::

    class C(B):

        def meth(self):
            super(C, self).meth() # Hope C is never rebound.

    class D(C):

        def meth(self):
            # ?!? issubclass(D,C), so it "works":
            super(C, self).meth()

Proposal: Add a ``__class__`` keyword which refers to the class
currently being defined (executed).  (But see open issues.)

::

    class C(B):
        def meth(self):
            super(__class__, self).meth()

Note that super calls may be further simplified by the "New Super"
PEP (Spealman).  The ``__class__`` (or ``__this_class__``) attribute came
up in attempts to simplify the explanation and/or implementation
of that PEP, but was separated out as an independent decision.

Note that ``__class__`` (or ``__this_class__``) is not quite the same as
the ``__thisclass__`` property on bound super objects.  The existing
``super.__thisclass__`` property refers to the class from which the
Method Resolution Order search begins.  In the above class D, it
would refer to (the current reference of name) C.


Rationale for ``__function__``
==============================

Functions (including methods) often want access to themselves,
usually for a private storage location or true recursion.  While
there are several workarounds, all have their drawbacks.

::

    def counter(_total=[0]):
        # _total shouldn't really appear in the
        # signature at all; the list wrapping and
        # [0] unwrapping obscure the code
        _total[0] += 1
        return _total[0]

    @annotate(total=0)
    def counter():
        # Assume name counter is never rebound:
        counter.total += 1
        return counter.total

    # class exists only to provide storage:
    class _wrap(object):

        __total = 0

        def f(self):
            self.__total += 1
            return self.__total

    # set module attribute to a bound method:
    accum = _wrap().f

    # This function calls "factorial", which should be itself --
    # but the same programming styles that use heavy recursion
    # often have a greater willingness to rebind function names.
    def factorial(n):
        return (n * factorial(n-1) if n else 1)

Proposal: Add a ``__function__`` keyword which refers to the function
(or method) currently being defined (executed).  (But see open
issues.)

::

    @annotate(total=0)
    def counter():
        # Always refers to this function obj:
        __function__.total += 1
        return __function__.total

    def factorial(n):
        return (n * __function__(n-1) if n else 1)


Backwards Compatibility
=======================

While a user could be using these names already, double-underscore
names ( ``__anything__`` ) are explicitly reserved to the interpreter.
It is therefore acceptable to introduce special meaning to these
names within a single feature release.


Implementation
==============

Ideally, these names would be keywords treated specially by the
bytecode compiler.

Guido has suggested [1]_ using a cell variable filled in by the
metaclass.

Michele Simionato has provided a prototype using bytecode hacks [2]_.
This does not require any new bytecode operators; it just
modifies the which specific sequence of existing operators gets
run.


Open Issues
===========

- Are ``__module__``, ``__class__``, and ``__function__`` the right names?  In
  particular, should the names include the word "this", either as
  ``__this_module__``, ``__this_class__``, and ``__this_function__``, (format
  discussed on the python-3000 and python-ideas lists) or as
  ``__thismodule__``, ``__thisclass__``, and ``__thisfunction__`` (inspired
  by, but conflicting with, current usage of super.``__thisclass__``).

- Are all three keywords needed, or should this enhancement be
  limited to a subset of the objects?  Should methods be treated
  separately from other functions?


References
==========

.. [1] Fixing super anyone?  Guido van Rossum
       https://mail.python.org/pipermail/python-3000/2007-April/006671.html

.. [2] Descriptor/Decorator challenge,  Michele Simionato
       http://groups.google.com/group/comp.lang.python/browse_frm/thread/a6010c7494871bb1/62a2da68961caeb6?lnk=gst&q=simionato+challenge&rnum=1&hl=en#62a2da68961caeb6


Copyright
=========

This document has been placed in the public domain.


id: pep-3118.rst
content:
PEP: 3118
Title: Revising the buffer protocol
Author: Travis Oliphant <oliphant@ee.byu.edu>, Carl Banks <pythondev@aerojockey.com>
Status: Final
Type: Standards Track
Created: 28-Aug-2006
Python-Version: 3.0
Post-History: `09-Apr-2007 <https://mail.python.org/archives/list/python-dev@python.org/thread/X7A5IP4VFJTVCQJSMBXGAH3BVB3A5C3T>`__

.. canonical-doc:: :external+python:ref:`bufferobjects`,
                   :external+python:c:type:`PyBufferProcs`,
                   :external+python:c:type:`PyMemoryView_FromObject`

   Not all features proposed here were implemented. Specifically:

   - ``PyObject_CopyToObject`` was not added.
   - :ref:`3118-struct-additions` were not added, except for ``?`` (``_Bool``).
   - ``PyObject_GetMemoryView`` is named
     :external+python:c:type:`PyMemoryView_FromObject`.

   This PEP targets Python 3.0, which was released more than a decade ago.
   Any proposals to add missing functionality should
   be discussed as new features, not treated as finishing the implementation
   of this PEP.


Abstract
========

This PEP proposes re-designing the buffer interface (``PyBufferProcs``
function pointers) to improve the way Python allows memory sharing in
Python 3.0

In particular, it is proposed that the character buffer portion
of the API be eliminated and the multiple-segment portion be
re-designed in conjunction with allowing for strided memory
to be shared.   In addition, the new buffer interface will
allow the sharing of any multi-dimensional nature of the
memory and what data-format the memory contains.

This interface will allow any extension module to either
create objects that share memory or create algorithms that
use and manipulate raw memory from arbitrary objects that
export the interface.


Rationale
=========

The Python 2.X buffer protocol allows different Python types to
exchange a pointer to a sequence of internal buffers.  This
functionality is *extremely* useful for sharing large segments of
memory between different high-level objects, but it is too limited and
has issues:

1. There is the little used "sequence-of-segments" option
   (bf_getsegcount) that is not well motivated.

2. There is the apparently redundant character-buffer option
   (bf_getcharbuffer)

3. There is no way for a consumer to tell the buffer-API-exporting
   object it is "finished" with its view of the memory and
   therefore no way for the exporting object to be sure that it is
   safe to reallocate the pointer to the memory that it owns (for
   example, the array object reallocating its memory after sharing
   it with the buffer object which held the original pointer led
   to the infamous buffer-object problem).

4. Memory is just a pointer with a length. There is no way to
   describe what is "in" the memory (float, int, C-structure, etc.)

5. There is no shape information provided for the memory.  But,
   several array-like Python types could make use of a standard
   way to describe the shape-interpretation of the memory
   (wxPython, GTK, pyQT, CVXOPT, PyVox, Audio and Video
   Libraries, ctypes, NumPy, data-base interfaces, etc.)

6. There is no way to share discontiguous memory (except through
   the sequence of segments notion).

   There are two widely used libraries that use the concept of
   discontiguous memory: PIL and NumPy.  Their view of discontiguous
   arrays is different, though.  The proposed buffer interface allows
   sharing of either memory model.  Exporters will typically use only one
   approach and consumers may choose to support discontiguous
   arrays of each type however they choose.

   NumPy uses the notion of constant striding in each dimension as its
   basic concept of an array. With this concept, a simple sub-region
   of a larger array can be described without copying the data.
   Thus, stride information is the additional information that must be
   shared.

   The PIL uses a more opaque memory representation. Sometimes an
   image is contained in a contiguous segment of memory, but sometimes
   it is contained in an array of pointers to the contiguous segments
   (usually lines) of the image.  The PIL is where the idea of multiple
   buffer segments in the original buffer interface came from.

   NumPy's strided memory model is used more often in computational
   libraries and because it is so simple it makes sense to support
   memory sharing using this model.  The PIL memory model is sometimes
   used in C-code where a 2-d array can then be accessed using double
   pointer indirection:  e.g. ``image[i][j]``.

   The buffer interface should allow the object to export either of these
   memory models.  Consumers are free to either require contiguous memory
   or write code to handle one or both of these memory models.

Proposal Overview
=================

* Eliminate the char-buffer and multiple-segment sections of the
  buffer-protocol.

* Unify the read/write versions of getting the buffer.

* Add a new function to the interface that should be called when
  the consumer object is "done" with the memory area.

* Add a new variable to allow the interface to describe what is in
  memory (unifying what is currently done now in struct and
  array)

* Add a new variable to allow the protocol to share shape information

* Add a new variable for sharing stride information

* Add a new mechanism for sharing arrays that must
  be accessed using pointer indirection.

* Fix all objects in the core and the standard library to conform
  to the new interface

* Extend the struct module to handle more format specifiers

* Extend the buffer object into a new memory object which places
  a Python veneer around the buffer interface.

* Add a few functions to make it easy to copy contiguous data
  in and out of object supporting the buffer interface.

Specification
=============

While the new specification allows for complicated memory sharing,
simple contiguous buffers of bytes can still be obtained from an
object.  In fact, the new protocol allows a standard mechanism for
doing this even if the original object is not represented as a
contiguous chunk of memory.

The easiest way to obtain a simple contiguous chunk of memory is
to use the provided C-API to obtain a chunk of memory.


Change the ``PyBufferProcs`` structure to ::

    typedef struct {
         getbufferproc bf_getbuffer;
         releasebufferproc bf_releasebuffer;
    } PyBufferProcs;

Both of these routines are optional for a type object

::

    typedef int (*getbufferproc)(PyObject *obj, PyBuffer *view, int flags)

This function returns ``0`` on success and ``-1`` on failure (and raises an
error). The first variable is the "exporting" object.  The second
argument is the address to a bufferinfo structure.  Both arguments must
never be NULL.

The third argument indicates what kind of buffer the consumer is
prepared to deal with and therefore what kind of buffer the exporter
is allowed to return.  The new buffer interface allows for much more
complicated memory sharing possibilities.  Some consumers may not be
able to handle all the complexity but may want to see if the
exporter will let them take a simpler view to its memory.

In addition, some exporters may not be able to share memory in every
possible way and may need to raise errors to signal to some consumers
that something is just not possible.  These errors should be
``PyErr_BufferError`` unless there is another error that is actually
causing the problem. The exporter can use flags information to
simplify how much of the PyBuffer structure is filled in with
non-default values and/or raise an error if the object can't support a
simpler view of its memory.

The exporter should always fill in all elements of the buffer
structure (with defaults or NULLs if nothing else is requested). The
PyBuffer_FillInfo function can be used for simple cases.


Access flags
------------

Some flags are useful for requesting a specific kind of memory
segment, while others indicate to the exporter what ki


id: pep-3119.rst
content:
PEP: 3119
Title: Introducing Abstract Base Classes
Author: Guido van Rossum <guido@python.org>, Talin <viridia@gmail.com>
Status: Final
Type: Standards Track
Created: 18-Apr-2007
Python-Version: 3.0
Post-History: 26-Apr-2007, 11-May-2007


Abstract
========

This is a proposal to add Abstract Base Class (ABC) support to Python
3000.  It proposes:

* A way to overload ``isinstance()`` and ``issubclass()``.

* A new module ``abc`` which serves as an "ABC support framework".  It
  defines a metaclass for use with ABCs and a decorator that can be
  used to define abstract methods.

* Specific ABCs for containers and iterators, to be added to the
  collections module.

Much of the thinking that went into the proposal is not about the
specific mechanism of ABCs, as contrasted with Interfaces or Generic
Functions (GFs), but about clarifying philosophical issues like "what
makes a set", "what makes a mapping" and "what makes a sequence".

There's also a companion :pep:`3141`, which defines ABCs for numeric
types.


Acknowledgements
----------------

Talin wrote the Rationale below [1]_ as well as most of the section on
ABCs vs. Interfaces.  For that alone he deserves co-authorship.  The
rest of the PEP uses "I" referring to the first author.


Rationale
=========

In the domain of object-oriented programming, the usage patterns for
interacting with an object can be divided into two basic categories,
which are 'invocation' and 'inspection'.

Invocation means interacting with an object by invoking its methods.
Usually this is combined with polymorphism, so that invoking a given
method may run different code depending on the type of an object.

Inspection means the ability for external code (outside of the
object's methods) to examine the type or properties of that object,
and make decisions on how to treat that object based on that
information.

Both usage patterns serve the same general end, which is to be able to
support the processing of diverse and potentially novel objects in a
uniform way, but at the same time allowing processing decisions to be
customized for each different type of object.

In classical OOP theory, invocation is the preferred usage pattern,
and inspection is actively discouraged, being considered a relic of an
earlier, procedural programming style.  However, in practice this view
is simply too dogmatic and inflexible, and leads to a kind of design
rigidity that is very much at odds with the dynamic nature of a
language like Python.

In particular, there is often a need to process objects in a way that
wasn't anticipated by the creator of the object class.  It is not
always the best solution to build in to every object methods that
satisfy the needs of every possible user of that object.  Moreover,
there are many powerful dispatch philosophies that are in direct
contrast to the classic OOP requirement of behavior being strictly
encapsulated within an object, examples being rule or pattern-match
driven logic.

On the other hand, one of the criticisms of inspection by classic
OOP theorists is the lack of formalisms and the ad hoc nature of what
is being inspected.  In a language such as Python, in which almost any
aspect of an object can be reflected and directly accessed by external
code, there are many different ways to test whether an object conforms
to a particular protocol or not.  For example, if asking 'is this
object a mutable sequence container?', one can look for a base class
of 'list', or one can look for a method named '__getitem__'.  But note
that although these tests may seem obvious, neither of them are
correct, as one generates false negatives, and the other false
positives.

The generally agreed-upon remedy is to standardize the tests, and
group them into a formal arrangement.  This is most easily done by
associating with each class a set of standard testable properties,
either via the inheritance mechanism or some other means.  Each test
carries with it a set of promises: it contains a promise about the
general behavior of the class, and a promise as to what other class
methods will be available.

This PEP proposes a particular strategy for organizing these tests
known as Abstract Base Classes, or ABC.  ABCs are simply Python
classes that are added into an object's inheritance tree to signal
certain features of that object to an external inspector.  Tests are
done using ``isinstance()``, and the presence of a particular ABC
means that the test has passed.

In addition, the ABCs define a minimal set of methods that establish
the characteristic behavior of the type.  Code that discriminates
objects based on their ABC type can trust that those methods will
always be present.  Each of these methods are accompanied by an
generalized abstract semantic definition that is described in the
documentation for the ABC.  These standard semantic definitions are
not enforced, but are strongly recommended.

Like all other things in Python, these promises are in the nature of a
friendly agreement, which in this case means that while the
language does enforce some of the promises made in the ABC, it is up
to the implementer of the concrete class to insure that the remaining
ones are kept.


Specification
=============

The specification follows the categories listed in the abstract:

* A way to overload ``isinstance()`` and ``issubclass()``.

* A new module ``abc`` which serves as an "ABC support framework".  It
  defines a metaclass for use with ABCs and a decorator that can be
  used to define abstract methods.

* Specific ABCs for containers and iterators, to be added to the
  collections module.


Overloading ``isinstance()`` and ``issubclass()``
-------------------------------------------------

During the development of this PEP and of its companion, :pep:`3141`, we
repeatedly faced the choice between standardizing more, fine-grained
ABCs or fewer, coarse-grained ones.  For example, at one stage, PEP
3141 introduced the following stack of base classes used for complex
numbers: MonoidUnderPlus, AdditiveGroup, Ring, Field, Complex (each
derived from the previous).  And the discussion mentioned several
other algebraic categorizations that were left out: Algebraic,
Transcendental, and IntegralDomain, and PrincipalIdealDomain.  In
earlier versions of the current PEP, we considered the use cases for
separate classes like Set, ComposableSet, MutableSet, HashableSet,
MutableComposableSet, HashableComposableSet.

The dilemma here is that we'd rather have fewer ABCs, but then what
should a user do who needs a less refined ABC?  Consider e.g. the
plight of a mathematician who wants to define their own kind of
Transcendental numbers, but also wants float and int to be considered
Transcendental.  :pep:`3141` originally proposed to patch float.__bases__
for that purpose, but there are some good reasons to keep the built-in
types immutable (for one, they are shared between all Python
interpreters running in the same address space, as is used by
mod_python [16]_).

Another example would be someone who wants to define a generic
function (:pep:`3124`) for any sequence that has an ``append()`` method.
The ``Sequence`` ABC (see below) doesn't promise the ``append()``
method, while ``MutableSequence`` requires not only ``append()`` but
also various other mutating methods.

To solve these and similar dilemmas, the next section will propose a
metaclass for use with ABCs that will allow us to add an ABC as a
"virtual base class" (not the same concept as in C++) to any class,
including to another ABC.  This allows the standard library to define
ABCs ``Sequence`` and ``MutableSequence`` and register these as
virtual base classes for built-in types like ``basestring``, ``tuple``
and ``list``, so that for example the following conditions are all
true::

    isinstance([], Sequence)
    issubclass(list, Sequence)
    issubclass(list, MutableSequence)
    isinstance((), Sequence)
    not issubclass(tuple, MutableSequence)
    isinstance("", Sequence)
    issubc


id: pep-3131.rst
content:
PEP: 3131
Title: Supporting Non-ASCII Identifiers
Author: Martin von LÃ¶wis <martin@v.loewis.de>
Status: Final
Type: Standards Track
Created: 01-May-2007
Python-Version: 3.0
Post-History:


Abstract
========

This PEP suggests to support non-ASCII letters (such as accented characters,
Cyrillic, Greek, Kanji, etc.) in Python identifiers.

Rationale
=========

Python code is written by many people in the world who are not
familiar with the English language, or even well-acquainted with the
Latin writing system.  Such developers often desire to define classes
and functions with names in their native languages, rather than having
to come up with an (often incorrect) English translation of the
concept they want to name. By using identifiers in their native
language, code clarity and maintainability of the code among
speakers of that language improves.

For some languages, common transliteration systems exist (in particular, for the
Latin-based writing systems).  For other languages, users have larger
difficulties to use Latin to write their native words.

Common Objections
=================

Some objections are often raised against proposals similar to this one.

People claim that they will not be able to use a library if to do so they have
to use characters they cannot type on their keyboards.  However, it is the
choice of the designer of the library to decide on various constraints for using
the library: people may not be able to use the library because they cannot get
physical access to the source code (because it is not published), or because
licensing prohibits usage, or because the documentation is in a language they
cannot understand.  A developer wishing to make a library widely available needs
to make a number of explicit choices (such as publication, licensing, language
of documentation, and language of identifiers).  It should always be the choice
of the author to make these decisions - not the choice of the language
designers.

In particular, projects wishing to have wide usage probably might want to
establish a policy that all identifiers, comments, and documentation is written
in English (see the GNU coding style guide for an example of such a policy).
Restricting the language to ASCII-only identifiers does not enforce comments and
documentation to be English, or the identifiers actually to be English words, so
an additional policy is necessary, anyway.

Specification of Language Changes
=================================

The syntax of identifiers in Python will be based on the `Unicode standard annex
UAX-31 <https://www.unicode.org/reports/tr31/>`__, with elaboration and changes
as defined below.

Within the ASCII range (U+0001..U+007F), the valid characters for identifiers
are the same as in Python 2.5.  This specification only introduces additional
characters from outside the ASCII range.  For other characters, the
classification uses the version of the Unicode Character Database as included in
the ``unicodedata`` module.

The identifier syntax is ``<XID_Start> <XID_Continue>*``.

The exact specification of what characters have the XID_Start or
XID_Continue properties can be found in the `DerivedCoreProperties
file <https://www.unicode.org/Public/4.1.0/ucd/DerivedCoreProperties.txt>`__
of the Unicode data in use by Python (4.1 at the time this
PEP was written). For reference, the construction rules
for these sets are given below. The XID_* properties are derived
from ID_Start/ID_Continue, which are derived themselves.

``ID_Start`` is defined as all characters having one of the general
categories uppercase letters (Lu), lowercase letters (Ll), titlecase
letters (Lt), modifier letters (Lm), other letters (Lo), letter
numbers (Nl), the underscore, and characters carrying the
Other_ID_Start property. ``XID_Start`` then closes this set under
normalization, by removing all characters whose NFKC normalization
is not of the form ID_Start ID_Continue* anymore.

``ID_Continue`` is defined as all characters in ``ID_Start``, plus
nonspacing marks (Mn), spacing combining marks (Mc), decimal number
(Nd), connector punctuations (Pc), and characters carrying the
Other_ID_Continue property. Again, ``XID_Continue`` closes this set
under NFKC-normalization; it also adds U+00B7 to support Catalan.

All identifiers are converted into the normal form NFKC while parsing;
comparison of identifiers is based on NFKC.

A non-normative HTML file listing all valid identifier characters for
Unicode 4.1 can be found at
https://web.archive.org/web/20081016132748/http://www.dcl.hpi.uni-potsdam.de/home/loewis/table-3131.html.

Policy Specification
====================

As an addition to the Python Coding style, the following policy is
prescribed: All identifiers in the Python standard library MUST use
ASCII-only identifiers, and SHOULD use English words wherever feasible
(in many cases, abbreviations and technical terms are used which
aren't English). In addition, string literals and comments must also
be in ASCII. The only exceptions are (a) test cases testing the
non-ASCII features, and (b) names of authors. Authors whose names are
not based on the Latin alphabet MUST provide a Latin transliteration
of their names.

As an option, this specification can be applied to Python 2.x.  In
that case, ASCII-only identifiers would continue to be represented as
byte string objects in namespace dictionaries; identifiers with
non-ASCII characters would be represented as Unicode strings.

Implementation
==============

The following changes will need to be made to the parser:

1. If a non-ASCII character is found in the UTF-8 representation of
   the source code, a forward scan is made to find the first ASCII
   non-identifier character (e.g. a space or punctuation character)

2. The entire UTF-8 string is passed to a function to normalize the
   string to NFKC, and then verify that it follows the identifier
   syntax. No such callout is made for pure-ASCII identifiers, which
   continue to be parsed the way they are today. The Unicode database
   must start including the Other_ID_{Start|Continue} property.

3. If this specification is implemented for 2.x, reflective libraries
   (such as pydoc) must be verified to continue to work when Unicode
   strings appear in ``__dict__`` slots as keys.

Open Issues
===========

John Nagle suggested consideration of `Unicode Technical Standard #39
<https://www.unicode.org/reports/tr39/>`__,
which discusses security mechanisms for Unicode identifiers.
It's not clear how that can precisely apply to this PEP; possible
consequences are

* warn about characters listed as "restricted" in xidmodifications.txt
* warn about identifiers using mixed scripts
* somehow perform Confusable Detection

In the latter two approaches, it's not clear how precisely the
algorithm should work. For mixed scripts, certain kinds of mixing
should probably allowed - are these the "Common" and "Inherited"
scripts mentioned in section 5? For Confusable Detection, it seems one
needs two identifiers to compare them for confusion - is it possible
to somehow apply it to a single identifier only, and warn?

In follow-up discussion, it turns out that John Nagle actually
meant to suggest `UTR#36 <https://www.unicode.org/reports/tr36/>`__,
level "Highly Restrictive".

Several people suggested to allow and ignore formatting control
characters (general category Cf), as is done in Java, JavaScript, and
C#. It's not clear whether this would improve things (it might
for RTL languages); if there is a need, these can be added
later.

Some people would like to see an option on selecting support
for this PEP at run-time; opinions vary on what precisely
that option should be, and what precisely its default value
should be. `Guido van Rossum commented
<https://mail.python.org/pipermail/python-3000/2007-May/007925.html>`__
that a global flag passed to the interpreter is not acceptable, as it would
apply to all modules.

Discussion
==========

`Ka-Ping Yee summarizes discussion and further ob


id: pep-3125.rst
content:
PEP: 3125
Title: Remove Backslash Continuation
Author: Jim J. Jewett <JimJJewett@gmail.com>
Status: Rejected
Type: Standards Track
Created: 29-Apr-2007
Post-History: 29-Apr-2007, 30-Apr-2007, 04-May-2007


Rejection Notice
================

This PEP is rejected.  There wasn't enough support in favor, the
feature to be removed isn't all that harmful, and there are some use
cases that would become harder.


Abstract
========

Python initially inherited its parsing from C.  While this has been
generally useful, there are some remnants which have been less useful
for Python, and should be eliminated.

This PEP proposes elimination of terminal ``\`` as a marker for line
continuation.


Motivation
==========

One goal for Python 3000 should be to simplify the language by
removing unnecessary or duplicated features.  There are currently
several ways to indicate that a logical line is continued on the
following physical line.

The other continuation methods are easily explained as a logical
consequence of the semantics they provide; ``\`` is simply an escape
character that needs to be memorized.


Existing Line Continuation Methods
==================================


Parenthetical Expression - ``([{}])``
-------------------------------------

Open a parenthetical expression.  It doesn't matter whether people
view the "line" as continuing; they do immediately recognize that the
expression needs to be closed before the statement can end.

Examples using each of ``()``, ``[]``, and ``{}``::

    def fn(long_argname1,
           long_argname2):
        settings = {"background": "random noise",
                    "volume": "barely audible"}
        restrictions = ["Warrantee void if used",
                        "Notice must be received by yesterday",
                        "Not responsible for sales pitch"]

Note that it is always possible to parenthesize an expression, but it
can seem odd to parenthesize an expression that needs parentheses only
for the line break::

    assert val>4, (
        "val is too small")


Triple-Quoted Strings
---------------------

Open a triple-quoted string; again, people recognize that the string
needs to finish before the next statement starts. ::

    banner_message = """
        Satisfaction Guaranteed,
        or DOUBLE YOUR MONEY BACK!!!





                                        some minor restrictions apply"""


Terminal ``\`` in the general case
----------------------------------

A terminal ``\`` indicates that the logical line is continued on the
following physical line (after whitespace).  There are no particular
semantics associated with this.  This form is never required, although
it may look better (particularly for people with a C language
background) in some cases::

    >>> assert val>4, \
            "val is too small"

Also note that the ``\`` must be the final character in the line.  If
your editor navigation can add whitespace to the end of a line, that
invisible change will alter the semantics of the program.
Fortunately, the typical result is only a syntax error, rather than a
runtime bug::

    >>> assert val>4, \
            "val is too small"

    SyntaxError: unexpected character after line continuation character

This PEP proposes to eliminate this redundant and potentially
confusing alternative.


Terminal ``\`` within a string
------------------------------

A terminal ``\`` within a single-quoted string, at the end of the
line.  This is arguably a special case of the terminal ``\``, but it
is a special case that may be worth keeping. ::

    >>> "abd\
     def"
    'abd def'

* Pro: Many of the objections to removing ``\`` termination were
  really just objections to removing it within literal strings;
  several people clarified that they want to keep this literal-string
  usage, but don't mind losing the general case.

* Pro: The use of ``\`` for an escape character within strings is well
  known.

* Contra: But note that this particular usage is odd, because the
  escaped character (the newline) is invisible, and the special
  treatment is to delete the character.  That said, the ``\`` of
  ``\(newline)`` is still an escape which changes the meaning of the
  following character.


Alternate Proposals
===================

Several people have suggested alternative ways of marking the line
end.  Most of these were rejected for not actually simplifying things.

The one exception was to let any unfinished expression signify a line
continuation, possibly in conjunction with increased indentation.

This is attractive because it is a generalization of the rule for
parentheses.

The initial objections to this were:

- The amount of whitespace may be contentious; expression continuation
  should not be confused with opening a new suite.

- The "expression continuation" markers are not as clearly marked in
  Python as the grouping punctuation "(), [], {}" marks are::

      # Plus needs another operand, so the line continues
      "abc" +
          "def"

      # String ends an expression, so the line does not
      # not continue.  The next line is a syntax error because
      # unary plus does not apply to strings.
      "abc"
          + "def"

- Guido objected for technical reasons.  [#dedent]_ The most obvious
  implementation would require allowing INDENT or DEDENT tokens
  anywhere, or at least in a widely expanded (and ill-defined) set of
  locations.  While this is of concern only for the internal parsing
  mechanism (rather than for users), it would be a major new source of
  complexity.

Andrew Koenig then pointed out [#lexical]_ a better implementation
strategy, and said that it had worked quite well in other
languages. [#snocone]_ The improved suggestion boiled down to:

    The whitespace that follows an (operator or) open bracket or
    parenthesis can include newline characters.

    It would be implemented at a very low lexical level -- even before
    the decision is made to turn a newline followed by spaces into an
    INDENT or DEDENT token.

There is still some concern that it could mask bugs, as in this
example [#guidobughide]_::

    # Used to be y+1, the 1 got dropped.  Syntax Error (today)
    # would become nonsense.
    x = y+
    f(x)

Requiring that the continuation be indented more than the initial line
would add both safety and complexity.


Open Issues
===========

* Should ``\``-continuation be removed even inside strings?

* Should the continuation markers be expanded from just ([{}]) to
  include lines ending with an operator?

* As a safety measure, should the continuation line be required to be
  more indented than the initial line?


References
==========

..  [#dedent] (email subject) PEP 30XZ: Simplified Parsing, van Rossum
    https://mail.python.org/pipermail/python-3000/2007-April/007063.html

..  [#lexical] (email subject) :pep:`3125` -- remove backslash
    continuation, Koenig
    https://mail.python.org/pipermail/python-3000/2007-May/007237.html

..  [#snocone] The Snocone Programming Language, Koenig
    http://www.snobol4.com/report.htm

..  [#guidobughide] (email subject) :pep:`3125` -- remove backslash
    continuation, van Rossum
    https://mail.python.org/pipermail/python-3000/2007-May/007244.html


Copyright
=========

This document has been placed in the public domain.


id: pep-3333.rst
content:
PEP: 3333
Title: Python Web Server Gateway Interface v1.0.1
Author: Phillip J. Eby <pje@telecommunity.com>
Discussions-To: web-sig@python.org
Status: Final
Type: Informational
Created: 26-Sep-2010
Post-History: 26-Sep-2010, 04-Oct-2010
Replaces: 333


Preface for Readers of PEP 333
==============================

This is an updated version of :pep:`333`, modified slightly to improve
usability under Python 3, and to incorporate several long-standing
de facto amendments to the WSGI protocol.  (Its code samples have
also been ported to Python 3.)

While for procedural reasons [6]_, this must be a distinct PEP, no
changes were made that invalidate previously-compliant servers or
applications under Python 2.x.  If your 2.x application or server
is compliant to PEP \333, it is also compliant with this PEP.

Under Python 3, however, your app or server must also follow the
rules outlined in the sections below titled, `A Note On String
Types`_, and `Unicode Issues`_.

For detailed, line-by-line diffs between this document and PEP \333,
you may view its SVN revision history [7]_, from revision 84854 forward.


Abstract
========

This document specifies a proposed standard interface between web
servers and Python web applications or frameworks, to promote web
application portability across a variety of web servers.


Original Rationale and Goals (from PEP 333)
===========================================

Python currently boasts a wide variety of web application frameworks,
such as Zope, Quixote, Webware, SkunkWeb, PSO, and Twisted Web -- to
name just a few [1]_.  This wide variety of choices can be a problem
for new Python users, because generally speaking, their choice of web
framework will limit their choice of usable web servers, and vice
versa.

By contrast, although Java has just as many web application frameworks
available, Java's "servlet" API makes it possible for applications
written with any Java web application framework to run in any web
server that supports the servlet API.

The availability and widespread use of such an API in web servers for
Python -- whether those servers are written in Python (e.g. Medusa),
embed Python (e.g. mod_python), or invoke Python via a gateway
protocol (e.g. CGI, FastCGI, etc.) -- would separate choice of
framework from choice of web server, freeing users to choose a pairing
that suits them, while freeing framework and server developers to
focus on their preferred area of specialization.

This PEP, therefore, proposes a simple and universal interface between
web servers and web applications or frameworks: the Python Web Server
Gateway Interface (WSGI).

But the mere existence of a WSGI spec does nothing to address the
existing state of servers and frameworks for Python web applications.
Server and framework authors and maintainers must actually implement
WSGI for there to be any effect.

However, since no existing servers or frameworks support WSGI, there
is little immediate reward for an author who implements WSGI support.
Thus, WSGI **must** be easy to implement, so that an author's initial
investment in the interface can be reasonably low.

Thus, simplicity of implementation on *both* the server and framework
sides of the interface is absolutely critical to the utility of the
WSGI interface, and is therefore the principal criterion for any
design decisions.

Note, however, that simplicity of implementation for a framework
author is not the same thing as ease of use for a web application
author.  WSGI presents an absolutely "no frills" interface to the
framework author, because bells and whistles like response objects and
cookie handling would just get in the way of existing frameworks'
handling of these issues.  Again, the goal of WSGI is to facilitate
easy interconnection of existing servers and applications or
frameworks, not to create a new web framework.

Note also that this goal precludes WSGI from requiring anything that
is not already available in deployed versions of Python.  Therefore,
new standard library modules are not proposed or required by this
specification, and nothing in WSGI requires a Python version greater
than 2.2.2.  (It would be a good idea, however, for future versions
of Python to include support for this interface in web servers
provided by the standard library.)

In addition to ease of implementation for existing and future
frameworks and servers, it should also be easy to create request
preprocessors, response postprocessors, and other WSGI-based
"middleware" components that look like an application to their
containing server, while acting as a server for their contained
applications.

If middleware can be both simple and robust, and WSGI is widely
available in servers and frameworks, it allows for the possibility
of an entirely new kind of Python web application framework: one
consisting of loosely-coupled WSGI middleware components.  Indeed,
existing framework authors may even choose to refactor their
frameworks' existing services to be provided in this way, becoming
more like libraries used with WSGI, and less like monolithic
frameworks.  This would then allow application developers to choose
"best-of-breed" components for specific functionality, rather than
having to commit to all the pros and cons of a single framework.

Of course, as of this writing, that day is doubtless quite far off.
In the meantime, it is a sufficient short-term goal for WSGI to
enable the use of any framework with any server.

Finally, it should be mentioned that the current version of WSGI
does not prescribe any particular mechanism for "deploying" an
application for use with a web server or server gateway.  At the
present time, this is necessarily implementation-defined by the
server or gateway.  After a sufficient number of servers and
frameworks have implemented WSGI to provide field experience with
varying deployment requirements, it may make sense to create
another PEP, describing a deployment standard for WSGI servers and
application frameworks.


Specification Overview
======================

The WSGI interface has two sides: the "server" or "gateway" side, and
the "application" or "framework" side.  The server side invokes a
callable object that is provided by the application side.  The
specifics of how that object is provided are up to the server or
gateway.  It is assumed that some servers or gateways will require an
application's deployer to write a short script to create an instance
of the server or gateway, and supply it with the application object.
Other servers and gateways may use configuration files or other
mechanisms to specify where an application object should be
imported from, or otherwise obtained.

In addition to "pure" servers/gateways and applications/frameworks,
it is also possible to create "middleware" components that implement
both sides of this specification.  Such components act as an
application to their containing server, and as a server to a
contained application, and can be used to provide extended APIs,
content transformation, navigation, and other useful functions.

Throughout this specification, we will use the term "a callable" to
mean "a function, method, class, or an instance with a ``__call__``
method".  It is up to the server, gateway, or application implementing
the callable to choose the appropriate implementation technique for
their needs.  Conversely, a server, gateway, or application that is
invoking a callable **must not** have any dependency on what kind of
callable was provided to it.  Callables are only to be called, not
introspected upon.


A Note On String Types
----------------------

In general, HTTP deals with bytes, which means that this specification
is mostly about handling bytes.

However, the content of those bytes often has some kind of textual
interpretation, and in Python, strings are the most convenient way
to handle text.

But in many Python versions and implementations, strings are Unicode,
rather than bytes.  This requires a careful balan


id: pep-3133.rst
content:
PEP: 3133
Title: Introducing Roles
Author: Collin Winter <collinwinter@google.com>
Status: Rejected
Type: Standards Track
Requires: 3115, 3129
Created: 01-May-2007
Python-Version: 3.0
Post-History: 13-May-2007


Rejection Notice
================

This PEP has helped push :pep:`3119` towards a saner, more minimalistic
approach.  But given the latest version of :pep:`3119` I much prefer
that.  GvR.


Abstract
========

Python's existing object model organizes objects according to their
implementation.  It is often desirable -- especially in
duck typing-based language like Python -- to organize objects by
the part they play in a larger system (their intent), rather than by
how they fulfill that part (their implementation).  This PEP
introduces the concept of roles, a mechanism for organizing
objects according to their intent rather than their implementation.


Rationale
=========

In the beginning were objects.  They allowed programmers to marry
function and state, and to increase code reusability through concepts
like polymorphism and inheritance, and lo, it was good.  There came
a time, however, when inheritance and polymorphism weren't enough.
With the invention of both dogs and trees, we were no longer able to
be content with knowing merely, "Does it understand 'bark'?"
We now needed to know what a given object thought that "bark" meant.

One solution, the one detailed here, is that of roles, a mechanism
orthogonal and complementary to the traditional class/instance system.
Whereas classes concern themselves with state and implementation, the
roles mechanism deals exclusively with the behaviours embodied in a
given class.

This system was originally called "traits" and implemented for Squeak
Smalltalk [#traits-paper]_.  It has since been adapted for use in
Perl 6 [#perl6-s12]_ where it is called "roles", and it is primarily
from there that the concept is now being interpreted for Python 3.
Python 3 will preserve the name "roles".

In a nutshell: roles tell you *what* an object does, classes tell you
*how* an object does it.

In this PEP, I will outline a system for Python 3 that will make it
possible to easily determine whether a given object's understanding
of "bark" is tree-like or dog-like.  (There might also be more
serious examples.)


A Note on Syntax
----------------

A syntax proposals in this PEP are tentative and should be
considered to be strawmen.  The necessary bits that this PEP depends
on -- namely :pep:`3115`'s class definition syntax and :pep:`3129`'s class
decorators -- are still being formalized and may change.  Function
names will, of course, be subject to lengthy bikeshedding debates.


Performing Your Role
====================

Static Role Assignment
----------------------

Let's start out by defining ``Tree`` and ``Dog`` classes ::

  class Tree(Vegetable):

    def bark(self):
      return self.is_rough()


  class Dog(Animal):

    def bark(self):
      return self.goes_ruff()

While both implement a ``bark()`` method with the same signature,
they do wildly different things.  We need some way of differentiating
what we're expecting. Relying on inheritance and a simple
``isinstance()`` test will limit code reuse and/or force any dog-like
classes to inherit from ``Dog``, whether or not that makes sense.
Let's see if roles can help. ::

  @perform_role(Doglike)
  class Dog(Animal):
    ...

  @perform_role(Treelike)
  class Tree(Vegetable):
    ...

  @perform_role(SitThere)
  class Rock(Mineral):
    ...

We use class decorators from :pep:`3129` to associate a particular role
or roles with a class.  Client code can now verify that an incoming
object performs the ``Doglike`` role, allowing it to handle ``Wolf``,
``LaughingHyena`` and ``Aibo`` [#aibo]_ instances, too.

Roles can be composed via normal inheritance: ::

  @perform_role(Guard, MummysLittleDarling)
  class GermanShepherd(Dog):

    def guard(self, the_precious):
      while True:
        if intruder_near(the_precious):
          self.growl()

    def get_petted(self):
      self.swallow_pride()

Here, ``GermanShepherd`` instances perform three roles: ``Guard`` and
``MummysLittleDarling`` are applied directly, whereas ``Doglike``
is inherited from ``Dog``.


Assigning Roles at Runtime
--------------------------

Roles can be assigned at runtime, too, by unpacking the syntactic
sugar provided by decorators.

Say we import a ``Robot`` class from another module, and since we
know that ``Robot`` already implements our ``Guard`` interface,
we'd like it to play nicely with guard-related code, too. ::

  >>> perform(Guard)(Robot)

This takes effect immediately and impacts all instances of ``Robot``.


Asking Questions About Roles
----------------------------

Just because we've told our robot army that they're guards, we'd
like to check in on them occasionally and make sure they're still at
their task. ::

  >>> performs(our_robot, Guard)
  True

What about that one robot over there? ::

  >>> performs(that_robot_over_there, Guard)
  True

The ``performs()`` function is used to ask if a given object
fulfills a given role.  It cannot be used, however, to ask a
class if its instances fulfill a role: ::

  >>> performs(Robot, Guard)
  False

This is because the ``Robot`` class is not interchangeable
with a ``Robot`` instance.


Defining New Roles
==================

Empty Roles
-----------

Roles are defined like a normal class, but use the ``Role``
metaclass. ::

  class Doglike(metaclass=Role):
    ...

Metaclasses are used to indicate that ``Doglike`` is a ``Role`` in
the same way 5 is an ``int`` and ``tuple`` is a ``type``.


Composing Roles via Inheritance
-------------------------------

Roles may inherit from other roles; this has the effect of composing
them.  Here, instances of ``Dog`` will perform both the
``Doglike`` and ``FourLegs`` roles. ::

  class FourLegs(metaclass=Role):
    pass

  class Doglike(FourLegs, Carnivor):
    pass

  @perform_role(Doglike)
  class Dog(Mammal):
    pass


Requiring Concrete Methods
--------------------------

So far we've only defined empty roles -- not very useful things.
Let's now require that all classes that claim to fulfill the
``Doglike`` role define a ``bark()`` method: ::

  class Doglike(FourLegs):

    def bark(self):
      pass

No decorators are required to flag the method as "abstract", and the
method will never be called, meaning whatever code it contains (if any)
is irrelevant.  Roles provide *only* abstract methods; concrete
default implementations are left to other, better-suited mechanisms
like mixins.

Once you have defined a role, and a class has claimed to perform that
role, it is essential that that claim be verified.  Here, the
programmer has misspelled one of the methods required by the role. ::

  @perform_role(FourLegs)
  class Horse(Mammal):

    def run_like_teh_wind(self)
      ...

This will cause the role system to raise an exception, complaining
that you're missing a ``run_like_the_wind()`` method.  The role
system carries out these checks as soon as a class is flagged as
performing a given role.

Concrete methods are required to match exactly the signature demanded
by the role.  Here, we've attempted to fulfill our role by defining a
concrete version of ``bark()``, but we've missed the mark a bit. ::

  @perform_role(Doglike)
  class Coyote(Mammal):

    def bark(self, target=moon):
      pass

This method's signature doesn't match exactly with what the
``Doglike`` role was expecting, so the role system will throw a bit
of a tantrum.


Mechanism
=========

The following are strawman proposals for how roles might be expressed
in Python.  The examples here are phrased in a way that the roles
mechanism may be implemented without changing the Python interpreter.
(Examples adapted from an article on Perl 6 roles by Curtis Poe
[#roles-examples]_.)

1. Static class role assignment ::

     @perform_role(Thieving)
     class Elf(Character):
       ...

   ``perform_role()`` accepts multiple arguments, suc


id: pep-3127.rst
content:
PEP: 3127
Title: Integer Literal Support and Syntax
Author: Patrick Maupin <pmaupin@gmail.com>
Discussions-To: python-3000@python.org
Status: Final
Type: Standards Track
Created: 14-Mar-2007
Python-Version: 3.0
Post-History: 18-Mar-2007


Abstract
========

This PEP proposes changes to the Python core to rationalize
the treatment of string literal representations of integers
in different radices (bases).  These changes are targeted at
Python 3.0, but the backward-compatible parts of the changes
should be added to Python 2.6, so that all valid 3.0 integer
literals will also be valid in 2.6.

The proposal is that:

a) octal literals must now be specified
   with a leading "0o" or "0O" instead of "0";

b) binary literals are now supported via a
   leading "0b" or "0B"; and

c) provision will be made for binary numbers in
   string formatting.


Motivation
==========

This PEP was motivated by two different issues:

- The default octal representation of integers is silently confusing
  to people unfamiliar with C-like languages.  It is extremely easy
  to inadvertently create an integer object with the wrong value,
  because '013' means 'decimal 11', not 'decimal 13', to the Python
  language itself, which is not the meaning that most humans would
  assign to this literal.

- Some Python users have a strong desire for binary support in
  the language.


Specification
=============

Grammar specification
---------------------

The grammar will be changed.  For Python 2.6, the changed and
new token definitions will be::

     integer        ::=     decimalinteger | octinteger | hexinteger |
                            bininteger | oldoctinteger

     octinteger     ::=     "0" ("o" | "O") octdigit+

     bininteger     ::=     "0" ("b" | "B") bindigit+

     oldoctinteger  ::=     "0" octdigit+

     bindigit       ::=     "0" | "1"

For Python 3.0, "oldoctinteger" will not be supported, and
an exception will be raised if a literal has a leading "0" and
a second character which is a digit.

For both versions, this will require changes to PyLong_FromString
as well as the grammar.

The documentation will have to be changed as well:  grammar.txt,
as well as the integer literal section of the reference manual.

:pep:`306` should be checked for other issues, and that PEP should
be updated if the procedure described therein is insufficient.

int() specification
--------------------

int(s, 0) will also match the new grammar definition.

This should happen automatically with the changes to
PyLong_FromString required for the grammar change.

Also the documentation for int() should be changed to explain
that int(s) operates identically to int(s, 10), and the word
"guess" should be removed from the description of int(s, 0).

long() specification
--------------------

For Python 2.6, the long() implementation and documentation
should be changed to reflect the new grammar.

Tokenizer exception handling
----------------------------

If an invalid token contains a leading "0", the exception
error message should be more informative than the current
"SyntaxError: invalid token".  It should explain that decimal
numbers may not have a leading zero, and that octal numbers
require an "o" after the leading zero.

int() exception handling
------------------------

The ValueError raised for any call to int() with a string
should at least explicitly contain the base in the error
message, e.g.::

    ValueError: invalid literal for base 8 int(): 09

oct() function
---------------

oct() should be updated to output '0o' in front of
the octal digits (for 3.0, and 2.6 compatibility mode).

Output formatting
-----------------

In 3.0, the string % operator alternate syntax for the 'o'
option will need to be updated to add '0o' in front,
instead of '0'.  In 2.6, alternate octal formatting will
continue to add only '0'.  In neither 2.6 nor 3.0 will
the % operator support binary output.  This is because
binary output is already supported by :pep:`3101`
(str.format), which is the preferred string formatting
method.


Transition from 2.6 to 3.0
---------------------------

The 2to3 translator will have to insert 'o' into any
octal string literal.

The Py3K compatible option to Python 2.6 should cause
attempts to use oldoctinteger literals to raise an
exception.


Rationale
=========

Most of the discussion on these issues occurred on the Python-3000
mailing list starting 14-Mar-2007, prompted by an observation that
the average human being would be completely mystified upon finding
that prepending a "0" to a string of digits changes the meaning of
that digit string entirely.

It was pointed out during this discussion that a similar, but shorter,
discussion on the subject occurred in January 2006, prompted by a
discovery of the same issue.

Background
----------

For historical reasons, Python's string representation of integers
in different bases (radices), for string formatting and token
literals, borrows heavily from C.  [1]_ [2]_ Usage has shown that
the historical method of specifying an octal number is confusing,
and also that it would be nice to have additional support for binary
literals.

Throughout this document, unless otherwise noted, discussions about
the string representation of integers relate to these features:

- Literal integer tokens, as used by normal module compilation,
  by eval(), and by int(token, 0).  (int(token) and int(token, 2-36)
  are not modified by this proposal.)

  * Under 2.6, long() is treated the same as int()

- Formatting of integers into strings, either via the % string
  operator or the new :pep:`3101` advanced string formatting method.

It is presumed that:

- All of these features should have an identical set
  of supported radices, for consistency.

- Python source code syntax and int(mystring, 0) should
  continue to share identical behavior.


Removal of old octal syntax
----------------------------

This PEP proposes that the ability to specify an octal number by
using a leading zero will be removed from the language in Python 3.0
(and the Python 3.0 preview mode of 2.6), and that a SyntaxError will
be raised whenever a leading "0" is immediately followed by another
digit.

During the present discussion, it was almost universally agreed that::

    eval('010') == 8

should no longer be true, because that is confusing to new users.
It was also proposed that::

    eval('0010') == 10

should become true, but that is much more contentious, because it is so
inconsistent with usage in other computer languages that mistakes are
likely to be made.

Almost all currently popular computer languages, including C/C++,
Java, Perl, and JavaScript, treat a sequence of digits with a
leading zero as an octal number.  Proponents of treating these
numbers as decimal instead have a very valid point -- as discussed
in `Supported radices`_, below, the entire non-computer world uses
decimal numbers almost exclusively.  There is ample anecdotal
evidence that many people are dismayed and confused if they
are confronted with non-decimal radices.

However, in most situations, most people do not write gratuitous
zeros in front of their decimal numbers.  The primary exception is
when an attempt is being made to line up columns of numbers.  But
since :pep:`8` specifically discourages the use of spaces to try to
align Python code, one would suspect the same argument should apply
to the use of leading zeros for the same purpose.

Finally, although the email discussion often focused on whether anybody
actually *uses* octal any more, and whether we should cater to those
old-timers in any case, that is almost entirely besides the point.

Assume the rare complete newcomer to computing who *does*, either
occasionally or as a matter of habit, use leading zeros for decimal
numbers.  Python could either:

a) silently do the wrong thing with their numbers, as it does now;

b) immediately disabuse them of the notion that this is viable syntax
   (and yes, the SyntaxWarning should be m


id: pep-3126.rst
content:
PEP: 3126
Title: Remove Implicit String Concatenation
Author: Jim J. Jewett <JimJJewett@gmail.com>,
        Raymond Hettinger <python@rcn.com>
Status: Rejected
Type: Standards Track
Created: 29-Apr-2007
Post-History: 29-Apr-2007, 30-Apr-2007, 07-May-2007


Rejection Notice
================

This PEP is rejected.  There wasn't enough support in favor, the
feature to be removed isn't all that harmful, and there are some use
cases that would become harder.


Abstract
========

Python inherited many of its parsing rules from C.  While this has
been generally useful, there are some individual rules which are less
useful for python, and should be eliminated.

This PEP proposes to eliminate implicit string concatenation based
only on the adjacency of literals.

Instead of::

    "abc" "def" == "abcdef"

authors will need to be explicit, and either add the strings::

    "abc" + "def" == "abcdef"

or join them::

    "".join(["abc", "def"]) == "abcdef"


Motivation
==========

One goal for Python 3000 should be to simplify the language by
removing unnecessary features.  Implicit string concatenation should
be dropped in favor of existing techniques. This will simplify the
grammar and simplify a user's mental picture of Python.  The latter is
important for letting the language "fit in your head".  A large group
of current users do not even know about implicit concatenation.  Of
those who do know about it, a large portion never use it or habitually
avoid it. Of those who both know about it and use it, very few could
state with confidence the implicit operator precedence and under what
circumstances it is computed when the definition is compiled versus
when it is run.


History or Future
-----------------

Many Python parsing rules are intentionally compatible with C.  This
is a useful default, but Special Cases need to be justified based on
their utility in Python.  We should no longer assume that python
programmers will also be familiar with C, so compatibility between
languages should be treated as a tie-breaker, rather than a
justification.

In C, implicit concatenation is the only way to join strings without
using a (run-time) function call to store into a variable.  In Python,
the strings can be joined (and still recognized as immutable) using
more standard Python idioms, such ``+`` or ``"".join``.


Problem
-------

Implicit String concatenation leads to tuples and lists which are
shorter than they appear; this is turn can lead to confusing, or even
silent, errors.  For example, given a function which accepts several
parameters, but offers a default value for some of them::

    def f(fmt, *args):
        print fmt % args

This looks like a valid call, but isn't::

    >>> f("User %s got a message %s",
          "Bob"
          "Time for dinner")

    Traceback (most recent call last):
      File "<pyshell#8>", line 2, in <module>
        "Bob"
      File "<pyshell#3>", line 2, in f
        print fmt % args
    TypeError: not enough arguments for format string


Calls to this function can silently do the wrong thing::

    def g(arg1, arg2=None):
        ...

    # silently transformed into the possibly very different
    # g("arg1 on this linearg2 on this line", None)
    g("arg1 on this line"
      "arg2 on this line")

To quote Jason Orendorff [#Orendorff]_:

    Oh.  I just realized this happens a lot out here.  Where I work,
    we use scons, and each SConscript has a long list of filenames::

        sourceFiles = [
            'foo.c'
            'bar.c',
            #...many lines omitted...
            'q1000x.c']

    It's a common mistake to leave off a comma, and then scons
    complains that it can't find 'foo.cbar.c'.  This is pretty
    bewildering behavior even if you *are* a Python programmer,
    and not everyone here is.


Solution
========

In Python, strings are objects and they support the __add__ operator,
so it is possible to write::

    "abc" + "def"

Because these are literals, this addition can still be optimized away
by the compiler; the CPython compiler already does so.
[#rcn-constantfold]_

Other existing alternatives include multiline (triple-quoted) strings,
and the join method::

    """This string
       extends across
       multiple lines, but you may want to use something like
       Textwrap.dedent
       to clear out the leading spaces
       and/or reformat.
    """


    >>> "".join(["empty", "string", "joiner"]) == "emptystringjoiner"
    True

    >>> " ".join(["space", "string", "joiner"]) == "space string joiner"
    True

    >>> "\n".join(["multiple", "lines"]) == "multiple\nlines" == (
    """multiple
    lines""")
    True


Concerns
========


Operator Precedence
-------------------

Guido indicated [#rcn-constantfold]_ that this change should be
handled by PEP, because there were a few edge cases with other string
operators, such as the %.  (Assuming that str % stays -- it may be
eliminated in favor of :pep:`3101` -- Advanced String Formatting.
[#elimpercent]_)

The resolution is to use parentheses to enforce precedence -- the same
solution that can be used today::

    # Clearest, works today, continues to work, optimization is
    # already possible.
    ("abc %s def" + "ghi") % var

    # Already works today; precedence makes the optimization more
    # difficult to recognize, but does not change the semantics.
    "abc" + "def %s ghi" % var

as opposed to::

    # Already fails because modulus (%) is higher precedence than
    # addition (+)
    ("abc %s def" + "ghi" % var)

    # Works today only because adjacency is higher precedence than
    # modulus.  This will no longer be available.
    "abc %s" "def" % var

    # So the 2-to-3 translator can automatically replace it with the
    # (already valid):
    ("abc %s" + "def") % var


Long Commands
-------------

    ... build up (what I consider to be) readable SQL queries [#skipSQL]_::

        rows = self.executesql("select cities.city, state, country"
                               "    from cities, venues, events, addresses"
                               "    where cities.city like %s"
                               "      and events.active = 1"
                               "      and venues.address = addresses.id"
                               "      and addresses.city = cities.id"
                               "      and events.venue = venues.id",
                               (city,))

Alternatives again include triple-quoted strings, ``+``, and ``.join``::

    query="""select cities.city, state, country
                 from cities, venues, events, addresses
                 where cities.city like %s
                   and events.active = 1"
                   and venues.address = addresses.id
                   and addresses.city = cities.id
                   and events.venue = venues.id"""

    query=( "select cities.city, state, country"
          + "    from cities, venues, events, addresses"
          + "    where cities.city like %s"
          + "      and events.active = 1"
          + "      and venues.address = addresses.id"
          + "      and addresses.city = cities.id"
          + "      and events.venue = venues.id"
          )

    query="\n".join(["select cities.city, state, country",
                     "    from cities, venues, events, addresses",
                     "    where cities.city like %s",
                     "      and events.active = 1",
                     "      and venues.address = addresses.id",
                     "      and addresses.city = cities.id",
                     "      and events.venue = venues.id"])

    # And yes, you *could* inline any of the above querystrings
    # the same way the original was inlined.
    rows = self.executesql(query, (city,))


Regular Expressions
-------------------

Complex regular expressions are sometimes stated in terms of several
implicitly concatenated strings with each regex component on a
different line and followed by a comment.  The plus ope


id: pep-3132.rst
content:
PEP: 3132
Title: Extended Iterable Unpacking
Author: Georg Brandl <georg@python.org>
Status: Final
Type: Standards Track
Created: 30-Apr-2007
Python-Version: 3.0
Post-History:


Abstract
========

This PEP proposes a change to iterable unpacking syntax, allowing to
specify a "catch-all" name which will be assigned a list of all items
not assigned to a "regular" name.

An example says more than a thousand words::

    >>> a, *b, c = range(5)
    >>> a
    0
    >>> c
    4
    >>> b
    [1, 2, 3]


Rationale
=========

Many algorithms require splitting a sequence in a "first, rest" pair.
With the new syntax, ::

    first, rest = seq[0], seq[1:]

is replaced by the cleaner and probably more efficient::

    first, *rest = seq

For more complex unpacking patterns, the new syntax looks even
cleaner, and the clumsy index handling is not necessary anymore.

Also, if the right-hand value is not a list, but an iterable, it
has to be converted to a list before being able to do slicing; to
avoid creating this temporary list, one has to resort to ::

    it = iter(seq)
    first = it.next()
    rest = list(it)


Specification
=============

A tuple (or list) on the left side of a simple assignment (unpacking
is not defined for augmented assignment) may contain at most one
expression prepended with a single asterisk (which is henceforth
called a "starred" expression, while the other expressions in the
list are called "mandatory").  This designates a subexpression that
will be assigned a list of all items from the iterable being unpacked
that are not assigned to any of the mandatory expressions, or an
empty list if there are no such items.

For example, if ``seq`` is a sliceable sequence, all the following
assignments are equivalent if ``seq`` has at least two elements::

    a, b, c = seq[0], list(seq[1:-1]), seq[-1]
    a, *b, c = seq
    [a, *b, c] = seq

It is an error (as it is currently) if the iterable doesn't contain
enough items to assign to all the mandatory expressions.

It is also an error to use the starred expression as a lone
assignment target, as in ::

    *a = range(5)

This, however, is valid syntax::

    *a, = range(5)

Note that this proposal also applies to tuples in implicit assignment
context, such as in a ``for`` statement::

    for a, *b in [(1, 2, 3), (4, 5, 6, 7)]:
        print(b)

would print out ::

    [2, 3]
    [5, 6, 7]

Starred expressions are only allowed as assignment targets, using them
anywhere else (except for star-args in function calls, of course) is an
error.


Implementation
==============

Grammar change
--------------

This feature requires a new grammar rule::

    star_expr: ['*'] expr

In these two rules, ``expr`` is changed to ``star_expr``::

    comparison: star_expr (comp_op star_expr)*
    exprlist: star_expr (',' star_expr)* [',']

Changes to the Compiler
-----------------------

A new ASDL expression type ``Starred`` is added which represents a
starred expression.  Note that the starred expression element
introduced here is universal and could later be used for other
purposes in non-assignment context, such as the ``yield *iterable``
proposal.

The compiler is changed to recognize all cases where a starred
expression is invalid and flag them with syntax errors.

A new bytecode instruction, ``UNPACK_EX``, is added, whose argument
has the number of mandatory targets before the starred target in the
lower 8 bits and the number of mandatory targets after the starred
target in the upper 8 bits.  For unpacking sequences without starred
expressions, the old ``UNPACK_ITERABLE`` opcode is kept.

Changes to the Bytecode Interpreter
-----------------------------------

The function ``unpack_iterable()`` in ceval.c is changed to handle
the extended unpacking, via an ``argcntafter`` parameter. In the
``UNPACK_EX`` case, the function will do the following:

* collect all items for mandatory targets before the starred one
* collect all remaining items from the iterable in a list
* pop items for mandatory targets after the starred one from the list
* push the single items and the resized list on the stack

Shortcuts for unpacking iterables of known types, such as lists or
tuples, can be added.


The current implementation can be found at the SourceForge Patch
tracker [SFPATCH]_. It now includes a minimal test case.


Acceptance
==========

After a short discussion on the python-3000 list [1]_, the PEP was
accepted by Guido in its current form.  Possible changes discussed
were:

* Only allow a starred expression as the last item in the exprlist.
  This would simplify the unpacking code a bit and allow for the
  starred expression to be assigned an iterator.  This behavior was
  rejected because it would be too surprising.

* Try to give the starred target the same type as the source
  iterable, for example, ``b`` in ``a, *b = 'hello'`` would be
  assigned the string ``'ello'``.  This may seem nice, but is
  impossible to get right consistently with all iterables.

* Make the starred target a tuple instead of a list.  This would be
  consistent with a function's ``*args``, but make further processing
  of the result harder.


References
==========

.. [SFPATCH] https://bugs.python.org/issue1711529
.. [1] https://mail.python.org/pipermail/python-3000/2007-May/007198.html


Copyright
=========

This document has been placed in the public domain.


id: pep-3155.rst
content:
PEP: 3155
Title: Qualified name for classes and functions
Author: Antoine Pitrou <solipsis@pitrou.net>
Status: Final
Type: Standards Track
Created: 29-Oct-2011
Python-Version: 3.3
Post-History:
Resolution: https://mail.python.org/pipermail/python-dev/2011-November/114545.html


Rationale
=========

Python's introspection facilities have long had poor support for
nested classes.  Given a class object, it is impossible to know
whether it was defined inside another class or at module top-level;
and, if the former, it is also impossible to know in which class it
was defined.  While use of nested classes is often considered poor
style, the only reason for them to have second class introspection
support is a lousy pun.

Python 3 adds insult to injury by dropping what was formerly known as
unbound methods.  In Python 2, given the following definition::

    class C:
        def f():
            pass

you can then walk up from the ``C.f`` object to its defining class::

    >>> C.f.im_class
    <class '__main__.C'>

This possibility is gone in Python 3::

    >>> C.f.im_class
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: 'function' object has no attribute 'im_class'
    >>> dir(C.f)
    ['__annotations__', '__call__', '__class__', '__closure__', '__code__',
    '__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__',
    '__eq__', '__format__', '__ge__', '__get__', '__getattribute__',
    '__globals__', '__gt__', '__hash__', '__init__', '__kwdefaults__',
    '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__',
    '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__',
    '__str__', '__subclasshook__']

This limits again the introspection capabilities available to the
user.  It can produce actual issues when porting software to Python 3,
for example Twisted Core where the issue of introspecting method
objects came up several times.  It also limits pickling support [1]_.


Proposal
========

This PEP proposes the addition of a ``__qualname__`` attribute to
functions and classes.  For top-level functions and classes, the
``__qualname__`` attribute is equal to the ``__name__`` attribute.  For
nested classes, methods, and nested functions, the ``__qualname__``
attribute contains a dotted path leading to the object from the module
top-level.  A function's local namespace is represented in that dotted
path by a component named ``<locals>``.

The repr() and str() of functions and classes is modified to use
``__qualname__`` rather than ``__name__``.

Example with nested classes
---------------------------

>>> class C:
...   def f(): pass
...   class D:
...     def g(): pass
...
>>> C.__qualname__
'C'
>>> C.f.__qualname__
'C.f'
>>> C.D.__qualname__
'C.D'
>>> C.D.g.__qualname__
'C.D.g'

Example with nested functions
-----------------------------

>>> def f():
...   def g(): pass
...   return g
...
>>> f.__qualname__
'f'
>>> f().__qualname__
'f.<locals>.g'


Limitations
===========

With nested functions (and classes defined inside functions), the
dotted path will not be walkable programmatically as a function's
namespace is not available from the outside.  It will still be more
helpful to the human reader than the bare ``__name__``.

As the ``__name__`` attribute, the ``__qualname__`` attribute is computed
statically and it will not automatically follow rebinding.


Discussion
==========

Excluding the module name
-------------------------

As ``__name__``, ``__qualname__`` doesn't include the module name.  This
makes it independent of module aliasing and rebinding, and also allows to
compute it at compile time.

Reviving unbound methods
------------------------

Reviving unbound methods would only solve a fraction of the problems this
PEP solves, at a higher price (an additional object type and an additional
indirection, rather than an additional attribute).


Naming choice
=============

"Qualified name" is the best approximation, as a short phrase, of what the
additional attribute is about.  It is not a "full name" or "fully qualified
name" since it (deliberately) does not include the module name.  Calling
it a "path" would risk confusion with filesystem paths and the ``__file__``
attribute.

The first proposal for the attribute name was to call it ``__qname__`` but
many people (who are not aware of previous use of such jargon in e.g. the
XML specification [2]_) found it obscure and non-obvious, which is why the
slightly less short and more explicit ``__qualname__`` was finally chosen.


References
==========

.. [1] "pickle should support methods":
   http://bugs.python.org/issue9276

.. [2] "QName" entry in Wikipedia:
   http://en.wikipedia.org/wiki/QName


Copyright
=========

This document has been placed in the public domain.


id: pep-3141.rst
content:
PEP: 3141
Title: A Type Hierarchy for Numbers
Author: Jeffrey Yasskin <jyasskin@google.com>
Status: Final
Type: Standards Track
Created: 23-Apr-2007
Python-Version: 3.0
Post-History: 25-Apr-2007, 16-May-2007, 02-Aug-2007


Abstract
========

This proposal defines a hierarchy of Abstract Base Classes (ABCs) (PEP
3119) to represent number-like classes. It proposes a hierarchy of
``Number :> Complex :> Real :> Rational :> Integral`` where ``A :> B``
means "A is a supertype of B". The hierarchy is inspired by Scheme's
numeric tower [#schemetower]_.

Rationale
=========

Functions that take numbers as arguments should be able to determine
the properties of those numbers, and if and when overloading based on
types is added to the language, should be overloadable based on the
types of the arguments. For example, slicing requires its arguments to
be ``Integrals``, and the functions in the ``math`` module require
their arguments to be ``Real``.

Specification
=============

This PEP specifies a set of Abstract Base Classes, and suggests a
general strategy for implementing some of the methods. It uses
terminology from :pep:`3119`, but the hierarchy is intended to be
meaningful for any systematic method of defining sets of classes.

The type checks in the standard library should use these classes
instead of the concrete built-ins.


Numeric Classes
---------------

We begin with a Number class to make it easy for people to be fuzzy
about what kind of number they expect. This class only helps with
overloading; it doesn't provide any operations. ::

    class Number(metaclass=ABCMeta): pass


Most implementations of complex numbers will be hashable, but if you
need to rely on that, you'll have to check it explicitly: mutable
numbers are supported by this hierarchy. ::

    class Complex(Number):
        """Complex defines the operations that work on the builtin complex type.

        In short, those are: conversion to complex, bool(), .real, .imag,
        +, -, *, /, **, abs(), .conjugate(), ==, and !=.

        If it is given heterogeneous arguments, and doesn't have special
        knowledge about them, it should fall back to the builtin complex
        type as described below.
        """

        @abstractmethod
        def __complex__(self):
            """Return a builtin complex instance."""

        def __bool__(self):
            """True if self != 0."""
            return self != 0

        @abstractproperty
        def real(self):
            """Retrieve the real component of this number.

            This should subclass Real.
            """
            raise NotImplementedError

        @abstractproperty
        def imag(self):
            """Retrieve the imaginary component of this number.

            This should subclass Real.
            """
            raise NotImplementedError

        @abstractmethod
        def __add__(self, other):
            raise NotImplementedError

        @abstractmethod
        def __radd__(self, other):
            raise NotImplementedError

        @abstractmethod
        def __neg__(self):
            raise NotImplementedError

        def __pos__(self):
            """Coerces self to whatever class defines the method."""
            raise NotImplementedError

        def __sub__(self, other):
            return self + -other

        def __rsub__(self, other):
            return -self + other

        @abstractmethod
        def __mul__(self, other):
            raise NotImplementedError

        @abstractmethod
        def __rmul__(self, other):
            raise NotImplementedError

        @abstractmethod
        def __div__(self, other):
            """a/b; should promote to float or complex when necessary."""
            raise NotImplementedError

        @abstractmethod
        def __rdiv__(self, other):
            raise NotImplementedError

        @abstractmethod
        def __pow__(self, exponent):
            """a**b; should promote to float or complex when necessary."""
            raise NotImplementedError

        @abstractmethod
        def __rpow__(self, base):
            raise NotImplementedError

        @abstractmethod
        def __abs__(self):
            """Returns the Real distance from 0."""
            raise NotImplementedError

        @abstractmethod
        def conjugate(self):
            """(x+y*i).conjugate() returns (x-y*i)."""
            raise NotImplementedError

        @abstractmethod
        def __eq__(self, other):
            raise NotImplementedError

        # __ne__ is inherited from object and negates whatever __eq__ does.


The ``Real`` ABC indicates that the value is on the real line, and
supports the operations of the ``float`` builtin. Real numbers are
totally ordered except for NaNs (which this PEP basically ignores). ::

    class Real(Complex):
        """To Complex, Real adds the operations that work on real numbers.

        In short, those are: conversion to float, trunc(), math.floor(),
        math.ceil(), round(), divmod(), //, %, <, <=, >, and >=.

        Real also provides defaults for some of the derived operations.
        """

        # XXX What to do about the __int__ implementation that's
        # currently present on float?  Get rid of it?

        @abstractmethod
        def __float__(self):
            """Any Real can be converted to a native float object."""
            raise NotImplementedError

        @abstractmethod
        def __trunc__(self):
            """Truncates self to an Integral.

            Returns an Integral i such that:
              * i>=0 iff self>0;
              * abs(i) <= abs(self);
              * for any Integral j satisfying the first two conditions,
                abs(i) >= abs(j) [i.e. i has "maximal" abs among those].
            i.e. "truncate towards 0".
            """
            raise NotImplementedError

        @abstractmethod
        def __floor__(self):
            """Finds the greatest Integral <= self."""
            raise NotImplementedError

        @abstractmethod
        def __ceil__(self):
            """Finds the least Integral >= self."""
            raise NotImplementedError

        @abstractmethod
        def __round__(self, ndigits:Integral=None):
            """Rounds self to ndigits decimal places, defaulting to 0.

            If ndigits is omitted or None, returns an Integral,
            otherwise returns a Real, preferably of the same type as
            self. Types may choose which direction to round half. For
            example, float rounds half toward even.

            """
            raise NotImplementedError

        def __divmod__(self, other):
            """The pair (self // other, self % other).

            Sometimes this can be computed faster than the pair of
            operations.
            """
            return (self // other, self % other)

        def __rdivmod__(self, other):
            """The pair (self // other, self % other).

            Sometimes this can be computed faster than the pair of
            operations.
            """
            return (other // self, other % self)

        @abstractmethod
        def __floordiv__(self, other):
            """The floor() of self/other. Integral."""
            raise NotImplementedError

        @abstractmethod
        def __rfloordiv__(self, other):
            """The floor() of other/self."""
            raise NotImplementedError

        @abstractmethod
        def __mod__(self, other):
            """self % other

            See
            https://mail.python.org/pipermail/python-3000/2006-May/001735.html
            and consider using "self/other - trunc(self/other)"
            instead if you're worried about round-off errors.
            """
            raise NotImplementedError

        @abstractmethod
        def __rmod__(self, other):
            """other % self"""
            raise NotImplementedError

        @abstractmethod
        def __lt__(self, other):
            """< on Reals defines a 


id: pep-3140.rst
content:
PEP: 3140
Title: str(container) should call str(item), not repr(item)
Author: Oleg Broytman <phd@phdru.name>,
        Jim J. Jewett <jimjjewett@gmail.com>
Discussions-To: python-3000@python.org
Status: Rejected
Type: Standards Track
Created: 27-May-2008
Post-History: 28-May-2008


Rejection
=========

Guido said this would cause too much disturbance too close to beta. See [1]_.


Abstract
========

This document discusses the advantages and disadvantages of the
current implementation of ``str(container)``.  It also discusses the
pros and cons of a different approach - to call ``str(item)`` instead
of ``repr(item)``.


Motivation
==========

Currently ``str(container)`` calls ``repr`` on items.  Arguments for it:

* containers refuse to guess what the user wants to see on
  ``str(container)`` - surroundings, delimiters, and so on;
* ``repr(item)`` usually displays type information - apostrophes
  around strings, class names, etc.

Arguments against:

* it's illogical; ``str()`` is expected to call ``__str__`` if it exists,
  not ``__repr__``;
* there is no standard way to print a container's content calling
  items' ``__str__``, that's inconvenient in cases where ``__str__`` and
  ``__repr__`` return different results;
* ``repr(item)`` sometimes do wrong things (hex-escapes non-ASCII strings,
  e.g.)

This PEP proposes to change how ``str(container)`` works.  It is
proposed to mimic how ``repr(container)`` works except one detail - call
``str`` on items instead of ``repr``.  This allows a user to choose
what results she want to get - from ``item.__repr__`` or ``item.__str__``.


Current situation
=================

Most container types (tuples, lists, dicts, sets, etc.) do not
implement ``__str__`` method, so ``str(container)`` calls
``container.__repr__``, and ``container.__repr__``, once called, forgets
it is called from ``str`` and always calls ``repr`` on the container's
items.

This behaviour has advantages and disadvantages.  One advantage is
that most items are represented with type information - strings
are surrounded by apostrophes, instances may have both class name
and instance data::

    >>> print([42, '42'])
    [42, '42']
    >>> print([Decimal('42'), datetime.now()])
    [Decimal("42"), datetime.datetime(2008, 5, 27, 19, 57, 43, 485028)]

The disadvantage is that ``__repr__`` often returns technical data
(like '``<object at address>``') or unreadable string (hex-encoded
string if the input is non-ASCII string)::

    >>> print(['ÑÐµÑÑ'])
    ['\xd4\xc5\xd3\xd4']

One of the motivations for :pep:`3138` is that neither ``repr`` nor ``str``
will allow the sensible printing of dicts whose keys are non-ASCII
text strings.  Now that Unicode identifiers are allowed, it
includes Python's own attribute dicts.  This also includes JSON
serialization (and caused some hoops for the json lib).

:pep:`3138` proposes to fix this by breaking the "repr is safe ASCII"
invariant, and changing the way ``repr`` (which is used for
persistence) outputs some objects, with system-dependent failures.

Changing how ``str(container)`` works would allow easy debugging in
the normal case, and retain the safety of ASCII-only for the
machine-readable  case.  The only downside is that ``str(x)`` and
``repr(x)`` would more often be different -- but only in those cases
where the current almost-the-same version is insufficient.

It also seems illogical that ``str(container)`` calls ``repr`` on items
instead of ``str``.  It's only logical to expect following code::

    class Test:
        def __str__(self):
            return "STR"

        def __repr__(self):
            return "REPR"


    test = Test()
    print(test)
    print(repr(test))
    print([test])
    print(str([test]))

to print::

    STR
    REPR
    [STR]
    [STR]

where it actually prints::

    STR
    REPR
    [REPR]
    [REPR]

Especially it is illogical to see that print in Python 2 uses ``str``
if it is called on what seems to be a tuple::

    >>> print Decimal('42'), datetime.now()
    42 2008-05-27 20:16:22.534285

where on an actual tuple it prints::

    >>> print((Decimal('42'), datetime.now()))
    (Decimal("42"), datetime.datetime(2008, 5, 27, 20, 16, 27, 937911))


A different approach - call ``str(item)``
=========================================

For example, with numbers it is often only the value that people
care about.

::

    >>> print Decimal('3')
    3

But putting the value in a list forces users to read the type
information, exactly as if ``repr`` had been called for the benefit of
a machine::

    >>> print [Decimal('3')]
    [Decimal("3")]

After this change, the type information would not clutter the ``str``
output::

    >>> print "%s".format([Decimal('3')])
    [3]
    >>> str([Decimal('3')])  # ==
    [3]

But it would still be available if desired::

    >>> print "%r".format([Decimal('3')])
    [Decimal('3')]
    >>> repr([Decimal('3')])  # ==
    [Decimal('3')]

There is a number of strategies to fix the problem.  The most
radical is to change ``__repr__`` so it accepts a new parameter (flag)
"called from ``str``, so call ``str`` on items, not ``repr``".  The
drawback of the proposal is that every ``__repr__`` implementation
must be changed.  Introspection could help a bit (inspect ``__repr__``
before calling if it accepts 2 or 3 parameters), but introspection
doesn't work on classes written in C, like all built-in containers.

Less radical proposal is to implement ``__str__`` methods for built-in
container types.  The obvious drawback is a duplication of effort - all
those ``__str__`` and ``__repr__`` implementations are only differ
in one small detail - if they call ``str`` or ``repr`` on items.

The most conservative proposal is not to change str at all but
to allow developers to implement their own application- or
library-specific pretty-printers.  The drawback is again
a multiplication of effort and proliferation of many small
specific container-traversal algorithms.


Backward compatibility
======================

In those cases where type information is more important than
usual, it will still be possible to get the current results by
calling ``repr`` explicitly.


References
==========

.. [1] Guido van Rossum, PEP: str(container) should call str(item), not
       repr(item)
       https://mail.python.org/pipermail/python-3000/2008-May/013876.html

Copyright
=========

This document has been placed in the public domain.


id: pep-3154.rst
content:
PEP: 3154
Title: Pickle protocol version 4
Author: Antoine Pitrou <solipsis@pitrou.net>
Status: Final
Type: Standards Track
Created: 11-Aug-2011
Python-Version: 3.4
Post-History: `12-Aug-2011 <https://mail.python.org/pipermail/python-dev/2011-August/112821.html>`__
Resolution: https://mail.python.org/pipermail/python-dev/2013-November/130439.html


Abstract
========

Data serialized using the pickle module must be portable across Python
versions.  It should also support the latest language features as well
as implementation-specific features.  For this reason, the pickle
module knows about several protocols (currently numbered from 0 to 3),
each of which appeared in a different Python version.  Using a
low-numbered protocol version allows to exchange data with old Python
versions, while using a high-numbered protocol allows access to newer
features and sometimes more efficient resource use (both CPU time
required for (de)serializing, and disk size / network bandwidth
required for data transfer).


Rationale
=========

The latest current protocol, coincidentally named protocol 3, appeared
with Python 3.0 and supports the new incompatible features in the
language (mainly, unicode strings by default and the new bytes
object).  The opportunity was not taken at the time to improve the
protocol in other ways.

This PEP is an attempt to foster a number of incremental improvements
in a new pickle protocol version.  The PEP process is used in order to
gather as many improvements as possible, because the introduction of a
new pickle protocol should be a rare occurrence.


Proposed changes
================

Framing
-------

Traditionally, when unpickling an object from a stream (by calling
``load()`` rather than ``loads()``), many small ``read()``
calls can be issued on the file-like object, with a potentially huge
performance impact.

Protocol 4, by contrast, features binary framing.  The general structure
of a pickle is thus the following::

    +------+------+
    | 0x80 | 0x04 |              protocol header (2 bytes)
    +------+------+
    |  OP  |                     FRAME opcode (1 byte)
    +------+------+-----------+
    | MM MM MM MM MM MM MM MM |  frame size (8 bytes, little-endian)
    +------+------------------+
    | .... |                     first frame contents (M bytes)
    +------+
    |  OP  |                     FRAME opcode (1 byte)
    +------+------+-----------+
    | NN NN NN NN NN NN NN NN |  frame size (8 bytes, little-endian)
    +------+------------------+
    | .... |                     second frame contents (N bytes)
    +------+
      etc.

To keep the implementation simple, it is forbidden for a pickle opcode
to straddle frame boundaries.  The pickler takes care not to produce such
pickles, and the unpickler refuses them.  Also, there is no "last frame"
marker.  The last frame is simply the one which ends with a STOP opcode.

A well-written C implementation doesn't need additional memory copies
for the framing layer, preserving general (un)pickling efficiency.

.. note::

   How the pickler decides to partition the pickle stream into frames is an
   implementation detail.  For example, "closing" a frame as soon as it
   reaches ~64 KiB is a reasonable choice for both performance and pickle
   size overhead.

Binary encoding for all opcodes
-------------------------------

The GLOBAL opcode, which is still used in protocol 3, uses the
so-called "text" mode of the pickle protocol, which involves looking
for newlines in the pickle stream.  It also complicates the implementation
of binary framing.

Protocol 4 forbids use of the GLOBAL opcode and replaces it with
STACK_GLOBAL, a new opcode which takes its operand from the stack.

Serializing more "lookupable" objects
-------------------------------------

By default, pickle is only able to serialize module-global functions and
classes.  Supporting other kinds of objects, such as unbound methods [4]_,
is a common request. Actually, third-party support for some of them, such
as bound methods, is implemented in the multiprocessing module [5]_.

The ``__qualname__`` attribute from :pep:`3155` makes it possible to
lookup many more objects by name.  Making the STACK_GLOBAL opcode accept
dot-separated names would allow the standard pickle implementation to
support all those kinds of objects.

64-bit opcodes for large objects
--------------------------------

Current protocol versions export object sizes for various built-in
types (str, bytes) as 32-bit ints.  This forbids serialization of
large data [1]_.  New opcodes are required to support very large bytes
and str objects.

Native opcodes for sets and frozensets
--------------------------------------

Many common built-in types (such as str, bytes, dict, list, tuple)
have dedicated opcodes to improve resource consumption when
serializing and deserializing them; however, sets and frozensets
don't.  Adding such opcodes would be an obvious improvement.  Also,
dedicated set support could help remove the current impossibility of
pickling self-referential sets [2]_.

Calling __new__ with keyword arguments
--------------------------------------

Currently, classes whose ``__new__`` mandates the use of keyword-only
arguments can not be pickled (or, rather, unpickled) [3]_.  Both a new
special method (``__getnewargs_ex__``) and a new opcode (NEWOBJ_EX)
are needed.  The ``__getnewargs_ex__`` method, if it exists, must
return a two-tuple ``(args, kwargs)`` where the first item is the
tuple of positional arguments and the second item is the dict of
keyword arguments for the class's ``__new__`` method.

Better string encoding
----------------------

Short str objects currently have their length coded as a 4-bytes
integer, which is wasteful.  A specific opcode with a 1-byte length
would make many pickles smaller.

Smaller memoization
-------------------

The PUT opcodes all require an explicit index to select in which entry
of the memo dictionary the top-of-stack is memoized.  However, in practice
those numbers are allocated in sequential order.  A new opcode, MEMOIZE,
will instead store the top-of-stack in at the index equal to the current
size of the memo dictionary.  This allows for shorter pickles, since PUT
opcodes are emitted for all non-atomic datatypes.


Summary of new opcodes
======================

These reflect the state of the proposed implementation (thanks mostly
to Alexandre Vassalotti's work):

* ``FRAME``: introduce a new frame (followed by the 8-byte frame size
  and the frame contents).

* ``SHORT_BINUNICODE``: push a utf8-encoded str object with a one-byte
  size prefix (therefore less than 256 bytes long).

* ``BINUNICODE8``: push a utf8-encoded str object with an eight-byte
  size prefix (for strings longer than 2**32 bytes, which therefore cannot
  be serialized using ``BINUNICODE``).

* ``BINBYTES8``: push a bytes object with an eight-byte size prefix
  (for bytes objects longer than 2**32 bytes, which therefore cannot be
  serialized using ``BINBYTES``).

* ``EMPTY_SET``: push a new empty set object on the stack.

* ``ADDITEMS``: add the topmost stack items to the set (to be used with
  ``EMPTY_SET``).

* ``FROZENSET``: create a frozenset object from the topmost stack items,
  and push it on the stack.

* ``NEWOBJ_EX``: take the three topmost stack items ``cls``, ``args``
  and ``kwargs``, and push the result of calling
  ``cls.__new__(*args, **kwargs)``.

* ``STACK_GLOBAL``: take the two topmost stack items ``module_name`` and
  ``qualname``, and push the result of looking up the dotted ``qualname``
  in the module named ``module_name``.

* ``MEMOIZE``: store the top-of-stack object in the memo dictionary with
  an index equal to the current size of the memo dictionary.


Alternative ideas
=================

Prefetching
-----------

Serhiy Storchaka suggested to replace framing with a special PREFETCH
opcode (with a 2- or 4-bytes argument) to declare known pickle chunks
explicitly. Large data may be pickled outside such ch


id: pep-3142.rst
content:
PEP: 3142
Title: Add a "while" clause to generator expressions
Author: Gerald Britton <gerald.britton@gmail.com>
Status: Rejected
Type: Standards Track
Created: 12-Jan-2009
Python-Version: 3.0
Post-History:
Resolution: https://mail.python.org/pipermail/python-dev/2013-May/126136.html


Abstract
========

This PEP proposes an enhancement to generator expressions, adding a
"while" clause to complement the existing "if" clause.


Rationale
=========

A generator expression (:pep:`289`) is a concise method to serve
dynamically-generated objects to list comprehensions (:pep:`202`).
Current generator expressions allow for an "if" clause to filter
the objects that are returned to those meeting some set of
criteria.  However, since the "if" clause is evaluated for every
object that may be returned, in some cases it is possible that all
objects would be rejected after a certain point.  For example::

    g = (n for n in range(100) if n*n < 50)

which is equivalent to the using a generator function
(:pep:`255`)::

    def __gen(exp):
        for n in exp:
            if n*n < 50:
                yield n
    g = __gen(iter(range(10)))

would yield 0, 1, 2, 3, 4, 5, 6 and 7, but would also consider
the numbers from 8 to 99 and reject them all since ``n*n >= 50`` for
numbers in that range.  Allowing for a "while" clause would allow
the redundant tests to be short-circuited::

    g = (n for n in range(100) while n*n < 50)

would also yield 0, 1, 2, 3, 4, 5, 6 and 7, but would stop at 8
since the condition (``n*n < 50``) is no longer true.  This would be
equivalent to the generator function::

    def __gen(exp):
        for n in exp:
            if n*n < 50:
                yield n
            else:
                break
    g = __gen(iter(range(100)))

Currently, in order to achieve the same result, one would need to
either write a generator function such as the one above or use the
takewhile function from itertools::

    from itertools import takewhile
    g = takewhile(lambda n: n*n < 50, range(100))

The takewhile code achieves the same result as the proposed syntax,
albeit in a longer (some would say "less-elegant") fashion.  Also,
the takewhile version requires an extra function call (the lambda
in the example above) with the associated performance penalty.
A simple test shows that::

    for n in (n for n in range(100) if 1): pass

performs about 10% better than::

    for n in takewhile(lambda n: 1, range(100)): pass

though they achieve similar results.  (The first example uses a
generator; takewhile is an iterator).  If similarly implemented,
a "while" clause should perform about the same as the "if" clause
does today.

The reader may ask if the "if" and "while" clauses should be
mutually exclusive.  There are good examples that show that there
are times when both may be used to good advantage. For example::

    p = (p for p in primes() if p > 100 while p < 1000)

should return prime numbers found between 100 and 1000, assuming
I have a ``primes()`` generator that yields prime numbers.

Adding a "while" clause to generator expressions maintains the
compact form while adding a useful facility for short-circuiting
the expression.


Acknowledgements
================

Raymond Hettinger first proposed the concept of generator
expressions in January 2002.


Copyright
=========

This document has been placed in the public domain.


id: pep-3156.rst
content:
PEP: 3156
Title: Asynchronous IO Support Rebooted: the "asyncio" Module
Author: Guido van Rossum <guido@python.org>
BDFL-Delegate: Antoine Pitrou <antoine@python.org>
Discussions-To: python-tulip@googlegroups.com
Status: Final
Type: Standards Track
Created: 12-Dec-2012
Python-Version: 3.3
Post-History: 21-Dec-2012
Replaces: 3153
Resolution: https://mail.python.org/pipermail/python-dev/2013-November/130419.html

Abstract
========

This is a proposal for asynchronous I/O in Python 3, starting at
Python 3.3.  Consider this the concrete proposal that is missing from
:pep:`3153`.  The proposal includes a pluggable event loop, transport and
protocol abstractions similar to those in Twisted, and a higher-level
scheduler based on ``yield from`` (:pep:`380`).  The proposed package
name is ``asyncio``.


Introduction
============

Status
------

A reference implementation exists under the code name Tulip.  The
Tulip repo is linked from the References section at the end.  Packages
based on this repo will be provided on PyPI (see References) to enable
using the ``asyncio`` package with Python 3.3 installations.

As of October 20th 2013, the ``asyncio`` package has been checked into
the Python 3.4 repository and released with Python 3.4-alpha-4, with
"provisional" API status.  This is an expression of confidence and
intended to increase early feedback on the API, and not intended to
force acceptance of the PEP.  The expectation is that the package will
keep provisional status in Python 3.4 and progress to final status in
Python 3.5.  Development continues to occur primarily in the Tulip
repo, with changes occasionally merged into the CPython repo.

Dependencies
------------

Python 3.3 is required for many of the proposed features.  The
reference implementation (Tulip) requires no new language or standard
library features beyond Python 3.3, no third-party modules or
packages, and no C code, except for the (optional) IOCP support on
Windows.

Module Namespace
----------------

The specification here lives in a new top-level package, ``asyncio``.
Different components live in separate submodules of the package.  The
package will import common APIs from their respective submodules and
make them available as package attributes (similar to the way the
email package works).  For such common APIs, the name of the submodule
that actually defines them is not part of the specification.  Less
common APIs may have to explicitly be imported from their respective
submodule, and in this case the submodule name is part of the
specification.

Classes and functions defined without a submodule name are assumed to
live in the namespace of the top-level package.  (But do not confuse
these with methods of various classes, which for brevity are also used
without a namespace prefix in certain contexts.)

Interoperability
----------------

The event loop is the place where most interoperability occurs.  It
should be easy for (Python 3.3 ports of) frameworks like Twisted,
Tornado, or even gevents to either adapt the default event loop
implementation to their needs using a lightweight adapter or proxy, or
to replace the default event loop implementation with an adaptation of
their own event loop implementation.  (Some frameworks, like Twisted,
have multiple event loop implementations.  This should not be a
problem since these all have the same interface.)

In most cases it should be possible for two different third-party
frameworks to interoperate, either by sharing the default event loop
implementation (each using its own adapter), or by sharing the event
loop implementation of either framework.  In the latter case two
levels of adaptation would occur (from framework A's event loop to the
standard event loop interface, and from there to framework B's event
loop).  Which event loop implementation is used should be under
control of the main program (though a default policy for event loop
selection is provided).

For this interoperability to be effective, the preferred direction of
adaptation in third party frameworks is to keep the default event loop
and adapt it to the framework's API.  Ideally all third party
frameworks would give up their own event loop implementation in favor
of the standard implementation.  But not all frameworks may be
satisfied with the functionality provided by the standard
implementation.

In order to support both directions of adaptation, two separate APIs
are specified:

- An interface for managing the current event loop
- The interface of a conforming event loop

An event loop implementation may provide additional methods and
guarantees, as long as these are called out in the documentation as
non-standard.  An event loop implementation may also leave certain
methods unimplemented if they cannot be implemented in the given
environment; however, such deviations from the standard API should be
considered only as a last resort, and only if the platform or
environment forces the issue.  (An example would be a platform where
there is a system event loop that cannot be started or stopped; see
"Embedded Event Loops" below.)

The event loop API does not depend on ``await/yield from``.  Rather, it uses
a combination of callbacks, additional interfaces (transports and
protocols), and Futures.  The latter are similar to those defined in
:pep:`3148`, but have a different implementation and are not tied to
threads.  In particular, the ``result()`` method raises an exception
instead of blocking when a result is not yet ready; the user is
expected to use callbacks (or ``await/yield from``) to wait for the result.

All event loop methods specified as returning a coroutine are allowed
to return either a Future or a coroutine, at the implementation's
choice (the standard implementation always returns coroutines).  All
event loop methods documented as accepting coroutine arguments *must*
accept both Futures and coroutines for such arguments.  (A convenience
function, ``ensure_future()``, exists to convert an argument that is either a
coroutine or a Future into a Future.)

For users (like myself) who don't like using callbacks, a scheduler is
provided for writing asynchronous I/O code as coroutines using the PEP
380 ``yield from`` or :pep:`492` ``await`` expressions.
The scheduler is not pluggable;
pluggability occurs at the event loop level, and the standard
scheduler implementation should work with any conforming event loop
implementation.  (In fact this is an important litmus test for
conforming implementations.)

For interoperability between code written using coroutines and other
async frameworks, the scheduler defines a Task class that behaves like a
Future.  A framework that interoperates at the event loop level can
wait for a Future to complete by adding a callback to the Future.
Likewise, the scheduler offers an operation to suspend a coroutine
until a callback is called.

If such a framework cannot use the Future and Task classes as-is, it
may reimplement the ``loop.create_future()`` and
``loop.create_task()`` methods.  These should return objects
implementing (a superset of) the Future/Task interfaces.

A less ambitious framework may just call the
``loop.set_task_factory()`` to replace the Task class without
implementing its own event loop.

The event loop API provides limited interoperability with threads:
there is an API to submit a function to an executor (see :pep:`3148`)
which returns a Future that is compatible with the event loop, and
there is a method to schedule a callback with an event loop from
another thread in a thread-safe manner.

Transports and Protocols
------------------------

For those not familiar with Twisted, a quick explanation of the
relationship between transports and protocols is in order.  At the
highest level, the transport is concerned with *how* bytes are
transmitted, while the protocol determines *which* bytes to transmit
(and to some extent when).

A different way of saying the same thing: a transport is an
abstraction for a socket (or si


id: pep-3143.rst
content:
PEP:               3143
Title:             Standard daemon process library
Author:            Ben Finney <ben+python@benfinney.id.au>
Status:            Deferred
Type:              Standards Track
Created:           26-Jan-2009
Python-Version:    3.x
Post-History:


========
Abstract
========

Writing a program to become a well-behaved Unix daemon is somewhat
complex and tricky to get right, yet the steps are largely similar for
any daemon regardless of what else the program may need to do.

This PEP introduces a package to the Python standard library that
provides a simple interface to the task of becoming a daemon process.

============
PEP Deferral
============

Further exploration of the concepts covered in this PEP has been deferred
for lack of a current champion interested in promoting the goals of the PEP
and collecting and incorporating feedback, and with sufficient available
time to do so effectively.

=============
Specification
=============

Example usage
=============

Simple example of direct ``DaemonContext`` usage::

    import daemon

    from spam import do_main_program

    with daemon.DaemonContext():
        do_main_program()

More complex example usage::

    import os
    import grp
    import signal
    import daemon
    import lockfile

    from spam import (
        initial_program_setup,
        do_main_program,
        program_cleanup,
        reload_program_config,
        )

    context = daemon.DaemonContext(
        working_directory='/var/lib/foo',
        umask=0o002,
        pidfile=lockfile.FileLock('/var/run/spam.pid'),
        )

    context.signal_map = {
        signal.SIGTERM: program_cleanup,
        signal.SIGHUP: 'terminate',
        signal.SIGUSR1: reload_program_config,
        }

    mail_gid = grp.getgrnam('mail').gr_gid
    context.gid = mail_gid

    important_file = open('spam.data', 'w')
    interesting_file = open('eggs.data', 'w')
    context.files_preserve = [important_file, interesting_file]

    initial_program_setup()

    with context:
        do_main_program()


Interface
=========

A new package, ``daemon``, is added to the standard library.

A class, ``DaemonContext``, is defined to represent the settings and
process context for the program running as a daemon process.


``DaemonContext`` objects
=========================

A ``DaemonContext`` instance represents the behaviour settings and
process context for the program when it becomes a daemon. The
behaviour and environment is customised by setting options on the
instance, before calling the ``open`` method.

Each option can be passed as a keyword argument to the ``DaemonContext``
constructor, or subsequently altered by assigning to an attribute on
the instance at any time prior to calling ``open``. That is, for
options named ``wibble`` and ``wubble``, the following invocation::

    foo = daemon.DaemonContext(wibble=bar, wubble=baz)
    foo.open()

is equivalent to::

    foo = daemon.DaemonContext()
    foo.wibble = bar
    foo.wubble = baz
    foo.open()

The following options are defined.

``files_preserve``
    :Default: ``None``

    List of files that should *not* be closed when starting the
    daemon. If ``None``, all open file descriptors will be closed.

    Elements of the list are file descriptors (as returned by a file
    object's ``fileno()`` method) or Python ``file`` objects. Each
    specifies a file that is not to be closed during daemon start.

``chroot_directory``
    :Default: ``None``

    Full path to a directory to set as the effective root directory of
    the process. If ``None``, specifies that the root directory is not
    to be changed.

``working_directory``
    :Default: ``'/'``

    Full path of the working directory to which the process should
    change on daemon start.

    Since a filesystem cannot be unmounted if a process has its
    current working directory on that filesystem, this should either
    be left at default or set to a directory that is a sensible âhome
    directoryâ for the daemon while it is running.

``umask``
    :Default: ``0``

    File access creation mask (âumaskâ) to set for the process on
    daemon start.

    Since a process inherits its umask from its parent process,
    starting the daemon will reset the umask to this value so that
    files are created by the daemon with access modes as it expects.

``pidfile``
    :Default: ``None``

    Context manager for a PID lock file. When the daemon context opens
    and closes, it enters and exits the ``pidfile`` context manager.

``detach_process``
    :Default: ``None``

    If ``True``, detach the process context when opening the daemon
    context; if ``False``, do not detach.

    If unspecified (``None``) during initialisation of the instance,
    this will be set to ``True`` by default, and ``False`` only if
    detaching the process is determined to be redundant; for example,
    in the case when the process was started by ``init``, by ``initd``, or
    by ``inetd``.

``signal_map``
    :Default: system-dependent

    Mapping from operating system signals to callback actions.

    The mapping is used when the daemon context opens, and determines
    the action for each signal's signal handler:

    * A value of ``None`` will ignore the signal (by setting the
      signal action to ``signal.SIG_IGN``).

    * A string value will be used as the name of an attribute on the
      ``DaemonContext`` instance. The attribute's value will be used
      as the action for the signal handler.

    * Any other value will be used as the action for the signal
      handler.

    The default value depends on which signals are defined on the
    running system. Each item from the list below whose signal is
    actually defined in the ``signal`` module will appear in the
    default map:

    * ``signal.SIGTTIN``: ``None``

    * ``signal.SIGTTOU``: ``None``

    * ``signal.SIGTSTP``: ``None``

    * ``signal.SIGTERM``: ``'terminate'``

    Depending on how the program will interact with its child
    processes, it may need to specify a signal map that includes the
    ``signal.SIGCHLD`` signal (received when a child process exits).
    See the specific operating system's documentation for more detail
    on how to determine what circumstances dictate the need for signal
    handlers.

``uid``
    :Default: ``os.getuid()``

``gid``
    :Default: ``os.getgid()``

    The user ID (âUIDâ) value and group ID (âGIDâ) value to switch
    the process to on daemon start.

    The default values, the real UID and GID of the process, will
    relinquish any effective privilege elevation inherited by the
    process.

``prevent_core``
    :Default: ``True``

    If true, prevents the generation of core files, in order to avoid
    leaking sensitive information from daemons run as ``root``.

``stdin``
    :Default: ``None``

``stdout``
    :Default: ``None``

``stderr``
    :Default: ``None``

    Each of ``stdin``, ``stdout``, and ``stderr`` is a file-like object
    which will be used as the new file for the standard I/O stream
    ``sys.stdin``, ``sys.stdout``, and ``sys.stderr`` respectively. The file
    should therefore be open, with a minimum of mode 'r' in the case
    of ``stdin``, and mode 'w+' in the case of ``stdout`` and ``stderr``.

    If the object has a ``fileno()`` method that returns a file
    descriptor, the corresponding file will be excluded from being
    closed during daemon start (that is, it will be treated as though
    it were listed in ``files_preserve``).

    If ``None``, the corresponding system stream is re-bound to the
    file named by ``os.devnull``.


The following methods are defined.

``open()``
    :Return: ``None``

    Open the daemon context, turning the current program into a daemon
    process. This performs the following steps:

    * If this instance's ``is_open`` property is true, return
      immediately. This makes it safe to call ``open`` multiple times on
      an instance.

    *


id: pep-3147.rst
content:
PEP: 3147
Title: PYC Repository Directories
Author: Barry Warsaw <barry@python.org>
Status: Final
Type: Standards Track
Created: 16-Dec-2009
Python-Version: 3.2
Post-History: 30-Jan-2010, 25-Feb-2010, 03-Mar-2010, 12-Apr-2010
Resolution: https://mail.python.org/pipermail/python-dev/2010-April/099414.html


Abstract
========

This PEP describes an extension to Python's import mechanism which
improves sharing of Python source code files among multiple installed
different versions of the Python interpreter.  It does this by
allowing more than one byte compilation file (.pyc files) to be
co-located with the Python source file (.py file).  The extension
described here can also be used to support different Python
compilation caches, such as JIT output that may be produced by an
Unladen Swallow (:pep:`3146`) enabled C Python.


Background
==========

CPython compiles its source code into "byte code", and for performance
reasons, it caches this byte code on the file system whenever the
source file has changes.  This makes loading of Python modules much
faster because the compilation phase can be bypassed.  When your
source file is ``foo.py``, CPython caches the byte code in a ``foo.pyc``
file right next to the source.

Byte code files contain two 32-bit big-endian numbers followed by the
marshaled [2]_ code object.  The 32-bit numbers represent a magic
number and a timestamp.  The magic number changes whenever Python
changes the byte code format, e.g. by adding new byte codes to its
virtual machine.  This ensures that pyc files built for previous
versions of the VM won't cause problems.  The timestamp is used to
make sure that the pyc file match the py file that was used to create
it.  When either the magic number or timestamp do not match, the py
file is recompiled and a new pyc file is written.

In practice, it is well known that pyc files are not compatible across
Python major releases.  A reading of import.c [3]_ in the Python
source code proves that within recent memory, every new CPython major
release has bumped the pyc magic number.


Rationale
=========

Linux distributions such as Ubuntu [4]_ and Debian [5]_ provide more
than one Python version at the same time to their users.  For example,
Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1,
with Python 2.6 being the default.

This causes a conflict for third party Python source files installed
by the system, because you cannot compile a single Python source file
for more than one Python version at a time.  When Python finds a ``pyc``
file with a non-matching magic number, it falls back to the slower
process of recompiling the source.  Thus if your system installed a
``/usr/share/python/foo.py``, two different versions of Python would
fight over the ``pyc`` file and rewrite it each time the source is
compiled.  (The standard library is unaffected by this, since multiple
versions of the stdlib *are* installed on such distributions..)

Furthermore, in order to ease the burden on operating system packagers
for these distributions, the distribution packages do not contain
Python version numbers [6]_; they are shared across all Python
versions installed on the system.  Putting Python version numbers in
the packages would be a maintenance nightmare, since all the packages
- *and their dependencies* - would have to be updated every time a new
Python release was added or removed from the distribution.  Because of
the sheer number of packages available, this amount of work is
infeasible.

(:pep:`384` has been proposed to address binary compatibility issues
of third party extension modules across different versions of Python.)

Because these distributions cannot share pyc files, elaborate
mechanisms have been developed to put the resulting pyc files in
non-shared locations while the source code is still shared.  Examples
include the symlink-based Debian regimes python-support [8]_ and
python-central [9]_.  These approaches make for much more complicated,
fragile, inscrutable, and fragmented policies for delivering Python
applications to a wide range of users.  Arguably more users get Python
from their operating system vendor than from upstream tarballs.  Thus,
solving this pyc sharing problem for CPython is a high priority for
such vendors.

This PEP proposes a solution to this problem.


Proposal
========

Python's import machinery is extended to write and search for byte
code cache files in a single directory inside every Python package
directory.  This directory will be called ``__pycache__``.

Further, pyc file names will contain a magic string (called a "tag")
that differentiates the Python version they were compiled for.  This
allows multiple byte compiled cache files to co-exist for a single
Python source file.

The magic tag is implementation defined, but should contain the
implementation name and a version number shorthand, e.g. ``cpython-32``.
It must be unique among all versions of Python, and whenever the magic
number is bumped, a new magic tag must be defined.  An example ``pyc``
file for Python 3.2 is thus ``foo.cpython-32.pyc``.

The magic tag is available in the ``imp`` module via the ``get_tag()``
function.  This is parallel to the ``imp.get_magic()`` function.

This scheme has the added benefit of reducing the clutter in a Python
package directory.

When a Python source file is imported for the first time, a
``__pycache__`` directory will be created in the package directory, if
one does not already exist.  The pyc file for the imported source will
be written to the ``__pycache__`` directory, using the magic-tag
formatted name.  If either the creation of the ``__pycache__`` directory
or the pyc file inside that fails, the import will still succeed, just
as it does in a pre-:pep:`3147` world.

If the py source file is missing, the pyc file inside ``__pycache__``
will be ignored.  This eliminates the problem of accidental stale pyc
file imports.

For backward compatibility, Python will still support pyc-only
distributions, however it will only do so when the pyc file lives in
the directory where the py file *would* have been, i.e. not in the
``__pycache__`` directory.  pyc file outside of ``__pycache__`` will only
be imported if the py source file is missing.

Tools such as ``py_compile`` [15]_ and ``compileall`` [16]_ will be
extended to create :pep:`3147` formatted layouts automatically, but will
have an option to create pyc-only distribution layouts.


Examples
---------

What would this look like in practice?

Let's say we have a Python package named ``alpha`` which contains a
sub-package name ``beta``.  The source directory layout before byte
compilation might look like this::

    alpha/
        __init__.py
        one.py
        two.py
        beta/
            __init__.py
            three.py
            four.py

After byte compiling this package with Python 3.2, you would see the
following layout::

    alpha/
        __pycache__/
            __init__.cpython-32.pyc
            one.cpython-32.pyc
            two.cpython-32.pyc
        __init__.py
        one.py
        two.py
        beta/
            __pycache__/
                __init__.cpython-32.pyc
                three.cpython-32.pyc
                four.cpython-32.pyc
            __init__.py
            three.py
            four.py

*Note: listing order may differ depending on the platform.*

Let's say that two new versions of Python are installed, one is Python
3.3 and another is Unladen Swallow.  After byte compilation, the file
system would look like this::

    alpha/
        __pycache__/
            __init__.cpython-32.pyc
            __init__.cpython-33.pyc
            __init__.unladen-10.pyc
            one.cpython-32.pyc
            one.cpython-33.pyc
            one.unladen-10.pyc
            two.cpython-32.pyc
            two.cpython-33.pyc
            two.unladen-10.pyc
        __init__.py
        one.py
        two.py
        beta/
            __pycache__/
                __init__.cpython-32.pyc
  


id: pep-3153.rst
content:
PEP: 3153
Title: Asynchronous IO support
Author: Laurens Van Houtven <_@lvh.cc>
Status: Superseded
Type: Standards Track
Created: 29-May-2011
Post-History:
Superseded-By: 3156

Abstract
========

This PEP describes an abstraction of asynchronous IO for the Python
standard library.

The goal is to reach an abstraction that can be implemented by many
different asynchronous IO backends and provides a target for library
developers to write code portable between those different backends.

Rationale
=========

People who want to write asynchronous code in Python right now have a
few options:

- ``asyncore`` and ``asynchat``
- something bespoke, most likely based on the ``select`` module
- using a third party library, such as Twisted_ or gevent_

Unfortunately, each of these options has its downsides, which this PEP
tries to address.

Despite having been part of the Python standard library for a long
time, the asyncore module suffers from fundamental flaws following
from an inflexible API that does not stand up to the expectations of a
modern asynchronous networking module.

Moreover, its approach is too simplistic to provide developers with
all the tools they need in order to fully exploit the potential of
asynchronous networking.

The most popular solution right now used in production involves the
use of third party libraries.  These often provide satisfactory
solutions, but there is a lack of compatibility between these
libraries, which tends to make codebases very tightly coupled to the
library they use.

This current lack of portability between different asynchronous IO
libraries causes a lot of duplicated effort for third party library
developers.  A sufficiently powerful abstraction could mean that
asynchronous code gets written once, but used everywhere.

An eventual added goal would be for standard library implementations
of wire and network protocols to evolve towards being real protocol
implementations, as opposed to standalone libraries that do everything
including calling ``recv()`` blockingly.  This means they could be
easily reused for both synchronous and asynchronous code.

.. _Twisted: http://www.twistedmatrix.com/
.. _gevent: http://www.gevent.org/

Communication abstractions
==========================

Transports
----------

Transports provide a uniform API for reading bytes from and writing
bytes to different kinds of connections.  Transports in this PEP are
always ordered, reliable, bidirectional, stream-oriented two-endpoint
connections.  This might be a TCP socket, an SSL connection, a pipe
(named or otherwise), a serial port...  It may abstract a file
descriptor on POSIX platforms or a Handle on Windows or some other
data structure appropriate to a particular platform.  It encapsulates
all of the particular implementation details of using that platform
data structure and presents a uniform interface for application
developers.

Transports talk to two things: the other side of the connection on one
hand, and a protocol on the other.  It's a bridge between the specific
underlying transfer mechanism and the protocol.  Its job can be
described as allowing the protocol to just send and receive bytes,
taking care of all of the magic that needs to happen to those bytes to
be eventually sent across the wire.

The primary feature of a transport is sending bytes to a protocol and
receiving bytes from the underlying protocol.  Writing to the
transport is done using the ``write`` and ``write_sequence`` methods.
The latter method is a performance optimization, to allow software to
take advantage of specific capabilities in some transport mechanisms.
Specifically, this allows transports to use writev_ instead of write_
or send_, also known as scatter/gather IO.

A transport can be paused and resumed.  This will cause it to buffer
data coming from protocols and stop sending received data to the
protocol.

A transport can also be closed, half-closed and aborted.  A closed
transport will finish writing all of the data queued in it to the
underlying mechanism, and will then stop reading or writing data.
Aborting a transport stops it, closing the connection without sending
any data that is still queued.

Further writes will result in exceptions being thrown.  A half-closed
transport may not be written to anymore, but will still accept
incoming data.

Protocols
---------

Protocols are probably more familiar to new users.  The terminology is
consistent with what you would expect from something called a
protocol: the protocols most people think of first, like HTTP, IRC,
SMTP... are all examples of something that would be implemented in a
protocol.

The shortest useful definition of a protocol is a (usually two-way)
bridge between the transport and the rest of the application logic.  A
protocol will receive bytes from a transport and translates that
information into some behavior, typically resulting in some method
calls on an object.  Similarly, application logic calls some methods
on the protocol, which the protocol translates into bytes and
communicates to the transport.

One of the simplest protocols is a line-based protocol, where data is
delimited by ``\r\n``.  The protocol will receive bytes from the
transport and buffer them until there is at least one complete line.
Once that's done, it will pass this line along to some object.
Ideally that would be accomplished using a callable or even a
completely separate object composed by the protocol, but it could also
be implemented by subclassing (as is the case with Twisted's
``LineReceiver``).  For the other direction, the protocol could have a
``write_line`` method, which adds the required ``\r\n`` and passes the
new bytes buffer on to the transport.

This PEP suggests a generalized ``LineReceiver`` called
``ChunkProtocol``, where a "chunk" is a message in a stream, delimited
by the specified delimiter.  Instances take a delimiter and a callable
that will be called with a chunk of data once it's received (as
opposed to Twisted's subclassing behavior).  ``ChunkProtocol`` also
has a ``write_chunk`` method analogous to the ``write_line`` method
described above.

Why separate protocols and transports?
--------------------------------------

This separation between protocol and transport often confuses people
who first come across it.  In fact, the standard library itself does
not make this distinction in many cases, particularly not in the API
it provides to users.

It is nonetheless a very useful distinction.  In the worst case, it
simplifies the implementation by clear separation of concerns.
However, it often serves the far more useful purpose of being able to
reuse protocols across different transports.

Consider a simple RPC protocol.  The same bytes may be transferred
across many different transports, for example pipes or sockets.  To
help with this, we separate the protocol out from the transport.  The
protocol just reads and writes bytes, and doesn't really care what
mechanism is used to eventually transfer those bytes.

This also allows for protocols to be stacked or nested easily,
allowing for even more code reuse.  A common example of this is
JSON-RPC: according to the specification, it can be used across both
sockets and HTTP [#jsonrpc]_.  In practice, it tends to be primarily
encapsulated in HTTP.  The protocol-transport abstraction allows us to
build a stack of protocols and transports that allow you to use HTTP
as if it were a transport.  For JSON-RPC, that might get you a stack
somewhat like this:

1. TCP socket transport
2. HTTP protocol
3. HTTP-based transport
4. JSON-RPC protocol
5. Application code

Flow control
============

Consumers
---------

Consumers consume bytes produced by producers.  Together with
producers, they make flow control possible.

Consumers primarily play a passive role in flow control.  They get
called whenever a producer has some data available.  They then process
that data, and typically yield control back to the producer.

Consumers typically implement buffer


