Merging the py3k-pep3137 branch back into the py3k branch.

No detailed change log; just check out the change log for the py3k-pep3137
branch.  The most obvious changes:

  - str8 renamed to bytes (PyString at the C level);
  - bytes renamed to buffer (PyBytes at the C level);
  - PyString and PyUnicode are no longer compatible.

I.e. we now have an immutable bytes type and a mutable bytes type.

The behavior of PyString was modified quite a bit, to make it more
bytes-like.  Some changes are still on the to-do list.

Author: gvanrossum

Doc/library/array.rst

@@ -56,8 +56,9 @@ The module defines the following type:
 .. function:: array(typecode[, initializer])
 
    Return a new array whose items are restricted by *typecode*, and initialized
-   from the optional *initializer* value, which must be a list, string, or iterable
-   over elements of the appropriate type.
+   from the optional *initializer* value, which must be a list, object
+   supporting the buffer interface, or iterable over elements of the 
+   appropriate type.
 
    If given a list or string, the initializer is passed to the new array's
    :meth:`fromlist`, :meth:`fromstring`, or :meth:`fromunicode` method (see below)
@@ -69,6 +70,10 @@ The module defines the following type:
 
    Obsolete alias for :func:`array`.
 
+.. data:: typecodes
+
+   A string with all available type codes.
+
 Array objects support the ordinary sequence operations of indexing, slicing,
 concatenation, and multiplication.  When using slice assignment, the assigned
 value must be an array object with the same type code; in all other cases,

Doc/library/exceptions.rst

@@ -405,7 +405,11 @@ module for more information.
 
    Base class for warnings related to Unicode.
 
-The class hierarchy for built-in exceptions is:
+.. exception:: BytesWarning
+
+   Base class for warnings related to :class:`bytes` and :class:`buffer`.
 
 
+The class hierarchy for built-in exceptions is:
+
 .. literalinclude:: ../../Lib/test/exception_hierarchy.txt

Doc/library/functions.rst

@@ -118,31 +118,44 @@ available.  They are listed here in alphabetical order.
    .. index:: pair: Boolean; type
 
 
-.. function:: bytes([arg[, encoding[, errors]]])
+.. function:: buffer([arg[, encoding[, errors]]])
 
-   Return a new array of bytes.  The :class:`bytes` type is a mutable sequence
+   Return a new array of bytes.  The :class:`buffer` type is an immutable sequence
    of integers in the range 0 <= x < 256.  It has most of the usual methods of
-   mutable sequences, described in :ref:`typesseq-mutable`, as well as a few
-   methods borrowed from strings, described in :ref:`bytes-methods`.
+   mutable sequences, described in :ref:`typesseq-mutable`, as well as most methods
+   that the :class:`str` type has, see :ref:`bytes-methods`.
 
    The optional *arg* parameter can be used to initialize the array in a few
    different ways:
 
    * If it is a *string*, you must also give the *encoding* (and optionally,
-     *errors*) parameters; :func:`bytes` then acts like :meth:`str.encode`.
+     *errors*) parameters; :func:`buffer` then converts the Unicode string to
+     bytes using :meth:`str.encode`.
 
    * If it is an *integer*, the array will have that size and will be
      initialized with null bytes.
 
    * If it is an object conforming to the *buffer* interface, a read-only buffer
      of the object will be used to initialize the bytes array.
 
-   * If it is an *iterable*, it must be an iterable of integers in the range 0
-     <= x < 256, which are used as the initial contents of the array.
+   * If it is an *iterable*, it must be an iterable of integers in the range
+     ``0 <= x < 256``, which are used as the initial contents of the array.
 
    Without an argument, an array of size 0 is created.
 
 
+.. function:: bytes([arg[, encoding[, errors]]])
+
+   Return a new "bytes" object, which is an immutable sequence of integers in
+   the range ``0 <= x < 256``.  :class:`bytes` is an immutable version of
+   :class:`buffer` -- it has the same non-mutating methods and the same indexing
+   and slicing behavior.
+   
+   Accordingly, constructor arguments are interpreted as for :func:`buffer`.
+
+   Bytes objects can also be created with literals, see :ref:`strings`.
+
+
 .. function:: chr(i)
 
    Return the string of one character whose Unicode codepoint is the integer

Doc/library/stdtypes.rst

@@ -1313,9 +1313,11 @@ Bytes and Buffer Methods
 
 Bytes and buffer objects, being "strings of bytes", have all methods found on
 strings, with the exception of :func:`encode`, :func:`format` and
-:func:`isidentifier`, which do not make sense with these types.  Wherever one of
-these methods needs to interpret the bytes as characters (e.g. the :func:`is...`
-methods), the ASCII character set is assumed.
+:func:`isidentifier`, which do not make sense with these types.  For converting
+the objects to strings, they have a :func:`decode` method.
+
+Wherever one of these methods needs to interpret the bytes as characters
+(e.g. the :func:`is...` methods), the ASCII character set is assumed.
 
 .. note::
 

Doc/library/warnings.rst

@@ -80,6 +80,10 @@ following warnings category classes are currently defined:
 | :exc:`UnicodeWarning`            | Base category for warnings related to         |
 |                                  | Unicode.                                      |
 +----------------------------------+-----------------------------------------------+
+| :exc:`BytesWarning`              | Base category for warnings related to         |
+|                                  | :class:`bytes` and :class:`buffer`.           |
++----------------------------------+-----------------------------------------------+
+
 
 While these are technically built-in exceptions, they are documented here,
 because conceptually they belong to the warnings mechanism.

Doc/whatsnew/3.0.rst

@@ -131,11 +131,6 @@ changes to rarely used features.)
   that if a file is opened using an incorrect mode or encoding, I/O
   will likely fail.
 
-* Bytes aren't hashable, and don't support certain operations like
-  ``b.lower()``, ``b.strip()`` or ``b.split()``.
-  For the latter two, use ``b.strip(b" \t\r\n\f")`` or
-  ``b.split(b" \t\r\n\f")``.
-
 * ``map()`` and ``filter()`` return iterators.  A quick fix is e.g.
   ``list(map(...))``, but a better fix is often to use a list
   comprehension (especially when the original code uses ``lambda``).
@@ -158,13 +153,11 @@ Strings and Bytes
 * There is only one string type; its name is ``str`` but its behavior
   and implementation are more like ``unicode`` in 2.x.
 
-* PEP 358: There is a new type, ``bytes``, to represent binary data
+* PEP 3137: There is a new type, ``bytes``, to represent binary data
   (and encoded text, which is treated as binary data until you decide
   to decode it).  The ``str`` and ``bytes`` types cannot be mixed; you
   must always explicitly convert between them, using the ``.encode()``
-  (str -> bytes) or ``.decode()`` (bytes -> str) methods.  Comparing a
-  bytes and a str instance for equality raises a TypeError; this
-  catches common mistakes.
+  (str -> bytes) or ``.decode()`` (bytes -> str) methods.
 
 * PEP 3112: Bytes literals.  E.g. b"abc".
 

Include/abstract.h

@@ -259,7 +259,7 @@ xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx*/
 	 string representation on success, NULL on failure.  This is
 	 the equivalent of the Python expression: repr(o).
 
-	 Called by the repr() built-in function and by reverse quotes.
+	 Called by the repr() built-in function.
 
        */
 
@@ -271,20 +271,7 @@ xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx*/
 	 string representation on success, NULL on failure.  This is
 	 the equivalent of the Python expression: str(o).)
 
-	 Called by the str() built-in function and by the print
-	 statement.
-
-       */
-
-     /* Implemented elsewhere:
-
-     PyObject *PyObject_Unicode(PyObject *o);
-
-	 Compute the unicode representation of object, o.  Returns the
-	 unicode representation on success, NULL on failure.  This is
-	 the equivalent of the Python expression: unistr(o).)
-
-	 Called by the unistr() built-in function.
+	 Called by the str() and print() built-in functions.
 
        */
 

Include/object.h

@@ -431,10 +431,8 @@ PyAPI_FUNC(int) PyObject_Print(PyObject *, FILE *, int);
 PyAPI_FUNC(void) _Py_BreakPoint(void);
 PyAPI_FUNC(void) _PyObject_Dump(PyObject *);
 PyAPI_FUNC(PyObject *) PyObject_Repr(PyObject *);
-PyAPI_FUNC(PyObject *) PyObject_ReprStr8(PyObject *);
-PyAPI_FUNC(PyObject *) _PyObject_Str(PyObject *);
 PyAPI_FUNC(PyObject *) PyObject_Str(PyObject *);
-PyAPI_FUNC(PyObject *) PyObject_Unicode(PyObject *);
+#define PyObject_Unicode PyObject_Str /* Compatibility */
 PyAPI_FUNC(int) PyObject_Compare(PyObject *, PyObject *);
 PyAPI_FUNC(PyObject *) PyObject_RichCompare(PyObject *, PyObject *, int);
 PyAPI_FUNC(int) PyObject_RichCompareBool(PyObject *, PyObject *, int);
@@ -478,7 +476,7 @@ PyAPI_FUNC(long) _Py_HashDouble(double);
 PyAPI_FUNC(long) _Py_HashPointer(void*);
 
 /* Helper for passing objects to printf and the like */
-#define PyObject_REPR(obj) PyString_AS_STRING(PyObject_ReprStr8(obj))
+#define PyObject_REPR(obj) PyUnicode_AsString(PyObject_Repr(obj))
 
 /* Flag bits for printing: */
 #define Py_PRINT_RAW	1	/* No string quotes etc. */

Include/opcode.h

@@ -65,7 +65,7 @@ extern "C" {
 
 #define RETURN_VALUE	83
 #define IMPORT_STAR	84
-#define MAKE_BYTES	85
+
 #define YIELD_VALUE	86
 #define POP_BLOCK	87
 #define END_FINALLY	88

Include/pydebug.h

@@ -11,6 +11,7 @@ PyAPI_DATA(int) Py_InteractiveFlag;
 PyAPI_DATA(int) Py_InspectFlag;
 PyAPI_DATA(int) Py_OptimizeFlag;
 PyAPI_DATA(int) Py_NoSiteFlag;
+PyAPI_DATA(int) Py_BytesWarningFlag;
 PyAPI_DATA(int) Py_UseClassExceptionsFlag;
 PyAPI_DATA(int) Py_FrozenFlag;
 PyAPI_DATA(int) Py_TabcheckFlag;