bpo-43950: Add documentation for PEP-657 (GH-27047)

Co-authored-by: Pablo Galindo <[email protected]>
Co-authored-by: Batuhan Taskaya <[email protected]>

Author: 3 people

Doc/c-api/code.rst

@@ -59,3 +59,11 @@ bound into a function.
 
     For efficiently iterating over the line numbers in a code object, use `the API described in PEP 626
     <https://www.python.org/dev/peps/pep-0626/#out-of-process-debuggers-and-profilers>`_.
+
+.. c:function:: int PyCode_Addr2Location(PyObject *co, int byte_offset, int *start_line, int *start_column, int *end_line, int *end_column)
+
+   Sets the passed ``int`` pointers to the source code line and column numbers
+   for the instruction at ``byte_offset``. Sets the value to ``0`` when
+   information is not available for any particular element.
+
+   Returns ``1`` if the function succeeds and 0 otherwise.

Doc/reference/datamodel.rst

@@ -1015,6 +1015,39 @@ Internal types
       If a code object represents a function, the first item in :attr:`co_consts` is
       the documentation string of the function, or ``None`` if undefined.
 
+      .. method:: codeobject.co_positions()
+
+         Returns an iterable over the source code positions of each bytecode
+         instruction in the code object.
+
+         The iterator returns tuples containing the ``(start_line, end_line,
+         start_column, end_column)``. The *i-th* tuple corresponds to the
+         position of the source code that compiled to the *i-th* instruction.
+         Column information is 0-indexed utf-8 byte offsets on the given source
+         line.
+
+         This positional information can be missing. A non-exhaustive lists of
+         cases where this may happen:
+
+         - Running the interpreter with :option:`-X` ``no_debug_ranges``.
+         - Loading a pyc file compiled while using :option:`-X` ``no_debug_ranges``.
+         - Position tuples corresponding to artificial instructions.
+         - Line and column numbers that can't be represented due to
+           implementation specific limitations.
+
+         When this occurs, some or all of the tuple elements can be
+         :const:`None`.
+
+         .. versionadded:: 3.11
+
+         .. note::
+            This feature requires storing column positions in code objects which may
+            result in a small increase of disk usage of compiled Python files or
+            interpreter memory usage. To avoid storing the extra information and/or
+            deactivate printing the extra traceback information, the
+            :option:`-X` ``no_debug_ranges`` command line flag or the :envvar:`PYTHONNODEBUGRANGES`
+            environment variable can be used.
+
    .. _frame-objects:
 
    Frame objects

Doc/whatsnew/3.11.rst

@@ -70,6 +70,83 @@ Summary -- Release highlights
 New Features
 ============
 
+.. _whatsnew311-pep657:
+
+Enhanced error locations in tracebacks
+--------------------------------------
+
+When printing tracebacks, the interpreter will now point to the exact expression
+that caused the error instead of just the line. For example:
+
+.. code-block:: python
+
+    Traceback (most recent call last):
+      File "distance.py", line 11, in <module>
+        print(manhattan_distance(p1, p2))
+              ^^^^^^^^^^^^^^^^^^^^^^^^^^
+      File "distance.py", line 6, in manhattan_distance
+        return abs(point_1.x - point_2.x) + abs(point_1.y - point_2.y)
+                               ^^^^^^^^^
+    AttributeError: 'NoneType' object has no attribute 'x'
+
+Previous versions of the interpreter would point to just the line making it
+ambiguous which object was ``None``. These enhanced errors can also be helpful
+when dealing with deeply nested dictionary objects and multiple function calls,
+
+.. code-block:: python
+
+    Traceback (most recent call last):
+      File "query.py", line 37, in <module>
+        magic_arithmetic('foo')
+        ^^^^^^^^^^^^^^^^^^^^^^^
+      File "query.py", line 18, in magic_arithmetic
+        return add_counts(x) / 25
+               ^^^^^^^^^^^^^
+      File "query.py", line 24, in add_counts
+        return 25 + query_user(user1) + query_user(user2)
+                    ^^^^^^^^^^^^^^^^^
+      File "query.py", line 32, in query_user
+        return 1 + query_count(db, response['a']['b']['c']['user'], retry=True)
+                                   ~~~~~~~~~~~~~~~~~~^^^^^
+    TypeError: 'NoneType' object is not subscriptable
+
+as well as complex arithmetic expressions:
+
+.. code-block:: python
+
+    Traceback (most recent call last):
+      File "calculation.py", line 54, in <module>
+        result = (x / y / z) * (a / b / c)
+                  ~~~~~~^~~
+    ZeroDivisionError: division by zero
+
+See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya
+and Ammar Askar in :issue:`43950`.)
+
+.. note::
+   This feature requires storing column positions in code objects which may
+   result in a small increase of disk usage of compiled Python files or
+   interpreter memory usage. To avoid storing the extra information and/or
+   deactivate printing the extra traceback information, the
+   :option:`-X` ``no_debug_ranges`` command line flag or the :envvar:`PYTHONNODEBUGRANGES`
+   environment variable can be used.
+
+Column information for code objects
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The information used by the enhanced traceback feature is made available as a
+general API that can be used to correlate bytecode instructions with source
+code. This information can be retrieved using:
+
+- The :meth:`codeobject.co_positions` method in Python.
+- The :c:func:`PyCode_Addr2Location` function in the C-API.
+
+The :option:`-X` ``no_debug_ranges`` option and the environment variable
+:envvar:`PYTHONNODEBUGRANGES` can be used to disable this feature.
+
+See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya
+and Ammar Askar in :issue:`43950`.)
+
 
 
 Other Language Changes

Misc/NEWS.d/next/Core and Builtins/2021-07-06-15-27-11.bpo-43950.LhL2-q.rst

@@ -0,0 +1,6 @@
+Code objects can now provide the column information for instructions when
+available. This is levaraged during traceback printing to show the
+expressions responsible for errors.
+
+Contributed by Pablo Galindo, Batuhan Taskaya and Ammar Askar as part of
+:pep:`657`.