'codecs' module docs improvements

[zuo]

BPO	19548
Nosy	@malemburg, @doerwalter, @ncoghlan, @vstinner, @ezio-melotti, @berkerpeksag, @vadmium
Files	codecs-doc.patch codecs-doc.v2.patch codecs-doc.v3.patch issue19548-codecs-doc.py34.patch: Updated version of patch for Python 3.4 branch issue19548-codecs-doc.v5.py3.4.patch default-branch-followup.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2015-01-06.14:42:05.079>
created_at = <Date 2013-11-11.01:29:13.621>
labels = ['type-feature', 'docs']
title = "'codecs' module docs improvements"
updated_at = <Date 2015-01-07.04:59:02.472>
user = 'https://bugs.python.org/zuo'

bugs.python.org fields:

activity = <Date 2015-01-07.04:59:02.472>
actor = 'ncoghlan'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2015-01-06.14:42:05.079>
closer = 'ncoghlan'
components = ['Documentation']
creation = <Date 2013-11-11.01:29:13.621>
creator = 'zuo'
dependencies = []
files = ['37485', '37525', '37530', '37553', '37610', '37618']
hgrepos = []
issue_num = 19548
keywords = ['patch']
message_count = 23.0
messages = ['202593', '202594', '202595', '202598', '202604', '203038', '203040', '203043', '203044', '203049', '207375', '219830', '232849', '233017', '233020', '233035', '233167', '233505', '233543', '233544', '233552', '233560', '233562']
nosy_count = 11.0
nosy_names = ['lemburg', 'doerwalter', 'ncoghlan', 'vstinner', 'ezio.melotti', 'zuo', 'docs@python', 'python-dev', 'berker.peksag', 'martin.panter', 'Zoinkity..']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue19548'
versions = ['Python 3.4', 'Python 3.5']

[zuo]

When learning about the 'codecs' module I encountered several places in the docs of the module that, I believe, could be improved to be clearer and easier for codecs-begginers:

1. Ad codecs.encode and codecs.decode descriptions: I believe it would be worth to mention that, unlike str.encode()/bytes.decode(), these functions (and all their counterparts in the classes the module contains) support not only "traditional str/bytes encodings", but also bytes-to-bytes as well as str-to-str encodings.

2. Ad 'codecs.register': in two places there is such a text: These have to be factory functions providing the following interface: factory([...] errors='strict') -- errors='strict' may be confusing (at the first sight it may suggest that the only valid value is 'strict'; maybe factory(errors=<error handler label>) with an appropriate description below would be better?).

3. Ad codecs.open: I believe there should be a reference to the built-in open() as an alternative that is better is most cases.

4. Ad codecs.BOM*: These constants define various encodings of the Unicode byte order mark (BOM). -- the world encodings seems to be confusing here; maybe These constants define various byte sequences being Unicode byte order marks (BOMs) for several encodings. They are used... would be better?

5. Ad 7.2.1. Codec Base Classes + codecs.IncrementalEncoder/codecs/IncrementalDecoder:

• Each codec has to define four interfaces to make it usable as codec in Python: stateless encoder, stateless decoder, stream reader and stream writer -- only four? Not six? What about incremental encoder/decoder???
• Comparing the fragments (and tables) about error halding methods (Codecs Base Classes, IncrementalEncoder, IncrementalDecoder) with similar fragment in the codecs.register description and with the codecs.register_error description I was confused: is it the matter of a particular codec implementation or of a registered error handler to implement a particular way of error handling? I believe it would be worth to describe clearly relations between these elements of the API. Also more detailed description of differences beetween error handling for encoding and decoding, and translation would be a good thing.

6. Ad 7.2.1.6. StreamReaderWriter Objects and 7.2.1.7. StreamRecoder Objects: It would be worth to say explicitly that, contrary to previously described abstract classes (IncrementalEncoder/Decoder, StreamReader/Writer), these classes are concrete ones (if I understand it correctly).

7. Ad 7.2.4. Python Specific Encodings:

• raw_unicode_encoding -- see: ticket bpo-19539.
• unicode_encoding -- Produce a string that is suitable as Unicode literal in Python source code but it is not a string; it's a bytes object (which could be used in source code using an ascii-compatibile encoding).
• bytes-to-bytes and str-to-str encodings -- maybe it would be nice to mention that these encodings cannot be used with str.encode()/bytes.decode() methods (and to mention again they can be used with the functions/method provided by the codecs module).

[zuo]

s/world/word
s/begginers/beginners

(sorry, it's late night here)

[zuo]

8. Again ad codecs.open: the default file mode is actually 'rb', not 'r'.

9. Several places in the docs -- ad: codecs.register_error, codecs.open, codecs.EncodedFile, Codec.encode/decode, codecs.StreamWriter/StreamReader -- do not cover cases of using bytes-to-bytes and/or str-to-str encodings (especially when using string/bytes and text/binary terms).

10. codecs.replace_errors -- bytestring should be replaced with bytes-like object (as in other places).

[zuo]

11. Ad encoding 'undefined': The sentence Can be used as the system encoding if no automatic coercion between byte and Unicode strings is desired. was suitable for Python 2.x, but not for Python 3.x'. I believe, this sentence should be removed.

[ncoghlan]

A few more:

• codec name normalisation (lower case, space to hyphen) is not mentioned
  in the codecs.register description

• search function registration is not reversible, which doesn't play well
  with module reloading

• codecs.CodecInfo init signature is not covered

[ncoghlan]

Another big one: the encodings module API is not documented in the prose docs, and nor is the interface between the default search function and the individual encoding definitions. There's some decent info in help(encoding) though.

The interaction with the import system could also be documented better - you can actually blacklist codecs by manipulating sys.modules and the encodings namespace, and you can search additional locations for codec modules by manipulating encodings.__path__ (even without it being declared as a namespace package)

[malemburg]

On 16.11.2013 14:25, Nick Coghlan wrote:

Nick Coghlan added the comment:

Another big one: the encodings module API is not documented in the prose docs, and nor is the interface between the default search function and the individual encoding definitions. There's some decent info in help(encoding) though.

The interaction with the import system could also be documented better - you can actually blacklist codecs by manipulating sys.modules and the encodings namespace, and you can search additional locations for codec modules by manipulating encodings.__path__ (even without it being declared as a namespace package)

Those were not documented on purpose, since they are an implementation
detail of the encodings package search function.

If you document them now, you'll set the implementation in stone,
making future changes to the logic difficult. I'd advise against
this to stay flexible, unless you want to open up the encodings
package as namespace package - then you'd have to add documentation
for the import interface.

[BreamoreBoy]

Could they be documented with a massive warning in red "Cpython implementation detail - subject to change without notice"? Or documented in a place that is only accessible to developers and not users? Or...???

[malemburg]

On 16.11.2013 15:03, Mark Lawrence wrote:

Mark Lawrence added the comment:

Could they be documented with a massive warning in red "Cpython implementation detail - subject to change without notice"? Or documented in a place that is only accessible to developers and not users? Or...???

The API is documented in encodings/init.py for developers.

[ncoghlan]

On 16 November 2013 23:33, Marc-Andre Lemburg <report@bugs.python.org> wrote:

On 16.11.2013 14:25, Nick Coghlan wrote:
Those were not documented on purpose, since they are an implementation
detail of the encodings package search function.

If you document them now, you'll set the implementation in stone,
making future changes to the logic difficult. I'd advise against
this to stay flexible, unless you want to open up the encodings
package as namespace package - then you'd have to add documentation
for the import interface.

Yes, that was what got me thinking along those lines, but to make that
possible, the contents of encodings/init.py would need to be moved
somewhere else. So this probably isn't on the table for 3.4.

[vadmium]

Addition to the list of improvements:

• Under codecs.IncrementalEncoder.reset() it mentions calling encode('', final=True). This call does not work as written for the byte encoders in my experience, because they do not accept empty text strings. Perhaps it should just say to use the final=True flag with no data.

[Zoinkity]

One glaring omission is any information about multibyte codecs--the class, its methods, and how to even define one.

Also, the primary use for codecs.register would be to append a single codec to the lookup registry. Simple usage of the method only provides lookup for the provided codecs and will not include regularly-accessible ones such as "utf-8". It would be enormously helpful to provide an example of proper, safe usage.