Add compact=True flag to json.dump/dumps

[AlexGordon]

BPO	29540
Nosy	@brettcannon, @rhettinger, @etrepum, @vstinner, @bitdancer, @benhoyt, @berkerpeksag, @serhiy-storchaka, @andrewnester
PRs	bpo-29863: Add json.COMPACT constant #72

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/etrepum'
closed_at = <Date 2017-03-14.05:11:28.407>
created_at = <Date 2017-02-13.02:04:34.589>
labels = ['3.7', 'type-feature', 'library']
title = 'Add compact=True flag to json.dump/dumps'
updated_at = <Date 2017-03-14.17:06:54.835>
user = 'https://bugs.python.org/AlexGordon'

bugs.python.org fields:

activity = <Date 2017-03-14.17:06:54.835>
actor = 'benhoyt'
assignee = 'bob.ippolito'
closed = True
closed_date = <Date 2017-03-14.05:11:28.407>
closer = 'rhettinger'
components = ['Library (Lib)']
creation = <Date 2017-02-13.02:04:34.589>
creator = 'Alex Gordon'
dependencies = []
files = []
hgrepos = []
issue_num = 29540
keywords = []
message_count = 21.0
messages = ['287663', '287685', '287707', '287716', '287739', '287740', '287746', '287773', '287774', '287786', '287827', '288026', '288030', '288161', '288162', '289543', '289547', '289564', '289581', '289603', '289608']
nosy_count = 10.0
nosy_names = ['brett.cannon', 'rhettinger', 'bob.ippolito', 'vstinner', 'r.david.murray', 'benhoyt', 'berker.peksag', 'serhiy.storchaka', 'Alex Gordon', 'andrewnester']
pr_nums = ['72']
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29540'
versions = ['Python 3.7']

[AlexGordon]

Broadly speaking, there are three main output styles for json.dump/dumps:

1. Default: json.dumps(obj)
2. Compact: json.dumps(obj, separators=(',', ':'))
3. Pretty-printing: json.dumps(obj, sort_keys=True, indent=4)

The 'compact' style is the densest, suitable if the JSON is to be sent over the network, archived on disk, or otherwise consumed by a machine. The pretty-printed style is for human consumption: configuration files, debugging, etc.

Even though the compact style is often desirable, the API for producing it is unforgiving. It's easy to accidentally write code like the following, which silently produces invalid nonsense:

    json.dumps(obj, separators=(':', ','))

I propose the addition of a new flag compact=True, that simply sets separators=(',', ':'). e.g.

    >>> obj = {"foo": 1, "bar": 2}
    >>> json.dumps(obj, compact=True)
    '{"foo":1,"bar":2}'

The defaults for separators= are good, so eventually compact= would relegate separators= to obscurity. Setting both compact=True and separators= at the same time should be an error.

[bitdancer]

An alternative would be to define a COMPACT constant in the json module equal to (',', ':').

[andrewnester]

I've just added PR implementing alternative version provided by R. David Murray as more simple and straight-forward one.

[serhiy-storchaka]

I'm -1. This adds the maintenance and learning burden and doesn't make the user code more clear. The reader needs to search what json.COMPACT means.

[rhettinger]

I concur with Serhiy and think this API change would be a net increase in complexity.

FWIW, network transmission size issues can already be handled more effectively "Accept-Encoding: gzip, deflate".

Bob, do you have any thoughts?

[etrepum]

I agree, in isolation it's a fine proposal, but the interface here is already a bit too complex and the benefit is pretty minimal. When the size really does matter, you can take care to set it correctly once and be done with it.

[AlexGordon]

The point is that, as a principle of good API design, the json module should not generate malformed JSON unless the user very explicitly asks for their JSON to be corrupted.

Python stands alone in having a JSON serializer that can produce strings such as {"k",[1:2:3]} if the user holds it wrong.

Outside of Python, the 'compact' encoding is just the normal way that JSON is encoded. It's the default almost everywhere else. I'm not suggesting Python should default to it also, but I am suggesting that it should be safe and easy to remove the extraneous whitespace.

Historically there were two values you might want to pass to separators. Aside from (',', ':'), the other was (',', ': ') when indent was not None, to suppress the trailing space at the end of each line. This is no longer necessary after 3.4.

After 3.4, separators= currently acts as a very complicated boolean flag, because the only value that makes sense to pass to it is (',', ':'). With a compact flag, users could ignore separators entirely and so the API would be made simpler and safer.

[berkerpeksag]

I'm +1 on the idea. Currently, a user needs to find this information in a wall of text: "To get the most compact JSON representation, you should specify (',', ':') to eliminate whitespace." It's easy to miss the lack of trailing space in (',', ':').

Perhaps the constant name could be changed to something more descriptive, but I don't have a better suggestion at the moment.

(Adding Brett since he also reviewed the PR.)

[vstinner]

"Add compact=True flag to json.dump/dumps"

Oh, fun, I implemented exactly this option in my perf project. Extract:

        def dump(data, fp, compact):
            kw = {}
            if compact:
                kw['separators'] = (',', ':')
            else:
                kw['indent'] = 4
            json.dump(data, fp, sort_keys=True, **kw)

So I like the idea of moving the compact=True feature directly into the json module ;-)

(I don't propose to change the default to indent=4, that's a personal choice ;-))

[brettcannon]

I'm a little confused as the PR proposes a COMPACT constant for the module but the issue proposes a new compact argument. Which are we actively considering?

[rhettinger]

Everyone can chime in with their thoughts, but Bob gets to make the decision on this one.