Please consider pushing tomli into stdlib

[mgorny]

Long story short, having a TOML parser in Python stdlib would be great and tomli seems to be the best implementation available right now, so also the best candidate for stdlib. Would you be interested in trying to push it?

The relevant CPython bug is: https://bugs.python.org/issue40059

[hukkin]

Long story short, having a TOML parser in Python stdlib would be great

Yep, I agree! Also I have no objections to adding Tomli there.

Would you be interested in trying to push it?

I'm not sure

• what concrete actions I'd need to take
• if I'm the right person for this
• if I have the energy

but yeah generally I don't object me or someone else pushing this forward.

[merwok]

Adding a module usually requires:

• writing a PEP to propose inclusion
• module developer agreeing to support it for some time in the standard library (developer usually becomes a python core dev)
• tests and docs should be integrated (no pytest for tests, sphinx for docs)
• a backport can be maintained, but it should follow the stdlib version and not diverge (see simplejson-json-simplejson example)
• the python core team may want to review the API and request changes (see ipaddress module for example)

So the process is not just two minutes, but also not impossible! I would suggest that you start by joining the discuss discussion and state that you’re willing to have your module adopted to the standard library. Other people can help make the case for inclusion (not the whole core team is well versed in packaging matters), and maybe offer to be responsible for maintenance.

(Note that the whole team collectively maintains the whole of Python, but some modules have dedicated maintainers who know the design and details well, and for a new module it’s expected to have one. In this specific case, as a parser for a specified format, it may be less important that you commit to maintain for the long term, and it is possible that someone who is both in pypa and python-dev will offer to be co-maintainer.)

[hukkin]

Thanks for your insight!

• writing a PEP to propose inclusion

This seems tedious. Any help much appreciated 😄

• module developer agreeing to support it for some time in the standard library

I can do this.

• tests and docs should be integrated (no pytest for tests, sphinx for docs)

I can do this.

• a backport can be maintained, but it should follow the stdlib version and not diverge

Will do this anyways to support non EOL Python versions less than the one where stdlib addition happens.

• the python core team may want to review the API and request changes

Fair enough.

I would suggest that you start by joining the discuss discussion and state that you’re willing to have your module adopted to the standard library.

I've actually been participating for quite some time already. And have stated I'm willing to have the module adopted to the stdlib.

---

If I've understood correctly, one additional obstacle is that since moving from BDFL to steering council, the process for adding/removing modules is not standardized yet, see python/steering-council#92

[merwok]

I've actually been participating for quite some time already. And have stated I'm willing to have the module adopted to the stdlib.

Ah good! Then I think you could send a call for help to write the PEP and make a fork of cpython with tomli added (or maybe not, there is no guarantee that the addition will be accepted).

If I've understood correctly, one additional obstacle is that since moving from BDFL to steering council, the process for adding/removing modules is not standardized yet

It’s not really an obstacle: there is a discussion in progress about the process, but that doesn’t mean additions are blocked, just that they use ad-hoc processes. zoneinfo and graphlib were added to 3.9!

[johnthagen]

If tomli is added to the standard library, I hope that it can be added using the simple name toml for long term consistency.

Packages supporting older Python versions could do something like:

try:
    import toml
except ImportError:
    import tomli as toml

Also note that there are currently plans to get control of the toml package name on PyPI: uiri/toml#361 (comment)

[hauntsaninja]

Hello @hukkin ! Happy new year and thank you for tomli and thank you for being so willing to help fill this gap in the Python ecosystem!

Over at https://bugs.python.org/issue40059, Brett indicated that the next step would be making a proposal to the SC.
With that in mind, I wrote up https://gist.github.com/hauntsaninja/9f136a5a60f63d8ca2cdfadb50edba44 as something we could send to python-dev.

I'd love to hear your thoughts :-) Would also appreciate any feedback from @pradyunsg (sorry for the somewhat random tag)!

[pradyunsg]

FYI: There's a bunch of discussion in https://discuss.python.org/t/adopting-recommending-a-toml-parser/4068.

[hauntsaninja]

Of course there's a 90 post thread that I've missed :-) Thanks for linking that.

https://discuss.python.org/t/adopting-recommending-a-toml-parser/4068/84 seems to change Brett's stance to "PEP is necessary", so I can help write up something more formal than my current (discursive and informal) proposal.

Not much new in the proposal that's not there in the thread, but might still be a good summary and the anecdotal evidence from grepping open source code might help sell API changes (particularly if we make breaking changes to uiri/toml). Comforting to know that the fallback package names I jotted down have all been suggested :-)

[hukkin]

Awesome work @hauntsaninja !

I added some notes to the gist's comment section and also pinged @gaborbernat there who showed interest in writing a PEP as well.

[hukkin]

Also tagging @encukou as they said they'd be able to co-maintain in the stdlib. I think we've got a pretty good team now. 😸

There's a few major decisions where the PEP must take a side. I assume it's best to plan these out either here or in the PEPs PR and return to https://discuss.python.org with the first draft complete?

Here's some decisions that come to mind:

A) What to do with future TOML spec versions?

1. TOML v1.0.0 forever
2. Keep up-to-date with the spec unless it makes breaking syntax changes (probably meaning TOML v2.0.0)
3. Keep up-to-date even if syntax breaking changes occur.

B) Which name?

1. toml
2. tomli
3. tomllib
4. tomlparser
5. toml, but under some other namespace. E.g. parser.toml, decoder.toml, formats.toml (didn't check if any of these are available).

Option 1. is special because the name is currently in use by https://github.com/uiri/toml so being able to use it requires that pradyunsg is able to claim the name. It also means we have to make breaking changes to the toml PyPI package and will break projects that use a pinned toml version (thanks encukou for pointing this out).

C) Should the standard library be able to dump(s) TOML?

1. No.
2. Yes, but keep it simple. Copy Tomli-W's API as is.
3. Yes, but keep it even simpler: remove the multiline_strings keyword argument from Tomli-W's API.
4. Yes, and allow output customization. Add a json-esque overridable TOMLEncoder class.

---

My personal favorites are probably: A2, B1 (if possible, else B3 or B4), and C1.

It should be noted that if we choose B1 (use toml name) then not adding dump capability means we need to break the existing toml package more.

EDIT: I know that @pradyunsg would want to keep write capability and TOMLEncoder API in the toml PyPI package, so not sure if they are happy to hand over the name to standard library if it ends up being read-only?

[merwok]

tomlparser is a nice name for a reading-only module. No expectations of writing and no name conflict.
A namespace was tried for concurrent.futures and it was a mistake (nothing else went there, and adjective for package name is always awkward to me).

It should implement the current spec at v1, could follow minor revisions as bugfixes, and add support for a possible v2 with a separate parser class (or parameters in the existing one if that’s clean and easy). Once in the stdlib, it should keep support for v1 working.