Filters decorated with @async_variant are not pickleable

[dairiki]

The jinja2.async_utils.async_variant decorator puts the wrong __name__ and __qualname__ on the decorated function. One result of this is that functions which it decorates are not pickleable.

Here's a doctest script which illustrates the issue.

>>> import pickle
>>> import jinja2.filters

>>> pickle.dumps(jinja2.filters.do_first)
Traceback (most recent call last):
    ...
_pickle.PicklingError: Can't pickle <function sync_do_first at 0x...>: it's not the same object as jinja2.filters.sync_do_first

# Here's the crux:
>>> jinja2.filters.do_first.__name__
'sync_do_first'

Environment:

• Python version: 3.10 — but I don't think that's important
• Jinja version: git main (09907ef) — but I think this affects all versions >= 3

I will submit a PR with one possible fix shortly.

[dairiki]

I've just discovered, as well, that instances of Undefined (and subclasses) are not pickleable.

The reason for this seems to be two-fold:

1. The post-class-definition deletion of the __slots__ attribute done here. If that is being done solely for aesthetics, perhaps that is unnecessary.
2. jinja2.utils.missing is not pickleable, owing to its type being inaccessible ("attribute lookup MissingType on jinaj2.utils failed").

(If you'd like a PR, let me know.)

[davidism]

Jinja environments and templates were never intended to be pickleable, there are other things besides these that make it difficult. I'm not sure what you're trying to do, but there's probably a better way to accomplish what you want.

[dairiki]

Jinja environments and templates were never intended to be pickleable, there are other things besides these that make it difficult. I'm not sure what you're trying to do, but there's probably a better way to accomplish what you want.

Yes, but why not fix the simple things?
There's no real reason (AFAICT) that do_first.__name__ shouldn't be "do_first". (It could be argued, even, that doing otherwise is an anti-pattern.)

All three of the items I've listed in this issue that prevent pickleability, are due to niggling details of implementation which could just as well be done in a way that would allow for pickleability.

I.e. I'm not asking for guarantees that all of Jinja should be pickleable, just trying to fix the bits that are unpickleable for no particularly good reason.

---

What's driving this for me: I'm starting to look into using multiprocessing (probably via multiprocessing) in the Lektor build process to speed up build times. Particularly on Windows, this seems to require passing things like the Lektor Environment instance to subprocesses — that's straightforward so long as everything is pickleable (the Lektor Environment contains a Jinja environment.)

[dairiki]

@davidism Thank you for merging #1613!

Would you accept PRs for the other two issues listed above? Namely:

• Avoid deleting __slots__ attribute (on Undefined and subclasses)?
• Actualize jinja.utils.MissingType singleton class or figure out some other way to make the jinja.utils.missing sentinel pickleable.

[Edit: Typo, sorry. jinja.utils... not lektor.utils... (fixed).]

[ETA: By "actualize" I mean make things so the type of jinja.utils.missing (which is jinja.utils.MissingType) is actually accessible by name. Care should be taken — e.g. a custom MissingType.__new__ — to ensure that it is a singleton. An alternative would be to add a custom __reduce__ to missing (which would return a callable that would import then return jinja.utils.missing.) What we'd really like here is something like PEP-661.]