How to convert annotations to Format.SOURCE in __annotate__?

Feature or enhancement

Let's say you have a dict of some custom annotations like I have in #122262

How users are expected to convert say an annotation dict of {'user': CustomUser[AuthToken], 'auth_callback': Callable[[CustomUser[T]], T]} to string?

There are several ways right now:

repr for simple types, which might not work for complex ones

cpython/Lib/annotationlib.py

Lines 466 to 501 in 536bc8a

    
           if format == Format.SOURCE: 
        
               # SOURCE is implemented by calling the annotate function in a special 
        
               # environment where every name lookup results in an instance of _Stringifier. 
        
               # _Stringifier supports every dunder operation and returns a new _Stringifier. 
        
               # At the end, we get a dictionary that mostly contains _Stringifier objects (or 
        
               # possibly constants if the annotate function uses them directly). We then 
        
               # convert each of those into a string to get an approximation of the 
        
               # original source. 
        
               globals = _StringifierDict({}) 
        
               if annotate.__closure__: 
        
                   freevars = annotate.__code__.co_freevars 
        
                   new_closure = [] 
        
                   for i, cell in enumerate(annotate.__closure__): 
        
                       if i < len(freevars): 
        
                           name = freevars[i] 
        
                       else: 
        
                           name = "__cell__" 
        
                       fwdref = _Stringifier(ast.Name(id=name)) 
        
                       new_closure.append(types.CellType(fwdref)) 
        
                   closure = tuple(new_closure) 
        
               else: 
        
                   closure = None 
        
               func = types.FunctionType( 
        
                   annotate.__code__, 
        
                   globals, 
        
                   closure=closure, 
        
                   argdefs=annotate.__defaults__, 
        
                   kwdefaults=annotate.__kwdefaults__, 
        
               ) 
        
               annos = func(Format.VALUE) 
        
               if _is_evaluate: 
        
                   return annos if isinstance(annos, str) else repr(annos) 
        
               return { 
        
                   key: val if isinstance(val, str) else repr(val) 
        
                   for key, val in annos.items() 
        
               }

but, it requires complex annotate object

cpython/Lib/typing.py

Lines 2955 to 2956 in 536bc8a

    
           def _convert_to_source(types): 
        
               return {n: t if isinstance(t, str) else _type_repr(t) for n, t in types.items()}

but it is a private API

I propose adding a public and documented API for that.

Linked PRs

gh-124412: Add helpers for converting annotations to source format #124551

The use cases for this would be cases where types get provided outside annotations (e.g., with the functional syntax for NamedTuple and TypedDict, and with make_dataclass), and you want the __annotate__ method to support the SOURCE format.

The implementation is repr() for most cases, but for types we want the fully qualified name instead of <type 'int'>, and there are a few other special cases. Currently we have typing._convert_to_source, which takes an annotations dict and uses typing._type_repr to repr each annotation.

Since we already have use cases from two standard library modules (typing and dataclasses), I think it makes sense to add something to annotationlib. I would suggest:

annotationlib.repr_annotations(dict), similar to the current typing._convert_source
annotationlib.repr_type(type), which works like typing._type_repr but without the special case for tuples

I want to investigate on this feature, because I still have a limited understanding of __annotate__ and all of its corner-cases.

Will probably work on this tomorrow 👍

Thanks! We'll also want to add the new functions to PEP 749.

We also need this in annotationlib itself, for the edge case where a user-created object has __annotations__ but not __annotate__. Currently, in that case SOURCE returns non-strings:

>>> class X:
...     @property
...     def __annotations__(self):
...         return {"x": int}
...         
>>> x = X()
>>> import annotationlib
>>> annotationlib.get_annotations(x, format=annotationlib.Format.SOURCE)
{'x': <class 'int'>}

I think the most reasonable behavior for this case is to return {'x': 'int'}.

pythongh-124412: Add helpers for converting annotations to source format

My initial reaction is, this seems like a least-worst option, and it may even be the least-worst option. I'd like to marinate on it further. I won't mind too much if you go ahead and merge without my blessing--as long as you don't mind me retroactively pushing back if I (eventually) arrive at some other conclusion.

I will say, shenanigans like this are an argument against calling this format SOURCE.

gh-124412: Add helpers for converting annotations to source format (#…

Merge branch 'main' into fix-help-double-prompt

@JelleZijlstra

@JelleZijlstra can this be closed now? Sorry, I was not quick enough to send a PR :( It was only partially ready yesterday, due to some other work.

Let's leave it open for a while to allow Larry to ruminate.

Jelle has convinced himself that we should change the name away from SOURCE, I think he preferred STRINGS which is fine.

It's a day later and this approach seems like a good idea. Ship it! We can always regret it later.

To be precise I propose STRING, since the other formats are also in the singular.

STRING is fine by me. My only counter-proposal is TEXT, which is also sufficiently singular. I don't have a strong opinion about it either way; I don't think TEXT does any better job of conveying the intended semantics of the format.

Navigation Menu

Repository navigation

How to convert annotations to `Format.SOURCE` in `annotate`? #124412

Feature or enhancement

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	if format == Format.SOURCE:
	# SOURCE is implemented by calling the annotate function in a special
	# environment where every name lookup results in an instance of _Stringifier.
	# _Stringifier supports every dunder operation and returns a new _Stringifier.
	# At the end, we get a dictionary that mostly contains _Stringifier objects (or
	# possibly constants if the annotate function uses them directly). We then
	# convert each of those into a string to get an approximation of the
	# original source.
	globals = _StringifierDict({})
	if annotate.__closure__:
	freevars = annotate.__code__.co_freevars
	new_closure = []
	for i, cell in enumerate(annotate.__closure__):
	if i < len(freevars):
	name = freevars[i]
	else:
	name = "__cell__"
	fwdref = _Stringifier(ast.Name(id=name))
	new_closure.append(types.CellType(fwdref))
	closure = tuple(new_closure)
	else:
	closure = None
	func = types.FunctionType(
	annotate.__code__,
	globals,
	closure=closure,
	argdefs=annotate.__defaults__,
	kwdefaults=annotate.__kwdefaults__,
	)
	annos = func(Format.VALUE)
	if _is_evaluate:
	return annos if isinstance(annos, str) else repr(annos)
	return {
	key: val if isinstance(val, str) else repr(val)
	for key, val in annos.items()
	}

	def _convert_to_source(types):
	return {n: t if isinstance(t, str) else _type_repr(t) for n, t in types.items()}

How to convert annotations to Format.SOURCE in __annotate__? #124412

Description

Feature or enhancement

Linked PRs

Activity

JelleZijlstra commented on Sep 24, 2024

sobolevn commented on Sep 24, 2024

JelleZijlstra commented on Sep 24, 2024

JelleZijlstra commented on Sep 25, 2024

larryhastings commented on Sep 25, 2024

larryhastings commented on Sep 25, 2024

sobolevn commented on Sep 26, 2024

JelleZijlstra commented on Sep 26, 2024

larryhastings commented on Sep 26, 2024

JelleZijlstra commented on Sep 27, 2024

larryhastings commented on Sep 29, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

How to convert annotations to `Format.SOURCE` in `annotate`? #124412