Make sure generated JSON Schemas are valid in tests

[Viicos]

Fixes #10173

Edit: fixes #5164.

Requires:

• Deprecate passing a dict to the Examples class #10181
• Breaking Change: Fix JSON Schema generation for constrained dates #10185
• Fix invalid JSON Schemas being generated for functions in certain scenarios #10188

With the schema validation:

real	0m31.669s
user	0m28.128s
sys	0m3.617s

Without:

real	0m29.951s
user	0m26.481s
sys	0m3.537s

Change Summary

Related issue number

Checklist

• The pull request title is a good summary of the changes - it will be used in the changelog
• Unit tests for the changes exist
• Tests pass on CI
• Documentation reflects the changes where applicable
• My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

[cloudflare-workers-and-pages]

Deploying pydantic-docs with    Cloudflare Pages

Latest commit:	c39d7da
Status:	✅  Deploy successful!
Preview URL:	https://916add11.pydantic-docs.pages.dev
Branch Preview URL:	https://10173-json-schema-tests.pydantic-docs.pages.dev

View logs

[codspeed-hq]

CodSpeed Performance Report

Merging #10182 will not alter performance

_{Comparing 10173-json-schema-tests (c39d7da) with main (5d99a65)}

Summary

✅ 17 untouched benchmarks

[Viicos]

Ok, one more case that I'm going to fix in a separate PR related to function arguments schemas.

[github-actions]

Coverage report

This PR does not seem to contain any modification to coverable code.

[sydney-runkle]

With the schema validation...

Perfect, thanks for including these performance comparisons here. I'm fine with this slight slowdown for what feels like a large amount of added value. This is a great quality check.

[sydney-runkle]

Looking great overall - just one small change request re the verbosity of the error message + a potential test for this new logic.

[sydney-runkle]

Perhaps this could be a bit more verbose:

1. Can we include the SchemaError information in this result?
2. Could we add a note about the fact that this is purely a testing feature, not a runtime pydantic check (at this point)?

Perhaps this is a bit excessive, but can we test this test? As in, run a test within a test, and check that this is raised for invalid schema? If not, no worries, but could you document an example of a failing test on this PR just to have as a reference for the blame later?

[Viicos]

1. Can we include the SchemaError information in this result?

Pytest will display the exception by default. This is the output you would get for a failing test:

tests/test_json_schema.py F                                                                                                                       [100%]
tests/test_json_schema.py:6476 test_fails - Failed: Failed to validate the JSON Schema against the Draft 2020-12 spec…                            [100%]
======================================================================= FAILURES ========================================================================
______________________________________________________________________ test_fails _______________________________________________________________________

args = (<pydantic.json_schema.GenerateJsonSchema object at 0x7b80723f5820>, {'metadata': {'pydantic_js_annotation_functions':...onSchema.__get_pydantic_json_schema__ of WithJsonSchema(json_schema={'type': 'invalid'}, mode=None)>]}, 'type': 'int'})
kwargs = {'mode': 'validation'}, json_schema = {'type': 'invalid'}

    def generate(*args: Any, **kwargs: Any) -> Any:
        json_schema = orig_generate(*args, **kwargs)
        if not request.node.get_closest_marker('skip_json_schema_validation'):
            try:
>               Draft202012Validator.check_schema(json_schema)

tests/conftest.py:166: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cls = <class 'jsonschema.validators.Draft202012Validator'>, schema = {'type': 'invalid'}
format_checker = <FormatChecker checkers=['date', 'email', 'idn-email', 'idn-hostname', 'ipv4', 'ipv6', 'regex', 'uuid']>

    @classmethod
    def check_schema(cls, schema, format_checker=_UNSET):
        Validator = validator_for(cls.META_SCHEMA, default=cls)
        if format_checker is _UNSET:
            format_checker = Validator.FORMAT_CHECKER
        validator = Validator(
            schema=cls.META_SCHEMA,
            format_checker=format_checker,
        )
        for error in validator.iter_errors(schema):
>           raise exceptions.SchemaError.create_from(error)
E           jsonschema.exceptions.SchemaError: 'invalid' is not valid under any of the given schemas
E           
E           Failed validating 'anyOf' in metaschema['allOf'][3]['properties']['type']:
E               {'anyOf': [{'$ref': '#/$defs/simpleTypes'},
E                          {'type': 'array',
E                           'items': {'$ref': '#/$defs/simpleTypes'},
E                           'minItems': 1,
E                           'uniqueItems': True}]}
E           
E           On schema['type']:
E               'invalid'

../../.pyenv/versions/3.12.4/envs/pydanticdev/lib/python3.12/site-packages/jsonschema/validators.py:317: SchemaError

During handling of the above exception, another exception occurred:

    def test_fails():
>       TypeAdapter(Annotated[int, WithJsonSchema({'type': 'invalid'})]).json_schema()

tests/test_json_schema.py:6478: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pydantic/type_adapter.py:142: in wrapped
    return func(self, *args, **kwargs)
pydantic/type_adapter.py:549: in json_schema
    return schema_generator_instance.generate(self.core_schema, mode=mode)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = (<pydantic.json_schema.GenerateJsonSchema object at 0x7b80723f5820>, {'metadata': {'pydantic_js_annotation_functions':...onSchema.__get_pydantic_json_schema__ of WithJsonSchema(json_schema={'type': 'invalid'}, mode=None)>]}, 'type': 'int'})
kwargs = {'mode': 'validation'}, json_schema = {'type': 'invalid'}

    def generate(*args: Any, **kwargs: Any) -> Any:
        json_schema = orig_generate(*args, **kwargs)
        if not request.node.get_closest_marker('skip_json_schema_validation'):
            try:
                Draft202012Validator.check_schema(json_schema)
            except SchemaError:
>               pytest.fail('Failed to validate the JSON Schema against the Draft 2020-12 spec')
E               Failed: Failed to validate the JSON Schema against the Draft 2020-12 spec

tests/conftest.py:168: Failed
                                   Summary of Failures                                    
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃  File                       ┃  Function    ┃  Function Line  ┃  Error Line  ┃  Error   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│  tests/test_json_schema.py  │  test_fails  │  6477           │  6478        │  Failed  │
└─────────────────────────────┴──────────────┴─────────────────┴──────────────┴──────────┘
Results (1.65s):
         1 failed
      5846 deselected

2. Could we add a note about the fact that this is purely a testing feature, not a runtime pydantic check (at this point)?

Added.

Perhaps this is a bit excessive, but can we test this test?

Seems like there are ways to do so, but they are pretty involved. I added a test with an expected failure.

[sydney-runkle]

Nice work, thanks for incorporating the feedback so promptly!