`model_dump(mode="json")` can output non-serializable NaN values

[whao89]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

Hi,

See example: I'm expecting that the NaN behavior of model_dump_json() and model_dump(mode="json") to be the same. The dictionary output of model_dump(mode="json") is not strictly JSON serializable. I included the Decimal field as an analogy for the expected behavior.

Thanks!

Example Code

from decimal import Decimal
from pydantic import BaseModel

class M(BaseModel):
    a: float
    b: Decimal

M(a=float("nan"), b=Decimal("12345")).model_dump(mode="json")
#  Out[5]: {'a': nan, 'b': '12345'}

M(a=float("nan"), b=Decimal("12345")).model_dump_json()
#  Out[6]: '{"a":null,"b":"12345"}'

Python, Pydantic & OS Version

pydantic version: 2.8.2
        pydantic-core version: 2.20.1
          pydantic-core build: profile=release pgo=true
                 install path: /home/ubuntu/[redacted]/lib/python3.11/site-packages/pydantic
               python version: 3.11.8 (main, Feb 26 2024, 21:39:34) [GCC 11.2.0]
                     platform: Linux-5.15.0-1066-aws-x86_64-with-glibc2.31
             related packages: typing_extensions-4.12.2 fastapi-0.111.1 mypy-1.11.0
                       commit: unknown

[sydney-runkle]

@whao89,

Thanks for reporting this. This does look inconsistent - probably a good first issue for someone looking to dig into pydantic-core a bit 👍

[Vasanth-96]

@sydney-runkle can you please help me figuring out what could go wrong with the #10102 .

[PrettyWood]

pydantic is a python library that uses pydantic-core behind the scene for most of the validation and serialization work.
In your PR @Vasanth-96 you make the fix in the python world meaning you get the json from rust and load it, which is not as efficient as directly making the fix in the rust world
So the right place for the fix is in pydantic-core

[sreeprasannar]

Interesting.. I tried

model_dump = M(a=float("nan"), b=Decimal("12345")).model_dump(mode="json")
json.dumps(model_dump)

and I see:

'{"a": NaN, "b": "12345"}'

This is definitely different from the second output in model_dump_json but I can do json.loads('{"a": NaN, "b": "12345"}' to get a dict back.

Haven't worked with pydantic-core before but I can give it a try

[sreeprasannar]

after more exploration in the linked issue above, this is currently how one could get the desired behavior @whao89 expects:

from pydantic import BaseModel, ConfigDict
import math

class M(BaseModel):
    model_config = ConfigDict(
        ser_json_inf_nan = "constants"
    )
    a: float

M(a=float("nan")).model_dump(mode="json") # {'a': nan}
M(a=float("nan")).model_dump_json() # '{"a":NaN}'

j = json.loads(M(a=float("nan")).model_dump_json())
assert math.isnan(j['a'])

j = M(a=float("nan")).model_dump(mode="json")
assert math.isnan(j['a'])

[davidhewitt]

If I recall correctly, in pydantic/pydantic-core#1062 I added the "constants" mode of ser_json_inf_nan because it is potentially nonstandard behaviour but also at the same time very useful. We had to keep the default of "null" for compatibility with V2.

Similarly, it would be a breaking change for .model_dump(mode="json") to respect the config (at the moment it just passes the float value through unchanged), as the default of "null" would immediately transform all float values to None.

IMO this is a sad situation where we need to wait for V3, and in V3:

• We should make .model_dump(mode="json") respect the ser_json_inf_nan setting.
• We should consider changing the default of ser_json_inf_nan to "constants", as that then matches json.dumps better (which allows these constants with an opt-out allow_nan=False).