Python⇒Speed: Loading Pydantic models from JSON without running out of memory
You have a large JSON file, and you want to load the data into Pydantic. Unfortunately, this uses a lot of memory, to the point where large JSON files are very difficult to read. What to do?
Assuming you’re stuck with JSON, in this article we’ll cover:
- The high memory usage you get with Pydantic’s default JSON loading.
- How to reduce memory usage by switching to another JSON library.
- Going further by switching to dataclasses with slots.
