The talk involved a demo running CPU & I/O bound workloads (standalone script and flask endpoints) with GIL enabled and disabled, and comparing the performance.

The benchmarking was done using Python 3.14 RC2 (the talk was before Python 3.14 released).

I originally presented this at PyCascades 2025, using Python 3.13. This time, I updated the benchmarks to Python 3.14 RC2, and added few updates regarding free threaded mode. This post is a recreation of the talk content in text format.

Access Presentation Slides Here · Benchmarking Scripts

GIL

GIL (Global Interpreter Lock) is a mutex that protects access to Python objects, preventing multiple threads from executing Python bytecodes at once.

GIL simplified CPython’s internals such as memory management. This is because code can be written with the assumption that at any single point in time, only one thread will be executing Python bytecode.

The Problem: Impact on Multithreading

The GIL can be a bottleneck in CPU-bound and multithreaded code. In CPU-bound multithreaded programs, threads often compete for the GIL, leading to context switching and reduced performance. I/O bounded programs are less affected, as threads can release the GIL during I/O operations.

Historically, multiprocessing was the primary way to achieve parallelism in Python for CPU-bound tasks, but with inter-process communication overhead.

PEP 703: Making the GIL Optional in CPython

PEP 703 proposed making the Global Interpreter Lock optional in CPython. It introduces “free-threaded build” of Python where the GIL is disabled. The PEP aims to unlock true parallel execution for Python threads, especially for CPU-bound workloads.

It was experimental in Python 3.13, eventually evolving into an optional (or possibly default-disabled) feature in future releases.

Challenges of Free-Threaded Python

• Backward Compatibility: Many C extensions are written assuming the GIL will exist. These extensions may not be thread-safe without the GIL, leading to potential crashes or data corruption.
• Performance Overhead: Free-threaded builds may run single-threaded code slower. This is due to the need for additional locking mechanisms to ensure thread safety (since multiple threads will be executing at a single point in time).

PEP 779 – Criteria for supported status for free-threaded Python:

The PEP outlines criteria for when free-threaded Python can be considered a supported feature. This included requirements that the free threaded build needs to meet across performance, stability, etc.

PEP 779 evaluated the free-threaded build against these criteria in Python 3.13 and the criteria were satisfied.

“The free-threaded build of Python is now supported and no longer experimental. This is the start of phase II where free-threaded Python is officially supported but still optional." - Python 3.14 Release Notes

The performance penalty on single-threaded code in free-threaded mode is now roughly 5-10%.

Setting Up Free Threaded Python

The official guide to set-up free-threaded Python is available here.

Windows/macOS users can use the installers from Python.org that include free-threaded builds.

For Ubuntu, free-threaded Python 3.14 can be installed from the deadsnakes PPA:

sudo add-apt-repository ppa:deadsnakes
sudo apt-get update
sudo apt-get install python3.14-nogil

Benchmarks

This section presents benchmarks comparing performance of standard Python 3.14 (with GIL) and free-threaded Python 3.14 (without GIL).

• For benchmarking, I created two virtual environments:
  • One with standard Python 3.14 (with GIL)
  • Another with free-threaded Python 3.14 (3.14t, without GIL, python3.14t -m venv .venv)

Different workloads were benchmarked. When running the benchmarks, the GIL status was checked to ensure the correct interpreter was used.

• GIL status can be checked using any of the following methods:

import sys

print(sys._is_gil_enabled())  # True if GIL is enabled, False otherwise

OR:

import sysconfig

sysconfig.get_config_var("Py_GIL_DISABLED")  # True if GIL is disabled, False otherwise

• The above is wrapped in check_gil.py file (used in benchmarks):

import sys
import sysconfig


def show_gil_status():
  gil_status = sys._is_gil_enabled()

  print(F"{sys._is_gil_enabled()=}")

  if gil_status:
    print("\nRunning in default interpreter - GIL is enabled\n")
  else:
    print("\nRunning in Free threaded interpreter - GIL is disabled\n")


def check_free_threading_support():
  print(f"\n{sysconfig.get_config_vars().get('Py_GIL_DISABLED')=}\n")


if __name__ == '__main__':
  check_free_threading_support()

  show_gil_status()

First benchmark - Single Threaded Program

The first benchmark involves running a single threaded program that sums numbers from 1 to 1 million, repeated 100 times and timed using timeit module.

import timeit

from check_gil import show_gil_status


def single_threaded_benchmark():
  show_gil_status()

  # Sum of numbers from 1 to 1 million
  code_to_test = """

total = 0
for i in range(1_000_000):
    total += i

"""

  # Use timeit to measure execution time - Run the code 100 times
  execution_time = timeit.timeit(code_to_test, number=100)

  print(f"Execution time: {execution_time:.6f} seconds")


if __name__ == '__main__':
  single_threaded_benchmark()

Benchmark Results: In my system, the free threaded execution was ~15% slower than default interpreter. Default interpreter completed in ~3s, while free-threaded took ~3.5s.

Execution time of individual runs are added in the next section, in the Observations table.

I/O Bound Program

Next, let’s benchmark an I/O bound program that performs file write and read operations.

We execute the code in single threaded mode as well as multi-threaded mode using ThreadPoolExecutor and repeat both with GIL enabled and disabled.

The code creates a temporary file, writes data to it in chunks, forces flush and fsync to ensure data is committed to disk, then reads the data back in chunks. This simulates a realistic I/O workload.

import os
import tempfile
import time
from concurrent.futures import ThreadPoolExecutor, wait, ALL_COMPLETED

from check_gil import show_gil_status

FILE_MB = 24  # total temp file size per op

CHUNK_KB = 64  # write/read granularity

SINGLE_OPS = 8  # Single threaded executions

MULTI_OPS = 12  # Multi-threaded executions
THREADS = 6

# ===== Derived =====
_CHUNK_BYTES = CHUNK_KB * 1024
_TOTAL_CHUNKS = (FILE_MB * 1024 * 1024) // _CHUNK_BYTES
BUF = b"A" * _CHUNK_BYTES


def _write_pass(f, buf):
  """Write the buffer repeatedly across the file, forcing flush+fsync each time to ensure data is committed to disk."""
  f.seek(0)

  # make sure each write is flushed and synced to disk - to increase IO load.
  for _ in range(_TOTAL_CHUNKS):
    f.write(buf)
    f.flush()
    os.fsync(f.fileno())


def _read_pass(f):
  """Read the entire file back chunk by chunk to simulate sequential disk read workload."""
  f.seek(0)
  remaining = _TOTAL_CHUNKS
  while remaining > 0:
    data = f.read(_CHUNK_BYTES)
    if not data:
      break
    remaining -= 1


def io_bound():
  """Perform one full write+read cycle on a temporary file with forced disk syncs, then clean up the file."""
  tmp = tempfile.NamedTemporaryFile(prefix="io_bench_", delete=False)
  tmp_name = tmp.name
  tmp.close()
  try:
    with open(tmp_name, "wb", buffering=0) as f0:
      f0.truncate(FILE_MB * 1024 * 1024)

    flags = os.O_RDWR | os.O_CREAT

    fd = os.open(tmp_name, flags, 0o600)
    f = os.fdopen(fd, "r+b", buffering=0)

    try:
      _write_pass(f, BUF)
      _read_pass(f)
    finally:
      f.close()

  finally:
    try:
      os.remove(tmp_name)
    except Exception:
      pass
  return "IO task done!"


def single_threaded():
  for _ in range(SINGLE_OPS):
    io_bound()


def multi_threaded():
  with ThreadPoolExecutor(max_workers=THREADS) as executor:
    futs = [executor.submit(io_bound) for _ in range(MULTI_OPS)]
    wait(futs, return_when=ALL_COMPLETED)


def run_benchmark():
  t0 = time.time()
  single_threaded()
  print(f"Single thread Exec Time: {time.time() - t0:.2f}s")

  t1 = time.time()
  multi_threaded()
  print(f"Multi thread Exec Time: {time.time() - t1:.2f}s")


if __name__ == '__main__':
  show_gil_status()
  print(f"[Config] FILE_MB={FILE_MB}, CHUNK_KB={CHUNK_KB} "
        f"SINGLE_OPS={SINGLE_OPS}, MULTI_OPS={MULTI_OPS}, THREADS={THREADS}")
  run_benchmark()

Benchmark Results: In case of this I/O bound operation, both interpreters performed comparably. Single threaded execution took ~2.9s in both interpreters. Multi-threaded execution took ~2.4s in both interpreters.

CPU Bound Program

The task involves performing a computationally intensive operation. We calculate primes upto the upper limit of 2 million.

The task is done using two modes:

1. Single-threaded mode: The prime counting function is executed in a single thread.
2. Multi-threaded mode: The prime counting function is divided into chunks, and each chunk is processed in a separate thread using ThreadPoolExecutor. This way, we can see the multi-threading related performance changes.

import math
import time
import os
from concurrent.futures import ThreadPoolExecutor, as_completed

# CPU bound task: count primes up to UPPER

# Upper limit for prime search - 2 Million
UPPER = 2_000_000

# Chunk size for multithreading
CHUNK_SIZE = 50_000


def is_prime(n: int) -> bool:
  if n < 2:
    return False
  if n % 2 == 0:
    return n == 2
  r = int(math.isqrt(n))

  # check if number is divisible by any odd number up to sqrt(n)
  for f in range(3, r + 1, 2):
    if n % f == 0:
      return False
  return True


def count_primes_range(a: int, b: int) -> int:
  # Count primes in [a, b)
  cnt = 0
  # handle 2 explicitly, then iterate odds
  if a <= 2 < b:
    cnt += 1
  start = max(a, 3)
  if start % 2 == 0:
    start += 1
  for n in range(start, b, 2):
    if is_prime(n):
      cnt += 1
  return cnt


# Calculate primes in single-threaded mode - 3 - 2 Million in a single thread
def single_threaded(upper: int) -> tuple[int, float]:
  t0 = time.perf_counter()
  cnt = count_primes_range(2, upper)
  dt = time.perf_counter() - t0
  return cnt, dt


# Calculate primes in multi-threaded mode using threads. Each thread processes a chunk of 50,000 (CHUNK_SIZE).
def multithreaded_threads(upper: int, max_workers: int | None = None) -> tuple[int, float, int]:
  # Split [2, upper) into chunks and process with threads.

  if max_workers is None:
    max_workers = max(2, os.cpu_count() or 4)

  # Build chunks
  chunks = []
  lo = 2
  while lo < upper:
    hi = min(lo + CHUNK_SIZE, upper)
    chunks.append((lo, hi))
    lo = hi

  t0 = time.perf_counter()
  total = 0
  with ThreadPoolExecutor(max_workers=max_workers) as ex:
    futures = [ex.submit(count_primes_range, a, b) for a, b in chunks]
    for fut in as_completed(futures):
      total += fut.result()
  dt = time.perf_counter() - t0
  return total, dt, max_workers


def main():
  print(f"Computing prime count in [2, {UPPER})")
  cnt1, t1 = single_threaded(UPPER)
  print(f"[Single-threaded] primes={cnt1:,}  time={t1:.2f}s")

  cnt2, t2, k = multithreaded_threads(UPPER)
  print(f"[Multithreaded (threads={k})] primes={cnt2:,}  time={t2:.2f}s")

  # Sanity check: both modes should count the same
  print("Results equal:", cnt1 == cnt2)


if __name__ == "__main__":
  main()

Benchmark Results: In single threaded mode, default interpreter completed in ~2.7s, while free-threaded took ~3.1s. For multi-threaded execution, default interpreter took ~2.7s, while free-threaded completed in ~0.8s - a significant improvement.

When using free-threaded Python, we got performance penalty in single-threaded code, but significant performance in multi-threaded mode.

WSGI App with Flask

Flask app is run in development mode with Flask’s built-in server (multi-threaded by default). So the WSGI server can be thought of as a multi-threaded server.

We have two API Endpoints defined - 1 running I/O bound operation & other running CPU bound task.

We benchmark the endpoints using load testing tool k6. k6 creates multiple virtual users (VUs) simulating real-users of the application and makes requests to the endpoints.

import threading

from flask import Flask

import tempfile
import os

from check_gil import show_gil_status

app = Flask(__name__)


# CPU-bound task
def heavy_computation():
  # sum of squares from 0 to 10 million
  return sum(i * i for i in range(10 ** 7))


# Routes
@app.route('/')
def root():
  return f"{threading.active_count()=}"


# Routes
@app.route('/cpu')
def cpu_bound():
  print(threading.active_count())
  heavy_computation()
  return "CPU task done!"


@app.route('/io')
def io_bound():
  print(threading.active_count())

  with tempfile.TemporaryFile() as f:
    f.write(b"Hello, World!" * 10000000)
    f.seek(0)
    f.read()
    os.remove(f.name)  # Added for more IO work

  with tempfile.TemporaryFile() as f:
    f.write(b"Hello, World!" * 10000000)
    f.seek(0)
    f.read()
    os.remove(f.name)  # Added for more IO work

  with tempfile.TemporaryFile() as f:
    f.write(b"Hello, World!" * 10000000)
    f.seek(0)
    f.read()
    os.remove(f.name)  # Added for more IO work

  return "IO task done!"


if __name__ == '__main__':
  show_gil_status()

  # By default, Flask's built-in server is multi-threaded.
  # This can be verified by visiting the root endpoint (/).
  app.run()

Benchmarks were done using the k6. The benchmark scripts used are available here

I configured k6 to run for 20 seconds with 15 virtual users (VUs) making requests to the Flask app endpoints. The benchmark was done using k6 run benchmark_io.js and k6 run benchmark_cpu.js commands.

Results: For I/O bound endpoint, both interpreters performed comparably - ~31-32 requests completed in 20 seconds. For CPU bound endpoint, free-threaded Python showed significant improvement - ~91 requests completed in 20 seconds, compared to ~47 requests with GIL enabled.

The results indicate that in our WSGI app, free-threaded Python can handle CPU-bound endpoints more efficiently in a multi-threaded environment, while I/O-bound workloads remain largely unaffected by the presence or absence of the GIL.

Observations

The below table shows the consolidated benchmark results:

For flask app, the benchmark lists the number of requests completed in 20 seconds (higher the better).

Summary of benchmarks:

• Single threaded - Worse performance in free-threaded mode.
• I/O Bound programs - Minimal/no impact. Both interpreters perform comparably.
• For CPU Bound, Multi-threaded Programs, free-threaded Python shows significant performance improvements.

The future of Free Threaded Python

1. In 2024, CPython 3.13 was released with support for a --disable-gil build time flag. There are two ABIs for CPython, one with the GIL and one without. Extension authors target both ABIs.
2. We are here now (PEP 779 Acceptance): “The free-threaded build of Python is now supported and no longer experimental".
3. After 2–3 releases, (i.e., in 2026–2027), CPython is released with the GIL controlled by a runtime environment variable or flag. The GIL is enabled by default. There is only a single ABI.
4. After another 2–3 release (i.e., 2028–2030), CPython switches to the GIL being disabled by default. The GIL can still be enabled at runtime via an environment variable or command line flag.

Migration Checklist

Free-threaded Python:

• Great for CPU bound Tasks + Threading
• Great for scenarios where parallelisation is needed - but multiprocessing is too complex / not suitable

Precautions:

• Always benchmark using your own workload before making a switch.
• Watch out for library compatibility issues/bugs. Proper testing is needed before production adoption with your workloads.

Conclusion

• Free-threaded Python in 3.14 shows significant performance improvements for CPU-bound multithreaded workloads, while maintaining comparable performance for I/O-bound tasks. A new concurrency mode is unlocked for CPU bound programs without overhead of inter-process communication

• Single-threaded performance sees a slight decrease in performance. The default interpreter works best for single-threaded programs. The performance penalty will be reduced further in future releases.

• Performance is comparable for I/O-bound workloads in both modes.

PS: I published my book “Deep Dive Python” last month. Get the book to learn Python concepts straight from real open-source code.

I started using PyCharm IDE back in 2019 for my work, and I’ve been hooked on it ever since. Since then, I’ve explored other AI-powered IDEs like Cursor and Windsurf, actively experimenting with each new release to see what they offer.

At PyCon US this year, I got to meet the JetBrains PyCharm team in person. They gave me a quick demo of Junie, and I knew immediately I wanted to give it a shot.

Below, I’ll share the detailed steps I took while developing Pyzlides, along with a few tips on making the most of AI-assisted coding.

Initial Setup

For AI coding tools, context is the most important. I always spend lot of time to create the initial prompt which will be given to the agent. I try to make it as detailed as possible with examples and intended usage. Initially, I bootstrapped the environment using uv and installed the dependencies in the environment. After a bit of brainstorming and help from ChatGPT to decide the project dependencies, I decided to use the below dependencies:

• ReportLab: For generating the PDF slides.
• Pillow: For handling images in slides.
• PyYAML: For parsing the configuration file.

I installed these libraries in the environment and this was my major point of contribution. After this point, the entire building was done by Junie and I was only involved with writing prompts.

Getting started with Junie

I used the Junie premium subscription for this project. Below is a screenshot of Junie’s chat interface:

Junie provides a simple, familiar chat interface to interact with. From the interface, we have two conversation modes:

1. Code mode: Junie directly generates or edits code.

2. Ask mode: Useful for discussing the codebase, understanding unfamiliar code structure, or clarifying doubts without making direct code changes.

Additionally, using the + button, we can attach relevant files to give Junie more context about the project.

Junie includes a Brave Mode toggle. When enabled, Junie executes terminal commands (such as creating directories, running the project, etc.) without asking for user confirmation. I recommend using Brave Mode only with simpler projects where you fully understand and trust the codebase. Otherwise, it poses a risk of unintended actions or prompt injection.

Recent Junie versions introduced another toggle: Think More, prompting the AI model to reflect more carefully during code generation, improving the overall quality of its responses.

Setting context

As we’ve previously discussed, AI coding assistants perform best when given ample context. Junie provides helpful ways to ensure it understands our project’s codebase clearly:

• .aiignore: From the + menu, we can generate an AI-ignore file, similar to .gitignore. Files added here won’t be modified by Junie.

• Guidelines: Also available through the + button and example prompts are project-specific instructions. Junie uses these as guidelines to stay aligned with project conventions. I will share the guidelines I created later in this post.

Junie Settings

Before we dive deeper into Pyzlides, let’s quickly look at Junie’s settings:

• Models:
  I chose Claude Sonnet 4 as my model, as it’s currently one of the best for coding tasks.

• Connect MCP Servers:
  MCP servers allow Junie to interact with external resources. They can be configured globally in settings or specifically for a project by adding them to the /.junie/mcp directory in the project’s root folder.
  For instance, if we integrate Stripe, connecting the Stripe MCP server ensures Junie understands and accurately utilizes Stripe’s latest API operations, significantly improving the quality of generated code.

• Action Allowlist:
  Here, we can specify terminal commands Junie may run directly without asking for our confirmation.

Init Pyzlides

To help Junie understand the project clearly, I prepared an initial prompt using ChatGPT. The complete prompt is available in the Pyzlides repo here.

Here’s a brief version of the initial prompt:

Pyzlides: Python Presentation Creator

Overview

Pyzlides is a Python library that lets us create PDF presentation slides directly from Python code. Slides are generated in standard presentation resolution, and elements like headers, body text, images, and layouts can be defined using simple Python classes. Pyzlides supports custom themes, layouts, and ordering slides through a configuration file.

Features

1. Declarative Slide Creation: Define slides using Python classes like Head1, BodyText, Img, etc.
2. Custom Layouts: Use custom layouts like Center or Bottom for positioning elements.
3. Configurable Themes: Define themes such as colors, fonts, and backgrounds via a config file.
4. Slide Ordering: Specify slide sequence in the configuration file.
5. PDF Output: Generate presentations as PDF files.
6. Extendable: Easily extend slide functionality by creating new element types or layouts.

---

Dependencies

• ReportLab: PDF slide generation.
• Pillow: Image handling.
• PyYAML: Config file parsing.

Core Classes

1. Base Slide Element Classes

   • Head1(text: str): Large header text.

   • BodyText(text: str): Paragraph-style text.

   • Img(path: str, caption: str): Image with optional caption.

   • Bold(text: str): Bold formatting.

   • Center(element): Center-align element.

   • Bottom(element): Bottom-align element.

To begin, I clicked on the example prompt button in Junie to generate project instructions. Junie created detailed guidelines automatically and stored them in the .junie directory. This file includes key features, project structure, and expectations. You can view it here.

Start Coding

One aspect I liked about Junie was its ability to first generate a clear implementation plan before writing any code. This made complex tasks manageable and easy to review.

Once Junie starts, it updates progress as each step completes. Below you can see Junie’s initial implementation plan for Pyzlides.

Below is the expanded interface when the task was completed by Junie. The pane on the right shows the steps taken for the implementation.

After finishing a task, Junie suggests helpful follow-up actions like adding files to Git or reverting unwanted changes.

The initial implementation by Junie was already impressive. There were minor alignment issues, so I asked Junie to fix these in the next prompt:

the pages are made in portrait mode - change it to landscape mode.

also, bolding does not seem to work.

increase the text size - it is a presentation, not just a simple pdf file.

also make necessary changes so that this can be published as a library. people can install this library and then add config and slide files and use the lib to generate pdf. sample slide files and config can be moved to examples folder if needed. keep the library source directory clean

Once fixed, I asked Junie to add more advanced features and create example slides to showcase Pyzlides capabilities clearly:

create multiple elements to support:

title slide - just a title in the centre - horizontally and vertically

image backgrounds - support images as slide backgrounds.

fix text getting cut out to the next sections.

add H1 to h3 tags - different type of sections.

in the example presentation, add slides to illustrate various types - title only, title with text, title with image, title with image and text, dark mode presentation, white mode presentation, custom background image presentation.

make folders inside example folder for each example. add more elements to the definition as needed - to support a fully featured slides functionality.

also, inline text colours can be specified to text elements.

After this, I asked Junie to add features like code snippets and grid layouts. After these enhancements, the library was nearly complete.

Finishing Up

There were a few remaining issues, primarily with text overflowing slides. I prompted Junie to handle these scenarios clearly:

1. detect when texts flow out of the view - and then fix it to go to next line. regenerate all examples after this change. also test this with long headings, text, etc. there should be a bit of default margins on top botoom left and right.

2. also, add a code component - that accept a code in markdown block and provide syntax highlighted output in text. you can use the already installed pygments library for the same.

Next, I asked Junie to address a few bugs from previous implementations:

• syntax highlight is not working.
• when there is too much text, split it into multiple slides automatically. currently the contents are being cut out - for example in the text and code example.
• also the newline feature is not working as expected. since title texts are starting from centre of the page, some text is still getting cut out horizontally - relook the calculation part

Throughout this project, I enabled Brave Mode. This allowed Junie to automatically run example code, generate PDFs, and proactively identify and fix issues.

To make Pyzlides easy for new users, I also had Junie add a README to guide them through using the library. At this stage, I noticed an unused layout section in the config files. I asked Junie to remove this:

i can see that the layouts section in config is unused. remove this from example config files and Config class defined in core

Junie’s response was prompt and accurate:

The layouts section was removed from the Config class in core.py and from all example config files, confirming it was unused. Functionality was verified by successfully generating presentations from both the basic-presentation and grid-layout examples without any errors.

Summary

The entire Pyzlides project—including fixes and improvements—was completed with fewer than 10 prompts. Below is the final file structure:

project_root/
├── .gitignore
├── pyproject.toml
├── pyzlides.py
├── readme.md
├── uv.lock
│
├── .junie/
│   └── guidelines.md
│
├── pyzlides/
│   ├── __init__.py
│   ├── __main__.py
│   ├── cli.py
│   └── core.py   # Core library functionality (1010 lines)
│
└── examples/
    ├── README.md
    │
    ├── images/
    │   ├── bg_black1.jpg
    │   ...
    │
    ├── basic-presentation/
    │   ...

These are the core files of the library:

• pyzlides.py: Main command-line entry point for generating presentations.

• pyzlides/

  • cli.py: Handles command-line interface functionality.
  • core.py: Contains the core implementation and logic of Pyzlides. Includes the component definitions, parsing logic and pdf generation code.

My conclusion: I’m impressed with Junie - especially loved the planning mode. Few would guess an AI coding agent could accomplish such a detailed task within minutes.

I haven’t fully refined or battle-tested Pyzlides yet, but it’s already an excellent starting point. You can review the generated PDFs in the examples folder, and see for yourself :)

Notes for AI-Assisted Coding

• Use Brave Mode cautiously, especially in complex or sensitive projects.
• Provide clear, detailed context for the best results.
• Always verify the plan generated by the agent and course correct whenever needed.
• Consider tools like GitIngest to enhance context. Also, connect MCP servers when available for deeper integrations.

Links

• Pyzlides GitHub project: GitHub - adarshdigievo/pyzlides

• Junie Docs: Junie, the AI coding agent by JetBrains

• Junie Playbook: PyCharm Junie Playbook - JetBrains Guide

• Download Pycharm : PyCharm: The only Python IDE you need

I’d like to thank the JetBrains team for collaborating on this project. I look forward to using Junie in future experiments.

At this PyCon, I decided to try something different. I skipped most of the talks and focused on the hallway track - just walking around, meeting people, and asking questions.

I was also presenting a poster at PyCon - on using Python to improve at chess: presentation materials here.

Before the conference, I had prepared a few questions I wanted to ask the Python people:

• What do you think is the future of programming?

• Will AI take over programming jobs in the next 2–3 years?

• What AI tools do you use in your daily work?

• What advice do you have for Python developers?

Most of the people I talked to are familiar names in the Python community. I’ve read their blog posts, watched their talks, or used their open-source tools. There were many more people I wish I had a chance to talk to, but I’m glad I got to speak with the ones I did.

One thing surprised me - very few people said AI will replace developers. I expected more.

Later I realized why. Most of the folks I spoke to are working on deep, core programming problems - like optimizing CPython, or building dev tools. These are areas where current AI tools aren’t that useful. LLMs haven’t seen much of that code. So maybe it’s not that surprising after all.

Here’s what they said:

Guido van Rossum — Dutch programmer

Guido said AI won’t replace programmers.

He’s not a fan of “vibe coding” (letting AI write code you don’t understand). He is not impressed with the quality of the code generated by current tools. But he does use GitHub Copilot (for autocomplete) and he expects AI assistants to get much better in the future - something like a supercharged Copilot.

Anthony Shaw — PSF Fellow, Author of CPython Internals

Anthony said AI is like the new Stack Overflow.

Earlier, people copied code from blogs or seniors. Now it’s from ChatGPT.
AI is useful for boilerplate or things you already know. He uses Copilot Chat for regular stuff and agent mode when he’s working on something unfamiliar (like frontend work).

His advice: Don’t blindly rely on AI. If you don’t keep learning, the gap between juniors and seniors will grow fast.

Anthony's Website · My favorite project – VSCode Pets

Anthony Sottile - Maintainer of pre-commit, flake8, and pytest

He doesn’t use AI tools.

He said his problems are too complex for AI. It’s easier for him to just write the code than explain it to a chatbot.

He does agree that AI is good for boring or repetitive code.

Anthony's GitHub · YouTube – Anthony Writes Code

Reuven M. Lerner - Python Author & Trainer

Reuven said companies should not replace junior devs.

He said juniors are like seeds. If companies don’t invest in them now, they’ll struggle later when they only have senior devs and AI bots.

He uses AI mostly for writing—like reviewing posts or generating images. He rarely uses it for coding.

Reuven's Website – lerner.co.il · Great courses and newsletters for Python devs

Trey Hunner — Python Morsels

Trey said AI won’t replace developers.

But it might change how we work.
Developers might end up reading and reviewing more code instead of writing everything from scratch.

He doesn’t use Copilot or coding tools. He copies from ChatGPT when needed and uses a tool called Typingmind to talk to AI.

Trey's Website · Python Morsels

It is also worth a mention that Trey taught us the Cabo card game during the evenings after PyCon.

Samuel Colvin — Founder of Pydantic

Samuel had a different view.
He said AI will replace some programmers.

Companies will use AI to do more with fewer people.
He sees AI as a way to speed things up and make work more efficient.

He uses tools like trae, cline, and others to boost his workflow.

Samuel's Website · GitHub

Simon Willison — Django co-creator, Creator of Datasette

Simon said AI won’t replace developers. But it will create new types of work and improve productivity.

He uses Claude, ChatGPT, and his own LLM CLI tool.
For personal scripts, he uses AI to get things done quickly, and reviews the code properly when it matters.

Simon's Website · GitHub

Mariatta Wijaya — Python Core Developer

Mariatta said AI is more helpful for senior developers.

Seniors can use AI to write code quickly and then fix what’s wrong.
For juniors, it depends. If the task is small, AI can help. But juniors still need to understand what the code does.

She uses Copilot in her daily work.

She also recommends this talk from PyTexas:
Heather’s Talk – How to Think Like a Senior Engineer

Mariatta's Website · GitHub

Final Thoughts

Most Python people agree: AI won’t replace developers anytime soon.

My thoughts: it will definitely change how we work.

If you’re using AI without understanding, you’ll likely fall behind.
If you keep learning and adapt, you’ll do fine.

My personal favorite moment

One of the highlights of PyCon US for me was meeting Paul Everitt.

He gave me a sneak peek of the Python documentary trailer before it was released on YouTube. And even better—he handed me a chocolate Cornetto from the PyCharm booth while telling stories. Instant core memory.

Paul is one of the most enthusiastic and genuine people I met at the conference. A natural storyteller.

A few things I didn’t know about him until then:

• He managed navy.mil while serving in the U.S. military—one of the earliest websites on the internet.

• He was a keynote speaker at the first-ever Python conference in 1994.

• He helped incorporate the Python Software Foundation.

• He was Guido’s manager during his Plone days.

The kind of person you meet once and never forget.

That’s all from my hallway track this year.

“Python Thread Safety: Using a Lock and Other Techniques”: https://realpython.com/python-thread-lock/

Why thread safety?

Below is a classic example of creating a singleton class is Python:

class SingletonClass(object):
  def __new__(cls):
    if not hasattr(cls, 'instance'):
      cls.instance = super(SingletonClass, cls).__new__(cls)
    return cls.instance

The SingletonClass is supposed to have only 1 object, but the code can lead to the creation of more than one instances when executed in a multithreaded environment.

Read the tutorial to spot such race conditions and learn to fix them using Python’s synchronization primitives.

Read now at Real Python Website: https://realpython.com/python-thread-lock/

I share interesting Python snippets from open-source projects illustrating Python language features in my newsletter, “Python in the Wild”.

Subscribe to the newsletter on Substack or Linkedin to receive new Pythonic posts to your email 💌🚀.

In this article, we will:

1. Explore Thread local storage: Python threading module’s solution for thread-specific/thread-private values.
2. See its real-world example from Open-Source ( usage in the Peewee ORM library).
3. Look at the CPython source code to see how thread local storage is implemented under the hood.

Background

Threads share the memory space of their parent process. This will allow us to seamlessly access and share variables, data structures, etc., across threads. But this comes with its own challenges. There may be scenarios where we need to isolate variables and might need to store data specific to each thread. Thread local storage can be leveraged in this case.

We can use local() found in the threading module to define thread-local variables.

import threading
import time

# Create a thread-local storage object (1)
thread_local = threading.local()


def init_data(number):
    thread_local.number = number * 100


def show_data():
    print(f"Thread {threading.current_thread().name} has number {thread_local.number}")


def worker(number):
    init_data(number)

    for _ in range(3):
        time.sleep(1)
        show_data()


thread1 = threading.Thread(target=worker, name="A", kwargs={"number": 1})
thread2 = threading.Thread(target=worker, name="B", kwargs={"number": 2})

thread1.start()
thread2.start()

thread1.join()
thread2.join()

The program output will be:

Thread A has number 100
Thread B has number 200
Thread A has number 100
Thread B has number 200
Thread A has number 100
Thread B has number 200

As seen in comment #(1), a thread-local storage object was created and assigned to the variable thread_local. Arbitrary attributes can be assigned to this variable, which are specific to the thread that performs the assignment and is isolated from others.

In the example, each thread stores its own number attribute to the thread_local object and accesses the thread-specific value during their concurrent execution.

Usage in the Wild

Peewee, a Python ORM, utilizes thread-local data in its ThreadSafeDatabaseMetadata to support dynamic database switches at runtime in multithreaded applications.

The source code can be found here.

# File: peewee/playhouse/shortcuts.py

class ThreadSafeDatabaseMetadata(Metadata):
    """
    Metadata class to allow swapping database at run-time in a multi-threaded
    application. To use:

    class Base(Model):
        class Meta:
            model_metadata_class = ThreadSafeDatabaseMetadata
    """

    def __init__(self, *args, **kwargs):
        # The database attribute is stored in a thread-local.
        self._database = None
        self._local = threading.local()
        super(ThreadSafeDatabaseMetadata, self).__init__(*args, **kwargs)

    def _get_db(self):
        return getattr(self._local, "database", self._database)

    def _set_db(self, db):
        if self._database is None:
            self._database = db
        self._local.database = db

    database = property(_get_db, _set_db)

In multithreaded applications using peewee ORM, database switching at runtime without using ThreadSafeDatabaseMetada can lead to errors. If multiple threads work in parallel and one thread changes the connection parameters, this can lead to errors such as writing to a wrong DB, inconsistent writes (in case of non-atomic DB operations), etc.

ThreadSafeDatabaseMetada solves this by keeping the database attributes in a thread-local object (self._local). In this way, dynamic changes to the database will only affect the thread that made the change. Other threads will keep working with their existing databases.

Should I use it?

Thread-local storage should be used when:

You are writing a multi-threaded application (obviously) and:

• If some variables are used by the current thread only and are not relevant to the main thread/other threads.
• You find out that changes made by one thread can lead to unintended side effects in other concurrent threads.

Generally, if you are working with multiple threads and there is shared mutable data:

You should check if sharing these data across threads is actually needed.

1. If sharing can be avoided, use thread-local storage to make the data specific to each thread.

2. Otherwise, implement locks or other synchronization primitives to enforce thread safety.

Note: Context variables from the contextvars standard library module can be used as an alternative to threading.local(). They work with multithreading as well as asyncio to store context-specific information. In the case of asyncio programs, context variables allow each coroutine task to have its own set of variables isolated from other (asyncio) tasks.

Behind the scenes

The Python implementation of threading.local can be found in the /Lib/_threading_local.py path in CPython source code.

The _localimpl class is used to store thread-local values.

class _localimpl:
    """A class managing thread-local dicts"""
    __slots__ = 'key', 'dicts', 'localargs', 'locallock', '__weakref__'

    def __init__(self):
        # The key used in the Thread objects' attribute dicts.
        # We keep it a string for speed but make it unlikely to clash with
        # a "real" attribute.
        self.key = '_threading_local._localimpl.' + str(id(self))
        # { id(Thread) -> (ref(Thread), thread-local dict) }
        self.dicts = {}

The dicts attribute maps the id of a thread to a tuple. The tuple is two-membered, containing a reference to the thread and the actual dictionary storing thread local values (# { id(Thread) -> (ref(Thread), thread-local dict) }).

Looking at a few other methods of the class:

class _localimpl:
    """A class managing thread-local dicts"""
    def get_dict(self):
        """Return the dict for the current thread. Raises KeyError if none
        defined."""
        thread = current_thread()
        return self.dicts[id(thread)][1] # (1) returning the local dict of current thread

    def create_dict(self):
        """Create a new dict for the current thread, and return it."""
        localdict = {}
        key = self.key
        thread = current_thread()
        idt = id(thread)
        ...
        wrthread = ref(thread, thread_deleted)
        thread.__dict__[key] = wrlocal
        self.dicts[idt] = wrthread, localdict  # (2) Populating the `dicts` with a new thread
        return localdict

The comments #(1) & #(2) illustrate operations on the dicts attribute discussed previously.

(1): Returns the local data dictionary corresponding to the current thread accessing it.

(2): This part initializes the local dict for a new thread.

Then, we have the actual local callable, which we call as threading.local() to initialize a thread-local object.

@contextmanager
def _patch(self):
    impl = object.__getattribute__(self, '_local__impl')
    try:
        dct = impl.get_dict()  # (3) this will return local dict of current thread -/
                               # see its definition in the above snippet
    except KeyError:
        ...
        # calls _localimpl`s create_dict to create & init a new dict
    with impl.locallock:
        object.__setattr__(self, '__dict__', dct) # (4) <The magic> Temporarily replaces -/
        # the instance's __dict__ attribute with the thread-specific dictionary.
        
        yield


class local:
    __slots__ = '_local__impl', '__dict__'

    def __new__(cls, /, *args, **kw):
        ...

        impl = _localimpl()  # (1) - wraps the _localimpl object in an attribute
        impl.localargs = (args, kw)
        impl.locallock = RLock()  # (2) - Lock for thread safety
        object.__setattr__(self, '_local__impl', impl)
        # We need to create the thread dict in anticipation of
        # __init__ being called, to make sure we don't call it
        # again ourselves.
        impl.create_dict()
        return self

    def __getattribute__(self, name):
        with _patch(self):
            return object.__getattribute__(self, name)

    def __setattr__(self, name, value):
        if name == '__dict__':
            raise AttributeError(
                "%r object attribute '__dict__' is read-only"
                % self.__class__.__name__)
        with _patch(self):
            return object.__setattr__(self, name, value)

    ...

Let’s look at various parts of the code one by one.

# (1): An object of _localimpl class is created and later stored in _local__impl attribute

# (2): RLock() is used on dict operations - Concurrent writes by multiple threads may cause ‘lost writes’ since dict is not thread-safe.

# (3): In the try/except block, the current thread’s local data is fetched and assigned to the variable named dct.

# (4): The magic happens here. Here is a quick refresher on class attributes before we delve further:

A class has a namespace implemented by a dictionary object. Class attribute references are translated to lookups in this dictionary, e.g., C.x is translated to C.dict[“x”] (although there are a number of hooks which allow for other means of locating attributes).

The _patch function we are currently in is a context manager. Using the line marked as # (4), it patches the namespace dictionary of the local class with the dot variable (which stores the current-thread specific data).

Since the __getattribute__ and __setattr__ dunders of the class local use the _patch context manager, attribute access performed inside the context will use the thread-local dictionary (dct) replacing the class’s actual namespace dictionary.

Note: The Lib/_threading_local.py starts with the below note:

Note that this module provides a Python version of the threading.local class. Depending on the version of Python you’re using, there may be a faster one available. You should always import the local class from threading.

The code we looked at might not be the one running in our Python installations. I think newer Pythons versions are using C implementations of the thread local functionality for efficiency.

I share interesting Python snippets 🐍 like this from open-source projects illustrating Python language features in my newsletter, “Python in the Wild”.

Subscribe to the newsletter on Substack or Linkedin to receive new Pythonic posts to your email 💌🚀.

Both of these use Timsort algorithm under the hood, to perform the sorting operation.

In this article, we will explore the evolution of sorting in Python from Python 2 to 3 and will look into the cmp_to_key function from functools.

Background

Below are the type annotations for sort and sorted, from the Python typeshed library.

@overload
def sorted(
    __iterable: Iterable[SupportsRichComparisonT],
    *,
    key: None = None,
    reverse: bool = False
) -> list[SupportsRichComparisonT]:
    ...


@overload
def sorted(
    __iterable: Iterable[_T],
    *,
    key: Callable[[_T], SupportsRichComparison],
    reverse: bool = False
) -> list[_T]:
    ...


class list(MutableSequence[_T]):
    @overload
    def sort(
        self: list[SupportsRichComparisonT], *, key: None = None, reverse: bool = False
    ) -> None:
        ...

    @overload
    def sort(
        self, *, key: Callable[[_T], SupportsRichComparison], reverse: bool = False
    ) -> None:
        ...

Both sort and sorted accept an optional keyword-only argument key. It should be a callable and if supplied, will be used to calculate the comparison key.

From Python docs:

key specifies a function of one argument that is used to extract a comparison key from each list element (for example, key=str.lower). The key corresponding to each item in the list is calculated once and then used for the entire sorting process.

Now lets take a look at the Python 2 docs for the sorted built-in:

 sorted(iterable[, cmp[, key[, reverse]]])

    Return a new sorted list from the items in iterable.

    The optional arguments cmp, key, and reverse have the same meaning as those for the list.sort() method (described in section Mutable Sequence Types).

    cmp specifies a custom comparison function of two arguments (iterable elements), 
    which should return a negative, zero or positive number depending on whether the first argument is considered smaller than, equal to, or larger than the second argument.
    
    ...

There is an additional argument cmp compared to what we have seen in Python 3.

The docs also mentions:

In general, the key and reverse conversion processes are much faster than specifying an equivalent cmp function. This is because cmp is called multiple times for each list element while key and reverse touch each element only once. Use functools.cmp_to_key() to convert an old-style cmp function to a key function.

The cmp argument was removed in Python 3 and the cmp_to_key helper to convert comparison functions to new-style key functions has been added to the functools module.

Usage in the Wild

If we have a comparison function (which takes two arguments and returns their comparison result), this can be converted to a sort key by using cmp_to_key function, provided in the functools module.

The below example from the pretty_midi, a MIDI processing library illustrates the use of cmp_to_key.

Link to source code

class PrettyMIDI(object):
    """A container for MIDI data in an easily-manipulable format."""

    ...

    def write(self, filename):
        """Write the MIDI data out to a .mid file."""

        def event_compare(event1, event2): # <--- (1)
            """Compares two events for sorting.

            Events are sorted by tick time ascending. Events with the same tick
            time ares sorted by event type. Some events are sorted by
            additional values. For example, Note On events are sorted by pitch
            then velocity, ensuring that a Note Off (Note On with velocity 0)
            will never follow a Note On with the same pitch.

            Parameters
            ----------
            event1, event2 : mido.Message
               Two events to be compared.
            """

            secondary_sort = {
                "set_tempo": lambda e: (1 * 256 * 256),
                "time_signature": lambda e: (2 * 256 * 256),
                "key_signature": lambda e: (3 * 256 * 256),
                "lyrics": lambda e: (4 * 256 * 256),
                "text_events": lambda e: (5 * 256 * 256),
                "program_change": lambda e: (6 * 256 * 256),
                "pitchwheel": lambda e: ((7 * 256 * 256) + e.pitch),
                "control_change": lambda e: (
                    (8 * 256 * 256) + (e.control * 256) + e.value
                ),
                "note_off": lambda e: ((9 * 256 * 256) + (e.note * 256)),
                "note_on": lambda e: ((10 * 256 * 256) + (e.note * 256) + e.velocity),
                "end_of_track": lambda e: (11 * 256 * 256),
            }
            # If the events have the same tick, and both events have types
            # which appear in the secondary_sort dictionary, use the dictionary
            # to determine their ordering.
            if (
                event1.time == event2.time
                and event1.type in secondary_sort
                and event2.type in secondary_sort
            ):
                return (secondary_sort[event1.type](event1) - 
                        secondary_sort[event2.type](event2))

            # Otherwise, just return the difference of their ticks.
            return event1.time - event2.time

        # Create track 0 with timing information
        timing_track = mido.MidiTrack()

        ...  # Code processing the timing_track

        # Sort the (absolute-tick-timed) events.
        timing_track.sort(key=functools.cmp_to_key(event_compare)) # <--- (2)

        ...

The event_compare function (pointed in comment (1)) provides a specific comparison logic for MIDI events, ensuring they are sorted correctly by their tick time and type, along with additional criteria for certain event types. Then, functools.cmp_to_key is used to convert this comparison function into a key function ( (pointed in comment (2))).

Should You Use It

As cmp_to_key was aimed to help transition from Python 2 style cmp functions to key functions, its usage is not very common in newer projects. However, it can come in handy when we already have a comparison function defined in our code, to compare two objects and need to sort an iterable of these objects. In such a case, cmp_to_key will allow us to reuse our comparison logic (defined in our comparison function) without defining a new key function.

Behind the Scenes

Below is the source code of cmp_to_key from the functools module

def cmp_to_key(mycmp):
    """Convert a cmp= function into a key= function"""
    class K(object):
        __slots__ = ['obj']
        def __init__(self, obj):
            self.obj = obj
        def __lt__(self, other):
            return mycmp(self.obj, other.obj) < 0
        def __gt__(self, other):
            return mycmp(self.obj, other.obj) > 0
        def __eq__(self, other):
            return mycmp(self.obj, other.obj) == 0
        def __le__(self, other):
            return mycmp(self.obj, other.obj) <= 0
        def __ge__(self, other):
            return mycmp(self.obj, other.obj) >= 0
        __hash__ = None
    return K

cmp_to_key intelligently maps the original comparison function to a key function suitable for sorting, by implementing all the comparsion dunders.

It returns a new callable (class K) with all the comparison dunders which will be called during the sorting process to calculate the sort key. Since the sorting process compare each of the calculated keys, the comparison dunders will get invoked, and they will return based on the result from the original comparison function.

For example, during sorting, when K object < another K object needs to be checked, Python will actually call the lt dunder method on K. This method simply calls the original cmp function to compare the obj attributes of the two K instances.

I share interesting Python snippets 🐍 like this from open-source projects illustrating Python language features in my newsletter, “Python in the Wild”.

Subscribe to the newsletter on Substack or Linkedin to receive new Pythonic posts to your email 💌🚀.

host = "localhost:8000"
print(host.split(":"))  # Output: ['localhost', '8000']

host = "localhost"
print(host.split(":"))  # Output:['localhost']

host = ""
print(host.split(":"))  # Output: ['']

partition is helpful when we need to get precisely two parts from a string using a separator.

server_name = "localhost:5000"
print(server_name.partition(":"))  # Output: ('localhost', ':', '5000')

server_name = "localhost"
print(server_name.partition(":"))  # Output: ('localhost', '', '')

server_name = ""
print(server_name.partition(":"))  # Output: ('', '', '')

Using partition instead of split saves us from additional checks on the result’s length when exactly two parts are required.

Flask uses the partition method to determine the host and port to use when running the Flask app. You can see the source code here.

A simplified version of the code is added as below:

def run(
    self,
    host: str | None = None,
    port: int | None = None,
    debug: bool | None = None,
    load_dotenv: bool = True,
    **options: t.Any,
) -> None:
    ...

    server_name = self.config.get("SERVER_NAME")
    sn_host = sn_port = None

    if server_name:
        sn_host, _, sn_port = server_name.partition(":")
        # The `partition` method will always return a three-membered tuple

    if not host:
        if sn_host:
            host = sn_host
        else:
            host = "127.0.0.1"

    if port or port == 0:
        port = int(port)
    elif sn_port:
        port = int(sn_port)
    else:
        port = 5000

In the example above, the server_name string is partitioned into host and port values.

I share interesting Python snippets 🐍 like this from open-source projects illustrating Python language features. Subscribe to my newsletter to receive new posts to your email 💌🚀.

TIL - “Today I learned” series of Posts are short notes of new things I encounter with Python.

While browsing the source of SQLAlchemy, I came across a custom list class

The custom class is created by inheriting from List from typing, rather than the builtin list or the UserList class provided by collections

It works.

from typing import List


class CustomList(List):
    pass


print(CustomList.__mro__)

# (<class '__main__.CustomList'>, <class 'list'>, <class 'typing.Generic'>, <class 'object'>)

The builtin list can be seen as a parent class of CustomList.

Normal list operations also work as expected.

from typing import List


class CustomList(List):
    pass


c = CustomList()
c.append("foo")
print(c)

# ['foo']

Now, what if we try to instantiate List directly?

list_obj1 = list()  # Works
list_obj2 = List()  # TypeError: Type List cannot be instantiated; use list() instead

Let’s see why this happens.

The List class is defined in typing as an alias of the list class.

# CPython - typing.py (https://github.com/python/cpython/blob/4afa7be32da32fac2a2bcde4b881db174e81240c/Lib/typing.py#L2609)

List = _alias(list, 1, inst=False, name='List')

This _alias creates List as a child instance of _BaseGenericAlias. See that the inst attribute is set to False above.

Now, let’s take a look at the _BaseGenericAlias class:

# CPython - typing.py (https://github.com/python/cpython/blob/4afa7be32da32fac2a2bcde4b881db174e81240c/Lib/typing.py#L1107)

class _BaseGenericAlias(_Final, _root=True):
    """The central part of internal API.

    This represents a generic version of type 'origin' with type arguments 'params'.
    There are two kind of these aliases: user defined and special. The special ones
    are wrappers around builtin collections and ABCs in collections.abc. These must
    have 'name' always set. If 'inst' is False, then the alias can't be instantiated,
    this is used by e.g. typing.List and typing.Dict.
    """
    def __init__(self, origin, *, inst=True, name=None):
        self._inst = inst
        self._name = name
        self.__origin__ = origin
        self.__slots__ = None  # This is not documented.

    def __call__(self, *args, **kwargs):
        if not self._inst:
            raise TypeError(f"Type {self._name} cannot be instantiated; "
                            f"use {self.__origin__.__name__}() instead")
        ...

The call dunder is overridden to raise a type error if the _inst attribute of the alias is False. The __call__ function gets invoked when we call the children of the _BaseGenericAlias class, in our case, the List.

As you can see in the docstring of _BaseGenericAlias:

If ‘inst’ is False, then the alias can’t be instantiated, this is used by e.g. typing.List and typing.Dict.

Summarizing,

• typing.List behaves like a list when used for inheritance
• But can’t be used directly, just like the list for instantiation.

This is my account of meeting Guido Van Rossum and his advice on becoming a Python Core Developer.

This interaction is a few months old and happened during Pycascades in Vancouver (18-19 March 2023). Yesterday a random thought about this interaction crossed my mind and I thought I should write a blog post on this.

Pycascades & Python Conferences

I flew from India to Vancouver to Speak at Pycascades to speak on “Metaprogramming in Python using Metaclasses”

I would broadly classify tech conferences into two types: Busy vs. Relaxed conferences.

A typical busy conference will have a large number of attendees, multiple parallel tracks, and a whole lot of things going around at the same time. Relaxed conferences have a maximum of a few hundred attendees, there is only a single track and everything occurs at a relaxed pace.

I would classify Pycascades '23 as a relaxed conference. In a typical busy conference, you will have to line up to meet your favorite tech rockstar. But here that was not the case. This was the first “relaxed” conference I have been to and was a bit surprising.

Meeting Guido

I had my presentation at 3 PM on day 1 of the conference. I mostly spend my day in the speaker-ready room doing some last-minute preparations. After completing my talk I went on to get some snacks during the evening break.

There he was. I did a quick Google image search and to double-check, I even asked a volunteer to confirm if that was Guido himself.

I don’t exactly remember when I heard about Guido for the first time. This might be from my high school where our computer teacher instilled in us a love for programming or in the first semester of my Bachelors where we had an introductory course in Python.

I mustered up enough courage to talk to him and thanks to Pycascades being a relaxed conference, there was no one around waiting to talk to him. He told me that he listened to the last few slides of my talk and wanted to know how I got into metaprogramming.

This was a part of his character. He always asks ‘Why’. Before answering a question, he wanted to know the motive or the origin of the question. I guess this curiosity is one factor that sets apart great developers from good developers.

Here is a brief of the conversation we had on that day(reworded and mostly taken from my memory).

Guido: I missed the initial parts of your talk. I came when you reached slide X(a slide towards the end). How did you get into the idea of metaprogramming?

Me: We have a knowledge-sharing session in our company. I presented an initial version of this talk there. Also, I love teaching others and I explore new topics from Realpython and blogs.

Guido: At your day job, what are you currently working on? Are you using Python?

Me: Yes. I am building a GraphQL API powering the backend of a travel application.

Guido: GraphQL? Is that the Facebook library?

Me: Yes. We are using the Graphene-Python library.

Guido: I see. So there are Python wrappers around it.

Then I asked him a few questions which came to my mind:

Q1: I asked a question regarding alternate Python implementations. Something along the lines of “Do you see Pypy overtaking CPython anytime in the future?”

Guido: PyPy or any other implementations won’t be replacing CPython. Alternate implementations are always trying to catch up with the CPython releases. With the Faster CPython Project, future releases have improved performance reducing the need for alternative implementations, if speed is the concern. Jit compiler

Q2: Will you be coming to India for our Pycon event (Pycon India)

Guido: I most probably won’t be there since I am not able to travel that long. (I don’t exactly remember the wording)

Q3: I work in a young team of Backend devs using Python. Do you have any advice for getting better at Python Programming?

(I only remember a part of this answer)

Guido:

1. Be a part of local Python communities
2. Read more code than you write

Expanding on point 2, reading good code is a great way to learn new patterns and expand our thinking approaches towards problems.

I recently came across “death and gravity” blog with two great resources for the same.

Learn by reading code: Python standard library design decisions explained and Struggling to structure code in larger programs? Great resources a beginner might not find so easily

I stopped when I ran out of questions to ask.

Initially I had told him that I love teaching. He asked me to connect with the RealPython team (who were present at the conference) and collaborate with them on some tutorials.

Then we took a selfie (The cover image of this post). Also, I was surprised no one else was doing the same (relaxed conference!).

I noted down important points of the conversation to share with my teammates, which became useful when writing this.

How to be a CPython Core Developer

On day 2, the final day of the conference, I arrived at the venue (SFU Vancouver campus) with a resolve not to miss any of the talks like I did yesterday owing to my talk preparation.

On the morning break, I saw Guido talking to a volunteer regarding Pycon. I was ready with my new set of questions.

I was looking to advance my Python knowledge and was looking to go deeper into CPython implementation. I had my copy of Cpython Internal Book and have taken a peek into the Python Devguide and wanted to get an idea of the next steps to take from the creator himself.

If you are new to the below terms, please refer to the listed resources

I went ahead and asked Guido, “How can I become a core developer”?

The hall was a bit crowded and noisy at that time. Guido insisted we move to a quieter side and talk. He grabbed a snack, went to a quiet corner overlooking the Vancouver harbor and asked:

Why?

Why do you want to be a core dev? Becoming one is a decision many would regret.

I didn’t quite understand this at that time. It made sense recently when I was reading the BDFL retirement mail thread from the Python mailing list archive at 3 AM. Also see this article from Brett Cannon: The social contract of Open source

Going back to the topic:

I somehow evaded his whys and got the answer from him.

Here are some of the points he mentioned as a part of his answer:

• Be active in discussion forums.

Python mailing lists

Python Discourse

• Look for issues or new features to be added to the standard library

Since I was already using Python, he asked me to identify bugs/improvements and try to work on those.

Fun fact: Currently, more than 63% of CPython source is written in Python. Knowledge of C is not required to contribute to CPython.

• There are little bugs everywhere He said that there is a lack of expert contributors. Even core developers won’t know all parts of the code and there are parts everyone is afraid of.

I was happy and relieved hearing this (There are little bugs everywhere, even in CPython). Maybe because I can justify the bugs I create using this statement.

• Never give up

Once started, we should go on taking up issues and continue contributing.

• Multiple Ways to Contribute:

Along with contributing code, we can also take part in discussions, writing docs, or reviewing PRs.

Then came the surprising move. He wanted to check how ChatGPT would answer this (no I am not making this up). He pulled in his laptop, logged in to ChatGPT, and asked the same question.

I do not remember exactly the answer it gave, I remember it had a link to the dev guide and some other instructions. He found out that there was a mistake in the answer and reported that and ChatGPT responded with a revised(and correct) answer.

Lessons Learnt

• Go to conferences:

Pycascades was a very enriching experience for me. I would recommend everyone to go to conferences. I connected & exchanged ideas with many great folks (VSCode-Python Team, RealPython Team, Mariatta, and so many more awesome people)

• Be nice:

There were common traits I observed whenever I spoke with most of the experts during relaxed as well as busy conferences.

They all are humble, welcoming, willing to help and genuinely care about others.

Looking back on this journey, everything was worth it - the 16-hour long flight, taking a week-long leave from work (and handling the production credentials to juniors)

Thanks to Pysacsades

It isn’t fair to end this post without thanking Pycascades Team for giving me a speaker invite and covering a part of the travel expenses.

Special thanks to Eliza, Ben, Jolene, and the entire Pycascades team for running this awesome conference.

Before getting started with Thread safety, let’s look into race conditions.

Race Conditions

From Wikipedia:

A race condition or race hazard is the condition of an electronics, software, or other system where the system’s substantive behavior is dependent on the sequence or timing of other uncontrollable events. It becomes a bug when one or more of the possible behaviors is undesirable.

Race conditions can occur especially in logic circuits, multithreaded, or distributed software programs.

For an example of a race condition on a real-world web app, see this disclosed security bug: https://hackerone.com/reports/759247

Here the hacker could redeem a gift card (intended for single use) multiple times.

A developer may think of implementing a gift card redemption flow like below:

1. Get a gift card code from the user
2. Validate the code: Check if it exists in DB, is not expired, and is not used.
3. If validation succeeds, add the gift card value to the user’s wallet and mark the gift card as used in the DB.

Think of the case where a user sends multiple requests at the same time. All parallel requests arriving before the execution of step 3, could succeed and can result in adding the gift card amount to the user’s wallet once per request. This might have happened in the case of the above bug.

Threading in Python

Threading is a method to implement concurrency in Python programs. Due to the presence of CPython’s GIL, threading is helpful in the case of I/O Bound programs.

threading module and concurrent.future’s ThreadPoolExecutor helper is commonly used to implement multithreading in Python.

Let’s dive into some Python examples after exploring the levels of thread safety.

Levels of thread safety

From Wikipedia, levels of thread safety:

Thread safe: Implementation is guaranteed to be free of race conditions when accessed by multiple threads simultaneously.

Conditionally safe: Different threads can access different objects simultaneously, and access to shared data is protected from race conditions.

Not thread-safe: Data structures should not be accessed simultaneously by different threads.

Race conditions in Python Threaded Programs

Example 1: Incrementing a counter

import concurrent.futures

count = 0


def increase_counter():
    global count
    for i in range(1000):
        count += 1

with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    for _ in range(10_000):
        executor.submit(increase_counter)

print(F"{count=}")

The expected value of the count is 10,000 * 1000.

Running with Python 3.9, the program output varies per execution and gives outputs like below:

1. count=2163149
2. count=5642376

From this, we can conclude that the program is not thread-safe, since it gives unexpected results when run using multiple threads.

In the above examples, there are at most 10 parallel threads working since we have given max_workers=10

On adjusting max_workers value,

When max_workers=3, the output varies like below:

1. count=10000000
2. count=9999000

On reducing max_workers, which is the number of threads running in parallel, there is a visible reduction in errors, and at times, the expected output is also returned.

Consider the above scenario where each thread reads a variable and increments its value with 1. Between reads and writes, context switches to different threads can happen. With multiple threads, the following scenario may be possible:

1. Thread_1 reads the value of count, gets 0
2. At the same time thread 2 reads count, and gets 0
3. Thread_1 sets count=count+1, as 1.
4. Thread_3 reads count, gets 1.
5. Thread_3 sets count=count+1, as 2
6. Thread_2 sets count=count+1, as 1 (since thread_2 holds a stale value of count, i.e., 0)

This explains the counter value being far less than expected.

Example 2: Singleton class in Python

Let’s take an example of a Singleton class created using Python. Singleton pattern is used to make sure that only one instance is created for a class. This can be used in cases where the class initialization performs some steps which should be performed exactly once. Singletons are useful in cases where there are restrictions on certain resource usage such as concurrent connection limits.

Below is a commonly used example for a Singleton class in Python:

class SingletonClass:
    def __new__(cls):
        if not hasattr(cls, 'instance'):
            cls.instance = super(SingletonClass, cls).__new__(cls)
        return cls.instance


obj1 = SingletonClass()
obj2 = SingletonClass()

print(obj1 is obj2) # True
print(id(obj1)) # 2402721138768
print(id(obj2)) # 2402721138768

In the above example, it is possible because two threads can evaluate the if check for instance (if not hasattr(cls, 'instance')) at near similar times and proceed to create two new objects.

Example 3: The print function

from concurrent import futures

def printer():
    print('testing thread safety of print')


with futures.ThreadPoolExecutor(max_workers=10) as executor:
    for _ in range(10):
        executor.submit(printer)

Output can vary from expected:

testing thread safety of print
testing thread safety of print
testing thread safety of printtesting thread safety of print
testing thread safety of print

testing thread safety of printtesting thread safety of print
testing thread safety of print
testing thread safety of print

testing thread safety of print

Before printing ending newlines, context switches happen sometimes printing output from another thread.

Making programs thread-safe

Synchronization primitives

The below synchronization primitives can be used to make a program thread safe.

Lock

A lock can be used for exclusive access to a resource. Once a thread acquires a lock, no other threads can acquire it (and proceed further) unless the lock is released.

RLock

RLock is a re-entrant lock. It allows when a holding thread requests the lock again. The simple lock is not aware of the current locking thread and this can lead to deadlocks.

import threading

num = 0
lock = threading.Lock()

lock.acquire()
num += 1
lock.acquire() # This will block. The program goes into a deadlock after this.
num += 2
lock.release()

# With RLock, that problem doesn't happen.
lock = threading.RLock()

lock.acquire()
num += 3
lock.acquire() # This won't block.
num += 4
lock.release()
lock.release() # We need to call release once for each call to acquire

For brevity, let’s skip the other primitives (Semaphore & Barrier, Events & Conditions).

Lock acquisition and release can be managed using context managers.

lock = threading.Lock()

lock.acquire()
num += 1
lock.release()

is equivalent to:

lock = threading.Lock()

with lock:
    num += 1

Thread safe counter implementation

We can use a lock at places of code where race conditions(simultaneous modifications in this case) should be avoided.

import concurrent.futures
from threading import Lock

count = 0
lock = Lock()


def increase_counter():
    global count
    for i in range(1000):

        lock.acquire()
        count += 1
        lock.release()

        """
        or:
        with lock:
            count += 1
        """


with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    for _ in range(10_000):
        executor.submit(increase_counter)

print(F"{count=}") # count=10000000, for all executions

Thread-safe singleton implementation

import threading


class Singleton:
  _instance = None
  _lock = threading.Lock()

  def __new__(cls):
      if cls._instance is None:
        with cls._lock:
          if cls._instance is None:
              cls._instance = super().__new__(cls)
          
      return cls._instance

After acquiring the lock (using with), we are rechecking the presence of an instance to prevent the below-race condition.

Consider two threads thread1 and thread2.

1. Both execute new dunder of singleton class at the same time.
2. cls_.instance is currently None, so both thread tries to acquire the lock
3. thread1 gets the lock while thread2 is blocked waiting.
4. thread1 creates the singleton instance and completes execution inside the context manager. Once it exits the with block, the lock (cls._lock`) is released.
5. Now, thread2 is unblocked and enters the context manager body by acquiring the lock. Since the None check for cls._instance fails, thread2 leaves execution and the already existing instance is returned.

Thread safe print implementation

Similar to the previous examples, we can use a lock here.

from concurrent import futures
from threading import Lock

lock = Lock()

def printer():
    with lock:
        print('testing thread safety of print')


with futures.ThreadPoolExecutor(max_workers=10) as executor:
    for _ in range(100):
        executor.submit(printer)

Context switching & Thread switch interval

The switch interval in Python specifies how long the Python interpreter will allow a Python thread to run, after which it is forced for a context switch.

Since only one thread runs at a time due to GIL, the interpreter tries to switch threads after this interval.

The default interval is 5 milliseconds.

>>> import sys
>>> sys.getswitchinterval()
0.005

We can set custom thread switch interval using sys.setswitchinterval

If the threads are waiting (e.g.: for IO operations) this switching allows for a parallel execution-like effect. If threads are running any CPU-bound operations, this context switch causes a performance penalty.

Eg: Network IO blocked tasks:

import time

import requests

start = time.perf_counter()
req1 = requests.get("https://reqres.in/api/users?delay=3")
req2 = requests.get("https://reqres.in/api/users?delay=3")
end = time.perf_counter()

print(end-start, " seconds") # 6.9710999 seconds

Using threads:

import threading
import time
import requests


def make_request():
    return requests.get("https://reqres.in/api/users?delay=3")


start = time.perf_counter()

t1 = threading.Thread(target=make_request)
t2 = threading.Thread(target=make_request)

t1.start()
t2.start()

t1.join()
t2.join()

end = time.perf_counter()

print(end-start, " seconds") # 3.494618 seconds

In case of CPU Blocked tasks:

import time

start = time.perf_counter()

s1 = sum(range(100000000))
s2 = sum(range(100000000))

end = time.perf_counter()

print(end-start, " seconds") # 3.8891145 seconds

Using threads:

import threading
import time


def perform_computation(): 
    return sum(range(100000000))


start = time.perf_counter()

t1 = threading.Thread(target=perform_computation)
t2 = threading.Thread(target=perform_computation)

t1.start()
t2.start()

t1.join()
t2.join()

end = time.perf_counter()

print(end-start, " seconds") # 3.9622078999999997 seconds

As expected, the use of threading slowed down this CPU-bound operation.

Revisiting the increment_counter example

On running the non-thread-safe increment_counter() used above using Python 3.11 always gives the expected result and race conditions are not seen.

We can use dis module to inspect the Python bytecode for the increase_counter function

import dis

count = 0


def increase_counter():
    global count
    for i in range(1000):
        count += 1

dis.dis(increase_counter)

Python 3.9 output:

9 0 LOAD_GLOBAL 0 (range)
2 LOAD_CONST 1 (1000)
4 CALL_FUNCTION 1
6 GET_ITER
>> 8 FOR_ITER 12 (to 22)
10 STORE_FAST 0 (i)

10 12 LOAD_GLOBAL 1 (count)
14 LOAD_CONST 2 (1)
16 INPLACE_ADD
18 STORE_GLOBAL 1 (count)
20 JUMP_ABSOLUTE 8
>> 22 LOAD_CONST 0 (None)
24 RETURN_VALUE

Python 3.11 output:

7 0 RESUME 0

9 2 LOAD_GLOBAL 1 (NULL + range)
14 LOAD_CONST 1 (1000)
16 PRECALL 1
20 CALL 1
30 GET_ITER
>> 32 FOR_ITER 12 (to 58)
34 STORE_FAST 0 (i)

10 36 LOAD_GLOBAL 2 (count)
48 LOAD_CONST 2 (1)
50 BINARY_OP 13 (+=)
54 STORE_GLOBAL 1 (count)
56 JUMP_BACKWARD 13 (to 32)

9 >> 58 LOAD_CONST 0 (None)
60 RETURN_VALUE

We can see that INPLACE_ADD opcode has been replaced with BINARY_OP in Python 3.11. I guess that the BINARY_OP is now a thread-safe operation.

References

Medium article by Jordan Gillard

RealPython Article on Threading

Article by Saurabh Chaturvedi

Superfastpython - Thread Safe print

Launched the initial version of my Python REPL-themed portfolio website (with an ASCII art photo gallery) 🚀.

Home Page

Website: adarshd.dev or adarsh.pizza 🍕

Source code: github.com/adarshdigievo/Portfolio-REPL/

Internals

• UI layer and Python layer - both executed in the client browser

The site can be considered to have two components. One is the terminal Ui which is powered by Xterm.js. Accepting user input, handling the events, and printing the results are handled using Xterm functions.

The second component is the Python interpreter which runs the code and returns the result to xterm for printing in terminal UI. This is created using Python’s built-in InteractiveInterpreter class found in code module of stdlib.

• Loading the portfolio data

The portfolio data like skills, experience, etc. are fetched from the free API from Gitconnected and follow the open Jsonresume standard

• Metaprogramming

I love metaprogramming and have (over)used some metaprogramming concepts on the site. I wanted the variables storing my portfolio to be dynamic. This is possible with Descriptors where we can call custom getters when an attribute is accessed.

Currently, I have mapped profile data variables passed to the REPL like SKILLS, EXPERIENCE, etc into attributes of the class named ProfileData and have created descriptor attributes using ProfileFetchDescriptor. This descriptor is responsible for calling Gitconnected API and reloading the data if it is expired.

I have stored all profile data fields in an enum named ProfileFields which is used multiple times when I need to list all available pre-loaded variables in the REPL.

attrs = {
        field.name.lower(): ProfileFetchDescriptor(field) for field in ProfileFields
    }

    # Dynamically generate a class named ProfileData. Each member attribute of the class is a descriptor
    ProfileData = type("ProfileData", (), attrs)

• Redirecting stdout

I needed to get the output of code executed by InteractiveIntrpetor into a string so that it can be printed on the xterm ui. For that, redirect_stdout and redirect_stderr functions from contextlib were used.

import io
from contextlib import redirect_stdout

with io.StringIO() as buf, redirect_stdout(buf):
    print('redirected')
    output = buf.getvalue()

This output is then written on the xterm terminal.

• Ascii Art powered photo Gallery

In the gallery code, I have used ascii magic library to convert images into colored text and it is printed directly to an xterm terminal.

To store image metadata, I have added a metadata json file which includes the path to image resources and the caption to display on each image. The page then does an infinite iteration over the image data displaying each of them one by one. I have not decided to cache the images in Python due to memory worries.

• Opening links

To open links to my blog, photo gallery, etc, another descriptor is used. The descriptor uses webbrowser.open() to open the corresponding link. This Python functionality is supported out of the box in Pyscript. I have created an object of ProfileLinks class named VISIT

Code here

link_attrs = {
        field.upper(): ProfileLinkDescriptor(field)
        for field in ProfileLinkDescriptor.link_map
    }
    
    ProfileLinks = type("ProfileLinks", (), link_attrs)

    VISIT = (
        ProfileLinks()
    )  # This class contains descriptor attributes which open corresponding webpages on access. Ex: VISIT.BLOG
    
    interpreter.locals |= {"VISIT": VISIT}  # Load the class to interpreter locals

Learnings

Random new things I learned from this project:

• Eruda - JS debugging on mobile browsers

The website had some weird input issues on Chrome-based Android browsers while testing. While searching for a tool to debug JS on mobile browsers, I found Eruda. Eruda will emulate the desktop browser dev console with almost all features like js console logs, network requests, etc and this was super helpful to debug on mobile devices.

• Pyscript

Pyscript as their tagline says is ‘Programming for the 99%’. Pyscript makes it a lot easier to deploy an application with no manual configuration steps needed like pip installs and can be easily hosted as a static site. It is Python for the front end and will be well-suited for data applications (visualizations, graphics, etc).

• Pyodide

The magic of Pyscript exists thanks to Pyodide and webassembly. Pyodide is a python runtime that runs in Webassembly. Pyscript uses Pyodide internally and it can be seen as a convenient wrapper over pyodide. It gets the python code from <py-script> tags and sends them to the interpreter (which is Pyodide by default, and can be customized) for execution. Refer makePyscript() or pyExec() in the Pyscript source code to see more on the same. Pyscript exposes the power of Pyodide in a simple and convenient manner.

I found the function and variable interoperability b/w Python and JS super convenient. We can call the functions/variables defined in Python using javascript and vice versa. I have used this in the photo gallery page, to print python ascii-magic library output to xterm terminal.

See the example:

// js file

var term = new Terminal({
    cursorBlink: false,
    fontSize:6,
    cols:200,
    rows:150
});

term.open(document.getElementById('terminal'));

# python file

from js import term  # the js object is now available in Python
  
# can access all js functions defined in xterm js library from Python!
term.clear()
term.write('some_string')

Here is the C Soruce code of the function to_js() which is used to convert Python calls as seen above to js and this is the function toPy() used to convert js objects to Python.

Planned updates for the next versions

Web (frontend) development is hard. There are a lot of layout & compatibility issues across multiple platforms and browsers.

When I tested the home page on mobile devices for the first time, inputs were not working on Chromium-based mobile browsers, but were working fine on Firefox focus. I finally ended up with code like this:

if (/Android|webOS|iPhone|iPad|iPod|BlackBerry/i.test(navigator.userAgent)) {
    // onKey event is not firing on android chromium based browser. This workaround is applied in that case.
    console.log('mobile device') }

Other tasks planned for coming versions:

• Support multiline inputs (to support function and class definitions)

I guess I might need to switch to InteractiveConsole class of the code module instead of the currently used InteractiveIntrepretor class.

• Switch to py-terminal

Switch to py-terminal plugin of Pyscript and perform all term-related tasks (managing history, xterm events, etc) in Python code itself. I can skip the load and init steps of xterm using js since pyscript now support xterm terminal out of the box. Looking to implement something along the lines of this Pyscript plugin(emscripten-shell)

• Use a custom on-screen keyboard using kioskboard.js, in case of mobile devices to support arrow keys, command history, etc.

• Develop a visual version of the website

• Support pastes in REPL

• ast based variable parsing

Currently, the profile data variables are mapped to the descriptor attributes using a simple and hacky string.contains() call here. I plan to first parse the input code into AST and replace the areas of variable access to use the descriptors.

Prior Art

• Terminal UI was Adapted from this Codepen.

Ascii Photo Gallery

Visit the site: adarshd.dev

Photo gallery: adarshd.dev/gallery.html

Source code: github.com/adarshdigievo/Portfolio-REPL/

Let’s check:

The code in the below examples works as-is. No additional imports are required.

We will be using isinstance function to check if ‘everything’ is an object.

From Python Docs:

builtin function isinstance(object, classinfo):

Return True if the object argument is an instance of the classinfo argument, or of a (direct, indirect, or virtual) subclass thereof. If object is not an object of the given type, the function always returns False

1. int

>>> dummy_int = 3
>>> isinstance(dummy_int, object)
True

Testing the class ‘int’ itself

>>> isinstance(int, object)
True

2. Testing a user defined class

>>> class DummyClass:
... 	pass

>>> dummy_obj = DummyClass()
>>> isinstance(dummy_obj, object)
True
>>> isinstance(DummyClass, object)
True

3. list

>>> l = [1, 2, 3, 4]
>>> isinstance(l, object)
>>> True

4. Functions

>>> def dummy_function():
...     pass
...
>>> isinstance(dummy_function, object)
True

Testing the ‘isinstance’ function

>>> isinstance(isinstance, object)
True

5. Exceptions and tracebacks

>>> import sys
>>> try:
...     raise Exception()
... except Exception as e:
...     exc = e
...     traceback = sys.exc_info()[2]
...

>>> isinstance(traceback, object)
True
>>> isinstance(exc, object)
True

6. Modules

>>> import math
>>> math
<module 'math' (built-in)>
>>> isinstance(math, object)
True

Why:

From Python docs:

class object:

Return a new featureless object. object is a base for all classes. It has methods that are common to all instances of Python classes. This function does not accept any arguments.

The object class is the base class for all other classes in Python. This means that all Python classes inherit from the object class, either directly or indirectly. The object class provides a number of methods that are common to all objects in Python, such as the init() method, and the setattr() method. These methods allow objects to be created, accessed, and modified.

We can verify that ‘object’ is the base class of other classes by using the base dunder.

>>> import math
>>> math
<module 'math' (built-in)>
>>> type(math).__base__
<class 'object'>

>>> dummy_int = 10_000
>>> type(dummy_int)
<class 'int'>
>>> type(dummy_int).__base__
<class 'object'>

>>> type(type)
<class 'type'>
>>> type.__base__
<class 'object'>