S.Lott -- Software Architect - Python

A get_object_size() Function -- Again

2025-10-17T13:08:00-04:00

BLUF

You rarely need this. Python handles memory management for you. Except in an edge case where you have a lot of objects to work with.

How many is a lot? Enough that your app crashes with MemoryError exception. Or, is consuming so much memory other processes have trouble working. Or, it's slow because of all the garbage collection going on.

Background

See A get_object_size() function. This shows a totally non-recursive and not-too-smart approach.

The previous example avoids recursion. This is not really helpful. While structures can be very large, they are rarely deeply nested. The ordinary Python stack limit would prevent us from walking a structure with over 1,000 layers of nesting. Even creating a test case is a pain in the neck.

What I Did

This function computes the total size of just about anything. This includes all the built-in collections. It also includes "custom classes", both the __slots__ and the non-__slots__ variants.

from collections.abc import Sequence, Mapping, Set, Iterable, Callable
import itertools
import sys
from textwrap import shorten
from typing import Any


def get_object_size(
    some_object: Any,
    additional_types: Callable[[Any], int | None] | None = None,
    verbose: bool = False,
) -> int:
    """
    Computes the size of the given object.
    This expands on the recipe cited in the documentation for :py:func:`sys.getsizeof`.

    :param some_object: Any Python object.
    :param additional_types: A function that can return the size for an object for a type not handled here.
    :param verbose: True to print object information as the size is computed.
    :return: aggregate size of the object and all the related objects.

    The sizes are **highly** implementation specific.

    The types handled here are the built-in collections
    defined in :py:mod:`collections.abc`:
    ``str``, ``Sequence``, ``Set``, ``Mapping``.
    Additionally, this will look at any instance of class derived from :py:class:`object`,
    handling the default ``__dict__`` as well as ``__slots__``.

    >>> get_object_size("Hello, world!")
    54
    >>> get_object_size("!")
    42
    >>> get_object_size(list(range(10)))
    416
    """
    default_size = sys.getsizeof(0)
    seen: set[int] = set()

    def component_size(obj: Any) -> int:
        nonlocal seen
        if id(obj) in seen:
            return 0
        seen.add(id(obj))

        if verbose:
            print(
                f"{id(obj):8x} {type(obj)}, {shorten(repr(obj), 32)}", file=sys.stderr
            )

        items: Iterable[int] = iter([])
        match obj:
            case str():
                pass
            case Sequence() | Set() as sequence:
                items = map(component_size, sequence)
            case Mapping() as mapping:
                items = itertools.chain(
                    map(component_size, mapping.keys()),
                    map(component_size, mapping.values()),
                )
            case object() as obj_dict if hasattr(obj, "__dict__"):
                items = itertools.chain(
                    map(component_size, obj_dict.__dict__.keys()),
                    map(component_size, obj_dict.__dict__.values()),
                )
            case object() as obj_slot if hasattr(obj, "__slots__"):
                values = (
                    getattr(obj_slot, name)
                    for name in obj_slot.__slots__
                    if hasattr(obj_slot, name)
                )
                items = map(component_size, values)
            case _:
                if additional_types and (obj_size := additional_types(obj)) is not None:
                    items = iter([obj_size])

        base = [sys.getsizeof(obj, default_size)]
        sizes = itertools.chain(base, items)
        return sum(sizes)

    return component_size(some_object)

This variant walks an entire structure recursively. It creates iterable generators with size details.

You won't often need this. But. I've posted it here so I won't lose it.

And. I like to think through alternative implementations. One of these is probably faster.

A get_object_size() Function [Updated]

2025-10-16T13:08:00-04:00

BLUF

You rarely need this. Python handles memory management for you. Except in an edge case where you have a lot of objects to work with.

Background

One of the significant benefits of using Python is memory management. Python creates and disposes of objects as needed, without a lot of complicated-looking code. The reference counting often works flawlessly.

Often.

Sometimes there are circular references, and objects can't (trivially) be collected. Object A refers to object B and (sadly) object B also refers to object A. They both have non-zero reference counts.

Weak references can sort this out, preventing a core leak that leads to unreliable software.

Use Case

Python has a way to reduce the size of objects. It alters a few minor details of how an object works, but is generally transparent.

This is an example of a rare case where you are doing memory management in Python.

You can use the __slots__ feature of a class to name the attributes that are present, and prevent the creation of the usual __dict__ structure to hold the attributes.

When playing with this, it's often easiest to use the @dataclass(slots=True) decorator apply this to a class.

This will save memory.

How much memory?

It's hard to say, because you need to know the "without slots=True" and "with slots=True" sizes. Generally, slots=True will be smaller. If you've chosen the right class to shrink, you may find improved performance, also.

To determine the savings, we need to know the actual size of the collection of objects that are crushing the life out of our application and leading to MemoryError exceptions.

The Object Size Problem

The sys module has a function, sys.getsizeof(), that will provide the size of an object.

This is the object in isolation.

Important

sys.getsizeof() doesn't include contained objects

What does it matter of it doesn't include contained objects?

Consider a list[int].

>>> import sys
>>> small_list = list(range(10))
>>> large_list = list(range(10_000))
>>> sys.getsizeof(small_list)
136
>>> sys.getsizeof(large_list)
80056

Okay. Superficially, it seems like each integer in the small list takes up about 13 bytes.

Weirdly, each integer in the large list seems to take about 8 bytes.

This can't be right.

Consider an int.

>>> sys.getsizeof(42)
28

Okay. That's really, really weird.

The list[int] size is the size of the list object. It doesn't include the 10 (or 10,000) int objects that are members of the list.

Total memory for the short list, then is \(136 + 10 \times 28 = 416\). Total memory for the large list would be \(80,\!056 + 10,\!000 \times 28 = 360,\!056\).

The Documentation Says

Check the documentation for sys.getsizeof(). You'll see this.

"See recursive sizeof recipe for an example of using getsizeof() recursively to find the size of containers and all their contents."

The documentation doesn't say "And read all the comments and integrate all those ideas into one function."

What I Did

This function computes the total size of just about anything. This includes all the built-in collections. It also includes "custom classes", both the __slots__ and the non-__slots__ variants.

from collections import deque
from collections.abc import Sequence, Mapping, Set, Callable, Iterator
import sys
from textwrap import shorten
from typing import Any


def get_object_size(
    some_object: Any,
    additional_types: Callable[[Any], int | None] | None = None,
    verbose: bool = False,
) -> int:
    """
    Computes the size of the given object.
    This expands on the recipe cited in the documentation for :py:func:`sys.getsizeof`.

    :param some_object: Any Python object.
    :param additional_types: A function that can return the size for an object for a type not handled here.
    :param verbose: True to print object information as the size is computed.
    :return: aggregate size of the object and all the related objects.

    The sizes are **highly** implementation specific.

    The types handled here are the built-in collections
    defined in :py:mod:`collections.abc`:
    ``str``, ``Sequence``, ``Set``, ``Mapping``.
    Additionally, this will look at any instance of class derived from :py:class:`object`,
    handling the default ``__dict__`` as well as ``__slots__``.

    >>> get_object_size("Hello, world!")
    54
    >>> get_object_size("!")
    42
    >>> get_object_size(list(range(10)))
    416
    """
    default_size = sys.getsizeof(0)
    seen = set()
    size = 0
    elements = deque([some_object])
    while elements:
        obj = elements.popleft()
        if id(obj) in seen:
            continue
        seen.add(id(obj))

        if verbose:
            print(
                f"{id(obj):8x} {type(obj)}, {shorten(repr(obj), 32)}", file=sys.stderr
            )

        size += sys.getsizeof(obj, default_size)
        match obj:
            case str():
                pass
            case Sequence() | Set():
                elements.extend(iter(obj))
            case Mapping():
                elements.extend(obj.keys())
                elements.extend(obj.values())
            case object() if hasattr(obj, "__dict__"):
                elements.extend(obj.__dict__.keys())
                elements.extend(obj.__dict__.values())
            case object() if hasattr(obj, "__slots__"):
                elements.extend(
                    getattr(obj, name) for name in obj.__slots__ if hasattr(obj, name)
                )
            case _:
                if additional_types and (obj_size := additional_types(obj)) is not None:
                    size += obj_size
    return size

Note that this walks an entire structure without actually being recursive. If you've got a complicated application, and a very deeply-nested data structure, the overhead of a lot of stack frames may be unmanageable.

(There are other optimization approaches to this problem.)

This assumes that a collection always contains heterogeneous types. This means computing the size of each item in the list.

This uses a big deque, which can involve impossible overhead, also.

In some cases, you may need to create a more complicated special-purpose benchmark app that builds your big data structure using your distinct storage alternatives. Use your special benchmark test-bed to uncover the implementation that meets all the criteria for storage use and CPU time.

The data that is used for the benchmark would need to reflect real-world data with respect to string lengths, and collection sizes. Creating synthetic data for an object size benchmark can be a challenge. See Synthetic Data. And, also see Synthetic Data Tools.

You won't often need this. But. I've posted it here so I won't lose it.

TODO

Handle numpy types, also.

OO Design Principles: GRASP patterns

2025-10-04T09:46:00-04:00

Some quotes to provide context.

"I read several times that ideally I should avoid inheritance with ABCs and if I do, then from a standard class. That is, I should avoid creating a typical base and subclass construct that is used in a strategy pattern, for example."

"If I look at a principle like SOLID, where we have the Liskov Substitution Principle, the interface separation principle, and the dependency inversion principle, they all rely on a base class, which of course is created by inheriting from ABC."

"Python, on the other hand, does not have the strong need to be object-oriented, but most of the principles assume that we are working with classes."

"Working with Liskov Substitution can make it difficult to navigate through an IDE."

BLUF

The GRASP patterns can be helpful reminders.

Some GRASP patterns sound like design principles, and seem to overlap with the SOLID principles.

Other GRASP patterns seem like summaries, with a number of fine-grained implementation choices.

Controller and Pure Fabrication seem most helpful.

The "Gang of Four" Object-Oriented Design Patterns seem to be more useful and have more supporting details.

GRASP Patterns

See https://www.geeksforgeeks.org/system-design/grasp-design-principles-in-ooad/

Controller

See also the "Model-View-Control" pattern.

There's an internal model of reality. A description of the real-world stuff. Maybe this is simply CRUD rules over a database (as if a database is reality -- it's only a model.) Maybe this is an OO model that gets serialized into something persistent when the application exits (you know, like a word processor or spreadsheet.) Maybe this is an OO model of a game where there's a state-of-play saved when you specifically ask to save. Maybe this is a real-time controller of a device like a cruise control on a car where the model has the car's current speed and the target speed (and not much else, TBH.) Maybe this is an anchoring position indicator with a position where the boat is expected to be and the most recent few minutes of GPS positions indicating if it's within a safe distance of the anchor or not and what direction the boat seems to be going (i.e., it is adrift?)

There's a view of reality, displaying the state of the model. You fiddle with the view to update the model. Maybe look at the number on the dashboard or push buttons on the cruise control. Maybe type a bunch of stuff into fields on a form.

And there's the controller.

"The controller is defined as the first object beyond the UI layer that receives and coordinates ("controls") a system operation. "

This applies widely. In some applications -- programs like grep and sed -- the UI is really tiny and not interactive.

The input is command-line parsing (via argparse or click or whatever).
The output is stdout and stderr (via logging or print() or whatever).
There's a model (the file being processed, the text in that file, the patterns and commands).
There's a tiny controller that gets the input, then iterates through the model's visible state changes, writing the outputs.

This is one essence of Clean Design. Separating the inputs and outputs from the model and control.

Creator

There's potential overlap between Creator and Information Expert. Creators have the information required to create an instance of a collaborator. Which sometimes means the creator is also an expert.

Indirection

Sometimes helpful. The Gang-of-Four design patterns subdivide this many ways. You might have an Adapter or Facade to wrap one (or more) collaborators. You might have a State or Strategy class hierarchy that wraps up alternative implementation choices into a tidy structure.

Note there are two layers of indirection. Objects can have indirect access to other objects, perhaps mediated by the current state. Classes should have indirect access to other class definitions, per the SOLID Dependency Inversion Principle.

Information Expert

The SOLID Single Responsibility Principle, restated. Nice to see everyone agrees on this.

Low Coupling

This is a desirable feature of object design. This is more of a principle that echoes the SOLID Interface Segregation Principle.

High Cohesion

This is also a very desirable feature of object design. This isn't (explicitly) a SOLID design principle, but lurks in she shadows of the SOLID Single Responsibility Principle. In order to have a single Responsibility, there must be a cohesive design to the class.

Polymorphism

This is a mechanism as well as a large number of patterns. The mechanism is a way to reuse code in a class hierarchy. It saves repeating things when we have related classes.

The reason why we have a SOLID Liskov Substitution Principle is to manage polymorphism.

The entire Gang-of-Four design patterns book is an extended set of patterns for applying polymorphism.

Protected Variations

This is a restatement of the SOLID Open/Closed Principle. Open to extension (via variations.) Closed to modification (the "protected" part of this.)

Pure Fabrication

I often call these fabricated classes part of the "Solution Domain" distinct from the "Problem Domain."

The model is part of the Problem Domain. It represents the real-world things in software space. This should map to the problem domain with a high degree of fidelity.

The view presents this to users. It's part of the UI, which is part of the solution. It can overlap with the problem domain, however, and may happen to match the model. A CRUD application, for example, may display the data more-or-less directly and the view may match the model.

In some cases, the view twists and transforms the model into what people think they want to see. Think of the aggregated data in a data warehouse. Details are elided. It's a summary -- it's not the reality. A great deal of twisting and turning happens during ETL and aggregation processing to provide a view the users can understand and act on.

A controller, however, has nothing to do with the problem domain. It's purely part of the solution domain. It reflects the purpose behind the software -- show status, update things, allow interaction, summarize, whatever.

Conclusion

The GRASP patterns can be helpful reminders of different kinds of solutions.

They overlap with -- and can provide some amplification for -- the SOLID design patterns.

The Pure Fabrication and Controller GRASP patterns seem to be more significant than the others.

OO Design Principles: CUPID

2025-10-03T09:46:00-04:00

Some quotes to provide context.

"I read several times that ideally I should avoid inheritance with ABCs and if I do, then from a standard class. That is, I should avoid creating a typical base and subclass construct that is used in a strategy pattern, for example."

"If I look at a principle like SOLID, where we have the Liskov Substitution Principle, the interface separation principle, and the dependency inversion principle, they all rely on a base class, which of course is created by inheriting from ABC."

"Python, on the other hand, does not have the strong need to be object-oriented, but most of the principles assume that we are working with classes."

"Working with Liskov Substitution can make it difficult to navigate through an IDE."

BLUF

Very important: Composable, Predictable, Domain-based.

These are consistent with the idea that software is knowledge capture.

Maybe less important: Unix-Philosophy and Idiomatic. While nice ideas, they feel a bit redundant.

CUPID

See https://cupid.dev/properties/

Composable

Highly desirable to have small pieces that compose into something useful. This fits with SOLID Interface Segregation Principle, and provides an answer to "but, why?" We segregate the interfaces to create something composable.

Unix-Philosophy

This also implies composability. It goes a step further to incorporate the idea of the SOLID Single Responsibility Principle. The Unix design patterns provides concrete examples of how to decompose a large problem into smaller pieces; each piece does one thing well.

Predictable

This decomposes into three distinct aspects:

A component behaves as expected. This is sometimes called the Principle of Least Astonishment.
A component's behavior is consistent and deterministic.
A component's behavior needs to be observable, also.

These are all critical features, especially observability.

Idiomatic

Yes, follow language idioms. Please do not boldly go where no programmer has gone before.

This seems to go without saying. It does help fill up the acronym, though. And, perhaps, it's necessary advice.

Domain-based

The solution code should mirror the problem domain. This is really important, and not present in other sets of design principles.

Indeed, I don't think it can be emphasized enough.

OO modeling and design isn't about optimization or code reuse. It's about fidelity to the problem domain.

Code reuse is nice to have. Providing common behavior among the problem domain objects that are being modelled is the point.

Conclusion

These are nice. They're very useful in conjunction with SOLID design principles.

It helps to have guideposts to help clarify an underlying "why" we design software.

OO Design Principles: SOLID

2025-10-02T09:46:00-04:00

Some quotes to provide context.

"I read several times that ideally I should avoid inheritance with ABCs and if I do, then from a standard class. That is, I should avoid creating a typical base and subclass construct that is used in a strategy pattern, for example."

"If I look at a principle like SOLID, where we have the Liskov Substitution Principle, the interface separation principle, and the dependency inversion principle, they all rely on a base class, which of course is created by inheriting from ABC."

"Python, on the other hand, does not have the strong need to be object-oriented, but most of the principles assume that we are working with classes."

"Working with Liskov Substitution can make it difficult to navigate through an IDE."

BLUF

Apply the design principles with a kind of priority mind-set.

Interface Segregation. Always.
Liskov Substitution where you actually have base classes or protocols.
Open/Closed. Always.
Dependency "Inversion" (I prefer "Injection"). Think of this as an optimization.
Single Responsibility. Here is where various GRASP patterns apply at this point here to help implement a "single" responsibility.

Remember, GRASP are implementation patterns not quite the same as design principles.

SOLID Principles

I find the SOLID acronym is confusing. Mostly because the principles are presented in a funny order. While the "Single Responsibility" principle is first, and seems most important, it's really more of a summary.

ISP

The "Interface Segregation" principle is -- perhaps -- the most important.

Important

Interface Segregation Principle

ISP == Cut the fluff seen by collaborators.

In a C++ or Java context, the more that's exposed to a collaborator, the wider the ripples from a change.

In a Python context, a too-rich interface is just annoying to understand, maintain, extend, and test.

One metric for paring down an interface is the number of mock objects needed for an isolated unit test.

(Some Facade and Adapter designs will have a lot of mocks because they're wrapping complicated things in a simple interface. Don't apply the "minimize mocks" metric blindly.)

LSP

Remember, Python has duck typing. LSP is something that applies if -- and only if -- inheritance is being used. A bunch of classes can all implement a common protocol without being subclasses of each other.

An Abstract Base Class has the abstract methods marked so that an object cannot be created. When all of the methods are defined by some subclass, it's now concrete and an object can be created.

A concrete base class can be extended as needed, and any class in the hierarchy is usable. There won't be any "Can't instantiate abstract class" exception.

In order to make LSP work, it's very helpful for subclasses to add features to base classes. Base classes should be a minimal foundation; think of them as a generalization. The extensions should add features; they will be specializations. (It's not mandatory, it's just kind of tedious to write "do nothing" methods for a subclass to take away a base class feature.)

OCP

This requires some consideration of the sources of change and the kinds of extensions that might be necessary.

It's rather difficult to do well when working on a problem domain that's not well understood. After creating some code -- and living with it -- it's easier to see what's likely to change and what's likely essential. Once the change vectors are more clear, it becomes easier to see what parts of a class are likely to be extended.

Important

Open/Closed Principle

There are no "bug fix" changes where you modify a class.

Instead, think about extending a broken class with a subclass that has less buggy implementations of methods. The app then needs to use the subclass that's not as broken. Change happens through extension, not modification.

Doing this means "Dependency Injection Principle" needs to be used also.

DIP

Dependency "Inversion" really means don't have simple names of classes everywhere.

In Java and C++ collaborators with a class would have a reference to that class compiled into them. Change some class, and the collaborators all need to be recompiled. No one likes this.

In Python, the name resolution happens at run-time, and there's no additional overhead from making a change. (The name lookups are an overhead that's inherent in Python.)

In Python, the dependency injection means assigning a target class to a variable instead of simply using it.

class SomeThing:
    def __init__(self, this: str, that: int) -> None:
        self.this = this
        self.that = that

class Collaborator:
    what_to_build: type[SomeThing] = SomeThing
    def __init__(self, arg: str) -> None:
        self.something: SomeThing = self.what_to_build("this", arg)

The name of the thing to build is a class variable in the Collaborator class. Making a change to the class used internally by the Collaborator class is isolated to this variable.

We can make a subclass of Collaborator with a new value for what_to_build and change it's behavior. We can go further, of course, and have some centralized configuration that names the classes to use. That can be handy in very complicated applications where a lot of things are likely to change.

As a practical matter, very few things change. A small configuration object with a few critical class references is all that's really required.

SRP

The most difficult of of the SOLID design principles is identifying a "single" responsibility. The question of responsibility often requires some qualifiers. It's important to consider responsibilities from which collaborator's perspective.

A class may do a bunch of things internally. But -- viewed from outside -- it's a single, atomic behavior.

This is where the nine GRASP patterns can come in handy, to implement a class with a single responsibility.

Conclusion

Apply the SOLID principles carefully.

Always apply ISP, OCD, and DIP. Use LSP when there's inheritance involved. The SRP requires some careful thought, and -- from different perspectives -- can be awkwardly complicated.

The GRASP patterns can be helpful for implementation. Sometimes they are overly focused on Java and C++.

OO Design Principles: ABC's

2025-10-01T09:46:00-04:00

Some quotes to provide context.

"I read several times that ideally I should avoid inheritance with ABCs and if I do, then from a standard class. That is, I should avoid creating a typical base and subclass construct that is used in a strategy pattern, for example."

"If I look at a principle like SOLID, where we have the Liskov Substitution Principle, the interface separation principle, and the dependency inversion principle, they all rely on a base class, which of course is created by inheriting from ABC."

"Python, on the other hand, does not have the strong need to be object-oriented, but most of the principles assume that we are working with classes."

"Working with Liskov Substitution can make it difficult to navigate through an IDE."

BLUF

ABC's are only required as a way to do earlier validation that a class is complete.

Otherwise, they're not required.

They can be helpful when planning dependency injection. The SOLID Dependency Inversion Principle advises us to depend on abstractions, not concrete, specialized subclasses.

In Python, we don't need an abstraction at the base of a class hierarchy.

Purely OO Design?

First, I want to talk a little bit about OO design and "purely" OO languages.

Unlike Java or C++, Python is purely object-oriented. Not that it actually matters.

A functional style of Python depends on functions which are -- essentially -- callable objects. They have one method, __call__(). They have attributes. Because they descend from the object class, you can add attributes to a function. They're instances of a type, function.

Java and C++ have "primitive" types which aren't objects; they're not "purely" OO. Python doesn't have this quirk. Python is more purely object-oriented than Java. Which suggests any OO purity test doesn't really matter.

OO Design Principles and Functional Design Principles can all be used with Python. Indeed, some of the old COBOL design patterns can be used, too. (Not all, of course. COBOL had GOTO's, an ALTER statement, and a very weird PERFORM THRU that make it right weird to map to Python.)

OO Purity? Doesn't matter.

Let's move on to look at Python's ABC's. After that, we'll look at the SOLID principles in general.

To ABC or not to ABC?

C++ and Java (and many other languages) are built around separately-built binaries that are linked together. Some linkage can be done at compile time, some can be done at run time by the OS loader. Since they're separately-built binaries, everyone must agree on the binary interface. Change cannot be tolerated.

I'll emphasize that.

Emphasis

C++ and Java emphasize a style of design where interface changes cannot be tolerated.

Pragmatically, we now have computers that are so fast that recompiling a very large Java or C++ app is no longer a nightmare of waiting for hours. When these languages were designed, a development team might do one nightly build of everything. During the day, you were limited to compiling the classes you were working on. Nothing more. Getting the interfaces to be stable was an important risk reduction technique.

Abstract Base Classes were a way to minimize recompilation.

Important

Abstract vs. Concrete

There are abstract base classes and concrete base classes. The abc module introduces a whole bunch of stuff to support abstraction. Ordinary class Special(Die): inheritance from a concrete base class involves no abstraction, and no abc.

Python eschews strict class hierarchies, and replaces this with "Duck Typing". All an object requires is to have the method defined.

See The eval() Conundrum and Python-as-DSL. Way at the end is this snippet of code.

class Die:
    def __init__(self, faces: int) -> None:
        ...
    def __rmul__(self, n: int) -> "Die":
        ...
    def __add__(self, adj: int) -> "Die":
        ...
    def roll() -> int:
        ...
    @property
    def min(self) -> int:
        ...
    @property
    def max(self) -> int:
        ...

D4 = Die(4)
D6 = Die(6)
D8 = Die(8)
etc.

This means an object like D4 can be used with the * and + operators. In very limited ways.

The expression 6 * D4 is legal, where D4 * 6 is not. The way Duck Type works, there's a search for a method to implement *.

Consider 6 * D4.

Does 6 implement __mul__()? It does. However, when int.__mul__() is evaluated with a Die object, the result is NotImplemented.
Does Die implement __rmul__()? It does. When Die.__rmul__() is evaluated with an int object, the result is a new Die object.

(There's actually a first step, omitted for brevity. See the sidebar.)

Here's the bottom line.

The Duck Typing two-step search for matching method names doesn't respect the class hierarchy.

A Protocol formalizes the Duck-Typing Two-Step in a way that tools can be sure the whole

Summary:

When using only Python, most developers don't need to care about separately-compiled binaries. (When writing Rust or C extensions, of course, separately-compiled binaries are a big deal.)
Duck typing eliminates a requirement for ABCs.

Conclusion

Is an Abstract Base Class still helpful?

Yes.

When?

When you need it.

When is it necessary?

The use case for an ABC in Python is to push the Duck-Typing Two-Step so it happens earlier.

ABC's permit type hint checking to be sure the code is likely to work.
At run time, ABC's prevent instantiating an incomplete object. Code may crash earlier.

Most important: the exception is much more clear when a method is missing.

ABC's promote early detection of design problems.

When is it superfluous?

When your base class is concrete, don't waste time on an ABC. Just use a concrete base class and extend it as needed.

That's enough on ABC's for now. Let's move on to the SOLID principles.

Comma Comma

2025-06-11T07:58:00-04:00

Ugh.

A painful stretch of hours looking for a problem. Working on this: https://github.com/cloud-custodian/cel-python/wiki/Evaluation-Design. I need to address a performance problem and upgrade things generally to get them ready for 3.13 and 3.14.

You know how it goes, right?

I touched something "minor" and all kinds of acceptance tests broke beacuse I also broke something central.

The horrible realization was this:

There Was No Unit Test

When the problem is not found first by a unit test, it means there's a feature that's only tested by the acceptance test suite.

Where do we stand?

Here are the causes for despair.

We've got an acceptance test failure.
This reveals a gap in the unit tests.
This also reveals the stuff doesn't work because I touched something.

What to do?

Ideally? Fix the unit tests.

Pragmatically? Review the git change history to see what I've touched recently.

Aha. A trailing comma.

There was a {key: lambda x: some_expression, ...} data structure. I took the lambda out and replaced it with a proper def function. With a name.

def some_new_function(x: type) -> type:
    some_exression,

You know how this goes.

Looks good, who can see the ,?

Doesn't work. tuple() shows up in unexpected places.

Big Lesson

Fix the unit tests.

Domain-Specific Language

2025-03-11T15:52:00-04:00

Generally, I try to frown on Domain-Specific Languages. Often, a tidy set of related functions, or a group of class definitions with a few decorators can create something that's every bit as expressive as a DSL in native Python syntax.

There are a few cases where a DSL can be handy.

One of which is when embedding complicated content in Gherkin test cases.

(We'll get the context later. I want to focus on the problem at hand.)

We need to write Gherkin that looks like this:

GIVEN some configuration
WHEN we run the Character UI with input of "..."
THEN we see the right kind of responses in the log and what-not

What's essential here is the input to the Character UI (CUI). It's a sequence of characters.

In Python, it might be "o" + 2*"mu" + "i" + "qy".

The idea is to provide a meaningful sequence of commands for the given scenario.

Except, of course, Gherkin isn't Python.

We have two choices:

Use eval() to evaluate a Python expression.
Use a DSL.

Since this is a test scenario, the ridiculous arguments about eval() being "unsafe" are clearly worthy of ridicule. Any Evil Super Genius capable of writing a Gerkin test case that includes import os or import subprocess has access to the source code and doesn't need to subvert the Gherkin language test scenarios.

While eval() is appealing because it's simple, it's not always ideal. In particular, some of the scenarios are long, and the sequence of input commands needs some supplemental information to follow through the actions and the responses to work out what the expected intermediate and final states will be.

The Context

I'm working on a Rogue-Like game. The test scenarious involve walking around and collecting treasure, bashing monsters, and avoiding traps. The interactions aren't too complicated, but, I'd like these Gherkin-based acceptance tests to involve an absolute minimum of specialized test harness. A code tweak to seed the random number generator seems to be all that's appropriate.

In the long run, there's a potential to work through some clear definitions of the various features of the game using RDF. SPARQL queries might provide ways to locate the features of items or behaviors of monsters.

The Complication

What we want to have is something like this.

o                          ⍝ acknowledge splash
room 1≔ ⟨m l⟩×4 ⟨m u⟩×2 i  ⍝ to the room exit
hall 1≔ ⟨m u⟩×5 i          ⍝ to the next room entrance
done≔ q y

We've wrapped the essential commands in a bunch of syntax to make the commands a bit more clear. I can include a label at the left end of the line, and a "comment" on the right end.

Here's what's tricky.

The test case string will contain almost any ASCII character found on the keyboard. We don't want to have a lot of meta-punctuation that's also on the keyboard.

This bumps into the same problem the classic Regular Expression language suffers from. We want to match characters. And we need the same set of characters to have meta-level meanings. Sometimes . means the damn dot. Sometimes . means "any" character. So we have to use \. vs. . to express the distinction.

I decided to do this.

Use Unicode

I could wrestle with some "meta" characters for grouping and repetition. Indeed, I could parse the ordinar Regular Expression language. These are -- technically -- regular expressions that summarize long strings of characters for the test cases.

Or.

I could use a few Unicode characters separate from the ASCII keyboard characters that would be input.

⟨ and ⟩ (MATHEMATICAL LEFT ANGLE BRACKET, and MATHEMATICAL RIGHT ANGLE BRACKET)
× (MULTIPLICATION SIGN)
≔ and ⍝ (COLON EQUALS, and APL FUNCTIONAL SYMBOL UP SHOE JOT)
␛, ␠, ␤ for Escape, Space, and Newline.

These eight additional characters are not on the keyboard. The game will never use these as input. I don't need to use fancy escape sequences to distinguish these meta-characters from the ASCII characters that are being generated. These few characters are difficult to type: you have to pluck them off the Unicode character pop-up window that Macos offers you.

The code to handle the Gherkin doesn't have to fuss around with hyper-complex-looking regular expressions to parse this DSL. The allowed input characters are . and the meta-characters are simply presented as literal values in the regular expression. (As noted above, this DSL is a regular expression language. It's highly limited.)

Joomla Conversion

2025-02-08T08:52:00-05:00

Recently, we talked about extracting data from complex relational databases. This is -- in a way -- another case study for my Unlearning SQL book. This is a description of what comes next after the "low-level" conversion. Warning: it's complicated.

BLUF: Take the time to get rid of SQL processing.

In Part I, we loaded a database and queried the metadata. In Part II, we extracted the raw tables and loaded up a TAR Archive with NDJSON documents. In Part III, we prepared native Python objects that had a complete representation for the various kinds of tree structures. These include assets, categories, forum topics, image galleries, amongst other things. In Part IV, we talked about some applications to examine the converted data, looking for useful values, keys, and relationships.

We're going to skip a lot of the icky Joomla! details and focus on how to create something potentially useful.

The Goal

Recall from Part IV, we thought we had several steaming heaps of content on the legacy site. After exploration, we think we have the following:

A home page with a few articles.
A right sidebar with two articles.
A few content pages, each of which has links to a dozen or so narrowly-focused articles in a few categories.
The master collection of articles, neatly organized by category. There's a hierarchy here, a SQL nightmare we've avoided by restructuring the data.
The Kunena forums collection categories, topics, and messages. There's a hierarchy here, another SQL nightmare.
The JoomGallery collection of images. Hierarchy.
The Phoca collection of download files. You guessed it, another hierarchy.

These aren't the only hierarchues. Menus, modules, and assets have very tangled relationships, also. These are a SQL-query nightmare that we've turned into simple Python references among objects.

There's more, of course.

A hoard of Yahoo message-board posts which are not first-class parts of Joomla! but are first-class content.
Scans of the old paper newsletters. These, too, are not first-class parts of Joomla!, but are clearly very important content.

We'd like to dump all of this into a form that the Hugo tool can use to approximate the original site's content and structure. We're not going to spend too much time on the original look and feel; we can fuss with CSS to maybe match the color scheme.

What we've got are two separate kinds of things in the resulting site:

The "pages" which are Hugo Page Bundles with an _index.md and maybe some image resources. Each article becomes a page. In the case of the Home page -- which has multiple articles plastered onto it -- we will need a special-case template to include the bodies of multiple articles in one place. The Yahoo! messages are -- essentially -- articles that require some extra effort to convert.
The "collections" which are Hugo Sections, using an empty _index.md and a section-index generated by the template. The old newsletters are little more than downloads; these should be handled gracefully as a collection of Page Bundles.

We've also got some things we're going to set aside. Specifically, the right side-bar for articles is a waste of screen real-estate. It's not present for Forum or Photo Gallery.

Many of the Hugo themes have a 3-column look: the top-level menu is on the left, and the page table-of-contents is on the right. This seems to be somewhat more useful. One very spare Hugo theme is the Book theme, which seems like a good place to start.

The Processes

There are two separate kinds of migration processes:

Bulk migration of the collections. We have four, separate, unique subclasses.

The Converter Class Hierarchy
Create the top-level pages that match the various pages of articles on the legacy site.

The MakePage Class Hierarchy

Note that the pages depend on the bulk-conversion results. The new path structure and new file names, and other details are (more-or-less) encapsulated in the converter classes.

Things That Aren't Easy

While Hugo handles a large number of special cases and exceptions gracefully, we have legacy content that's a bit of a mess. Some of the mess may be my inability to ferret out all of the details of the Joomla! data model. Other aspects of the mess also seem to be a result of the way Joomla! decides what's "published" and what's not "published."

First, and most obvious, we have HTML content. We can -- if we want -- generate HTML pages and leave the details to Hugo. In the long run, we'd like to move away from HTML. We'd really like to emphasize Markdown and make HTML an exception.

To do this, we state each page uses markdown, and wrap the HTML in {{<html>}}...{{</html>}} "short tags". This is -- well -- ugly. It sequesters the HTML in those few places where it's used.

descriptions for galleries and downloads.
articles.
forum messages.

It means a lot of code like this:

print()
print("{{<html>}}", article.fulltext, "{{</html>}}")

This gets us started with "safe" HTML everywhere. We can see a great deal of the site with this hack.

Hugo leaves HTML comments in places where unsafe HTML shows up. We can look for  in the generated HTML and include needed wrappers.

Embedded Images and Links

There are two kinds of links that show up in articles, descriptions, and forum messages:

<a href="..."> tags
<img src="..."> tags

These have a variety of forms:

Proper external references with a scheme and/or a "netloc" (host name.)
Redundant internal references with a scheme and netloc of the server currently hosting the legacy content.
index.php?... queries.
#fragment fragements of the current page.
local/path/to/content paths into the legacy site content.
userupload/whatever paths into the local directory tree outside what Joomla! manages.

These devolve to three functions for link rewriting algorithms.

A filter to distinguish between "don't bother", "query", and "path" cases. A URL with a scheme or netloc (or both) is ignored. A URL that's only a fragment is also ignored. (We could try to clean up the fragments, but, there aren't many and they require divining the author's intent, something that's hard to automate.)
A function to rewrite the Joomla! queries into paths into the new content structure.
A function to examine the various paths that are used and restate these as part of the new content structure. Because we have four kinds of collections, plus the local filesystem references, we have a number of "search" functions for this case:
- search galleries
- search articles
- search downloads
- search forums
- search image archive files

Yes. This is a large pain. While there is some overlap, each collection is unique with unique names and a distinct resulting tree in the new content. No, there's no trivial way to impose a single, unifying, "one-ring-to-rule-them-all" content structure. The whole point is to respect the unique features of each category of content.

What Else? Oh, Right, Section Index

The Book theme doesn't (by default) include section indexes as a default structure.

If there's a layout/_defaults/section.html, this is used for those _index.md pages that are clearly the top of a section tree.

We don't need to do anything more than define the template for the index. Here's what we started with:

{{ define "main" }}
  <main>
    {{ .Content }}

    {{ $pages := .Sections }}
    {{ $paginator := .Paginate $pages 25 }}

      <ul>
    {{ range $paginator.Pages }}
      <li><a href="{{ .RelPermalink }}">{{ .LinkTitle }}</a></li>
    {{ end }}
      </ul>

    {{ template "_internal/pagination.html" . }}
  </main>
{{ end }}

This doesn't sort things properly, so we need to add metadata with weighting.

Finally, Broken Links

We have two origins for broken links.

Stuff we couldn't find in the database. Or, more properly, things which appear to be named in the database, but we can't find anywhere.
Stuff we thought we found, but it still didn't work in Hugo. These are essentially bugs, and we're still working through the last five.

The "stuff we could never find" includes things that were likely removed from the legacy site. Since we didn't take the time to work out all the "pubish this--don't publish that" rules, we've likely included things which were "unpublished" but not deleted.

Other things are <a href="href=">Something</a> kinds of HTML. That's just broken. There aren't many of these, and they need to be addressed manually.

Conclusion

Note the complexity of the migration.

There's not much that can be done to magically simplify all the special cases.

The time spent getting the database out of SQL and into Python objects gave us pleasantly simple Python objects to work with.

The class hierarchies evolved slowly. While it seems clear from the UML diagrams that these are "logical" designs, they didn't happen first. The initial design was not so clear and simple, leading to lots of redundant and inter-dependent code.

There's still a fair number of ultra-long methods that need to be decomposed into shorter, easier-to-understand methods. The remaining bugs involve two lost files and three index.php?... references that the link rewriter didn't handle correctly.

Database Migration, Part IV

2025-01-21T07:21:00-05:00

We're talking about extracting data from complex relational databases. This is -- in a way -- another case study for my Unlearning SQL book.

We can now explore the details of the data, looking for the original content.

How Joomla! Works

We're avoiding looking at the Jooml! elephant, wandering around the parlor. Yes, we could reverse engineer the PHP to figure out how the database content is used to build the web site. We, however, don't much care about the details.

We have a pile of specific kinds of content we can see. This includes:

A home page with a few articles.
A right sidebar with some articles.
A few pages with dozen or so narrowly-focused articles.
The master collection of articles, neatly organized by category.
Several separate pages of different kinds of links. These are actually articles with the links.
The Kunena Forums.
A Photo gallery that might be from RSGallery2 or JoomGallery.
The Phoca Downloads.

While there are a bunch of Joomla! Modules that seem to be used to define the site organization, there is also a (simpler) tree of Assets. For the most part, it appears we need a "grep"-like tool to dig through the database looking for content. Then we can work out what appears to be the owning asset, bypassing many Joomla! complications.

Doing a grep on the raw data is actually kind of easy.

The "raw" database has the following type definition:

DBTable = list[db_model.DBModel]

type Database = dict[str, DBTable]

We can, then, use database.values() to work all of the tables. And for each table, all of the string columns looking for relevant rows. It shapes up like this:

def row_match(pattern: Pattern, row: BaseModel) -> bool:
    for name in row.model_fields_set:
        val = getattr(row, name)
        match val:
            case str() as text if pattern.search(text):
                return True
            case _:
                pass
    return False

Since the rows are based on pydantic.BaseModel, we can introspect the columns and search the text-based columns to locate all rows that has a column matching the given pattern. We search all of them because there are columns with names like "title", "name", and "alias", all of which seem to have potentially relevant values, and we don't know precisely what the semantics of them are. Here's the containing function that wraps the row_match function:

@staticmethod
def execute(options: argparse.Namespace) -> None:
    """
    Search all str columns of all tables for the pattern.
    """

    # def row_match... (shown earlier)

    database = load_db(options.source)

    Grep.logger.info("grep pattern %r", options.pattern)
    pattern = re.compile(options.pattern)
    matcher = partial(row_match, pattern)
    for cls in database:
        # Could reuse :meth:`where` method for this
        for match in filter(matcher, database[cls]):
            print(type(match), shorten(repr(match), 128))

This is a method of a Grep command:

class Grep(Command):
    """Make a grep-like search through the content."""
    logger = logging.getLogger("Grep")

    @staticmethod
    def config_argparse(
        subparsers: argparse._SubParsersAction, defaults: dict[str, Any]
    ) -> None:
        parsers_grep = subparsers.add_parser("grep", help="grep all tables for a Regex")
        parsers_grep.add_argument("--pattern", "-p", action="store", type=str)
        Command.common_args(parsers_grep, defaults)
        parsers_grep.set_defaults(command=Grep.execute)

The abstract base class is defined like this:

class Command(abc.ABC):
    """CLI Command abstract base class."""

    logger: logging.Logger

    @staticmethod
    @abc.abstractmethod
    def config_argparse(
        subparsers: argparse._SubParsersAction, defaults: dict[str, Any]
    ) -> None: ...

    @staticmethod
    def common_args(parser: argparse.ArgumentParser, defaults: dict[str, Any]) -> None:
        parser.add_argument(
            "source", action="store", type=Path, default=defaults.get("source")
        )

    @staticmethod
    @abc.abstractmethod
    def execute(options: argparse.Namespace) -> None: ...

This provides a tidy package to wrap the grep command so we can create a CLI to poke around in the database looking for the Home Page content, the various pages with narrowly-focused articles, and the specific articles with the links.

This isn't quite enough to locate all of the various forums and galleries. But it gets us started examining the content. There's more -- of course -- but it's all outside the realm of SQL processing.

The Path

There are several steps on this path:

SQL legacy data.
Python extract of SQL data.
Python structures without SQL complications of foreign keys. And -- more important -- with proper hierarchies.

The migration path so far

These first few transformations aren't the goal, of course.

The goal is a directory tree of markdown and images that Hugo can transform into a static web site. The rest of the exploration and migration isn't SQL-related at all. It's a fairly complicated matter of finding the content and restating it in a form Hugo can work with.

Conclusion

We started with a SQL database, and carefully set it aside. We wrote two small applications to get the data out of the SQL database.

scan_db.py -- extracts the table definitions and PlantUML descriptions from the database.
extract_db.py -- extracts the data, writing a TAR file of NDJSON documents with all the database rows.

Once we had the data in a neutral form -- specifically NDJSON documents -- we could create alternative models for the data and preparation steps to populate those models.

This model is an integration part of exploring the data. This means the exploration application evolves until it becomes the migration application.

We start with a skeleton of view_content.py. This is based on a number of Builder classes and a prepare_content() function to get raw data organized into what appears to be a useful model.

The steps in this view_content.py application (and the association model.py) are free of SQL complications.

The conversion process has at least three parts:

Locate the relevant objects. Often, an instance of the Assets class does this. A relevant Assets instance doesn't seem to be universal, though.
Convert the objects for use by a static site generator like Hugo. This turns out to be pretty complicated. There are a number of distinct cases for the different kinds of content: articles, images, downloads, and forum topics. However, since we're done with SQL, these complications don't involve database queries.
Write needed _index.md files so Hugo Sections and Page Bundles will mimic the legacy site's Joomla! presentation.

Database Migration, Part III

2025-01-14T07:21:00-05:00

We're talking about extracting data from complex relational databases. This is -- in a way -- another case study for my Unlearning SQL book.

In Part I, we loaded a database and queried the metadata. In Part II, we extracted the raw tables and loaded up a TAR Archive with NDJSON documents.

We can now move beyond the raw relational data into something more useful.

Things SQL Does Badly

One thing SQL does badly is model hierarchies. (This is one aspect of not handling graphs in general.)

In a hierarchy -- a Directed Acyclic Graph -- there are nodes. A node may have a parent. A mode may have one or more children. A node with no parent is the "root" of the tree. A node with no children is a "leaf" of the tree.

A tree

The point is that the relationships are transitive -- the root has children that have children dot dot dot that have leaves. No arbitrary <h1> to <h6> limit. (Pragmatically, you don't need very many levels. Common SQL hacks impose limits to so a simple SELECT statement and a programming languages like COBOL will work. The WITH clause permits indefinite hierarchies, at the cost of consuming time querying the rows from the database.)

Not Using SQL

In ordinary, in-memory data structures, it makes sense to define tree structures like this:

from pydantic import BaseModel, Field, Json, WrapValidator

class AppModel(BaseModel, from_attributes=True, arbitrary_types_allowed=True):
    @property
    def pk(self) -> int:
        raise NotImplementedError()

    def __str__(self) -> str:
        base = super().__str__()
        return f"{self.__class__.__name__} {base}"

class Assets(AppModel):
    """
    1239 rows
    """

    @property
    def pk(self) -> int:
        return self.id

    id: int  # <<PK>> range 1..1334, in [(1, 1), (2, 1), (3, 1), (4, 1), (5, 1), ... 1234 more]
    parent_id: int  # range 1..941, in [(895, 361), (699, 51), (478, 49), (1, 42), (35, 38), ... 117 more]
    lft: int  # range 1..2475, in [(1, 2), (3, 1), (9, 1), (11, 1), (13, 1), ... 1233 more]
    rgt: int  # <<unique>> range 2..2477, in [(2477, 1), (2, 1), (8, 1), (10, 1), (12, 1), ... 1234 more]
    level: int  # range 1..5, in [(4, 766), (3, 254), (2, 90), (5, 85), (1, 43), ... 1 more]
    name: str  # <<unique>> range 'com_actionlogs'..'root.1', in [('root.1', 1), ('com_admin', 1), ('com_banners', 1), ('com_cache', 1), ('com_checkin', 1), ... 1234 more]
    title: str  # range 'Ar n-Inin (Hull #331)'..'vtest1', in [('Uncategorised', 7), ('Whitby42 #172 [...]', 5), ('General', 3), ('Introduction', 3), ('2008 Rendezvous', 3), ... 1170 more]
    rules: str  # range '{"core.admin":[],"core.mana...'..'{}', in [('{}', 544), ('None', 348), ('{"core.delete":{"...', 81), ('{"core.delete":[]...', 74), ('{"core.delete":[]...', 65), ... 25 more]

    children: list["Assets"] = Field(default_factory=list, repr=False)
    parent: ref["Assets"] | None = Field(default=None, repr=False)
    gallery_catgs: list["Joomgallery_Catg"] = Field(default_factory=list, repr=False)
    galleries: list["Joomgallery"] = Field(default_factory=list, repr=False)
    categories: list["Categories"] = Field(default_factory=list, repr=False)
    content: list["Content"] = Field(default_factory=list, repr=False)
    modules: list["Modules"] = Field(default_factory=list, repr=False)

An Assets instance has children: list["Assets"]. Similarly, an Assets instance may have a weak reference to a parent, ref["Assets"] | None.

It's a weak reference because two mutual references -- parent -> child and child -> parent -- will create a circularity that defeats reference counting. Using weak references adds a bit of fussiness, but otherwise leads to objects that play well with others.

Some of these fields are nonsense. The rest describe the asset tree used by Joomla!

Having an explicit children list attached to each Assets saves going back to the database to do additional queries to find the children of a given asset. Further, it makes it very easy to "walk" the transitive closure of all children under an asset. And, it makes it very easy to locate the transitive closure of all parents of an asset.

How Do We Get There?

Building the the model objects is a two-step process.

Step, the first

Most of the attributes are seeded from db_model objects using a line like the following:

obj = Assets.model_validate(row)

The model_validate() moves data into a new instance of the Assets model. The from_attributes=True means attribute name matching is used; this means our AppModel classes must have attribute names that match the DBModel classes. These have have attribute names that match the original SQL. We have a reasonably transparent mapping because of this constraint.

Step, the second

The relationships don't resolve themselves.

We need to attach children to parents and parents to children. For this, we've defined a Builder class.

class AssetsBuilder(AppModelBuilder):
    """
    @startuml
    hide circle
    skinparam linetype ortho

    entity Assets
    Assets }o-- "parent" Assets
    @enduml
    """

    class AppTable(model.AppTable[model.Assets]):
        pass

    def __call__(self, table: DBTable) -> model.AppTable[model.Assets]:
        items = self.AppTable.build(model.Assets, table)
        for item in items.values():
            if item.parent_id in items:
                items[item.parent_id].add_child(item)
        return items

In SQL world, every Assets row has a parent_id column with a foreign key reference to another Assets. Or a null of some kind, maybe a database NULL, maybe a zero.

(There is not one standard answer to null representation. Don't @ me with it should be NULL. In this case, it isn't NULL, and it doesn't have to be a NULL. It's usually zero. Except in one case that seems to be the result of a bug of some kind.)

(We'll look at the AppTable.build later, for now I want to focus on the hierarchies.)

For each Assets object in items.values(), we need to see if it has a parent. If it does have a parent, we need to as the parent to add this child. This will do two things: add the child to the parent's children list, and also set the parent relationship for each of the children.

def add_child(self, item: "Assets") -> None:
    self.children.append(item)
    item.parent = ref(self)

The `AppTable` class

The final step in the Builder is a the AppTable; a handy structure to manage each collection of objects.

In the long run, this is not required.

In the short run -- where we can't navigate the database -- it's really handy for exploring.

T_AppModel = TypeVar("T_AppModel")


class AppTable[T_AppModel: AppModel](dict[Any, T_AppModel]):
    """
    A mapping from PK id to AppModel instance.
    """

    logger: ClassVar[logging.Logger]

    @classmethod
    def build(
        cls, row_cls: type[T_AppModel], db_table: Iterable[BaseModel]
    ) -> "AppTable[T_AppModel]":
        cls.logger = logging.getLogger(cls.__name__)
        app_table = AppTable[T_AppModel]()
        for row in db_table:
            obj = row_cls.model_validate(row, from_attributes=True)
            if obj.pk in app_table:
                cls.logger.error(
                    "Duplicate key %r, replacing %r", row, app_table[obj.pk]
                )
            app_table[obj.pk] = obj
        return app_table

    def where(
        self, filter_function: Callable[[T_AppModel], bool]
    ) -> Iterator[T_AppModel]:
        """
        A vaguely SQL-like search.
        """
        yield from filter(filter_function, self.values())

This is where we build a model.Assets object from the database db_model.Assets object. Further, we index them by the stated PK so we don't need to search.

The where() method lets us provide a lambda that searches the rows for matching instances.

featured = list(self.content.Content.where(lambda c: c.featured == 1))

This is equivalent to SELECT * FROM content WHERE featured = 1 in SQL. Except it's a lot faster. And a lot more flexible.

This is not heavily used. Most of what we need, we can find with ordinary foreign-key-to-primary-key relationships that use the native Python mappings. A few things, like specific assets that define Joomla! modules and content categories, must be found by name, and will use the where() method.

All the Things

Now that we can unravel the parent-child hierarchies, we can prepare the database for real work.

We'll transform the original SQL-like structures to a module-like namespace that has all the things we want, with their proper relationships. There are 18 tables that seem to have all the content we care about. For now, we're avoiding some of the installed Joomla! extensions.

def prepare_content(database: Database) -> SimpleNamespace:
    content = SimpleNamespace()

    content.Phocadownload_Categories = PhocaCategoryBuilder(content)(
        database["Phocadownload_Categories"]
    )
    content.Phocadownload = PhocaDownloadBuilder(content)(database["Phocadownload"])

    content.Kunena_Categories = KCategoryBuilder(content)(database["Kunena_Categories"])
    content.Kunena_Topics = KTopicBuilder(content)(database["Kunena_Topics"])
    content.Kunena_Messages = KMessageBuilder(content)(database["Kunena_Messages"])
    content.Kunena_Messages_Text = KMessageTextBuilder(content)(
        database["Kunena_Messages_Text"]
    )
    content.Kunena_Attachments = KAttachmentBuilder(content)(
        database["Kunena_Attachments"]
    )

    content.Assets = AssetsBuilder(content)(database["Assets"])

    content.Rsgallery2_Galleries = RSGalleryBuilder(content)(
        database["Rsgallery2_Galleries"]
    )
    content.Rsgallery2_Files = RSFileBuilder(content)(database["Rsgallery2_Files"])

    content.Joomgallery_Catg = JGCatgBuilder(content)(database["Joomgallery_Catg"])
    content.Joomgallery = JGalleryBuilder(content)(database["Joomgallery"])

    content.Categories = CategoriesBuilder(content)(database["Categories"])
    content.Content = ContentBuilder(content)(database["Content"])

    content.Modules = ModulesBuilder(content)(database["Modules"])
    content.Menu = MenuBuilder(content)(database["Menu"])
    content.Weblinks = WeblinksBuilder(content)(database["Weblinks"])

    # The following are of dubious value...
    content.Modules_Menu = ModulesMenuAssoc(content)(database["Modules_Menu"])
    content.Content_Frontpage = ContentFPBuilder(content)(database["Content_Frontpage"])

    return content

Each Builder applies several transformative steps:

Build model objects from db_model objects for the relevant few DBTable objects.
Make AppTable dictionaries from object.pk to object.
Make trees for objects with parent-child relationships.
Resolve other inter-object references.

Once we've got a proper namespace full of objects, we can start to explore it to find the relevant pieces.

Are are the lines we've drawn to distinguish the various parts of our processing.

scan_db.py -- extracts the table definitions and PlantUML descriptions from the database.
extract_db.py -- extracts the data, writing a TAR file of NDJSON documents with all the database rows.
view_content.py -- Builder classes and prepare_content() function to get raw data organized.

The steps in view_content are free of SQL complications.

In the next section we'll look at the conversion process.

There will be three parts:

Locate the relevant objects
Convert the objects for use by a static site generator like Hugo. This turns out to be pretty complicated. However, since we're done with SQL, the complications don't involve database queries.
Write needed _index.md files the mimic the legacy site's Joomla! presentation.

Database Migration, Part II

2025-01-07T13:21:00-05:00

We're talking about extracting data from complex relational databases. This is -- in a way -- another case study for my Unlearning SQL book.

In Part I, we loaded a database and queried the metadata. From this we created Python Table and Column objects that we used to record what we know about the data. These class could also emit metadata in other formats. The other formats include

PlantUML ERD diagrams.
Pydantic class definitions.

So far, these have given us a sense of what the data is.

We've fiddled with the PUML file(s) to create ERD's that seem to capture our initial understandings.

We've got a db_model.py file full of class definitions we can use for further work.

Stumble 2, Extract

We can write a database extract (and database reloader) to work with the NDJSON extracts. Then we can kiss MariaDB goodbye, and stop the service from running on our laptop.

The database metadata includes a lot of tables. We don't want all of them. It's hard to be sure exactly which ones we need, so it pays to be flexible.

What makes sense to me is creating a list of relevant tables in the scan_db.py application. Then we can run it as often as we uncover another table that seems relevant.

The extract can use a function like this to find the tables in the db_model module.

def class_iter(module: ModuleType) -> Iterator[type]:
    other_imported_names = {
        "Decimal",
        "ClassVar",
        "Any",
        "BaseModel",
        "Field",
        "DBModel",
        "__class__",  # SimpleNamespace
    }
    for name in dir(module):
        object = getattr(module, name)
        match object:
            case typing._AnyMeta():  # type: ignore
                pass
            case type() if name not in other_imported_names:
                print("DEBUG", name)
                yield object

This yields the class definitions. Here's the entire list of classes in the module.

classes = list(class_iter(db_model))

Since each of our classes has a query and a from_query() method, getting the data for a given table looks like this:

def get_data(
    connection: mariadb.Connection, cls: type[db_model.DBModel]
) -> list[db_model.DBModel]:
    try:
        with connection.cursor() as crsr:
            crsr.execute(cls.query)
            data = [cls.from_query(row) for row in crsr.fetchall()]
            # pprint.pprint(data)
            return data
        print(cls.__name__, len(data))
    except AttributeError:
        print(f"***UNEXPECTED {cls.__name__}")
        # print(cls.query)
        raise

Execute the table's query. Convert the table's rows to the pydantic model instances. Return the list of instances.

Producing an line in an NDJSON file is delightfully simple with Pydantic.

row.model_dump_json(indent=None)

While we can easily make a bunch of NDJSON files, it offends me to have a whole directory full of files that we're only going to read.

Stumble 3, The Working Files

My first preference was to pickle the data. It's easy to create a dictionary that maps table name to a list of row instances.

We have a common base class

class DBModel(BaseModel):
    query: ClassVar[str]

    @classmethod
    def from_query(cls, row: tuple[Any, ...]) -> "DBModel":
        raise NotImplementedError()

This means the database is

type Database = map[str, list[DBModel]]

We can pickle this mapping and recover the entire thing.

It's really quite elegant. And pretty fast, too.

Big Problem

There's a big problem.

The data is essentially wired to specific class definitions. Change the class too much, and the data no longer loads from the pickle.

Since this is exploratory, we won't know anything up front. We need more flexibility.

Course Correction

Pickle didn't work. What's next?

Make a TAR Archive (compressed) with all the NDJSON members. The extra CPU of compression is more than offset by the reduced time to do physical I/O on a smaller file.

Here's the TAR write:

def save_data(
    content_path: Path,
    archive: tarfile.TarFile,
    cls: type[db_model.DBModel],
    data: list[db_model.DBModel],
) -> None:
    detail = (content_path / cls.__name__).with_suffix(".ndjson")
    with open(detail, "w") as detail_file:
        for row in data:
            print(row.model_dump_json(indent=None), file=detail_file)
    info = archive.gettarinfo(detail, arcname=cls.__name__)
    print(info.name, info.size)
    with open(detail, "rb") as detail_file:
        archive.addfile(info, detail_file)
    detail.unlink()

The idea is to write a table of data to a file at the detail path, add this to the open TAR archive, and then delete the detail entry. This leaves us with a TAR file filled with the extracted database rows. Further, it's in JSON notation, so we can fiddle with the schema.

The original SQL backup was 167,885,194 bytes.

The useful subset of data, compressed, is 28,815,360 bytes. 17% of the original. About 1/5 the size.

Simply rebuilding the original db_model collections goes quickly. And I can make small changes without breaking things.

It turns out, I don't want to make small changes.

The Raw Database Model

The real model is derived from the class definitions in the db_model module. I don't need the SQL query. Or the from_query() method. The db_model module is full of classes that have these features, but doesn't need them.

To move on in the data pipeline, I need to reload data using db_model class definitions. Later, we'll start transforming this data as we undo the mischief of normalization. Loading the data for exploration is done by this function:

def load_db(source_path: Path) -> Database:
    logger = logging.getLogger("load_db")
    database: Database = {}

    with tarfile.open(source_path, "r") as archive:
        for item in archive.getmembers():
            cls = getattr(db_model, item.name)
            raw_file = archive.extractfile(item)
            if raw_file:
                reader = io.TextIOWrapper(raw_file)
                rows = DBTable(cls.model_validate_json(line) for line in reader)
                database[item.name] = rows
            else:
                logger.error("archive item %r as no content", item)
    return database

I can read and validate the NDJSON documents with the following generator expression.

(cls.model_validate_json(line) for line in reader)

We can use the Pydantic model_validate_json() method to create my target object. I can now adjust attribute definitions in a limited way, and add new attributes.

First, however, we need to take a look at the DBTable class.

The DBTable Collection

For the purposes of reading the db tables back in from the TAR archive, we have these two definitions:

class DBTable(list[db_model.DBModel]):
    pass


type Database = dict[str, DBTable]

Yes, DBTable is a list. It could do more. It turns out, nothing more is needed.

Once we've got a dictionary full of lists of data, we need to restructure it into a more useful form. This means drawing some more lines to distinguish the various parts of our processing.

scan_db.py -- extracts the table definitions and PlantUML descriptions from the database.
extract_db.py -- extracts the data, writing a TAR file of NDJSON documents with all the database rows.

Reading and "preparing" the data for deeper analysis is a separate application.

It took a few mistakes to learn that the db_model schema must match the database. We really can't tweak it. We need to build a model derived from this model.

In the next section we'll define the model.AppModel class for objects derived from the db_table.DBModel objects. These AppModel classes can have a number of additional fields and distinct annotated types and validation rules. This makes it easy to build them using a line like the following:

obj = row_cls.model_validate(row, from_attributes=True)

the model_validate() moves data into the row_cls model. The from_attributes=True means attribute name matching is used. This means our AppModel classes must have attribute names that match the DBModel classes. These have have attribute names that match the original SQL. We have a reasonably transparent mapping because of this constraint.

Database Migration, Part I

2024-12-31T13:21:00-05:00

Let's talk about extracting data from complex relational databases. This is -- in a way -- another case study for my Unlearning SQL book.

This case study is about legacy database preservation: we want the data. We don't want the code.

Let's reach back 20 years, when packages like Joomla! were -- essentially -- the only way to have interactive, peer-maintained content. Facebook barely existed before 2005. Yahoo! Groups was what we had been using to share information as a kind of "social media" up until 2008, when the Joomla! site was started.

We have web content reaching back almost two decades, most of it in a Jooma! database. Some of the Joomla! articles are extracts stretching from the Yahoo! groups. Currently, the user base interacts through Facebook, rarely touching this complicated web content.

We want to preserve what's in Joomla! and migrate it into a simpler publishing system like Hugo (https://gohugo.io) We intend to sacrifice some of the Kunena features. However. Since no one is interacting via the Kunena forums, this isn't a real sacrifice.

Let's work through the conversion of data from a relational database to a directory of Markdown files. One stumbling step at a time.

Stumble 1: What do we have?

We have a snapshot of the database. It's MariaDB/MySQL, and the snapshot is a big SQL script. We can install MariaDB on our laptop and run the script.

We have the data.

What do we have?

The Joomla! PHP world has a bunch of admin apps and tools to peek at the database. We're not interested in these because they run on the server, which we'd like to disconnect from.

First, we have tables

SELECT TABLE_NAME, TABLE_ROWS
FROM   information_schema.TABLES
WHERE  TABLE_SCHEMA = "testdb"
AND    TABLE_TYPE = "BASE TABLE"

We can extract some details and helps us discover the Joomla! naming convention. There's a prefix in front of each table name. j930_ and jos_ seem to be prefixes used by previous admins to preserve some data for a test instance (or something.) While we don't know this, the overlapping names and smaller (or zero) row counts suggest these don't matter. It's the j500_ tables that matter.

It's best to fuss about with this initial peeking in a notebook, just uncover the metadata, and see what's going on.

Plan to abandon the notebook.

Here's how we prefer to deal with the schema

@dataclass
class Table:
    name: str
    num_rows: int | None = field(default=None)
    columns: dict[str, Column] = field(default_factory=dict)

    query: ClassVar[str] = """
        SELECT TABLE_NAME, TABLE_ROWS
        FROM   information_schema.TABLES
        WHERE  TABLE_SCHEMA = "testdb"
        AND    TABLE_TYPE = "BASE TABLE"
    """

    @classmethod
    def from_query(cls, row: tuple[Any, ...]) -> "Table":
        return Table(name=row[0], num_rows=row[1])

A reusable function can execute the query, and then use the from_query() method to build rows of the Table class.

The column metadata is a separate dataclass:

@dataclass
class Column:
    name: str
    type_name: str
    size: int
    python_type: str | None
    optionality_type: OptionalityType
    domain_type: DomainType | None = field(default=None)
    not_used: bool = field(default=False)
    val_min: Any = field(default=None)
    val_max: Any = field(default=None)
    val_common: list[tuple[Any, int]] = field(default_factory=list)
    val_cardinality: int = field(default=0)

    query: ClassVar[str] = """
        SELECT COLUMN_NAME, IS_NULLABLE, DATA_TYPE, CHARACTER_MAXIMUM_LENGTH
        FROM information_schema.COLUMNS
        WHERE TABLE_SCHEMA = "testdb"
        AND   TABLE_NAME = ?
    """

    @classmethod
    def from_query(cls, row: tuple[Any, ...]) -> "Column":
        assert row[2].upper() in PYTHON_TYPE, f"unknown type {row[2]}"
        return Column(
            name=row[0],
            type_name=row[2],
            size=row[3],
            python_type=PYTHON_TYPE.get(row[2].upper()),
            optionality_type=(
                OptionalityType.REQUIRED if row[1] == "NO" else OptionalityType.OPTIONAL
            ),
        )

The various val_... attributes are populated later. We need to query the data to get the minimum value, maximum value, the five most common values, and a general sense of the overall cardinality (is each value unique?)

We can see what the columns mean when we see sample data.

We'll do this with methods that are part of the Table dataclass.

def rows(self, connection: mariadb.Connection) -> Iterator[dict[str, Any]]:
    query = f"""SELECT * FROM {self.name}"""
    with connection.cursor() as c:
        c.execute(query)
        column_names = [col[0] for col in c.description]
        for rt in c.fetchall():
            row_dict = dict(zip(column_names, rt))
            yield row_dict

def set_domain(self, connection: mariadb.Connection) -> None:
    raw_domains = collections.defaultdict(collections.Counter)
    for row in self.rows(connection):
        for name in self.columns.keys():
            raw_domains[name][row[name]] += 1
    for name, col in self.columns.items():
        col.set_domain(raw_domains[name])

These were not shown above to keep the initial definition of Table clear.

The set_domain() method for a Table gets all of the data, and then -- column by column -- sets the data domain for the column. For vast tables, these has to be approached in a different. For this databsae, with under 10,000 rows in any given table, fetching all the rows works out quite nicely.

This relies on a set_domain() method for the Column class. Like this:

def set_domain(self, frequencies: collections.Counter[Any]) -> None:
    values = list(filter(None, frequencies.keys()))
    if values:
        self.val_min = min(values)
        self.val_max = max(values)
        self.val_common = frequencies.most_common(5)
        self.val_cardinality = len(frequencies)
        if all(f == 1 for val, f in frequencies.items() if val is not None):
            self.domain_type = DomainType.UNIQUE
        else:
            self.domain_type = DomainType.NON_UNIQUE
    else:
        self.not_used = True

There are two enum class definitions that are part of this, also.

class DomainType(StrEnum):
    UNIQUE = "unique"
    NON_UNIQUE = "non-unique"


class OptionalityType(StrEnum):
    OPTIONAL = "nullable"
    REQUIRED = "non-nullable"

With this, we can build a schema -- a collection of Table definitions -- from the database. We can then view the beast as a whole.

Which means what?

How do we Explore?

Step 1 is to build some ERD diagrams.

We can add a method to Table to expose it as a Plant UML entity:

def as_puml(self) -> str:
    buffer = io.StringIO()
    with contextlib.redirect_stdout(buffer):
        print(f"entity {self.name} {{ /' {self.num_rows} rows '/")
        for col in (c for c in self.columns.values() if not c.not_used):
            flag = (
                "* "
                if col.domain_type == DomainType.UNIQUE
                and col.optionality_type == OptionalityType.REQUIRED
                else ""
            )
            print(
                f"  {flag}{col.name} {col.type_name}({col.size}) /' {col.optionality_type}, {col.domain_type}, range {col.value_range}, {col.value_common} '/"
            )
        print("}")
    return buffer.getvalue()

The output is a block of text like this:

entity j500_assets { /' 1239 rows '/
  * id INT(4) /' non-nullable, unique, range 1..1334, [(1, 1), (2, 1), (3, 1), (4, 1), (5, 1), ... 1234 more] '/
  parent_id INT(4) <<FK>> /' non-nullable, non-unique, range 1..941, [(895, 361), (699, 51), (478, 49), (1, 42), (35, 38), ... 117 more] '/
  --
  lft INT(4) /' non-nullable, non-unique, range 1..2475, [(1, 2), (3, 1), (9, 1), (11, 1), (13, 1), ... 1233 more] '/
  rgt INT(4) /' non-nullable, unique, range 2..2477, [(2477, 1), (2, 1), (8, 1), (10, 1), (12, 1), ... 1234 more] '/
  level INT(4) /' non-nullable, non-unique, range 1..5, [(4, 766), (3, 254), (2, 90), (5, 85), (1, 43), ... 1 more] '/
  name VARMYSQL(200) /' non-nullable, unique, range 'com_actionlogs'..'root.1', [('root.1', 1), ('com_admin', 1), ('com_banners', 1), ('com_cache', 1), ('com_checkin', 1), ... 1234 more] '/
  title VARMYSQL(400) /' non-nullable, non-unique, range 'Ar n-Inin (Hull #331)'..'vtest1', [('Uncategorised', 7), ('Whitby42 #172 [...]', 5), ('General', 3), ('Introduction', 3), ('2008 Rendezvous', 3), ... 1170 more] '/
  rules VARMYSQL(20480) /' non-nullable, non-unique, range '{"core.admin":[],"core.mana...'..'{}', [('{}', 544), ('None', 348), ('{"core.delete":{"...', 81), ('{"core.delete":[]...', 74), ('{"core.delete":[]...', 65), ... 25 more] '/
}
note bottom: 1239 rows

This isn't too pretty, but when the PlantUML tool finishes with it, it's a tidy little box in an ERD. The long /'...'/ comments are not shown in the diagram. They're helpful in the file because they show us the domain of values in a column.

Once we have all of the entities in a .puml file, we can insert relationships. We can also partition the tables into packages to try and discern which ones have interesting content, and which ones are operational overheads.

What We Didn't Do

An important part of this is to not -- emphatically not -- build an ORM layer. We don't really want to try and get ORM class definitions wrapped around a legacy database. It's technically possible. The tables are small, so there may not be profound performance problems.

It's much, much easier to extract that data from the database, and build native Python objects.

What We Will Do

The goal is to have a db_model module with Pydantic BaseModel definitions for the tables we want to preserve. As we'll see in the next section, we can query the database and populate the Pydantic class definitions. We can then dump these Python objects into NDJSON files so we can explore without the overheads of SQL or MariaDB.

The relational model -- and the requirement to normalize -- has decomposed relatively straight-forward objects into a table of tables with primary keys, foreign keys, and equijoin operations. We want to undo the normalization and recreate a more sensible structure in native Python. We really want to have nested for statements without have to create cursors and execute queries.

We want to be able to create dictionaries without the overhead of defining an index.

The Database Model

The starting position is some Pydantic class definitions for the database tables. This is another method of the Table class.

def as_dataclass(self) -> str:
    """Actually, as pydantic DBModel subclass..."""
    buffer = io.StringIO()
    with contextlib.redirect_stdout(buffer):
        column_subset = [c for c in self.columns.values() if not c.not_used]
        keys = set(
            col.name
            for col in column_subset
            if col.domain_type == DomainType.UNIQUE
            and col.optionality_type == OptionalityType.REQUIRED
        )
        print(f"class {self.class_name}(DBModel):")
        print('    """')
        print(f"    {self.num_rows} rows")
        print('    """')
        for col in column_subset:
            annotation = (
                f"{col.python_type}"
                if col.optionality_type == OptionalityType.REQUIRED
                else f"{col.python_type} | None"
            )
            key = "<<PK>> " if col.name in keys else ""
            print(
                f"    {col.name}: {annotation}  # {key}range {col.value_range}, in {col.value_common}"
            )
        wrapped_names = [f"`{col.name}`" for col in column_subset]
        print()
        print('    query: ClassVar[str] = """')
        print(f"        SELECT {', '.join(wrapped_names)}")
        print(f"          FROM {self.name}")
        print('    """')
        print()
        print("    @classmethod")
        print(
            f"    def from_query(cls, row: tuple[Any, ...]) -> '{self.class_name}':"
        )
        print(f"        return {self.class_name}(")
        for position, col in enumerate(column_subset):
            print(f"            {col.name}=row[{position}],")
        print("        )")
    return buffer.getvalue()

It writes the definition as Python code. We can assemble a db_model class from these. Once we have that we're in a position to extract the data and build NDJSON files.

This first part, then, is an application with a name like scan_db.py to emit the UML, and the db_model.

We draw a line under this module because it deals with the available metadata. It doesn't do the full extract. Nor does it explore the data prior to migration. The database metadata analysis is something we'd like to isolate, and run rarely.

Unit Testing

While -- in principle -- this is one-0ff software, test cases are essential. We don't need 100% code or logical path coverage. But, we do need enough coverage that we can refactor with confidence.

Documentation

The PUML files document the source database.

We should create a docs directory and put some notes in there about what this is and how to use it.

So far, we have a sense of what the data is.

We've fiddled with the PUML file(s) to create ERD's that seem to capture our initial understandings.

We've got a db_model.py file full of class definitions we can use for further work.

We can write a database extract (and database reloader) to work with the NDJSON extracts. Then we can kiss MariaDB goodbye, and stop the service from running on our laptop.

Better than grep

2024-09-26T09:50:00-04:00

In the process of writing Unlearning SQL, I had a need to extract SQL blocks from Python programs. Of course, I tried grep. It wasn't ideal.

Note

Book is available here:

SQL blocks are -- ideally -- triple-quoted strings. Of course, so are docstrings. This creates additional problems.

The first pass is to try to use grep to track down the triple-quoted blocks. This is a complex regular expression because it spans multiple lines.

The output is a file of text with SQL stateements, docstrings, and quote marks all over the place. It requires much manual cleanup.

Doing the cleanup of the initial grep output is right awful. I need to somehow preserve a multi-line triple-quoted string that starts with a SQL reserved word, and ignore multi-line triple-quoted strings that don't begin with something obvious SQL.

I give up.

Python RE

Using Python to extract the initial strings and create a data structure is a good second step. I can use the same regular expression with the Python re module. I can use Path.glob() instead of shell globbing.

I can now apply a second regular expression to the list-of-strings object to look for SQL words. (There aren't many: CREATE, DROP, INSERT, UPDATE, SELECT, DELETE.)

This is much nicer. But. I can do better.

Python AST

This process is reading valid, tested Python code. The standard library's ast module defines the abstract syntax tree for Python code. This module can identify literal strings in code, and docstrings.

How does this work?

There are three parts.

Define an ast.NodeVisitor subclass.
Parse the module's source to create an AST.
Use the visitor to collect the strings.

We'll look at each in some detail.

First, a note for those unfamiliar with the Visitor design pattern. A tree structure is recursive. Examining each node of the tree can be done with a recursive function to visit a node, then visit all descendents of that node. It's not a complicated function, but, there's a complication.

The node in an abstract syntax tree tend to be a union of a wide variety of types. There will be an ast.Module, an ast.ClassDef, ast.FunctionDev, etc., etc. A "simple" recursive function needs to treat each class distinctly to properly visit all of the children.

The Visitor pattern delegates part of the work to a visitor object that is presented each node of the tree. The visitor object can have methods for each unique type of node, avoiding a complex match-case statement, or a complex if isinstance(node, Whatever)-elif chain.

The Visitor

We're interested in one feature of the abstract syntax tree of Python: the ast.Constant objects where the constant's value is a string. Ideally, this will focus on those strings that start with a SQL keyword.

Here's the starting point:

class StringVisitor(ast.NodeVisitor):
    def __init__(self) -> None:
        self.sql_blocks = []
    def visit_Constant(self, node: ast.AST) -> None:
        match node.value:
            case str() if sql_like(node.value):
                self.sql_blocks.append(range(node.lineno+1, node.end_lineno))
            case _:  # Ignore all other types
                pass

The visit_Constant() method is invoked for each ast.Constant object in the tree. This will find nodes that are strings and pass the sql_like() filter.

This net is a little too fine. It can will capture docstring comments that happen to look SQL-like. It's essentially the same as the grep output with two improvements:

No baffling regular expression.
The output can be a more useful data structure, not a file of lines of text.

While nicer in some respects, it's incomplete. The next step is to add the required feature to ignore docstring comments. There two choices:

Write a better sql_like() function.
Exclude any string constant that is the first line of the body of a module, class, or function definition.

Option 1 can can involve finding a SQL parser to see if a string really is pure SQL. Or, it can require writing a better regular expression to locate likely SQL statements.

Here's the unit test case:

>>> src_text = """
... def f(c):
...     '''
...         Select the right answers.
...     '''
...     r = c.execute('''
...         SELECT * FROM DUMMY
...     ''')
... """

The docstring comment starts with a SQL keyword. Ugh. It seems kind of daunting to locate a suitable SQL parser. The regular expression to distinguish casual use of SQL-like keywords seems hopeless complicated. There's something better: exclude docstring constants.

Exclusion Rules

The ast.NodeVisitor implementation has a handy feature. This permits an application to choose to visit or skip the subsidiary nodes of an object. When a class overrides a visit_XXX() method, the override can call the self.generic_visit(node) to visit all the children. If the overriding method does not evaluate self.generic_visit(node), the children are not examined.

This is a bit too strict.

Skipping all children of a module, class definition or function definition isn't helpful. The rest of the children could have SQL code. It's important to skip only the very first line of code when this is a string constant. The rest of the code needs to be visited.

I decided to accumulate an "ignore these" set of nodes. This set of nodes will be the first line of code that's also a string constant. The visit_Constant() can then politely decline these nodes.

Here's the visitor class looks:

class StringVisitor(ast.NodeVisitor):
    def __init__(self) -> None:
        self.docstrings = set()  # Docstring nodes at the start of module, class, def
        self.sql_blocks = []
    def exclude_docstring(self, node: ast.AST) -> None:
        if isinstance(node.body[0], ast.Expr) and isinstance(node.body[0].value, ast.Constant):
            self.docstrings.add(node.body[0].value)
        self.generic_visit(node)
    visit_Module = exclude_docstring
    visit_ClassDef = exclude_docstring
    visit_FunctionDef = exclude_docstring
    def visit_Constant(self, node: ast.AST) -> None:
        match node.value:
        case str() if sql_like(node.value):
            if node not in self.docstrings:
                self.sql_blocks.append(range(node.lineno+1, node.end_lineno))
            case _:  # Ignore all other types
                pass

First, I've added a a set of nodes to exclude. (Nodes are immutable, and have a hash value, that's why a set works well for this.)

Second, there's a generic visit function, exclude_docstring(), that handles ast.Module, ast.ClassDef and ast.FunctionDef classes. I can define the needed visti_XXX() methods for these classes to all use the generic exclusion method.

Finally, I need to make sure any ast.Constant node wasn't already excluded because it was the first string in a definition.

With this, I can now parse the source, and apply the visitor. The sql_blocks attribute will have a list of range() objects that point to the SQL statements.

The Parsing and Visiting

Here's the final bit.

tree = ast.parse(src_text, "<test>")
sv = StringVisitor()
sv.visit(tree)
print(sv.sql_blocks)

This will parse the code, then visit the ast.

When this is done, it has a list of range() objects. These ranges can be used to highlight the proper lines of code from the source file. Your application may be different, of course, you may want to simply write the literals to a TOML configuration file, for example.

"Why," you might ask, "do you have range() objects instead of the code?"

The Ultimate Goal

In my specific (and unique) use case, I need to be sure the minted code highlighter will properly format SQL code.

The goal is a book, written using RST markup, with properly colorized SQL examples. That part is easy, what's hard is making sure all examples are unit-tested and really, really work. (I'm fussy like that.)

This means including the source text from a .py file into the book's content.

But. (Big Sigh.)

Minted can't cope with a mixture of Python and SQL, and can't properly color code a few isolated SQL lines plucked from a Python context. (This shouldn't have been too surprising; after I ranted and raved about it, I realized minted must colorize the whole file before a few lines can be selected from it. Context matters when highlighting syntax.)

In my application, I'm creating a parallel file with all non-SQL lines prefixed with the "--" SQL comment marker. This leaves the SQL behind to be seen by minted and properly colored. The Python lines are hidden from minted.

The book can then use an .. literalinclude:: directive that has properly highlighted SQL. The Python file is unit tested, which gives me confidence in the SQL file.

Modern Python Cookbook and Type Hints

2024-09-18T16:03:00-04:00

Modern Python Cookbook — with lots and lots of recipes — is something you might need. Find the results of checking all these recipes here: https://www.amazon.com/Modern-Python-Cookbook-updated-techniques/dp/1835466389

I (reluctantly) switched from using mypy to using pyright to check all of these recipes carefully. The type alias (PEP 695) syntax wasn’t fully supported.

I didn’t find any new problems with pyright, but I did tweak my workflow a little bit to use more of pyright’s linting features.

Follow this for status on mypy https://github.com/python/mypy/issues/15238.

What gets checked?

Using a vague phrase like "everything" is -- obviously -- not quite as helpful as some details.

There are three broad categories of examples:

REPL examples.
Code examples.
Jupyter Lab examples.

Clearly, the REPL examples can't be checked. However. Many REPL examples depend on code definitions that can include checkable hints. The idea is to provide code outside the REPL and then use REPL doctest test cases to demonstrate how the code works. Which means many REPL examples have type hints that aren't shown in detail in the recipe.

All of the ordinary code examples are checked.

The Jupyter Lab examples in Chapter 12 are a mixed bag. Some examples involve an importable module, which can be checked. The notebook itself, though, is difficult to check. One could convert the notebook to a script and the type check the script. This only works when the notebook is careful about not reusing variables.

So that means all the example code in almost all of the chapters is put through pyright to be sure there's nothing obviously sketchy.

I'd like to switch back to mypy. I don't have a great reason, it just seems like it's vaguely "better", measured on some axis I can't articulate.

Modern Python Cookbook, 3e

2024-08-01T13:41:00-04:00

Book Announcement:

https://www.packtpub.com/en-us/product/modern-python-cookbook-9781835466384

There are about 130 recipes in here. Some new. Some revised. All examined (and tested) for Python 3.12.

For the Python folks who are just starting as well as those looking to pick up some more skills, this is for you

The recipes cover a wide variety of topics. It covers both functional and object-oriented programming.

Here's a run-down of the chapters (yes, it's a long list):

Numbers, Strings, and Tuples
Statements and Syntax
Function Definitions
Built-In Data Structures Part 1: Lists and Sets
Built-In Data Structures Part 2: Dictionaries
User Inputs and Outputs
Basics of Classes and Objects
More Advanced Class Design
Functional Programming Features
Working with Type Matching and Annotations
Input/Output, Physical Format, and Logical Layout
Graphics and Visualization with Jupyter Lab
Application Integration: Configuration
Application Integration: Combination
Testing
Dependencies and Virtual Environments
Documentation and Style

Most chapters have recipes that include beginners as well as advanced developers. For example, the first chapter covers the differences between true division and floor division. Some folks get this deeply, where for others, it can feel cryptic at first.

Some chapters are based on others, making them less beginner-oriented. The Functional Programming Features chapter is based on the Function Definitions chapter. The More Advanced Class Design chapter builds on the Basics of Classes and Objects chapter.

Since the whole book is tested (both doctests and unit tests), I'm particularly fond of the Testing chapter. I have read posts from too many folks saying that testing isn't a thing in school, and at work, they're left on their own to work out what and how to test. I think I can help a little with some recipes on using docstrings, testing functions with exeptions, using unittest, using pytest, and mocking external resources.

I’m hoping you’ll find this useful for growing a deeper understanding of Python programming.

Synthetic Data Tool

2024-07-25T14:22:00-04:00

See Synthetic Data.

I've updated the repository with a "Noisy Data" feature.

This will generate bulk data with invalid field values.

It helps with testing ETL pipelines to be sure they will scale to the expected volumes.

Clone https://github.com/slott56/DataSynthTool

Read https://slott56.github.io/DataSynthTool/_build/html/index.html

Functional SQL in Pure Python

2024-07-16T08:14:00-04:00

I've published a framework for doing SQL-like programming in Pure Python -- no database required.

Here: https://github.com/slott56/functional-SQL. See the functional-SQL documentation.

This allows us to transform SQL:

SELECT n.name, v.c2
FROM names_table n, values_table v
WHERE n.code = v.c1

To pure Python:

Select(name=lambda cr: cr.n.name, value=lambda cr: cr.v.c2)
.from_(n=names_table, v=values_table)
.where(lambda cr: cr.n.code == cr.v.c1)

Yes, the Python is longer and cluttered with lambdas.

This produces the same results using a similar algorithm.

Most important, this works with table-like collections of Any class of Python objects.

This implements the essential SQL query algorithm:

having filter()
group-by reduce()
where filter()
select map()
from itertools.product()

This From-Select-Where-GroupBy-Having(Tables) design pattern is very handy. A lot of people think of processing data following this template. There's no reason, however, to inject the overhead of schema and database.

DataSynthTool Repository

2024-07-01T19:22:00-04:00

I've published a framework for making tools to leverage formal schema definitions to synthesize bulk data for performance tuning.

Here: https://github.com/slott56/DataSynthTool.

See the original talk.

See the documentation

Synthetic Data

2024-06-29T09:22:00-04:00

Book? Second draft (with tech review comments addressed) off to editors.

Boat? Still on the hard during Hurricane Season.

Synthetic Data

I've had a passing interest in data synthesis for decades.

Early on in my career, I figured out how the Z/OS IEDBG utility worked. See https://www.ibm.com/docs/en/zos/3.1.0?topic=utilities-iebdg-test-data-generator-program. It synthesized test data according to a number of mainframe-centric rules.

When I started doing DBA work (Ingres, Oracle, DB2, etc.) the need for bulk synthetic data became more profound. Folks would debate optimization questions without the benefit of facts. They'd suppose the optimizer might use an index or a 2NF demormalization might save some time.

But.

Without data, it's all second-guessing the DBMS algorithms.

Note

This is not synthetic data for generative machine learning models.

That's a possible application, but the focus is on databases, ETL, and data analytics.

New Development

The real "shoal water without a chart" problem is new development.

You don't have legacy data with which to benchmark anything.

You only have hand-waving ideas of what the data might look like once the application sees some adoption.

And those are often wishful thinking. (Worse, they can be outright lies by folks seeking investment money.)

What To Do?

All design is driven by data. (Remember, when you switch application code, you preserve the data; it's the real value.)

All scalability problems are data-related.

You can use ordinary "Big-O" complexity analysis to design algorithms that are optimal, but you don't know any actual performance metrics without hardware, software, and -- what's really hard -- data.

The only thing you can do is follow this plan:

Write your Proof-of-Concept, Spike solution.
Synthesize realistic volumes of data with realistic relationships among the values and realistic distributions of values.
Benchmark the performance of the POC/Spike with actual data on actual hardware.
Continuously monitor performance, establishing new benchmarks with better algorithms or data structures.

Without actual performance benchmarks, you're creating two scalability problems:

The immediate problem of "can't start using it because it's too slow."
The subsequent problem of "users are complaining that it's slow."

Okay, How Do I Do It?

Which brings us to tools that synthesize data.

This is not synthetic data for generative machine learning models.

This is synthetic data for database, bulk ETL, and ordinary statistical analysis performance testing.

I've got a bunch of stuff that I'll be posting to Git with an approach that I think might be useful to others.

It based on stuff I've done before. It includes the results of some lessons learned.

More to come.

Git Nightmare

2024-05-07T08:01:00-04:00

My sailing blog, Team Red Cruising is very large: 859 postings over the last few years. 2,334 image files.

This is a LOT of content.

A few of the files (were) Movies, which tend to create immense files.

The whole mess was so big

How Big Was It?

It was so big, the git push command crashed. Repeatedly.

I researched a lot of Stack Overflow answers on dealing with big files. Maybe they were helpful. Maybe they were misleading. I tried a lot of things.

The problem was using the following approach.

First. Get it all organized.
Then. Do a single massive commit of everything.
Now. The git push.

This -- it turns out -- is not a good idea. There are too many files and too many huge files.

Beating to Weather

It seemed sensible to try and preseve the "one big commit with everything in it."

This was -- of course -- a mistake.

It's just too big to preserve. And there's no good reason to preserve it.

It had some HUGE movie and PDF files that are better kept separate from the blog content. Some of the pictures were exported at full size, leading to about 100 image files of 22Mb or larger. The resulting pack files are huge. Far too big to process.

Trying to get the git gc and git repack commands to create some useful approach to uploading one big commit was (in retrospect) a waste of time and brain calories.

Image Resizing

This was fun. A little program using pillow to do Image.reduce() on the 100 or so egregiously large files.

def png_reduce(counts: Counter, path: Path, new: Path) -> None:
    target_image_size =  4_851_306  # about 2202 x 2202.

    counts[path.suffix] += 1
    if not path.suffix.lower() in {".png"}:
        return
    try:
        with Image.open(path) as image:
            if image.width * image.height < target_image_size:
                counts['small'] += 1
                return
            counts['reduce'] += 1
            factor = int((image.width * image.height) / target_image_size + 0.5)
            target_width = int(image.width / factor + 0.5)
            target_height = int(image.height / factor + 0.5)
            reduced = image.reduce(factor)
            # Save reduced Image in separate directory. These can then be moved to replace the originals.
            target = new / path.name
            reduced.save(target)
        print(
            f"Reduce {path.name} {path.lstat().st_size // 1024 // 1024}M:"
            f" factor={factor} from {image.width}×{image.height} to {target_width}×{target_height}"
        )
    except UnidentifiedImageError as ex:
        counts['exception'] += 1
        print(path.name, ex)

The target image size was a kind of guess. I divided the range of sizes into 64 bins. I counted the number of files in each bin to see where the various size lumps occurred. The bin with 4,851,306 seemed to be on the line between a lot of small files and a few large files.

Tack to a New Course

What's the alternative?

Here's what worked.

git reset back to empty. (There was only one commit, so this was easy.)
Put in the overhead files: README.rst, requirements.txt, the Pelican configuration files, Makefile, etc.
Commit this. And push.
Put in the text content files, all 859 of them. Plus the handful of non-blog pages and photo albums.
Commit this. And push.
The images can be broken into 5 batches of about 400 files. Commit and push each of these.

At the end of the day, it's all there, but it took 7 commits and 7 separate git push operations to get there.

It went really fast. Error free. No drama.

Doubly aggravating because this could have been completed two days ago.

The Finish Line

Now, I can do ordinary git pull and git push from any computer.

I can login to the hosting service from any computer, do a git pull and a make publish, and the site is updated.

The point is to be able to use an iPad to edit content and leverage my hosting service to do a little more than serve static HTML.

DBLock Context Manager

2024-03-10T08:01:00-04:00

Consider, for a moment, the shelve and dbm packages for storing things in a “database.” Built-in. Lightweight. The database is essentially a mapping from identifiers to objects. It can be quite nice.

The shelve module directly puts Python objects in a file. It’s an ideal database structure for Python, with relatively little overhead.

If you don’t like using pickle, you can use the underlying dbm with something like Pydantic for class definitions. This means explicitly serializing a representation of object state as bytes before stuffing them into the dbm-managed mapping. Pydantic class definitions can deserialize the bytes and recover the object's state.

With a little effort at designing keys, these provide a persistent mapping for arbitrarily complex objects. (Simple UUID's are nice, but sometimes it helps to provide a key with two parts: collection name and identifier.)

Why?

It lets you read and write objects without the complexity of an ORM layer and a SQL database.

This is often really helpful. But. What about concurrent writes in a multiprocessing context? The shelve page is clear:

The shelve module does not support concurrent read/write access to shelved objects. (Multiple simultaneous read accesses are safe.) When a program has a shelf open for writing, no other program should have it open for reading or writing. Unix file locking can be used to solve this, but this differs across Unix versions and requires knowledge about the database implementation used.

(Emphasis mine.) Okay. We get it; this is BYOL™ -- Bring Your Own Locking.

Which — it turns out — isn’t trivial. See https://en.wikipedia.org/wiki/Readers–writer_lock for information on multiple readers and single writers.

The Single Writer Locking Problem

We can use the fcntl module to lock files. This module (and the Linux OS) offers exclusive locks and shared locks.

(For Windows folks, get a package like portalocker or use the functions in pywin32.)

The objectives are these:

Take out an exclusive lock for transactions that will update the database.
Take out a shared lock for transactions that read the database.

This Shared vs. Exclusive locking is elegant, but also habors a small problem.

Consider a web server. Each GET transaction acquires a shared lock, reads the data, prepares the response, and releases the shared lock. Because the shared lock prevents a writer from getting an exclusive lock, the data is untouched for the duration of the transaction. There's no possibility of data file corruption mid-transaction because of a concurrent write.

Each POST/PUT/PATCH/DELETE transaction acquires an exclusive lock, reads and writes the data, prepares a response, and releases the exclusive lock. This forces the writer to wait for readers to finish. It prevents any reader from seeing incomplete or uncreadable files.

The underlying dbm file is updated in one smooth, atomic event. Everyone sees the data in a consistent state at all times.

Yes, it's coarse-grained whole-database level locking. The point was to avoid the overheads of a huge SQL server. A RESTful service can read and write local files. Emphasis on the read. Why have a super-elaborate database server to provide rows and tables that have to be assembled into JSON documents? Maybe just read the JSON document from the database and reply with it.

But.

There's a problem: a parade of readers can prevent the writer from getting in.

time	action	lock count
T0	reader 1 starts	1 shared lock
T1	reader 2 starts	2 shared locks
T2	writer 3 waiting for an exclusive lock...
T3	reader 2 finishes	1 shared lock
T4	reader 4 starts	2 shared locks
T5	reader 1 finishes	1 shared lock

And so it goes: a sequence of overlapping readers will starve the writer. This is called Livelock, and — while rare — it’s not impossible.

Preventing Livelock

One algorithm for preventing livelock is to have a “pending writer queue” that the readers have to acknowledge.

Think of a velvet rope to get into the most exclusive club in town.

The writer talks to the bouncer, and the line of readers is stopped. No one gets in. Once the club is empty of readers, the writer is allowed in and has the place to themself. When they’re done writing, then they’re out of there, so the readers can crowd the place again.

This is sometimes called the “preferred writer” solution. It’s unfair by design because many web requests are read requests; write requests are less common. Overall performance depends on getting any write out of the way as soon as possible. There are other variants that are more equitable, but also a bit more complicated.

To prevent livelock, we need some kind of shared queue to broadcast to all concurrent processes that there’s a writer waiting.

We can do this using two lock files:

A “queue” lock. This is always Exclusive. It's always acquired first. In effect, it's a one-element queue.
The “working” lock. The working lock is either Shared or Exclusive.

The queue lock defines a mutual exclusion zone of code (called a mutex) where at most one process is able to run. The idea is readers enter the mutex, get their shared working lock, and leave the mutex. Once they have their shared lock, they can loiter, doing whatever it is they need to do.

When a writer enters the mutex, they have to wait for their exclusive working lock before they leave the mutex. This stops the readers.

Here are some play-by-play views for a number of scenarios.

Reader following Readers

Readers can get shared access freely.

time	action	queue lock	working lock
T0	reader 1 acquire queue	1 ex
T1	reader 2 waiting	1 ex
T2	reader 1 acquire working	1 ex	1 sh
T3	reader 1 release queue	0 ex	1 sh
T4	reader 2 acquire queue	1 ex	1 sh
T5	reader 2 acquire working	1 ex	2 sh
T6	reader 2 release queue	0 ex	2 sh
T7	reader 1 release working	0 ex	1 sh
T8	reader 2 release working	0 ex

And so it goes, readers acquiring and releasing working locks.

Reader following Writer

If there's a writer, the reader is forced to wait until the writer releases their exclusive lock.

time	action	queue lock	working lock
T0	writer 1 acquire queue	1 ex
T1	reader 2 waiting	1 ex
T2	writer 1 acquire working	1 ex	1 ex
T3	writer 1 release queue	0 ex	1 ex
T4	reader 2 acquire queue	1 ex	1 ex
T5	reader 2 waiting	1 ex	1 ex
T7	writer 1 release working	1 ex
T8	reader 2 acquire working	1 ex	1 sh
T9	reader 2 release queue	0 ex	1 sh
T10	reader 2 release working	0 ex

Yes. Traffic will back up waiting for a writer. If this is a problem, then finer-grained locking is required. This can lead to the possibility of deadlocks; proceed with caution and consider sharding the data to avoid contention for locks,

Writer following Reader

If there's a reader, the writer is forced to wait before they can get their exclusive lock.

time	action	queue lock	working lock
T0	reader 1 acquire queue	1 ex
T1	write 2 waiting	1 ex
T2	reader 1 acquire working	1 ex	1 sh
T3	reader 1 release queue	0 ex	1 sh
T4	writer 2 acquire queue	1 ex	1 sh
T5	writer 2 waiting	1 ex	1 sh
T7	reader 1 release working	1 ex
T8	writer 2 acquire working	1 ex	1 ex
T9	writer 2 release queue	0 ex	1 ex
T10	writer 2 release working	0 ex

Clearly, if there are a lot of readers, the writer waits a long time for them all to finish. Some more clever lock definitions permit an upper bound on the number of locks that can be acquired.

Our goal, however, is simplicity.

Python Implementation

This is intended to be used with Flask.

from pathlib import Path
from flask import Flask

class DBLock:

    def __init__(self, app: Flask | None = None) -> None:
        if app:
            self.init_app(app)

    def init_app(self, app: Flask) -> None:
        self.lock_path = Path(cast(str, app.config.get("DB_LOCK_FILENAME", "dblock")))
        self.queue_path = self.lock_path.with_suffix(".dbqueue")
        self.thread_local = threading.local()

This could be refactored to work outside a Flask-specific context.

The thread_local storage is required to track each thread's unique open file handles.

The essence is acquiring a lock and releaing a lock. The "lock mode" is one the fcntl.LOCK_EX or fcntl.LOCK_SH values.

def acquire(self, lock_mode: int) -> Self:
    if "lock_file" not in self.thread_local.__dict__:
        # Enter Queue Mutex to acquire a database lock.
        queue_file = self.queue_path.open("w+")
        fcntl.flock(queue_file, fcntl.LOCK_EX)
        self.thread_local.lock_file = self.lock_path.open("w+")
        fcntl.flock(self.thread_local.lock_file, lock_mode)
        # Exit Queue Mutex. Permits another thread (or process) to acquire a lock.
        fcntl.flock(queue_file, fcntl.LOCK_UN)
    return self

def release(self) -> None:
    if "lock_file" in self.thread_local.__dict__:
        fcntl.flock(self.thread_local.lock_file, fcntl.LOCK_UN)
        self.thread_local.lock_file.close()
        delattr(self.thread_local, "lock_file")

The acquire and release are the algorithm described above. An exclusive lock defines a system-wide Mutex for all threads and processes. The working lock is either shared or exclusive.

The cleanup on release undoes the lock, closes the file to release any OS resources, and also purges the thread_local to make sure there's no confusion about the state.

Some useful overheads:

close = release

def is_locked(self) -> bool:
    # print(f"is_locked: {self.thread_local.__dict__=}")
    return "lock_file" in self.thread_local.__dict__

If we provide a close() method, then the contextlib.closing() context manager can be used.

The is_locked() method can be helpful to know the state of the lock. It's far better to use the with statement to define a context that eliminates any doubt.

While this can be used with contextlib functions, it seems helpful to provide explicit context management.

def __enter__(self) -> Self:
    return self

def __exit__(
    self,
    exc_type: type[Exception],
    exc_val: Exception,
    exc_tb: TracebackException,
) -> Literal[False]:
    self.release()
    return False

And. Two convenience methods to avoid having to muck around with fcntl.LOCK_SH and fcntl.LOCK_EX.

def shared(self) -> Self:
    """
    Context manager, equivalent to::

        with dblock.acquire(fcntl.LOCK_SH):
            pass
    """
    self.acquire(fcntl.LOCK_SH)
    return self

def exclusive(self) -> Self:
    """
    Context manager, equivalent to::

        with dblock.acquire(fcntl.LOCK_EX):
            pass
    """
    self.acquire(fcntl.LOCK_EX)
    return self

The goal is to have relatively lightweight code like the following.

Some Flask app setup:

dblock = DBLock()
dblock.init_app(app)

Within a GET view function:

with dblock.shared():
    with dbm.open(some_file) as db:
        item = SomeClass.model_validate_json(db[your_key_here])

Within a POST/PUT/PATCH/DELETE view function:

with dblock.exclusive():
    with dbm.open(some_file, flag='c') as db:
        db[item.id] = item.model_dump_json().encode('utf-8')

By acquiring an exclusive access lock, all changes will be saved reliably and predictably: an atomic state change.

And yes, the explicit model_validate_json() and model_dump_json() is wordy. I use a DB class to conceals these details.

Conclusion

We can use dbm as a dictionary-like repository of objects serialized as JSON.

We have the benefits of a fancy relational database without the overheads.

Functional Python and Lambdas

2024-02-09T08:01:00-05:00

I saw a confusing post on https://fosstodon.org.

I won't link to it, but I will quote it because it repeats some common misconceptions.

I have some iterator chain (in reality longer and more complex than this example).
And now in one or more steps, I need to add an extra operator.
Going from `foo.map(thing).filter(condition).reduce(collector)` to

foo.map({
  thing
  if condition {
      other_thing
  }
).reduce({
  setup
  collector
  logger
})

The design of Python's iterators make this very hard.

Because a lambda cannot easily contain multiple lines, conditions or statements.

There are two unrelated misconceptions here. One's an minor error, the other is a nuanced design choice. We'll look at the minor error first, since it's a common misconception about Python lambdas.

And. I cover these in Functional Python Programming

Lambda and Lines of Code

A Python lambda has a single expression. This precludes any statements.

An expression must be complete on a single logical line of code. Because of the way ( and ) must balance, it can span multiple physical lines of code.

a = list(
    map(
        lambda x: (
            (
                3 * x + 1
            )
            if
            (
                x % 2 == 1
            )
            else
            (
                x // 2
            )
        ),
        range(10)
    )
)

A lambda can be quite long. Use ( and ) to enclose it.

Design Choices

The Ruby foo.map(m).filter(f).reduce(r) has two renderings in Python.

Nested functions. This is bad, but we'll look at it.
Chains of generator expressions. I call them stacks. This is good.

Let's take a concrete example, one that we can unit test. We'll look at it first as nested functions to see how bad it can be.

Nested Functions

Here's are two functions to generate some complicated data.

hotpo = (
    lambda x:
        (x * 3 + 1) if (x % 2 == 1)
        else (x // 2)
)
hotpo_run = (
    lambda x, r:
        r + [x] if x == 1
        else hotpo_run(hotpo(x), r + [x])
)

These are two functions that we've defined as lambdas for no particularly good reason.

Let's use a mapping to compute a range of values. This is a bit like range(1, 11).map(lambda...) in Ruby.

list(
    map(
        lambda n: hotpo_run(n, []),
        range(1, 11)
    )
)

We don't want the lists. We want the lengths of the lists.

list(
    map(len,
        map(
            lambda n: hotpo_run(n, []),
            range(1, 11)
        )
    )
)

Not that it matters much, but let's add a filter.

list(
    filter(
        lambda x: x > 0,
        map(len,
            map(
                lambda n: hotpo_run(n, []),
                range(1, 11)
            )
        )
    )
)

Note

BTW, the answer is [1, 2, 8, 3, 6, 9, 17, 4, 20, 7].

Without a cache of some kind. It takes quite a while to compute more than a few results.

Let's reduce this to find the largest value.

Clearly max() will work, but, for the sake of matching the Ruby, let's build max() as a reduce().

from functools import reduce
reduce(
    lambda a, b: a if a > b else b,
    filter(
        lambda x: x > 0,
        map(len,
            map(
                lambda n: hotpo_run(n, []),
                range(1, 11)
            )
        )
    )
)

The answer is 20. What's important is the function-application version of the Ruby.

foo.map(m).filter(f).reduce(r)

reduce(r, filter(f, map(m, source)))

Folks don't like reading the Python right-to-left. When you spread it into multiple lines it has to be read from bottom-to-top. Or maybe from inside to outside. This nested function version is not widely used.

We can do better with a stack of generator expressions.

Stacked Generators

We'll start with the original two lambdas, hotpo() and hotpo_run().

runs = map(
        lambda n: hotpo_run(n, []),
        range(1, 11)
    )
lengths = map(len, runs)
positive = filter(lambda x: x > 0, lengths)
maximum = reduce(
    lambda a, b: a if a > b else b,
    positive
)

This reads from start to finish in an understandable fashion.

We can easily add steps in the middle of this.

The downside of adding steps is the intermediate results have names.

When we want to insert a step, we have to also modify the step after to use the new intermediate results.

The upside is the intermediate results have names. These describe what's going on. I really like this approach.

Hey, Wait

Yes, this is related to the Collatz Conjecture. See https://projecteuler.net/problem=14.

The definition of the hotpo_run() function isn't conducive to creating a cache. We can rewrite it, easily, into a function that builds a list from a single argument value. This works better with a cache.

from functools import cache
hotpo_run = cache(
    lambda x: [1] if x == 1 else [x] + hotpo_run(hotpo(x))
)

This necessitates a change to the pipeline to deal with the slightly different parameter signature for the hotpo_run() function.

runs = map(
        hotpo_run,
        range(1, 11)
    )

The rest is the same. Second verse same as the first verse.

lengths = map(len, runs)
positive = filter(lambda x: x > 0, lengths)
maximum = reduce(
    lambda a, b: a if a > b else b,
    positive
)

This computes almost instantly.

What's important is the sequence of map()-filter()-reduce() functional operations is better expressed as a sequence of generator expression statements. I like to call it the "Stack of Generators" design pattern. It has much of the expressive power of Ruby, with all of the flexibility we desire.

Note

And yes, that's still not a solution to Euler 14.

Euler 14 wants this: "Which starting number, under one million, produces the longest chain?".

We need to change the result of the runs generator to be a (run, starting value) two-tuple.

runs = ((hotpo_run(n), n) for n in range(1, 11))

runs = map(lambda n: (hotpo_run(n), n), range(1, 11))

The lengths generator needs to be modified to be lambda r_s: (len(r_s[0]), r_s[1]).

Drop the filter(). It's only here to match the original conversation.

Replace the reduce() with a simpler max().

More Reasons to Stop Bash-ing

2023-12-19T08:01:00-05:00

There are many good reasons to use shell scripts. Mostly, a script can be useful when it's an alias that launches an application. Beyond that, I have doubts.

BLUF

Incumbency is a popular argument for bash.

It's not a good argument, however.

Use invoke and you'll be much happier.

Background

See When You Should Use Bash Over Python.

I'll start with the three "expressiveness" points.

Syntax: Python code is longer. While true, this isn't a reason to use bash. I have to reject this for two reasons.
- No one wins at code golf. Shorter code isn't better by any metric other than size. Bash syntax hides important details.
- The argument starts from the notion that there's a "better" way to express complicated structures, and the bash reflects better. Python, by being more explicit, is less good.
Runtime: Shell script interpreters are ubiquitous. True. Not a compelling argument, when we consider that bash scripts are untestable.
Semantics: There is a cognitive cost in converting bash to Python. Correct. Easy to avoid by avoiding the confusing and opaque bash abstractions.
- The argument (again) starts from the notion that the bash abstraction is a standard against which other languages -- by virtue of being different -- aren't as good.

The argument for bash is incumbency. Bash is installed, and because it's installed, it's better.

"Bash’s longevity is rooted in core strengths that still resonate in the technology industry". I suggest the longevity is due entirely to it's incumbency. It's not the best choice for anything. It's a handy default choice because it's already installed.

And. There's no easy way to unit test.

Let's move on to the other six reasons.

Some reasons for using bash

Here are the the detailed reasons for rejecting Python. Most of this isn't persuasive. It's mostly about the incumbency of bash.

Mastery Matters. Parts of this argument are true. Bash scripts seem to be uniformly bad because bad is a permitted style. They could be better, making use of clever things like functions and their obscure semantics.

This doesn't make bash better. It only says that a lot of people write bad scripts. A lot of bash-bashing stems from seeing so many bad scripts.
Bash is Everywhere. True. Incumbency may be helpful under certain situations. It's like learning how to compute logarithms so you can then add them to avoid multiplication. Yes. It does work. However. Calculators exist on this timeline; it's no longer 1617.
- "Sidestepping the discussions about the Python version to pick"? What discussion? Is this a "Python 2 v. Python 3" question? That's been answered.
- "the best way to install Python on a given environment"? Most linux distros have Python ready-to-go. That's best.
- "gymnastics to keep dependencies and environments in check"? This isn't hard, actually. Almost anything bash-related is in the standard library plus a few add-ons line psutil. For a very complicated application with a tall stack of poorly-chosen dependencies, there's work involved. That application with a complicated set of installs isn't doing bash-like things, though.
- "fragmentation of Python runtime versions"? What fragmentation? Python is popular, and evolves quickly. Is evolution to new versions some kind of problem?
- "mutually exclusive dependency matrix"? Callback to gymnastics. A tall stack of poorly-chosen dependencies is an edge case. It's not the sweet spot for admin tasks often written as bad, untestable bash scripts.
Secured Production Environments. This is hard. None of these difficulties are Python-related, however. It applies to every single application in the environment. Java requires installs for develolpers, too. So go Go and Rust. Air-gapped systems are hard to build.
- "whatever Python runtime environment you lock into a production environment"? Um. This is true for all applications. It has nothing to do with Python. This is configuration management. It's hard.
- "Running package managers safely inside a production environment is possible, but everything’s got a price". And the price is actually quite low. Further, this means building secured systems for software development. That's quite hard in all languages. The only language that wouldn't require extra downloads of new and useful packages would be Pascal, I think.
- "You could do [here documents] with a Python script, too, as long as it does not import any package not already installed in the system." Right. Most bash-related Python features are part of the standard library. This isn't daunting or even particularly difficult or complicated. And. With Python you can unit test.
Container Runtimes. See #3. This is bash incumbency and configuration management from point 3, repeated. It's still challenging.
The Universal Language of Platforms. The bash CLI is ubiquitous, it's ideal for bash. But it's not actually ideal in general. A Python library that offers access an application's API may be much easier to work with and involve far fewer weird leaps to make the CLI amendable to the relatively weak set of abstractions bash has available.
- "It would take a couple of days in syntactical and semantical translations to get a result with more lines of code that were less readable than its Bash counterpart". Again, the argument presumes the bash language is the gold standard. Starting with bash and enduring translating into Python involves a cost. It also had benefits, like the ability to test. Why not start with Python? "Less readable" is offered without further evidence. Again, this repeats the bash incumbency argument where smaller and older is inherently better.
  
  Further, the time spent writing Python is often time well spent getting the abstractions right, and understading use cases.
- "All original samples in the docs were written using the command-line interface." Incumbency. And maybe lazy documentation writers in the vendor organization. "All Internet forums reference the command-line interface". Sigh. The "All" is disputable, but the point remains that using Python takes some effort.
The End of the Line Is Not Scripted. (Not sure what this means.) There are two obstacles here, both of which seem specious at best.
- Mega CLI's. Just because a bash CLI is available does not make it "best."
  - "every bit of function be wrapped with command-line interfaces." While true, it ignores the fact that some packages are actually written in Python, and the bash interface is -- at best -- a hack for those folks who won't learn Python.
  - Bash is everywhere. Incumbency does not make it better. It only makes it incumbent.
  - Writing shell scripts is more accessible than writing a new application. A good straw-man. It throws Python scripting away as if we can't write a short, pithy, testable, reusable Python script.
  - "open-source juggernauts..." like awk, curl, openssl, jq, and yq involves two issues. First, some programs like openssl are better left as stand-alone binaries used by a Python script. Second, programs like awk, jq, and yq are the primary symptom of how unsuitable bash is for working with anything other than a trivial string of characters. Reliance on these add-on programs is one of the reasons why bash is so confusingly horrible.
- Operations Frameworks like Ansible, Terraform, and (not mentioned) Puppet. These require some scripting for integration. Having done it in Python, I can safely say Python works.
  
  And I could unit test it.
Unrelated to the two obstacles is this nugget: "tuned for five decades of minimum resource utilization". I don't think this is true at all.

The original Bourne sh wasn't very thrifty to begin with. It was constrained by the tiny size of early machines. And. The Linux technique of sharing the read-only code pages meant the costs could stay low. State management was environment variables and some OS settings (like the current working directory.) The bash program is bloatware by comparison to the Bourne shell. The use of the OS | operator forks subprocess after subprocess leading to crazy OS overheads for a "simple" app | awk | grep | sed > file operation.

Conclusion

"In objective terms, regarding task automation for Cloud operations, it is hard to argue against Bash".

No. Actually. It's really easy.

The bash scripting language is opaque. Objectively, the syntax rules are quite obscure with complicated line-ending and quoting rules. Objectively, it's really difficult to understand the semantics of the operators like ;, |, and &. Why is ; optional? Why can a line end with & or ; but not end with |?
Error-handling in bash is an unholy mess. Objectively, what does set -e do? Objectively, why are there so many return codes?
Unit testing is almost impossible. Objectively, no one should run a shell script without a test case.
Bash has almost no useful data structures beyond the string. Objectively, we can argue that there's a way to break strings on spaces to treat the string as an array. This is essentially Python .split() as the alternative data structure to the string.
Programs like expr and [ are used widely and very difficult to understand. Objectively, the man pages for these programs are quite complicated. What looks like an expression isn't really. It's input to a separate binary that produces a result used by the shell's if construct. Objectively, this is confusing and unpleasant.

Programs like awk, jq, yq are used widely and can be difficult to understand. They're -- technically -- separate binaries, part of the overall bash ecosystem of internal bash features and external binaries. They do permit a kind of functional style on bash programming which is nice. Objectively, this isn't all bad. Python, also, has functional programming features.

The ubiquity of the bash programming is undeniable. It's also terrible. Bash should be used cautiously.

When to use bash?

Use bash you need to launch a Python script. A bash script should be little more than an alias for a program written in a language that offers unit testing.

Understanding the Abstraction -- matplotlib Edition

2023-12-12T08:01:00-05:00

I wasted three days because I refused to get a grip on how matplotlib really works.

Most of the time, folks like me are happy and successful using the pyplot module. The Quickstart provides brilliant, working examples.

As my partner's grandfather used to say, "We're off to the races in a cloud of heifer dust."

The examples are easily rewritten for the data at hand. They work in Jupyter Lab. Boom. Done.

There's a little bit of technical detail in https://matplotlib.org/stable/users/explain/figure/interactive.html#jupyter-notebooks-jupyterlab. When I realized things weren't working. I followed each piece of advice, scruplously. They were not the cause of my problems. The root cause was failure to understand the abstraction.

Digging a Little Deeper

What is not painfully obvious is how the matplotlib architecture works. (It's not written in flaming letters 100 feet high.)

The pyplot module is pure genius. It works to shield us from a tech stack that's non-trivial.

Which means, when someone like me wants to do something that's not copy-and-paste from one of the hundreds of examples, I have to actually read the documentation. Carefully.

It took me three days to understand what the documentation said. Here's my timeline.

Day 1. Fuss around with my incorrect understanding of how graphics are created.

Day 2. Write the entire thing as a stand-alone command-line app, where the extemely robust, clever matplotlib architecture works. It works in spite of me using it utterly incorrectly.

Day 3. Blinding realization that for the last two days, I've been doing it wrong.

The Big Reveal

Just about everything that happens in matplotlib depends on an abstraction called an Artist being out of date from the canvas.

This is not obvious, and no one needs to know it except in the rare cases of an animation.

The rest of the time, we observe that changes to scale or axes or whatever lead to changes to the diagram that are just what were expected. The "out-of-date" business doesn't enter into our experience when we're making changes that update the static diagram we want.

Anyone (even me) can get things to work by simply creating axes, creating a "fill" (a Polygon, actually), and being happy. The plt.show() works.

But that's actually not right for the kinds of things I was trying to do.

Here's what I was working on.

This map is actually "grown" using some simple rules from a few seed points. The animation of that growth process is what I want.

This isn't as clever as Conway's Game of Life, but it is similar in a few respects. Mine, for example, involves random numbers. Is that desirable? Can the dependency be reduced and still lead to complicated structures?

I want to tinker with the rules.

(I have a version running in the Pythonista environment on my iPad. I want a version in JupyterLab that I can expand on more easily.)

Let's compare and contrast the two approaches

Seductive and Wrong

This is seductive and simple. It fits (to an extent) with previous examples. It seems so right. And it sometimes works. But it's so wrong.

Create 324 polygon outlines to paint the background grid.
Create 324 text labels to label the hexes.
As the generative algorithm runs, create colored polygon fill patches, showing how the 5 seed positions evolve into the 5 interlocking shapes. This starts with 5 filled polygons and grows to 200+ polygons through 48 generations. So that's \(35 + ... + 235 = 6,650\) filled polygons.

Wave after wave of increasing number of polygons. Sure, it's a lot of objects. I have a big laptop. We're good.

This has two problems.

It's slow.
If I save the animation as an HTML or JSHTML object, I get a cycling animation with the right number of images, but no content in any image.
In spite of the animation being empty, the final image looks good.
A few off-by-one errors.

What's wrong?

I'm patient and thorough. I tried a lot of things. I added Qt. I added ipympl. I restructured the animation as functions and callable objects. I used FuncAnimation and ArtistAnimation.

Nothing Worked. Nothing.

Spoiler Alert. That's how you know you're doing something fundamentally wrong. The thing works in general. But specific features are missing.

How Can That Be Wrong?

The foundational mis-understanding was trying to animate the appearance of various matplotlib Artist objects on the map.

I drew the grid. I drew the labels.

Then the colored hexes are supposed to appear, one at at time. I figured (wrongly) I would just draw the filled polygons.

See above. "Just about everything that happens in matplotlib depends on an abstraction called an Artist being out of date from the canvas."

Out-of-date?

Out-of-date!

State Change.

The Artist won't paint unless there's something new to paint.

On day three, I realized the truth.

It works like this:

Create 324 polygon outlines to paint the background grid.
Create 324 text labels to label the hexes.
Create 324 polygons filled with white.
As the generative algorithm runs, change the color in the polygon. Change the color. Change.

Change. The Artist is waiting for a change.

Don't create a wave of new polygons. Change the color of the polygons. It's simpler. It's faster. It works.

def paint(self, col: int, row: int, fill: str) -> None:
    """
    Update a hex's fill color.
    """
    for a in self.cells[col, row]:
        a.set(
            fill=True,
            color=fill
        )

Don't create a new polygon. Change the color of the polygon you have.

I still have no idea how the scale factors work when creating the JSHTML. I have eight mypy complaints because I'm not using matplotlib correctly. I have more work to do.

But. I have pictures that work. For the right reason.

It's Not THE Ternary Operator -- there are many

2023-12-05T08:01:00-05:00

I'm sick of reading about THE Ternary Operator.

There is not merely a single operator that is ternary. There are many operators that are ternary.

Here's one example:

>>> 6+1 >= 6 >= 6-1

The >= >= operator is ternary. It has 3 operands. Count them.

There are a 36 of these ternary operators.

"Oh, pish-tosh," you say. "It's an example of two binary operators."

Really?

In a sense, you're almost right, it's equivalent to 6+1 >= 6 and 6 >= 6-1. Which is three binary operators.

But it's not three binary operators. It's one ternary operator.

"Whoa. What about 5 < 6 < 7 < 8?" you reply triumphantly. "That has four operands!"

Right. It's quaternary. There a lot of ways to create quaternary (and higher) operators in Python. There are 216 of quaternary operators. 1296 quinary.

There is not A Ternary Operator. The phrase is meaningless. You can stop using it.

There's more

We're not done.

>>> (2*a + 1
...    for a in range(5)
...    if a % 2 == 0
... )

Is ternary.

I'll clarify:

2*a + 1 -- expression #1
range(5) -- expression #2
a % 2 == 0 -- expression #3

Admittedly, the two pieces of interstitial syntax are complicated.

for a in -- kind of big; and it names a bind variable.
if -- more typical for ternary operators.

But. There are three operands separated by punctuation.

It is, therefore, ternary.

And yes. When we have multiple for clauses or multiple if clauses, we clearly have quaternary and quinary operators. That's part of my point: there are a number of arities and a number of operators of each arity.

I could go on

A Big Pointless Beef (BPB™) is often this.

The ternary operator (by which I presume they mean the conditional expression) evaluates the middle first.

Which is kind of a "so what?"

Many things in Python are left-to-right.

But not everything is trivially left-to-right.

>>> noisy = lambda x: print(x) or x
>>> list(noisy(2*a + 1)
...   for a in noisy(range(5))
...   if noisy(a % 2 == 0)
... )
range(0, 5)  # expression 2, the range(...)
True  # expression 3, the if.
1  # expression 1, the 2*a+1.
False
True
5
False
True
9
[1, 5, 9]

The range(5) -- in the middle of this particular ternay operator -- is evaluated first. And only evaluated once, where the outer expressions are evaluated on right-to-left order over and over again.

We're not done

Consider, if you will,

>>> a = list(range(5))
>>> a[1:-1]
[1, 2, 3]

The a[1:-1] is ternary. It has three expressions. Count them yourself.

Also, a[:-1:2] and a[-1::-1]. All ternary subsets of a more general quaternary operator.

"That'a wrong! You can't call a slice part of an operator," you claim.

Perhaps I am pushing it. But it sure looks like a is one expressions, 1 is another and -1 is the third. And it sure looks like [ is one separator, : is another, and there's an extra closing punctuation mark of ].

"You've jumbled up indexing and slicing!" you claim. "They're clearly separate syntactic categories!"

Clearly? If I can only using slicing in the context of indexing, I'm not completely sold that these two concepts are separate and foreign. They seem pretty tightly coupled.

Summary

Stop writing (and saying) "The Ternary Operator". Please.

There are a lot of ternary operators.

If you don't like the Conditional Expression because it's too much like a list comprehension with that "evaluate something not on the left first" semantics, please say that you don't like The Section 6.13 Conditional Expression.

Please. Please try to be precise.

Otherwise, the rest of your rant on evaluation order looks like you haven't really taken the time to think things through. Maybe you have, but the use of "The Ternary Operator" dilutes your message.

Other languages use phrases like "the ternary operator." That doesn't really mean much. We're talking about Python, where there's more than one.

This is Awful

2023-11-14T08:01:00-05:00

This is a disheartening thing to read

As someone who doesn't do a ton of JSON parsing on the command line, I tend to forget jq syntax.

Two tools I really like are gron (make JSON greppable) from @tomnomnom and jqp ..., which provides a "tui playground for exploring jq."

98% of the time I end up being able to get what I need with gron + grep, then jqp is awesome for when I actually need jq :)

What's so bad?

With Python, all of this "syntax" and "greppable" and "interactive" goes away.

Stop using bash. Start using Python. Life is so much simpler. (And faster. And unit-testable.)

The Debugger

2023-10-10T18:21:00-04:00

See Python 3.12: what didn't make the headlines. This is very helpful.

It is a great list of 7 key features of Python 3.12.

With one tiny point I need to object to.

I don't like debuggers

This is a strongly-held position.

Debuggers are harmful.

I say this because I have had the misfortune to help more than one programmer who could not actually describe the semantics of the code.

They couldn't draw a picture. Write a sentence. Nothing.

They could only point at the interactive debugger session with hapless flailing, and "see, it should work" kind of noises.

This is emphatically bad.

Every time I would ask them to step away from the debugger and describe -- maybe on a whiteboard -- what the heck they thought was going on.

I could go on with horror stories of bad debugging.

I use debuggers

Back when C++ was my "stock-in-trade", I used the debugger.

Rarely.

And then, mostly, on core dump files to figure out where the program failed.

And to look at a few key variables to confirm the state of the computation.

Then.

I went back to the source, and looked for a logic path that lead to the wrong state. It wasn't often hard to find. And it didn't involve using the debugger for much more than finding the call frame, stack contents, and local variables.

What set me off

This:

...it's also removing a big "WTF" that all beginners will experience using the Python debugger with nobody in sight to explain to them what's going on.

I think there are no circumstances under which beginners should be using the debugger.

I think there are no circumstances under which anyone should use a debugger before they already know what's supposed to be going on.

The idea of "beginners" being surprised at the structure of stackframes is an oxymoron.

Beginners don't know about stack frames.

More-or-less, this is one definition of "beginner".

People who know about stack frames aren't beginners and can be trusted to understand the debugger.

The points in the blog posts are sound: better debugging, additional support for evaluating expressions.

The "audience of beginners" is my only quibble.

The Echo-Pipe Trap [Updated]

2023-08-30T09:00:00-04:00

This is a great question.

https://fosstodon.org/@JustineSmithies/110979871574705636

This works, they said.

echo -en "Firefox\0icon\x1fweechat" | fuzzel -d -w 100 -l 10

But. The superficial switch to subprocess.Popen() doesn't work.

Why?

The way echo works varies from shell to shell. When MacOSX changes to zsh, things can break. Or when you share it with someone else, who uses YetAnotherShell, things break.

Two choices:

Specify which shell.
Stop using the echo ... | (echo-pipe) construct.

What's Better?

The stdin parameter to Popen() can be used to provide the required stream of bytes.

from pathlib import Path
import subprocess

temp = Path.cwd() / "temp.tmp"
temp.write_text("Firefox\0icon\0x1fweechat")  # I think.

with temp.open() as echo_file:
    subprocess.Popen(['fuzzel', '-d', '-w', '100', '-l', '10'], stdin=echo_file)

Something like the above will avoid the echo-pipe construct.

But. It leaves a temporary file lying around. What to do?

Slightly Better

This will cleanup the file. And. You don't have to pick a name.

import tempfile
import subprocess

temp = tempfile.TemporaryFile(mode='w+')
with temp:
    temp.write("Firefox\0icon\0x1fweechat")  # I think.
    temp.seek(0)
    subprocess.Popen(['fuzzel', '-d', '-w', '100', '-l', '10'], stdin=temp)

Seems kind of long. And it involves an additional problem. A file.

But. There's a FILE!

Yes. We can create a pipe. I think it's kind of hideous, though.

import os
import subprocess

r, w = os.pipe()
readable = os.fdopen(r, 'r')
writeable = os.fdopen(w, 'w')
writeable.write("Firefox\0icon\0x1fweechat")  # I think.
writeable.close()
subprocess.Popen(['fuzzel', '-d', '-w', '100', '-l', '10'], stdin=readable)
readable.close()

However. It's not too long.

We can create a pleasant wrapper in the form of a context manager.

How's This?

This seems pleasant, if you do a lot of this sort of thing.

import os
import subprocess

class EchoPipe:
    def __init__(self, content):
        self.content = content

    def __enter__(self):
        r, w = os.pipe()
        self.readable = os.fdopen(r, 'r')
        writeable = os.fdopen(w, 'w')
        writeable.write(self.content)
        writeable.close()
        return self.readable

    def __exit__(self, exc_type, exc_value, traceback):
        self.readable.close()

Once you have the EchoPipe context, you can now write this.

with EchoPipe("Firefox\0icon\0x1fweechat") as echo_pipe:
    subprocess.Popen(['fuzzel', '-d', '-w', '100', '-l', '10'], stdin=echo_pipe)

Which is pretty close to the original terse shell stuff.

And this [Update]

Consider this...

subprocess.run(
    ['fuzzel', '-d', '-w', '100', '-l', '10'],
    input="Firefox\0icon\0x1fweechat",
    text=True
)

It a very useful variant. This is -- perhaps -- the best of them all.

I found the documentation is a bit hard to follow around this topic, so I was completely taken by surprise when I was shown this.

More Python Quirks Debunking

2023-08-15T09:00:00-04:00

Stuff I found on the internet that I have to disagree with.

(And no, I didn't ask for clarification. If the author posts things without supporting details it suggests they might lack the supporting details. I can be charitable and assume they don't really care about providing useful information, but are merely trolling for engagement. Yes. That's cruel. I can't see how you take the time to have an opinion and not provide support for it.)

A few of #Python 3 #quirks and #kludges

Global Interpreter Lock

Strong, dynamic typing

Massive Any hole in the type system

Verbose class definition

Declaration of instance attributes is their definitions in __init__()

Repetitious self and kludgy super().__init__()

Kludgy string-quotes to reference the class from within its definition

Kludgy TypeVar definition

Absence of structural typing

Need explicitly to convert iterator to list using list()

Usurping the id() name

Inner method may mutate referenced objects in the closure but may not mutate primitive values therein

Kludgy main() invocation

Disconcerting lack of type information in the holdover documentation from the P2 days

Some of these might be good points. Some of these seem to be nonsense. A bunch can't be interpreted, which I find madding.

1. Global Interpreter Lock

I'm not sure what this means. It's a solution to a specific problem. It's -- I suppose -- a candidate "quirk" because it's an unusual solution to the problem of assuring that data structure updates are atomic.

It saves us from having to use explicit locks all over the place when updating objects with complex state.

The GIL-less Python proposals will require a bit more care in defining structures useful in multithreaded environments. Is this a gain? It's disputable.

2. Strong, dynamic typing

Yep. There it is. Quirk? Kluge? Dunno. It seems like a brilliant solution to a long-standing problem.

3. Massive `Any` hole in the type system

Does this mean it's bad that mypy assumes Any for missing types? If you don't like the mypy assumptions, write type hints.

Does this mean it's bad that you can use Any to provide uninformative type hints? If you don't like Any, consider not using it as a type hint.

4. Verbose class definition

Quirk? I guess they've never seen Java.

Kluge? What would they prefer?

What would they omit is the real question. From the item 6, below, I'm guessing they don't like to have self listed explicitly.

5. Declaration of instance attributes is their definitions in `init()`

First -- and most important -- there aren't any C- or Java- style declarations. The instances are dynamic in every sense of the word. The __new__() method does almost nothing.

I'll buy this as a legit quirk. It's a consequence of the way attributes work, and logic is compelling and consistent.

It's trivial to include type hints in the class definition, separate from initialization in the __init__() method. I find this helpful.

class X:
    a: int
    b: float

    def __init__(self, a: str, b: str) -> None:
        self.a = int(a)
        self.b = float(b)

It's potentially misleading: the a and b appear to be class variables.

6. Repetitious `self` and kludgy `super().init()`

The use of self is not repetitious. It's explicit.

Not sure what's klugy about super().__init__(). I guess they don't like writing __init__() and prefer having this assumed, also.

This is -- to me -- flat our wrong.

7. Kludgy string-quotes to reference the class from within its definition

I guess they'd prefer to have an explicitly complicated-looking forward reference for a name. We could have a lot of class XYZ: defined_later() constructs to sort out circular references among classes.

I guess they don't like this.

class X:
    @classmethod
    def makes_X(cls: type["X"], *args, **kwargs) -> "X":
        ...

It seems like a tedious X = ForwardRef('X') would be required to define the name X before the actual class is defined. See #8, below, they don't like that syntax. Does this mean they want a new statement for forward references?

Or. It would require mypy to gaze more deeply at the parse tree to resolve circular references. I'm not sure what they think would be better.

8. Kludgy `TypeVar` definition

I'm guessing they want a new statement in the language instead of a function in the typing module.

Since types are explicitly optional, new statements to handle types seems wrong to me.

9. Absence of structural typing

This is confusing. The NamedTuple provides structural types.

I'm guessing they were hoping for some other classes to also behave like types in a structural system. It seems simplest to use NamedTuple and a functional style of programming.

10. Need explicitly to convert iterator to list using `list()`

This is nonsense. What if the iterator is a sequence of pairs that should be converted to a mapping with dict()?

11. Usurping the `id()` name

Don't get this. The print() name is also usurped by built-ins. There are a dozen built-in function names that usurp other names one might want to use. And all those keywords! The name class and def and return are all usurped by keywords.

12. Inner method may mutate referenced objects in the closure but may not mutate primitive values therein

Primitives can't be mutated.

Referenced objects can always be mutated.

It doesn't require an "inner" method. It's true for every function and method at all levels.

I'm guessing the idea of mutable vs. immutable objects could be a quirk.

13. Kludgy `main()` invocation

Kludge? Really? I guess they've never seen Java.

14. Disconcerting lack of type information in the holdover documentation from the P2 days

It's often helpful to provide an example of a documentation gap where type information is totally missing (or is only present in a stubs file where it's not automatically included by Sphinx). While I haven't seen any examples of missing type information in Python or standard library documentation, that doesn't mean much. I only write books about Python, I don't actually help maintain it.

Summary

There's a good point:

Declaration of instance attributes is their definitions in __init__()

The reset is a mixture of

too vague to comment on,
it's not clear what would be better, and
wrong.

Mostly the former. Few of the latter. (#10 seems to be the stand-out for wrong.)

It's important to think about these things when learning a language. Some discussion of alterantives from other languages would make these points a lot easier to interpret and understand.

However, it's also important to understand why soem things are present in a language. It's important to look a little more deeply at the language rules -- perhaps read the relevant PEP's -- to see what alternatives have been proposed and discarded.

In most cases, decisions aren't arbitrary, but reflect deeper considerations on the underlying semantics of the language and the implementation details of the compiler and/or run-time.

Python Quirks that aren't very quirky

2023-08-01T09:00:00-04:00

See https://writing.peercy.net/p/python-quirks

Superficially, most of these are true.

Looking a little more deeply, most of them are also presented in a somewhat misleading way. A few set up a good punch-line. The Inheritance one, for example, is funny.

If the point is to force a deeper investigation, I think the piece might not be helpful. I know too many people who would look at this list and say "See, Python is as bad as JavaScript." Or "That's why I only use perl." These are the sort of folks won't actually refer to the Python language reference manual to see what's going on.

One of these is a legitimate quirk. The rest involve a little bit of "don't look at the man behind the curtain" mixed with "don't read the documentation."

To ease my own mental anguish, I'll include a slightly deeper dive into these language feaures.

Generators.

sorted() creates a new list from the argument value. It's not a generator. Comparing the resulting list to the argument is unsurprising.

reversed() doesn't create a list. It is a generator. Since it can only be used once, one use of the generator y has a sequence of values. The other use of the generator y has no values.

I suppose the single-use-of-a-generator featrure could be called a quirk. Except it's well-documented, so, I'd argue that this simply exposes a language feature.
References.

The example fails to show how a was created. It's not obvious how the reused reference to a sublist was propogated throughout the list.

Missing:
```
a = [[0]] * 5
```
Assignment.

Not sure what the point of this is. It doesn't even seem quirky.

I guess they're astonished they can use something other than a trivial variable in a for statement.
Closures.

My guess on this is they were hoping the i variable would not be a single variable; instead, a fresh, new variable would be created by the generator expression. Perhaps other languages do this, and manufacture fresh, new variable bindings.

Python has a (relatively) simple rule for variables: Local, Enclosing, Global, and Built-in. There's no closure rule to create new variables. There are many good tutorials on the LEGB rule.
Inheritance. This one is kind of funny.

It's also, on reflection, unsurprising. The type type -- like all types -- is an object. Mostly because almost everything is an object. The object type -- like all types -- is a type.
Operator Chaining.

This isn't a quirk at all. This seems to be an exploration of precedence rules among operators. It seems to be a matter of definition among ==, and, and in operators.

Also, it's not clear what "chaining" means here. If 1 + 2 + 3 is what they mean by "operator chaining", then I think that may be the root cause of the confusion. These are all binary operators with intermediate results. Perhaps it can help to think of implicit ()'s around each binary operation.
Identity.

This is an optimization in the CPython interpreter to pre-allocate some integers. This is a proper quirk. I'm glad it's on this list.
NFKC Normalization.

See https://unicode.org/reports/tr15/.

This isn't Python. This is Unicode.
Default Arguments.

This is pretty well-known. Some newbies discover it, and then re-read the documentation that describes why this happens, and say "makes sense." Here's the rule: The mutable object is only created once; it's shared.

Most linters warn that this feature may not be doing what folks think it's doing.

This appears in many other places. For example, default values for fields of dataclasses cannot be mutable objects, because they'd be shared.
Whatever This Is.

This isn't a quirk, it's a bug. It was fixed in Python 3.11, though. So it's much less interesting. It remains a known bug until Python 3.10 end-of-life, 04 Oct 2026.
Python 2.

Python 2 has been at end-of-life since 01 Jan 2020. These kinds of things ceased to be interesting on that date.

In summary, and in conclusion, the identity of small integers is a legitimate quirk. I like it. The inheritance is funny. I like that, too.

Two Problems with Python

2023-07-25T09:00:00-04:00

I want to call out two huge problems with Python. I'm not the first to point these out, but they've been bothering me for a while.

Surprising Changes
Dependency Hell

I've provided them here to save folks from repeating these. They're now officially "known" and there's no point in repeating this again. Write your blog posts about something else, please.

Surprising Changes

Every language and library has changes. That's part of normal innovation and evolution of the language.

Some changes, however, were not communicated to me, personnally, and are therefore suprising, which makes them bad. Really bad.

Let's focus on linter tools as an example. Here's the scenario.

I have a code base. It's good. 100% compliant.
I ugprade the linter.
A new error is flagged. This was not an error before but somehow (big eyeroll) it's an error now.

This is a surprising change. No one told me.

The code is sketchy. It could be seen as ambiguous. Even though it passes all the unit tests.

Someone else may have learned a lesson about sketchy code, and embodied that lesson in the linter. But they didn't tell me.

Python had a surprise change, and the mere presence of a surprise means one thing:

Python is useless.

Dependency Hell

Every application has dependencies. That's part of building a language in a rich ecosystem with a lot of useful packages.

Some changes to these packages, while well-intentioned, can break a dependency with another package. Packages have inter-dependencies, which I find impossible to manage.

Here's the scenario.

I have a code base. It's good. 100% tests pass. Installs perfectly on all supported platforms.
Two packages, p==3.14 and q==2.78 both depend on x version 1.1
The authors of p updated to 4.0 and switched their dependency to to x version 2.0. The authors of q did not switch.
If I include p==4.0 and x==2.0 the q package breaks. I can't upgrade p.

Dependency Hell. Unresolvable Conflicts.

Any combination of packages will have numerous internal dependencies. The mere presence of these dependencies means one thing:

Python is useless.

Summary

Python is useless.

I cannot tolerate innovation.

If someone learns something and changes a linter, that's innovation: it breaks my code; I don't want it.

If someone creates a new version of an open-source package, that's innovation: it breaks my code; I don't want it.

This isn't to say that innovation is bad.

Innovation is good. When it occurs very slowly, and I'm able to personally vet each individual change for impact on my project(s).

The idea that every single open source package is innovating and learning at their own unique tempo is insanity. It makes Python useless.

An Implementation of Annotated Types

2023-07-11T09:00:00-04:00

The typing module includes the mysterious-looking Annotated type hint. See https://docs.python.org/3/library/typing.html#typing.Annotated for details.

What does this do?

Why do I need it?

Where can I see examples?

What does this do?

The Annotated type hint lets us append "details" to a type.

It might look like this

x: Annotated[int, MustBePrime()]

The annotated type has one origin type (which must be first) and a sequence of objects. Presumably, they are "annotations" of some kind. They can be anything. We can do a lot with them; we'll start with using them to narrow the domain of values.

The core x: int provides a large domain of possible values. Python's ints can be immense numbers, easily filling memory with digits.

The MustBePrime class is the kind of thing that might be used to narrow the domain of allowed values to prime numbers.

When does this value checking happen?

I'm glad you asked.

Use of annotated types is not part of the Python run-time. Annotated type arguments are essentially ignored. The origin type is used by tools like mypy.

Any further use of annotations is a thing your application or tool-chain will need to do.

An application can see the annotations for an object using the __annotations__ special attribute number, or use the typing.get_type_hints() function.

>>> from typing import Annotated, get_type_hints
>>> class MustBePrime:
...     pass
...

>>> class SomeApp:
...     x: Annotated[int, MustBePrime()]
...

>>> get_type_hints(SomeApp)
{'x': <class 'int'>}
>>> get_type_hints(SomeApp, include_extras=True)
{'x': typing.Annotated[int, <__main__.MustBePrime object at 0x7fde259a7be0>]}
>>> get_type_hints(SomeApp, include_extras=True)['x']
typing.Annotated[int, <__main__.MustBePrime object at 0x7fde259a7be0>]

We can see the annotated type hint for x.

This means, our application is free to "apply" the annotation in some way.

"Whoa! That's vague," you say. "There are no specific rules for annotated types?"

I agree.

The details are up to your app. Seriously. Define them in a way that makes sense.

Maybe you want your app looks like this:

>>> class SomeApp:
....    x: Annotated[int, MustBePrime()]
...     def __init__(self, arg_value: int) -> None:
...         self.x = arg_value
...

And you've got a use case in mind...

>>> sa = SomeApp(42)
Traceback (most recent call last):
   ...
ValueError: value 42 is not prime

The idea is that this specific app has an associated collection of annotations that are used during __init__() processing to further validate the supplied values.

The code to this is clearly part of SomeApp -- maybe a metaclass, maybe a superclass -- but clearly part of the app.

And the app will use the annotation as a kind of "plug-in" or "extension" or Strategy design pattern to do some additional processing at some point.

Our use case was part of __init__() processing. What does this look like?

An example app

We'll avoid metaclasses, and pretend that Annotated types are checked by an explict call to a method of the class. Let's say a superclass, named RuleCheck has a method that must be called at the end of __init__() to check compliance with annotations.

class SomeApp(RuleCheck):
    x: Annotated[int, MustBePrime()]

    def __init__(self, arg_value: int) -> None:
        self.x = arg_value
        self.check()

The idea here is that the class-level hints are carefully defined.

The __init__() merely slaps any old value in there.

And the self.check() then assures that all hints are actually true for the supplied values.

This means it will "apply" the annotation to the given value. In this case, it will either allow the value silently or raise an exception if there's a problem.

Here's the RuleCheck class.

from typing import Annotated, get_type_hints, get_args

class RuleCheck:
    def check(self) -> None:
        vars = get_type_hints(self.__class__, include_extras=True)
        for name in vars:
            match vars[name]:
                case Annotated:
                   base, *rules = get_args(vars[name])
                   for rule in rules:
                       rule(getattr(self, name))

Each annotated variable has the arguments to the annotation retrieved with typing.get_args(). Each of these annotations must be a callable object of some kind that can be applied to the attribute's value.

We leave the implementation of MustBePrime as an exercise for the reader.

Why do I need it?

You need it in a bunch of cases. Here are some ideas.

Type domain narrowing. We used "prime" as an example. You might want to use positive values, or values in a range. Or other properties that you'd like to make part of a type.
Documentation. You can imagine x: Annotated[str, title("Some Descriptive Information"), Positive()]. Since the documentation is not a comment or other ephermeral source text, you can use this to create a formal Schema for a class. Thing JSONSchema. (Or XSD if you're old.) You could use the title to beef up the exception messages, for example.
Other Processing. Let's not get crazy, but the following is possible.

x: Annotated[float, Title("Independent"), Range(0, 10)]
y: Annotated[float, DerivedFrom("x"), Function(lambda x: 2*x-1)]

The idea is that we might build a class where any change to x computes a value for y based on the annotation; and the value is cached as an attribute value, not a @property which is always recompued.

(Yes, @cache and @property can do this. This isn't necessarily a great idea. But it's possible.)

Building a type definition

Maybe we want this.

PosInt: TypeAlias = Annotated[int, MustBePositive()]
PrimePosInt: TypeAlias = Annotated[PosInt, MustBePrime()]

We've built a complicated type on top of another complicated type.

This permits us to -- for example -- improve the performance of MustBePositive with an attendant speedup of other, related objects.

File Parsing

This is an edge case. But. It applies to the vast number of files processed by COBOL programs.

x: Annotated[str, Start(0), Length(5)]
y: Annotated[str, Start(5), Length(10)]
z: Annotated[Decimal, Start(15), Length(10), Scale(2)]

We've provided the metadata for positions of the source data in a text document. A file with a line like "ABCDEZYXWVUTSRQ0000001299" could be parsed by a class that leveraged the annotations to pluck values out of the source string. It could apply conversion from mainframe encodings ("EBCDIC") and do decimal conversion.

Where can I see examples?

I have two examples, right now.

Pydantic v2 Annotated Validators: https://docs.pydantic.dev/latest/usage/validators/#annotated-validators

Wow is this cool.

Also.

TigerShark. https://github.com/slott56/TigerShark This is a pretty narrow problem domain. But, the Annotated type hints were a perfect solution to an ages-old problem. The X12 messages have complex more-or-less hierarchical structure. Messages have Loops (that can repeat), Segments, and individual Data Elements.

The definitions of the messages have complicated meta-data on size, encoding, data types, optionality, etc., and etc.

What we want is a top-level definition of a message that looks like this:

class MSG270(Message):
    """HIPAA Health Care Eligibility Inquiry X092A1-270"""
    ItemIsa_Loop: TypeAlias = Annotated[ISA_LOOP, Title('Interchange Control Header'), Usage('R'), Position(1), Required(True)]
    isa_loop: Annotated[list[ItemIsa_Loop], MinItems(1)]

The TypeAlias and Annotated type provide all the metadata for this message.

Looking elsewhere in the message module, we find this...

class ISA_LOOP_ISA(Segment):
    """Interchange Control Header"""
    _segment_name = 'ISA'

    isa01: Annotated[I01, Title('Authorization Information Qualifier'), Usage('R'), Position(1), Enumerated(*['00', '03'])]

    isa02: Annotated[I02, Title('Authorization Information'), Usage('R'), Position(2)]

    isa03: Annotated[I03, Title('Security Information Qualifier'), Usage('R'), Position(3), Enumerated(*['00', '01'])]

Again, the elements are defined (entirely) by annotations.

The base type? I01? A pool of common definitions.

I01: TypeAlias = Annotated[ID, MinLen(2), MaxLen(2)]

But wait! That still depends on a more foundational definition, ID.

ID: TypeAlias = str

The idea of this is to map the type information to type aliases, so anyone can follow the message definitions completely. The annotations are defined formally by the X12/EDI standards; the mapping to Python is through these foundational type aliases for Python types.

Also see https://pypi.org/project/TigerShark3/ if you have the urge to install it.

Behave Tests and Fixtures

2023-01-27T08:00:00-05:00

BLUF

Behave fixtures totally rock for testing complex applications.

I had been doing them wrong. Doing them right is simpler.

History

I'm a fan of the Gherkin language for specifying the behavior of software.

Scenario: Works for Me

Given a configuration
When a request is made
Then the response can be evaluated.

I love this.

What I particularly love is the way Behave's steps package provide implementations for the individual steps of the scenario.

The steps can be organized around technical needs, where the features are organized around the user's experience when operating the software.

The Non-Fixture Approach

For a long time, I used the Given step and an after_scenario() function in Behave's environment module to create and destroy fixtures.

Scenario: Test with Mock API

Given a server running on http://127.0.0.1:8000
And the server has the resource requested
When a client makes some random request or other
Then the response is a tidy bit of data the user just loves

The Given step would seed the context with details used to configure a tiny, specialized service built with Python's http.server module. This separate subprocess would provide an appropriate response for this scenario.

The mock server srequired creating a request handling class hierarchy with reusable and extensible choices for the various scenarios and features. Often only one or two paths would be handled, since that's all a scenario needed.

The context parameters were turned into command-line options. The mini server was started by the When step and stopped (eventually) after the scenario.

This was workable. But. Ultimately. Dumb.

Fixtures

Behave has a much, much better way to configure and manage fixtures. This is great for tests that databases or RESTful API servers or other, separate processes to collaborate with.

Fixtures.

@fixture.the_mock_server
Scenario: Test with Mock API

Given a server running on http://127.0.0.1:8000
And the server has the resource requested
When a client makes some random request or other
Then the response is a tidy bit of data the user just loves

There's one change to the scenario -- a tag with @fixture. to positively identify a fixture required for this scenario to make sense.

When reviewing the Gherkin with users, the @fixture tag is easy to explain. There are often other tags throughout the scenarios. A @slow tag, for example, might be used for those scenarios that involve throttling or timeouts. A @future tag for those options that aren't required but can be tested to observe development progress. For one project I had a @core tag that recapitulated the examples in the documentation --- these had to work exactly as shown.

Infrastructure

The fixture infrastructure has three parts.

Our specific fixture-managing generator function. This will create the fixture, yield something, and then destroy the fixture. This precisely parallels the way pytest fixtures work.
A before_tag() function in the environment to look for the tags and do any setup or logging required.
The fixture itself. This is our specialized test-case server based on http.server. It still uses a configuration file or command-line options -- or both -- to define some behavior required for the scenario.

What happens, then, is this.

The before_tag() function is evaluated for every tag on every step. If a tag starts with "fixture." then, something special needs to be done.
The before_tag() function will evaluate the behave.use_fixture() function to inject our specific fixture-managing generator function into the step processing.
The fixture will be created (and destroyed) as part of the scenario's execution.

(If you need details, see https://behave.readthedocs.io/en/stable/fixtures.html#fixture-cleanup-points)

The mapping from "fixture.this_special_api" to a generator function named this_special_api() is kind of trivial. So trivial that the examples in the Behave documentation suggest you look these up in a map in the simplest possible way.

TAG_IMPLEMENTATIONS = {
    "the_mock_server": server_fixture_generator,
    "the_other_server": another_fixture_generator,
    "the_timeout_server": the_timeout_server
}

def before_tag(context, tag):
    if tag.startswith("fixture."):
        _, name = tag.split('.')
        use_fixture(TAG_IMPLEMENTATIONS[name], context)

There's a use_fixture_by_tag() function that may be considered to be simpler than my example.

Now, we can add fixtures by writing a generator function to create (and destroy) the fixture and adding the new function to the TAG_IMPLEMENTATIONS mapping.

The fixture names are for users who might want to review the scenarios. They're subject to the same kind of negotiation the rest of the Gherkin terminology is. Sometimes, you'll tweak the wording as the user's understanding (and needs) evolve.

Cleanup

When you have serious problems in your test implementation, you'll have tiny cleanup issues.

For example, if your step implementation code is broken, the test can crash without having executed all the steps you anticipated.

This can mean a fixture isn't properly torn down. It's a rare, but annoying thing to happen.

See https://behave.readthedocs.io/en/stable/fixtures.html#ensure-fixture-cleanups-with-fixture-setup-errors for some solutions.

I'm a fan of leaving information about the fixture in the context, and using after_scenario() or after_feature() functions to kill long-running process in the rare case that a step failed.

The alternative, using add_cleanup(), is -- perhaps -- nicer, because it relies on a closure that doesn't clutter the context with these little, technical overheads. (I find closures a little awkward to debug, but, debugging is rarely needed for this.)

Books

Yes, this is for a book. Stay tuned. Later this year.

Testing RESTful web services in Django -- Tantalizingly Close.

2008-08-13T10:10:00-04:00

Here's what's great about Django coupled with the Django-REST Interface : It's almost all model. You define the model, write some tests. Add the URL mappings, write some tests using the built-in Django client.

We're almost there, but this doesn't work out perfectly. To do complete tests, we have to either subclass the Django Client to add "put" and "delete" or curry in methods for "put" and "delete". Then we can almost test our complete set of web services functions.

At this point, the core of the application is -- well -- done. It works, it handles the web services requests. We can then start folding in HTML pages for the endlessly negotiated human interface.

However, we're still not ready for deployment.

Authorization Differences

First, we haven't really got a solid security model in place. Sure, we can add @login_required decorators to any view functions. But that doesn't really secure the REST interface at all. That's where the going gets tough.

The Django-REST Collection has an 'authentication' attribute that checks passwords. It has an HttpDigestAuthentication class that handles more-secure password digests. This looks perfect for web services. But, it has two problems.

We don't have MD5 digests readily available. Django uses SHA1 digests of password only, not an MD5 digest of username:realm:password.
We can't easily test using digest authentication with the off-the-shelf Django test Client. Not only does the test client lack Put and Delete, but it can't handle HTTP Digest authentication, either.

Sigh. I thought we'd be done in 20 minutes . Turns out, I have to actually do some work.

Adding MD5 Digests

MD5 digests seem to work out best with the 'Profile' extension to the Django authorization application. The model is delightfully simple, just a single CharField to hold the MD5 hexdigest of username:realm:password.

One consequence is that we now have two password digests, the default SHA1 in the User model and our Web Services MD5 in the Profile extension. This means that our page for password resets must have a view that sets both passwords.

Testing Complications

In the long run, we have to provide WS client libraries. While the application is entirely RESTful, the marketplace expects an API library that they can install. We have to provide Python, .NET and Java libraries to invoke our service. This isn't very complex.

For Python, it would be simplest to leverage the urllib2 package. We can provide some classes which act as remote procedure call proxies; these classes have methods that invoke our REST services (GET, POST, PUT and DELETE) on various resources or collections.

Something like the following:

class MyProxy( object ):
    def __init__( self, host, port, username, password, realm ):
        self.urlBase= "http://%s:%s" % ( host, port )
        # Build Handler to support HTTP Digest Authentication...
        digest_handler = urllib2.HTTPDigestAuthHandler()
        if username is not None:
            digest_handler.add_password(realm, self.urlBase, username, password)
        # Build Handler to support HTTP Basic Authentication...
        basic_handler = urllib2.HTTPBasicAuthHandler()
        if username is not None:
            basic_handler.add_password(realm, self.urlBase, username, password)
        # Build Handler to treat 201 as a normal response, not an exception...
        error_handler= RESTHTTPHandler()
        self.server = urllib2.build_opener(digest_handler,basic_handler,error_handler)
    def request( self, method, uri ):
        assert method in ( "GET", "POST", "PUT", "DELETE" )
        data= urllib.urlencode( argDict )
        theReq= RESTRequest( method, self.urlBase + path, data )
        try:
            response= self.server.open( theReq )
            # fold in attributes that are compatible with Django HttpResponse
            response.status_code = response.code
            response.content= response.read()
            return response
        except:
            ... handle various kinds of IOError, HTTPError exceptions...
    def getSomeResource( self, key ):
        response= self.request( "GET", "/path/to/resource/%s" % key )
        ... examine response.content, maybe do simplejson decode or xml.etree parse...

The problem is that the Django test client and the urllib2 packages are wildly incompatible.

Okay, maybe not wildly , but seriously incompatible.

First, the Django Client's HttpResopnse includes attributes status_code and content. The urllib2.addinfourl response uses code and is -- itself -- a file-like object.

Second, and more important, the Django Client's HttpResponse is a dictionary full of headers. The urllib2.addinfourl is a file with an info() method that contains the headers.

Choices

We have a tantalizing set of alternatives.

Make urllib2's response look more like Django's response. This requires adding a few additional attributes, and a __getitem__ method. Not too difficult to do. But only because our unit tests are not very demanding.
Create a Facade over urllib2.addinfourl and django.http.HttpResponse that is independent of both, and can work with both as implementation classes. While cool-sounding, and easy to implement in our WS client package, we'd have to do a tiny bit of extra work in our unit tests to create a Facade-based client rather than use the default client.
Get a proper Python RESTful client. Like RESTClient or Python-rest-client . The approach in RESTful Python -- a decorator -- is another possibility.

The problem with #1 is that the Python client package we distribute will have this odd-looking design that adds a bunch of random-looking features to urllib2.addinfourl. A lot of explanation (like this Blog posting) doesn't remove the oddness. The Java and .Net packages will be fine.

The problem with #2 is that the Python client package will be even more complex than #1, with little recognizable value to anyone for the complexity.

There's no problem with #3. Indeed, this might be best in the long run.

The Schema Evolution Problem

2008-08-06T10:21:00-04:00

Fundamentally, we need to provide explicit version identification on a schema. This is technically easy, but organizationally nearly impossible.

Technically, we need to use some kind of version control software for our model and the resulting DDL. We need some meta-meta-data to track schema names and version numbers. If we like doing too much work, we can introduce a meta-meta-data table with schema name and version numbers. If we're lazy, there's an even simpler, more reliable approach.

Organizationally, we need the discipline to track every single schema change and determine the level of compatibility with application software. Garden-variety ALTER statements (to add columns, or extend the size of a column) won't break software; bumping the minor version number is fine. Adding new tables or views won't break software. Renames and drops, however, will break software and require a bump to the major version number.

What is a Schema?

First, a schema isn't the entire set of metadata in a single database instance. Even if your data is organized in one massive, flat schema with thousands of tables, you still have many smaller "schema" within that single SQL schema owned by "PROD" or "OPS" or "DBA" or "SYS" or whoever owns your production tables.

We'll distinguish between the practical, conceptual schema and the often-misused SQL schema. Sometimes they overlap, but this is rare.

Your smaller conceptual schemas are the "application-specific" subsets of your overall SQL schema. If you're smart, your SQL schemas match your conceptual schemas. If you're lazy, you have a single massive SQL schema and use table name prefixes to try and separate tables into smaller conceptual schemas.

Here's the bottom-line suggestion. Use SQL schema. Don't use table prefixes.

[In a Big IT organization, this can't happen because it would "break everything". Everyone depends on there being a single, flat anonymous schema. This isn't true, as new applications and maintenance to existing applications are an opportunity to restructure the SQL schema to match the actual use of the tables. Sadly, it only reduces future maintenance costs, so it doesn't have any current-year impact, so no one ever does this.]

What About the Applications?

The applications exist independent of the data. Stored procedures (A Configuration Management Nightmare ) are in the application model, not the data model, and evolve independently from the data schema. However, this isn't always understood, and stored procedures are often mis-managed.

An application could check the schema meta-meta-data to be sure that the application is compatible with the schemas it uses. It can be a simple query, and an exception gets thrown to indicate that the application can't start and run with the given mix of database schemas. We know that production programs shouldn't work with the new, upgraded integration test database. However, we also see this happen; sometimes they crash and we fix them, other times they don't crash, but don't produce right answers, either. Sigh.

There's a simpler approach, however, than a query.

Should the application and schema version numbers track? Should the application go through version 2.1.2 and 2.1.3 to indicate that it requires schema version 2.1? Not necessarily. There is not a tidy 1:1 mapping between software components and database schema objects. Generally, database schema objects are shared -- widely -- by software components. Version 2.1 of application X and version 4.2 of application Y may both depend on version 3.x of database schema Z.

How To Make This Work

Put the version number in the schema name.

First, don't create a bunch of XYZ_table1, XYZ_table2 names in a single, flat schema. Create table1 and table2 in schema XYZ. Use lots of schemas. That's why they're available to you.

[Yes, your historical, legacy applications didn't use schemas. I'm aware that this is new. Start now.]

Second, don't simply create a "timeless" XYZ schema, use the major release number as part of the name. Create an XYZ_2 schema. This will work for all 2.x versions of the schema.

When you move to version 3.1, create a new XYZ_3 schema. New. Migrate the data from the XYZ_2 schema. Then, rename XYZ_2 to XYZ_2_OLD, so that any program that improperly uses the old schema will throw an exception and die. When you need to recover the space, you can drop the XYZ_2_OLD schema, knowing that no program is expected to use it; any program that does use it, needs a fix.

Wait! That's potentially a lot of code to touch. Or, if your a mainframer, that's a lot of programs that need to be rebound to the new SQL. Yep. It is. It's a trivial administrative task. If you can't recompile or rebind your programs, you have serious quality issues that you must fix.

If you can't make simple SQL changes, you have serious flaws in your application software and your overall IT processes. You must fix these application design flaws and organizational process flaws. I'm sorry for pointing this out.

Implementation Steps

Name your schema. Allocate your tables to appropriate schema. More schemas is a better approach than fewer schemas. There's no performance penalty. Design and maintenance are simpler. When you design software, You don't design a single, massive, does-everything application, you write small, focused application programs. Your database, similarly, should be structured in small, conceptually simple modules.

For Java programmers, use iBatis to extract your SQL from your programs. The schema changes will be isolated to the iBatis configuration files, mostly.

For Python programmers, you can use SQLAlchemy to isolate most of the SQL from your overall application. Put each schema definition in a separate "models" file. Include the SQLAlchemy table definitions as well as the Python classes and the mappings. You can, without too much difficulty, include a few convenience functions that will create or drop-and-create the schema.

If you're creating Python/Django applications, consider including the schema version number on your Django application name. Your Django project folder for a given site might include things like someapp_1 and someapp_2. The older application (someapp_1) has one model, and the newer version (someapp_2) has the expanded, incompatible model.

Change Management

Rather than mess with an complex, risky in-place conversion, you are adding to the database. You can write a simple batch application to create the someapp_2 data objects from the someapp_1 objects. Once the data is migrated, you can switch the settings.py and the urls.py files to use someapp_2 instead of someapp_1. You can easily dry-run this conversion process in an integration test or staging instance of your web site. If it works there, you can do it again in production.

The best part about keeping the two schema in parallel for a time is the ability to fall-back to the previous version and try the conversion again after fixing the bugs. You're never replacing anything; you're simply adding a schema and directing the application programs at the new schema.

Stored Procedures Are A Configuration Management Nightmare (revised)

2008-08-03T16:44:00-04:00

I've been asked about the proper location of Stored Procedures more than once. I've come down very strongly in opposition to triggers and stored procedures.

First, PL/SQL is slow . Anecdotally, people claim that introducing PL/SQL made an app faster. I submit that they restructured the application significantly to create small, focused transactions, and that's what created the improvement. As a practical matter, you need to write focused, PL/SQL-like transaction methods in your Java programs. While technically possible, you can't casually execute SQL statements willy-nilly.

Second, it's hard to do configuration management on stored procedures. Not impossible, but very hard. The reasons are entirely organizational.

Recently I received an email that was nearly opaque, but seemed to indicate that the organization couldn't clone production to create another test, and couldn't rationalize the versions of their various stored procedures. I think they wanted a puff of Faerie Dust™ that would allow stored procedure X to determine if it was being used by package Y or package Z and behave differently in the different contexts. The request makes no sense -- this is just a version control issue. Clearly, there are two versions of X, but the emailer claimed there was one version of X and it had to determine it's behavior dynamically.

Conflation - The Organizational Root Cause

A stored procedure lives in the database. Consequently, it's conflated with persistent data and schema definitions and assigned -- for no good reason -- to the DBA's. These three things -- data, schema and processing -- have little to do with each other. They emphatically do not belong together. However, they're almost always conflated into a murky puddle of SQL.

Let's break these things apart.

Data: is the organization's actual data. Some (but not all) of the business records lives in managed databases. Some live in desktop application documents (word processing, spreadsheets, unmanaged desktop databases, etc.) Data is precious, perhaps the most precious thing in the organization.
Schema: (or metadata) is table, column, view and index definitions. It's also physical stuff like tablespaces, files, instances, etc. Some of this is important, some of it is subject to change without notice. Tablespace configuration parameters rarely matter except as an implementation detail.
Processing: is triggers, stored procedures and all of the application programs that live outside the database. Note that there is no crisp distinction between "low-level" and "high-level" processing. Many DBA's have tried to explain to me that CRUD rules are "low-level", but then they add some foreign-key relationships, after that they also need to add some many-to-many relationships and the intermediate bridge tables, then they start adding other things that are part of larger and more complex relationships. Stop! If you can't find a boundary easily, it doesn't really exist.

Version Control

Data -- typically -- has fairly loose version control. The RDBMS often has secret sequence numbers (SCN's) that are used internally to manage cache and synchronize physical files. These transaction sequence numbers are, effectively, a kind of version number for the data.

Often, we'll have a "last changed date" in a database record. This is a surrogate version number for the record. It tells us when the data changed. Most applications don't record a complete change log for the data, we simply update the change date. A few applications do create detailed change logs. In some cases, people try to leverage the database logging facilities to back into a formal change log for the data.

Schema is rarely under any kind of version control. Metadata is often the least disciplined part of the enterprise infrastructure. It's easy (really easy) to have formal version control over metadata. It's rarely done, however. For some reason, DBA's don't seem to use version control software.

Application software, typically, has the best version control. Many organizations use some formal version control software (CVS, Subversion or some commercial product like MKS, VSS or PVCS.) This can easily apply version control information to the source code (and even the resulting .class files.)

[Some organizations can't even put their application software under version control. This doesn't change the issue of conflating data, schema and application.]

For no good reason stored procedures are the province of the DBA's (who don't use version control software.) Consequently, the external application software (in Java, Python or whatever) may have version control information, but the stored procedures never have version control.

Schema Versions

The database schema (the tables, columns, indexes, views and sequence generators) has a version number. The version number for a schema -- like the version number for software -- defines "compatibility".

Schema version 2.1 and 2.2 are "compatible" in some sense. Schema versions 3.5 and 4.1 are incompatible.

What defines "compatibility"? Clearly, "compatible" means "compatible with SQL DML". If you've done standard database ALTER statements (adding columns, expanding the sizes of columns) or changing indexes or adding views, you haven't broken any SQL DML. The old SQL still works with the new schema. This is a 2.2 to 2.3 kind of change.

If you've dropped a column or table or view, or you've shortened a column, or changed the type of a column, then you've made a change which may break existing SQL. When you've make this kind of change, you'll need to bump the major version number.

You need two things:

Discipline.: This doesn't happen by default.
Some meta-meta-data.: A table that has schema names and version numbers is all you really need. It's nice to fold in "applicable dates" and "responsible person", etc., but not essential. In some cases, you can use database comments for this.

When you make database changes, you must create a script that (a) makes the change and (b) updates the database schema version table. That's about it.

What About Stored Procedures?

Why can't we annotate our stored procedures with version numbers and put them under version control like the rest of the database?

The question is rhetorical. Of course we can put stored procedures under version control. It just requires some discipline. And -- perhaps -- making stored procedures part of application software's responsibility, and not part of the DBA's job.

If we take stored procedures away from the DBA's, we need a formal turnover procedure for putting a particular suite of stored procedures into a database.

Separating the stored procedures from the schema via a formal turnover has some marvelous consequences.

You can reconstruct the stored procedures from your source code repository exactly the same way you extract your Python or Java. Indeed, you can make a complete software package with all of the various language elements. You can extract all of the procedure creates as a big script and run it any time you need to.
The database has two distinct parts: the Data, the Processing. These two are matched by schema version number. The DBA's are responsible for the data; the schema versions; the preservation of essential corporate information. The DBA's are also responsible for running the scripts that upgrade that portion of the application software that happens to live in the database. The DBA's aren't responsible for stored procedures.
The migration of a database from development to test is a two-part job. Move the schema and data from the developers to a test environment. Separately, run all of the scripts to build the proper software version that matches the schema of the data.
You have explicit compatibility checks. Version 2.x of schema and software are being used in production. Version 3.x of schema and software is in some kind of parallel test prior to conversion. Version 3.y of schema and software is in some early test; 3.z is in development.
You can begin to wean yourself away from the nightmare of stored procedure management. Once you take this out of the DBA's hands, you find that a consistent set of Python (or Java) packages that define the Model layer does everything that stored procedures and triggers do, only more simply and more maintainably.

What's So Hard?

It's very easy to put stored procedures under explicit, clear version control. With a little care, even a database schema can be put under version control.

What's so hard is actually making the organizational change. Ask around. The DBA's will tell you that they are overworked, because they're "forced" to write all the stored procedures and triggers. Forced? By whom?

Generally, the "organization" seems to mandate that everything SQL -- tables, columns, indexes, views, stored procedures and triggers -- pass through the DBA's. The distinction between data and processing is somehow lost. Splitting it up will often anger the manager of the DBA's, who'll make the case that no one else can be trusted to create stored procedures.

When testing stops because of version control issues, when production fails, it seems like the problem should be addressed. It's usually obvious that there are serious version control problems between the schema and stored procedures.

I only know that there's a long-standing, steadfast refusal to split the database into data and processing elements. The consequence of this is that stored procedures are unmaintainable, testing is nearly impossible, and production problems are rampant.

Consequently, I suggest that stored procedures and triggers never be used. Ever.

Denormalization or "What did you mean by that?"

2008-06-14T11:59:00-04:00

I use the word denormalization heavily, to make a point to a certain class of developers. Other developers object to the term, since it doesn't have a precise meaning.

The point I often have to make this:

3rd Normal Form is for Updates.
Data Warehousing is about Insert and Select; there are no Updates (to speak of).
Consequently, the traditional normalization rules (Third Normal Form a/k/a 3NF) doesn't apply to data warehousing.

My habit is to describe the star-schema (or snowflake schema) as "denormalized". This isn't really correct, but it does emphasize my point. I have to make this point emphatically because we have to get past the Data Cartel's Standard Objection: New Technology Won't Work. Most DBA's who are new to Data Warehousing and the star schema will exercise their veto authority over new technology, claim that the design is "inefficient" and stop (or delay) the project.

DBA Objections

DBA's can object in Passive-Aggressive (and Passive-Aggressive Part II ) mode -- where they don't have a better solution, they just have "concerns" about the standard DW solution. Here are some things I've heard.

It isn't normalized. Which is a WTF? kind of point. It isn't normalized for updates because there aren't any (to speak of). It's normalized for SELECT SUM(*) GROUP BY, which is the canonical dimensional query. I call this "denormalization" to make the point; perhaps I should call it star-schema normalization.
It doesn't use "natural keys" correctly. I'm pretty sure that natural keys don't actually exist. Almost everything is either an attribute (which can change) or a surrogate key (which isn't very likely to change). A changeable attribute isn't really a key, is it?

When writing ETL programs, we sometimes have a blurry edge when an external application assigns a truly permanent surrogate key. In these cases, the external surrogate is often something that the organization uses heavily -- as if it was a natural key. In other cases, they have a surrogate-like key that can (it turns out) change, making it just an attribute. In the warehouse, it's usually best to simply assign warehouse surrogates and not burn up brain calories trying to make too many distinctions in the source applications.
All those joins are inefficient. This can -- in the extreme case -- lead to The Uni-Table. This is the pre-joined ur-fact table that contains all dimensional attributes and all fact values. It works, but it repeats all of the dimensional attributes and it doesn't track dimensional change at all. Yes, I've seen it done.
It uses too much storage. This is just silly, but it comes up. Once, I caught the sysadmins and DBA's in a meeting where they were quibbling about log sizes so that they could micro-manage storage at the 100Gb increment. "There's four people in this meeting. At your hourly cost, I could have bought 400Gb at Circuit City." And the price of storage continues to plummet. Nowadays, I think I could buy a terabyte.
Fact updates can be inefficient. This is crazy, because changing a fact's measurement value is a single row update; it's fine if you're correcting errors. Changing a batch of fact's measurements is -- what? -- criminal mischief? Who changes batches of facts? Considerer deleting the incorrect ones and reloading correct ones.

Changing the association between a batch of facts and a dimension is even spookier. The historical fact is what you recorded. You don't get to change the facts; it's called perjury. If you're restating your books, you usually have new facts that apply to a historical time period.

Star Schema Normalization

To get past the DBA objections, we need to have several heart-to-heart conversations on star-schema normalization. Generally, these are painful because DBA's are overworked and sometimes underqualified. Examples on paper don't help much. Telling them to read Kimball does nothing. Loading up realistic sets of data is the only workable approach to showing them that the storage is manageable, the joins won't kill you, surrogate keys work, and the star schema is a "real" thing.

Once we've aired out an example, we then have to revisit the star schema thing over and over again. Most DBA's are so habituated to 3NF that they can't get past it to see that a star schema is an alternative normal form. Except in rare cases, the best we get is grudging tolerance. [In the rare cases where the DBA's embrace a star schema approach, no one needs me, except to validate the design.]

The basic 1NF and 2NF rules apply to the star schema normal form as well as transactional normal form. Arrays are still a bad idea in the relational world. Foreign attributes (those not functionally dependent on the key) are still a bad idea. However, 3NF is out the window -- derived data is a helpful thing.

Derived Data? What About Updates?

DBA standard objection #5 -- updates hurt -- often surfaces when discussing the approach of persisting derived data. This is a focused "denormalization" that unwinds just 3NF to avoid repeating a calculation. In the case of a data warehouse (load once, query an infinite number of times) all calculations done at load time are amortized across an infinite number of queries, making them delightfully efficient.

The "update" issue can't arise. Let's look at some common dimensions.

Time.: You don't change the day of the week for March 8, 1987. It is, was, and always will be Sunday.
Space.: Geographical boundaries change. However, this is the canonical Slowly Changing Dimension (SCD) problem that Kimball covers in detail. [If you have what Kimball calls a "type 3" SCD, you have the most common example of an update in a data warehouse; the change of status from "current" to "previous".]
Customer.: Your customers (either individuals in huge collections or other businesses in small collections) have numerous changes. However, they often have attributes which can't change as well as attributes which frequently change. For example, demographics change very slowly (if at all). Customers often requires more sophisticated "snowflake schema" techniques. There still aren't any updates, but there are SCD techniques for handling this.
Product.: Your products, product lines, product families, product groupings, solutions, technologies, platforms, services, etc., are all grouped by marketing in the randomest ways. These groupings and hierarchies and clusters and affinities are just ways that marketing tries to portray your company; and it changes with every whim and brain-fart. This is also a basic SCD issue; you simply add the alternative hierarchies and groupings and do alternate joins on the facts.
Cost Centers.: Your internal cost structure changes. Sometimes frequently. This is still SCD. No updates, just inserts.

Recent Example

Recently , I aired out some brain-dead non-solutions to a simple reporting problem where number of rows (500 new rows per hour) might have been the design problem being solved. The non-solution involved a five ways of avoiding a viable solution. As follow-up to my suggestions, I was given the following variations on DBA objection 1; each affixing blame somewhere else.

1.1. The customer cannot accept a "denormalized" table that pre-computes the values. [The customer is at fault.]

1.2. Since we can't directly use the "denormalized" table in your blog posting, the idea of denormalization is broken, and we can never talk about it in any form whatsoever. Temporary tables, materialized views and other techniques are off the table, a priori. [I'm at fault for not providing the expected solution, which involved some kind of Faerie Dust™ that would make a bad table process quickly.]

1.3. The organization can't learn anything new. Talking about "denormalization" would be new, and is therefore forbidden. The idea of persistent derived values is off the table, a priori. [The organization is at fault.]

At this point, any suggestion I might have has been trumped by the DBA's opposition to denormalization. Blame has been assigned everywhere. I think this is because I used the word "denormalization" incautiously and set myself up for three flavors of DBA objection #1 ("It isn't normalized.")

Perhaps, if I'd said "persistent derived values" instead of "denormalization" we might have gotten somewhere. Ideally, they would have suggested a temporary table or materialized view as an implementation technique. But, we stalled out at my incautious use of a loaded buzzword.

Genius Move -- Characteristic Functions

2008-06-07T13:54:00-04:00

The comment was eaten by Haloscan, but here's the text...

You need to read Rozhenstein on characteristic functions.

select
sum(case when a < .5 then 1 else 0 end) 'A'
,sum(case when a >= .5 and a < .75 then 1 else 0 end) 'B'
,sum(case when a >= .75 then 1 else 0 end) 'C'
,bar
from foo
group by bar

So, I googled it, and figured out what I'd been missing.

Skip the Math

The Google page on characteristic functions is heavy going. The issue here is to characterize the frequency distribution of some more-or-less random variable. This is a real close fit with the formal definition of a characteristic function.

When should we apply the characteristic function? Load time or query time? The comment showed it at query time. However, we could also do it at load time.

Here's the genius part.

If we define it as a separate function, we can defer this decision based on which implementation meets our performance guidelines.

We have this situation.

def c1( value ):
    a,b = divmod( int(value*100), 10 )
    if b == 0:
        return "== 0.%d" % ( a, )
    else:
        return "0.%d - 0.%d" % ( a, a+1 )

We can then use this during load or we can use it in a fetch loop. Quite cool. Very elegantly separated from other parts of the processing.

My Query Is Slow -- What To Do? Or Dumb-As-A-Post SQL (Revised)

2008-06-06T22:30:00-04:00

First, let me point out that the Data Cartel ("DBA" means Don't Bother Asking) won't release all the information I requested, so some of this is a guess.

We'll look at a number of dumb-as-a-post SQL techniques. This is proof -- if any were needed -- that bad SQL is worse than no SQL.

The table appears to have 2 columns, a date and a floating-point value in the range 0.0 to 1.0. Rows arrive at the rate of 500 an hour.

Someone wants a weekly summary (about 90,000 rows) binned into 10 ranges 0.0 to 0.1, 0.1 to 0.2, 0.2 to 0.3, etc. The algorithm might be slightly more complex (to separate \(n = 0.1\) from \(0.1 < n \leq 0.2\).)

[Again, the DBA steadfastly refuses to provide the use cases, so I'm doing a lot of this with minimal information. However, I did get a spreadsheet showing an Excel version of the algorithm. Not PL/SQL, not pure SQL, not Java, but Excel.]

What's the Issue?

The issue is that some programmers can't be trusted to find their ass groping with both hands. I was sent three versions of the obvious SQL query, each more contrived and senseless than the last.

I was asked -- really -- "What is a more scalable approach to the problem ?". "Scalable"? WTF? Scalable with respect to what? Rows? Physical I/O's? Elapsed Time? CPU use? User queries? Web page hits? Shots of Tequila? If I look at the Zachman Framework or the SEI Quality Measures Taxonomy , I can come up with at least a half-dozen more dimensions of potential "scalability".

[Yes, I asked. No, I didn't get an answer. "Scalability" appears to mean the same thing as "Better".]

I'm assuming that the volume has increased or something, and the old query isn't fast enough. Or something. There's a claim that some query is run 83 million times each week; that's 138 times per second, a number I just don't believe.

Fetishize a Feature

Someone has an Oracle Bulk Bind fetish. I've listened to this tripe before. There are probably places where it helps. I haven't seen any, but I haven't really made a study of the feature. Apparently, they couldn't get it to work for the required 80,000 rows. It gets what they call "the standard ORA-04030 error."

The sent me a copy of solution one: a big pile of PL/SQL including some BULK COLLECT stuff. PL/SQL they couldn't get it to work. I'm not sure what's going on here, but it's clearly the first dumb-as-a-post SQL programming technique: Fetishize a Feature. You pick something and stick with it.

Drown it in Documentation

Here's the best part of solution one. It didn't work. And they provided me with extensive documentation -- on the feature they couldn't get to work. I like that. So technique two is to quote a lot of documentation -- as if Drowning It In Documentation somehow make the feature start working.

I suppose I could try and debug it, but I really don't have the patience. There are simpler, provably faster techniques. Why debug something that is highly Oracle-specific, and doesn't seem to work very well?

Write More Code

Solution two was to purge the bulk-bind syntax from solution one and see if a big pile of PL/SQL will work. PL/SQL is a demonstrably slow platform. There are some anecdotal stories of applications that were made faster by replacing external application programs with PL/SQL. I believe that those stories involve comparing the performance of a Bentley with an Etap 37S. One's a car, the other's a boat. PL/SQL is faster when you change your application design to make better use of PL/SQL features.

In this case, the PL/SQL solution is a huge amount of code for something that is -- as far as I can tell -- a SELECT COUNT(*) GROUP BY operation. It's hard to be completely sure, since the code is bad, and obscures the intent.

Rather than summarize and simplify, they Wrote More Code.

Don't Do The Obvious

Another generally dumb technique is to avoid writing the obvious SQL because -- well -- I don't know why. I don't have the actual requirements. However, each example strives to produce one line of output with the frequency table spread out horizontally. This is fairly hard to do in SQL, and requires lots of copy and paste programming to repeat the CASE expressions over and over again.

The basic SELECT COUNT(*) GROUP BY produces a number of rows, each of which has a key and a count. This can be rotated into a horizontal configuration by a reporting program. For some reason, we're locked into a single form for the report, making it so we can't Do The Obvious.

Refuse to Change the Structure

Data structures and algorithms are two complementary sides of the same coin. You can't fix the algorithm without fixing the data structure, and vice versa. In this case, the table design was bad, but no one seemed prepared to fix it.

About a year ago, I had told a member of data cartel to read Ralph Kimball's Data Warehouse Toolkit. They claimed they read it. Since they got nothing out of it, I'm not sure what they meant by "read". Data warehouse folks know that you have to denormalize for reporting. A relentless focus on "normalization" -- when dealing with non-updatable reporting-only data -- is simply wrong.

In this case, the floating point numbers had to be split up into bins. The calculation must be done at load time, and must be a permanent part of the table.

TABLE data(
time DATETIME,
value FLOAT );

This isn't really sufficient for reporting. You need something more like the following:

TABLE data(
time DATETIME,
week INTEGER,
month INTEGER,
year INTEGER,
value FLOAT,
bin INTEGER );

The various derived values are all trivial to calculate at load time. Once they're calculated, your query reduces to a trivial SELECT bin, COUNT(*) FROM DATA GROUP BY bin. It isn't the absolutely fastest way to process the data, but it's a far, far sight faster than on-the-fly CASE expressions or PL/SQL loops.

The Hubris of Time Calculations

There's more that's wrong in the various examples I was sent. Specifically, they use "closed-ended date ranges". A serious mistake that is caused by simple hubris. Time is subtle and complex and easy to get wrong.

Here's their code.

time >= TO_DATE( '05/01/2008 00:00:00', 'MM/DD/YYYY HH24:MI:SS') AND
time <= TO_DATE( '05/08/2008 23:59:59', 'MM/DD/YYYY HH24:MI:SS');

It can't -- in general -- work.

There's a 1-second gap between the two times. You have use half-open intervals to avoid losing a row that happens to have a timestamp in the gap. [Don't waste time adding .999's, either, because the decimal value doesn't provide down-to-the-last bit way to encode the internal binary values.]

time >= TO_DATE('05/01/2008','MM/DD/YYYY')
AND time < TO_DATE('05/08/2008','MM/DD/YY' )

This has NO gap.

However, this still isn't very good. As shown in the table definitions above, you need to denormalize the time-stamp into the buckets you actually want to use for selection and grouping.

Real Speed

I don't have the table or sample data, so I can't compare my results with their performance numbers. However, their numbers are sad.

First, they couldn't get the bulk bind to work, but sent me the code, as if it mattered.

Second, their massive PL/SQL loop ran for an hour. Apparently, this is unacceptable, but they sent me the code, as if it mattered. Which is sad.

Third, their SQL GROUP-BY with all the CASE expressions ran in 12 minutes. I don't know if that's too long or uses too much memory or takes too many tequila shots.

The real SELECT COUNT(*) GROUP BY, with denormalized data, is fast. On my little 1Gb RAM, 1.7Ghz Dell, running Fedora Core 8 and using SQLite, a basic SELECT COUNT(*) processes 100,000 records in about 3 seconds.

That's about as fast as this little drip of code.

import collections
    count= collections.defaultdict(lambda:0)
    for row in q.execute().fetchall():
        b, exact = divmod( int(row[1]*100), 10 )
        band= "==0.%d"%(b,) if exact == 0 else "0.%d-0.%d"%(b,b+1)
        count[band] += 1
    print count

In SQLite, for 100,000 rows, this is the same speed as SQL. Why? Because we're not asking the database to do anything much more than fetch rows.

Interestingly, in Oracle, the SELECT COUNT(*) GROUP BY is much, much faster. Why? Because Oracle queries involve a context switch, where SQLite does not. A simple fetch loop in Oracle is relatively slow without using some kind of buffering.

The database fetch time still dominates what we're doing. A table design change, and doing all processing at load time will minimizes the query time.

How Many Bad Things Can We Do?

Let's enumerate them:

Fetishize a Feature
Drown It In Documentation
Write More Code
Refuse to Change the Structure
The Hubris of Time Calculation

All of these habits get in the way of a simple denormalization that makes the obvious query work at amazing speeds.

The Django World-View: Model+Admin First; Built-in Transparency and Trustworthiness

2008-03-24T18:24:00-04:00

See Michael Hugos "Think about screens and the data on them to simplify system development " for some helpful insight on what an "application" really is -- access to data. Simple transparency is lifted up as a critical value for software.

I liked the "If you don't believe it could be this simple, consider the reasons for your response" insight. I didn't like Hugos' sample response, "complex code that you can brag about". I don't think complexity for the sake of complexity is a real problem.

Complexity can be defined as everything that separates the user from their data. As Hugos' notes, the data model and the simplest, most direct presentation is the best design. Everything else is complexity that obscures the real purpose of the software.

Complexity comes from several sources. I've blogged about complexity before , ever since Mark Grand gave me the hint that complexity was part of the IT culture.

Here are seven kinds of complexity that get between a user and their data.

"The Conflict Is The Problem". The inherent conflicts in the relationship between developers and buyers or users make the problem appear complex. Often this is because buyers insist on their solution -- irrespective of the actual problem. Rather than describe the underlying problem, buyers describe a solution based on their favorite technology. They insist they have to do this because the job of the business analyst is to translate the business problem to technology terms -- usually oriented around a complex non-solution. After all, when you try to solve a business problem with a spreadsheet, you've created two business problems.
"Fear of Showing Weakness". Simplicity isn't valued. Some aspect of the problem is (or appears) complex, so we need lots of complex software. Many "business rules" are transient; an orientation around the decisions a person needs to make is more helpful than over-specifying something that handles 1% of the dollar value of an application. Rather than simply expose the data (and the business process) to the people, we overdesign "automation" that makes the exceptions and special cases a larger and more complex problem than they deserve to be.
"Quality vs. Quantity of Ideas". It's hard to let go of the first idea, no matter how bad it is. Someone with deep experience in legacy technology will often be the root cause of complexity. Just because batch processing was once the vogue doesn't mean it is essential or even necessary. Many, many things can be handled via an asynchronous message queue rather than an overnight batch process.
"Form vs. Function". If we fail to define the problem in the first place, we don't know what problem we're solving. We're left applying technology inappropriately, filling in the form of a solution, manufacturing complexity because we're vague on what the actual function should be. Rather than simply present data, we feel that application logic is "important" and should be part of the system; simple presentation of data isn't appropriate.
"When I Grow Up". If we don't have a mature process for solving problems, we're stuck applying inappropriate technology. Often we're forced into building something prematurely, and the previous problems all surface: we lock onto the first bad idea, we don't back down from that bad idea, it fills the form of software we think we understand.
"If I Had A Hammer". Tied in with the lack of quality ideas or well-defined problems, we make inappropriate use of tools or solution design patterns; we view all fastener problems as nails because we only understand hammers. We have very, very sophisticated application software development tools; we don't need to write mountains of code when we have sophisticated technology stacks like Linux/Apache/MySQL/Python and Django.
"How Hard Can It Be?" Failure to assess risks appropriately biases users against a simple solution. More programming seems -- in some views -- to be less risky. More automation isn't a solution. Appropriate controls are more important than volume of software.

Read Thibodeau's "D.C.'s tax system won plaudits but couldn't stop alleged insider thefts ". Complexity and technology aren't the answer. Good old-fashioned controls and audits are what matters. Audits and controls require transparency, not complexity.

How Essential Is Unit Testing? Or, How Do We Make It Essential?

2007-12-24T11:31:00-05:00

See Present Perfect for some thoughts on unit testing. See some other commentary on the discipline required to write Python programs in Gnome devs too lazy for python . I think I see the disconnect that makes testing appear to be too costly; I think that some basic "meta-quality attributes" are essential to understanding unit testing.

Here's the original C# vs. Python analysis in Monotonous .

Here's the famous quote: "Writing real applications in Python requires a discipline that unfortunately most people (including myself, at that time) are unwilling to adhere to, and this easily leads to buggy and hard to maintain programs. You have to be very diligent about unit tests and code coverage for every line of code, because you can’t rely on the compiler to catch errors for you."

I didn't take careful notes at a client meeting where this came up, so I don't have good quotes. The client balked at the very idea of Test Driven Development. In a larger presentation on testing (technology, environments, process, etc.) I had lifted up TDD as a direction that would benefit the developers. The response from the director of development was a series of "how would you do that?" questions.

These weren't practical "how to" questions. They were rhetorical "that isn't possible" statements, framed as questions. In order to portray TDD as impossible the questions quickly devolved into how TDD interacts with requirements gathering and business analysis; I couldn't successfully bracket the questions as part of the fringe of TDD. I think the disconnect was their certain knowledge that test cases come only from requirements and nowhere else.

It Hurts When I Do That

TDD is -- certainly -- a pain the neck. I think I see two complaints. First, it's a lot of "extra" code. I'm guessing that there's a "non-deliverable" view of test cases that pervades some people's thinking. I've been measuring the lines of code in both parts of a project, and the total volume of source is about 50% test cases and 50% operational.

Many years ago, we made a distinction between "deliverable" and "non-deliverable" software. We used to carefully segregate any non-deliverable software so that we could wring our hands over how to estimate the cost for it. Since it wasn't "deliverable," some managers felt we couldn't charge the customer for it; the logical conclusion was that we should exclude it from our project plans.

I maintained that "non-deliverable" is still "essential", so we must include it in our plans. The "compromise" was to inflate the estimated size of the deliverable to include a pro-rated version of the non-deliverable code. The claim was that non-deliverable software was half the cost of deliverable software. It had less documentation and less testing or some such.

I Can't Cope With That

The second complaint seems to be that programmers can't be trusted. Because they can't be trusted, we must have full definitions of all deliverables, complete up-front design, rigorous schedules for code creation, and a fungible, compressible schedule for testing.

We can't engage in an agile test-driven process for a number of reasons. First, and foremost, programmers are the root cause of scope creep. We all know that programmers will "gold-plate" the simplest thing; they'll spend years polishing and improving something of limited business value. [Why did we let them get started on something of limited value? Why aren't we willing to invest in making it work correctly every time?]

We can't trust programmers to do just enough design because they are lazy slobs and won't ever get anything to work. If we try to let them fire at half-cock, they'll never get anything useful accomplished. After all, we -- as managers -- have a vision of something trivially simple. The programmers keep introducing some technology nuance that makes a simple thing horribly complex and difficult. [Who -- specifically -- told us that introducing new technology will be simpler? Or did we just make that part up?]

We can't trust programmers to evolve code, design and test hand-in-hand. If we did, there'd be scope creep and they'd just play with the technology. Worse, of course, they'd miss the schedule.

We certainly can't trust programmers and end-users to collaborate. If we did, they would change the focus of the project, and the schedule might be missed. As managers, we don't fully understand the business value proposition; we don't completely get the technology, but we do understand the schedule. Since we really, truly, deeply understand the calendar, that is the one thing we can manage to. [Why is schedule more important than delivered features?]

Above all, we can't trust programmers to create test cases. Only end-users can create tests, and those tests must be married to the requirements. There's no reason to elaborate the tests to match the design, or elaborate the tests to match the details embodied in the code. Tests based on design or programming amount to letting a programmer do their own tests; programmers are untrustworthy; therefore this can't work.

The TDD Alternatives

I think there's just one disconnect underlying this. This disconnect manifests itself as two alternatives to TDD. The static language folks seem to like the idea that compiler type-checking is an alternative to testing. I suppose -- to a limited extent -- this is true. Rather than write a unit test to examine proper integration among classes or modules, we can trust the compiler.

Everyone knows -- or should know -- that the compiler is easily fooled. When using externally developed JAR files, we can easily compile against one version, and try to execute against a different version. All the compile-time type-checking in the world can't cope with mis-configuration. Eventually, we need Python-style dynamic testing.

When we can't trust our programmers, we have a number of clever alternatives to TDD. The primary approach is to define a process that imposes a waterfall approach to developing unit test cases in parallel with the code. When asked "How does TDD interact with requirements gathering?" no answer I could give was acceptable. What they wanted me to say was "Oh crap, you're right, I'm such an idiot. Test cases are only based on requirements, never design or programming." They wanted me to agree that programmers can't be allowed write their own test cases.

The Disconnect

I think both viewpoints stem from looking at testing as "final", "end-user" or "acceptance" testing. If testing is only for acceptance, then we should have other ways to test that the programming is correct and the design really works. The compiler should -- somehow -- validate our basic programming via static type analysis. The design, similarly, should be checked via some static analysis. That leaves testing to focus on the requirements and nothing else.

Additionally, we can't trust one person to interpret the requirements as test cases and as code. We must apply a second pair of eyeballs to the requirements to create the acceptance-oriented test cases.

It appears to me that TDD is dismissed as worthless because people don't see a need to test their designs or programming. Either they hope that static type analysis will do this, or they simply dismiss this testing as worthless.

It's hard to create a value proposition for testing the design and programming. It requires emphasizing a sense of distrust. And the level of distrust is already fairly high. After all, unit testing requires a level of discipline that programmers are unwilling to adhere to. The idea of adding testing at the design and code level only gets into complex philosophical discussions about "the role of requirements".

It's hard to break testing free from "End-User Acceptance." However, if we can portray testing as essential, it then becomes deliverable. Indeed, it becomes essential to establishing confidence in the software. It also becomes part of the documentation, since each API is demonstrated by at least a test case (in some cases, a whole test suite.)

Meta-Quality

The logical conclusion is a set of meta-quality attributes. Software quality attributes can be based on the SEI Quality Measures Taxonomy http://www.sei.cmu.edu/str/taxonomies/view_qm.html. This taxonomy includes need satisfaction, resource use, maintainability, adaptability and cost factors.

Meta-quality includes the quality attributes of the test cases. There are probably a number of quality attributes regarding things like 'traceability to requirements", "class coverage" and "method coverage" that determine how useful and complete the test cases are. Looking at "traceability", we examine how the test cases apply to end-user acceptance.

I look at "class coverage" as a way to to look at the design. This includes classical "class-in-isolation" unit tests, as well as module- (or "component" or "package") -level unit tests that examine a collection of classes to be sure that they interact properly. This makes limited use of mock objects, since this is looking at integration of classes and modules.

The "method coverage" is how we look at the programming. This includes appropriate test cases to exercise each method more-or-less in isolation. This level of testing makes heavy use of mock objects to be sure that the code in each method is actually correct.

I think that these meta-quality attributes of the test case code is as important as the quality attributes of the "operational" code.

User Interface Testing

2007-08-14T10:34:00-04:00

The question seemed simple, which testing framework is the simplest? The situation is complex. There's a web application, there are developers and there are testers. The developers develop, and the testers test. So far, not so complex.

Here's the complexity. The testers are pretty focused on manual point-and-click testing. They didn't like HttpUnit , declaring it too complex.

What's simpler than HttpUnit?

At first blush, my answer was to look at Selenium . This is a widely-used, easily automated toolset for browser and UI testing. But further conversation showed that this is the wrong approach.

They aren't deeply interested in the kind of cross-browser testing that Selenium does well. They're more interested in the essential functionality testing that HttpUnit does. They need to know that the application works with the given target browser. Articles like "Selenium rocks - and you don't need it " help to clarify this distinction between Selenium and HttpUnit

My next answer was to look at Twill . Articles like the Advogato "Introduction " are very compelling.

It turns out, though, the real problem isn't "complexity" per se . The real problem is that the testers aren't interested in writing sophisticated test scripts. They know the application, they know what they want to see, and they don't feel that programming is the best use of their time.

Unit Testing 101

This wasn't my idea, I'm just relaying the insight I got from the conversation. I was busy shilling shamelessly for Twill when the real solution surfaced.

The smart answer isn't to give the testers more tools. The testers (as currently managed) don't see a need for tools. The smart answer is to have the developers made officially responsible for unit tests, in HttpUnit (or Twill). The developers need to put the unit tests into the source tree along with everything else. They need to run the unit tests themselves.

The official "testers" are now freed from the "test everything" requirement. Instead, they can now do "guerilla testing" as well as review the unit test logs.

At some point in time -- and at a higher level in the organization -- the testers need to be encouraged to use powerful scripting and unit testing tools as force multipliers. They can claim that HttpUnit is too complex, but that's because they're looking at the wrong thing.

They need to see that they can only point and click so fast. A tool like Twill or HttpUnit can point and click a whole lost faster. Until they're rewarded for speed, they don't have any incentive to master a tool. Until they're given the incentive, every tool will be labeled as "too complex".

Another Dimensional Model Implementation

2007-05-26T01:14:00-04:00

The Cubulus project and Alexandru Toth 's page describe an "OLAP Aggregation Engine". It is very nice to see advanced work done on the dimensional model.

The cited research dates from 1999 (V. Markl, F. Ramsak, R. Bayer, "Improving OLAP Performance by Multidimensional Hierarchical Clustering", Proceedings of the Intl. Database Engineering and Applications Symposium , pp. 165-177, 1999.) I'm suspicious that it predates the "bit-mapped index".

It may be that this technique helps a lot with an RDBMS that doesn't support the star schema via bit-mapped indexes. It may be that this technique only helps a little with a more modern RDBMS.

However, the idea of a nice, tidy Python application that helps manipulate the dimensional model is a great thing.

Just for a moment, I though I'd found something SQLAlchemy doesn't do perfectly.

2007-05-18T17:40:00-04:00

After having written a number of application-specific object-relational mappers, I have been on the prowl for an elegant, enduring solution. I had started to come to grips with Django , and like much of the approach. Django has a tiny infrastructure feature (the settings.py file) which made it unpleasant to separate the ORM from the rest of the framework. (Not impossible, just fleetingly unpleasant.)

My first look at SQLAlchemy made it look over the top. However, after the PyCon 2007 presentation, I realized that the layers were cleanly separated, and I could use the ORM without messing about in the SQL-in-Python-Notation layer.

Then, I figured out ("PL/SQL vs. Java, Which One is Really Faster? ") that stored procedures were slow. Given that PL/SQL is slow, what else in the RDBMS world is slow? How much SQL is too much SQL, when speed matters? That answer is forthcoming -- I'm still fussing around with experiments.

Problem Child

The central issue started out as the all-too-common situation of Disjoint Subentities. This is where a single table has distinct classes of entities. The usual symptoms of this are indicators or NULL columns. Often, both are used. Sometimes, the indicator is omitted, and the pattern of NULLs has to be used to discriminate among the entity classes.

In this specific experiment, a single table has two subentities, each with different granularity. One subentity has to be summed to match the grain of the other. This gives us a union of two kinds of SQL queries: detailed and summary.

The detailed query, in SQLAlchemy, looks like this:

qrySingle= select(
    [stuff.c.groupName,stuff.c.amount,literal(1)],
    and_(stuff.c.status=='unmatched',
        stuff.c.subtype=='single'))

In SQL, this is

SELECT "stuff"."groupName", "stuff".amount, ?
FROM "stuff" WHERE "stuff".status = ? AND "stuff".subtype = ?

This is precisely the SQL that would be coded "by hand". The literals (1, 'unmatched' and 'single') are bound into the SQL at run-time.

The summary query looks like this in SQLAlchemy.

qryMulti= select(
     [stuff.c.groupName,func.sum(stuff.c.amount),literal(2)],
     and_(stuff.c.status=='unmatched',
             stuff.c.subtype=='multi'),
     group_by=[stuff.c.groupName])

And produces the following SQL.

SELECT "stuff"."groupName", sum("stuff".amount), ?
FROM "stuff"
WHERE "stuff".status = ? AND "stuff".subtype = ?
GROUP BY "stuff"."groupName"

This is all very pleasant. You can see that the literals (2, 'unmatched', 'multi') are bound in at run-time. This technique often leads to a speed-up because the SQL statement can be reused by the RDBMS. When coding by hand, this is easily overlooked.

Real World

In the "real world", that is, the world of my clients, this kind of query is distressingly common. And doing simulations and architectural recommendations is often made complex by having to cope with these kind of table designs.

To work with this table, I needed a union, and (for a brief time) SQLAlchemy couldn't generate the correct SQL.

Here's my union in SQLAlchemy.

invQry= union( qrySingle, qryMulti )

Here's the SQL which was generated. Note that the GROUP-BY vanished.

SELECT "stuff"."groupName", "stuff".amount, ?
FROM "stuff"
WHERE "stuff".status = ? AND "stuff".subtype = ?
UNION SELECT "stuff"."groupName", sum("stuff".amount), ?
FROM "stuff"
WHERE "stuff".status = ? AND "stuff".subtype = ?

Very disappointing. However, it's since been fixed. And the amazing speed of that fix is more reason to love SQLAlchemy and the folks who support it. Many thanks!

Now we can continue investigating which is faster: "Pure SQL" (i.e., complex stored procedures) or some programming language which uses SQL as necessary for persistence.

Dejavu and Python-based Dimensional Analysis

2007-03-13T10:14:00-04:00

Actually, the code looks like a clever expansion on my example, in PyCon 2007 (Revised) .

"But wait," you say. "Creating a pivot table in Python?"

Of course. Spreadsheets can create pivot tables from dimensionally normalized data. However, getting the data in this form is often challenging and if there is any manual operation at all, the data quality is immediately suspect.

To have perfect transparency -- with no possibility of manual transformations -- you need a simple application program which reliably, auditably, and testably produces the correct data. Further, you want to reduce the manual operations to formatting and presentation. The ideal solution is to produce the data in the required pivot table so that it can be loaded into a spreadsheet for display only.

With an object-relational mapper, you can write a tidy query to fetch raw data, and compute a aggregate along two dimensions. You then assemble result columns on one dimension and rows on the other dimension.

Elegant -- But Dirty -- Pool.

The Elegant Thing that makes this work pleasantly and simply in Python is being able to use a tuple as the key to a mapping. I can't say enough good things about this simple, elegant piece of Pythonic programming. You can easily handle complex, multi-column keys in each dimension of the pivot table, by simply creating a tuple of key values, and using a pair of tuples to locate the appropriate cell in a mapping.

Things like dimensional conformance often create a gnarly algorithm in Java or -- shudder -- COBOL. In Python, it's a tuple that you can use to locate the dimension value in a dictionary. It works for everything except the Customer dimension, which in some applications is too huge to retain in a simple in-memory mapping.

The graceful elegance of Python's Mapping Indexed By A Tuple™ (MXT) can really prevent a lot of brain-cramping bugs.

What a Data Warehouse Can Never Do

2007-01-12T14:40:00-05:00

In one form, the question is "How do we handle the [X] transaction in the warehouse?" Another form of the question is "What do we do when [Y] changes?" The third form is less clear, but essentially the same: "How do we maintain [Z] in the warehouse?"

All of these are questions that superficially cover change management, but we're not really talking about Kimball's Slowly Changing Dimension (SCD) design pattern. It turns out, we're talking about something more subtle and confusing.

System of Record

The real questions are System of Record (SoR) questions. In short, each question is a version of "Where's the authoritative copy, and how do I keep it current?"

The [X] transaction is, at least in theory, part of a source application, a System of Record. It is extracted from SoR, transformed, and loaded into the warehouse. The [X] transaction does not change when the warehouse is implemented. Unless, of course, there is no SoR.
The changes to [Y], similarly, should be made in the SoR. This is almost the same question as the "[X] transaction" question, but it's asked about a piece of data, not a named business process. The distinction reveals much about the processes which the warehouse must support.
The maintenance of [Z], clearly, should be made in the SoR. This is similar to the "changes to [Y]" question, but shows a different point of view on what data is and why it exists.

We'll look at these questions in a bit of depth.

Transactions in the Warehouse

When someone asks about the "[X] transaction," they're often summarizing a business process. In general, every business process is either informal, or formalized. Informal transactions are done manually using desktop tools: email, spreadsheet, word processing, etc. No piece of software captures, manages and enforces the transaction.

Formal transactions have three general patterns for their SoR: one SoR, many SoR's and a badly-chosen SoR. When there's one SoR, life is good. The transaction happens in some system (SAP, Oracle, QuickBooks, Aptiva, etc.) It propagates through the organization through ordinary Enterprise Application Integration (EAI) techniques. It winds up in the warehouse through ordinary Extract-Transform-Load (ETL) processing.

When there are multiple SoR's, we have some challenges. Sometimes, the relationship is horizontal: two peer business units have separate sources for similar data. One unit has SAP, the other has Aptiva. This means that there may be common data which must be conformed into a warehouse dimension. So far, so good.

Sometimes the relationship between SoR's is vertical: the parent company uses SAP, the subsidiary uses Great Plains. This means that there may be contradictions between the views of the common data. When data is moved up from the subsidiary, it may be aggregated: business entities are elided, and the data is difficult (or impossible) to conform.

Sometimes the relationship between SoR's is psychotic. This often leads to a badly-chosen SoR. A single organization can have the same data in two applications and neither can be trusted to be the SoR. They may have customer data in Siebel and JDE, and the data is different, and can only be reconciled manually. Sigh. No amount of Data Warehouse ETL can sort this out. The organization must pick something as the SoR, and revise their business processes to reflect that.

In summary, there are no transactions in the warehouse. Transactions happen in the SoR, and the results of those transactions are applied to the warehouse. You must pick an SoR.

Change in the Warehouse

Sometimes, there is no System of Record. There are two common cases: the data is maintained manually, and the data is maintained through a cryptic transaction buried in the legacy reporting application. When data is maintained manually, we have a rather difficult Master Data Management (MDM) issue because we don't have an official SoR. We're often in a bad position, here, because we're forced to stop data warehouse development work to put a SoR in place. This extra work can be hard to justify; managers say "we never needed a system for that before, why now?"

The answer is simple, but unpleasant. "It never worked before, either." People put in data warehouses because their legacy reporting tools are incorrect or inconsistent. One root cause of errors is lack of a public, well-understood truth because of manual or informal changes.

The cryptic transaction is the worst thing to ferret out. Let's say we have two applications, B and C, which each do parts of a business function. Further, each has it's little quirks, and we periodically must reconcile B and C's results against each other. How do we do this reconciliation when the two applications are largely disjoint except where they have to be reconciled?

The usual solution is to merge the data into a kind of data warehouse. However, when there are reconciliation problems, we hate to make a change to B or C, and re-run the complete ETL cycle. Instead we make the change directly in the warehouse. Who wants to duplicate this change in B or C? No one, so we back-propagate the change from the warehouse into the SoR's. In effect, we've made the warehouse the SoR.

In summary, change in the warehouse is limited to a historical snapshot of change in the SoR. Change happens in the SoR, and the results of the changes are applied to the warehouse. You must pick an SoR.

Maintenance in the Warehouse

The question of maintaining data in the warehouse usually stems from a warehouse design which involves something more complex than simple facts and dimensions. Generally, a bridge table (often for a hierarchy) becomes a source of confusion. Most business entities (dates, accounts, products, documents, etc.) are pretty clear in the source applications. The facts are usually obvious.

It's the reporting relationships that get confusing. Something like product family can be a very difficult thing to handle. Something like a bill of materials (BoM) or Organization Hierarchy (OH) can be even more complex.

In the product family case, the reporting is an organization fiction. It doesn't tie back to anything except how managers chunk information. In this case, the reporting hierarchy is entirely a feature of the warehouse itself. This is the pure Master Data Management problem, where business entities are grouped in the warehouse exclusively for the user's convenience.

In the BoM or OH case, however, the reporting hierarchy does have an independent existence. In the case of the BoM, it ties to engineering or product configuration. In the case of OH, it ties to some project structure or accounting structure. However, these hierarchical structures don't often exist in the same simple form that they do in the warehouse bridge table. And this leads to confusion on how we maintain the bridge table.

In summary, maintenance in the warehouse is limited to loading a historical snapshot of the relationships in the SoR. Maintenance happens in the SoR, and the results of the maintenance are applied to the warehouse. You must depend on an SoR.

Bridge Tables Maintenance

There are several varieties of Bridge Tables. We'll address hierarchy, since it seems to lead to the most confusion. We'll touch on minidimension and outrigger tables, also, since the same design pattern applies to those.

The essential worry about hierarchies stems from the fact that a hierarchy bridge table can have many more rows than the dimension it bridges. Generally, it's an \(n \log(n)\) kind of multiplication, where \(\log(n)\) is an estimate of the depth of the hierarchy.

As a practical matter, moving one child to another parent is a single row change in the original data. However, the expansion in the bridge table means that \(2d\) rows will change, where \(d\) is the depth of the node in the hierarchy. For some reason, this is intimidating.

There are two solutions:

Reload the entire bridge table with each source change. This is easy to implement but slow. If you use SCD change tracking, you'll have lots of nearly identical rows that are labeled with change dates because they were associated with a source node change.
Recompute just the changed parentage, updating only those rows of the bridge table. This is not significantly more complex. First, write a "find-all-parents" function, and apply this across every element of the source data to populate the bridge initially. Then, you can use the "find-all-parents" function to compute just the relevant bridge table changes when a source node changes.

A similar pattern is appropriate for minidimensions and outriggers, which are based on subsets of a dimension. The lazy approach is to rebuild these each time the dimension changes. A slightly more efficient approach is to derive just the changed rows from the changes in the dimension.

Bottom Line

Change doesn't happen in the warehouse. Change happens in the SoR. The warehouse merely captures the effect of that change.

Refactoring and Unit Testing

2006-10-11T00:19:00-04:00

I do a fair amount of manual refactoring. I've used WebSphere Studio (Eclipse) to do some automated refactoring, so I have some experience in using IDE's which exploit Java's static type-checking.

However, the question of type checking in a dynamic language is interesting. I don't use a sophisticated IDE for Python development. So, I have limited experience using an IDE to do refactoring in a dynamic language.

However, JB notes

"I'm having some heartburn about hierarchical type systems as a way of determining 1) conformability and 2) managing commitments of semantically equivalent behavior.

...

Seems a constraining way to do it, to me, not that I have a better alternative at the moment."

Duck Typing

Since Python relies on Duck Typing , refactoring takes on an interesting new dimension. We aren't constrained to simply shuffle methods up and down the class hierarchy. We are now able to -- well -- put a method just about anywhere.

Further, since Python doesn't have a simple, single-inheritance model, the "hierarchical" type system doesn't completely apply.

For these reasons, refactoring in Python is one potentially complex problem.

From what I understand of Ruby, you can override a class method without creating a subclass, essentially redefining a base class in some obscure way. This gives me the willies because it makes refactoring a problem without any sensible boundaries. Maybe I'm misunderstanding Ruby, and have this wrong.

Essential Use Cases

The principal refactoring use case involves moving a common method up the inheritance hierarchy. As a practical matter, this does happen once in a while.

An additional text-book refactoring use case arises when we're adding or removing whole methods in the subclass hierarchy.

The fringe use case is a variation on the theme of SubClass1.methodA looking a lot like Subclass2.methodA, but they're not the same. There are two interesting cases.

SC1.mA() is a superset of SC2.mA(). All of SC2.mA() gets refactored up, and ``SC1.mA()` overrides it to add features.
SC1.mA() overlaps with SC2.mA(). Some common functionality has to get extracted, moved up the hierarchy; SC1.mA() and SC2.mA() are rebuilt around this common kernel.
SC1.mA() has no usable relationship with SC2.mA(). What now? In some cases, a complete change in design may be called for. The mere presence of this situation is diagnostic of the designer having missed something.

Beyond the Fringe

Outside the fringe of ordinary refactoring are the New Design Pattern™ situations. Mostly, these are Strategy situations, where what looks -- initially -- like a variant method grows into a different approach as we learn more about the solution.

Consider a pair of ordinary Entity classes that look like different entities because of different behavior. However, they have the same attributes, and almost identical methods. The only difference is one algorithm. This can be done through inheritance, but sometimes that variant algorithm is only the tip of the iceberg, and there is more variability just below the surface.

At this point, we realize we need a Strategy hierarchy to contain the variant algorithms, not a hierarchy of ordinary Entities. How does refactoring work here, where we're moving the functionality out of a class hierarchy into a different class hierarchy? Is this even refactoring, or is it the more general case of redesign?

It doesn't feel like refactoring because we aren't shuffling methods up and down the class hierarchy. In Python, the Duck Typing means we don't actually need a proper hierarchy for the Strategy class definitions. Consequently, we're free to make significant structural changes that I don't think an IDE can ever help with.

Across the Spectrum

One potential problem with this is captured in the comment that "[the compiler] can’t help but see your code as a pile of text". By extension, then, the IDE can't do anything more than treat source as text, losing precious semantic information.

However, when doing a fundamental restructuring (from class methods to Strategy hierarchy), the source code information available to the compiler (or the IDE) isn't of much value until you've finished. Nothing helps you when you're in the middle of this. Until the semantic information exists, no IDE can help you manage and maintain the semantic information.

There are parts of design (and redesign) that are hard. I think anyone would agree that the earliest phases of noodling around about a problem are done without benefit of an IDE or formal semantics. When the design is merely conceptual, tools can't help.

I think refactoring includes a very broad spectrum. At one end, things are essentially mechanical; at the other end things, are completely conceptual. This isn't really a problem that needs a solution; it doesn't need tools. It's part of the game of moving from a good idea to software. Some parts of the good idea don't have formal semantics. Eventually, when formal semantics exist, tools can be applied.

In the case of Python, I suspect that IDE support for refactoring could only be feeble at best. The mechanical end of the spectrum is so easy that tools aren't required. At the conceptual end of the spectrum, tools don't help in the first place.

The Middle Ground

One might argue that simple, mechanical refactoring can be aided by the presence of static type declarations. However, my experience is that this covers only the most mundane of the refactoring use cases. In Python, we just move the method around.

In Python, there's a double whammy: checking types can't be done because the language traditionally lacked type declarations. Further -- and more important -- type checking doesn't need to be done because the language is dynamic. It can't be done, and even if it could, it didn't matter anyway.

This is overly simplistic, however. There is some type checking which can be done in Python. The epydoc package does considerable analysis of source as part of writing documentation. It spots unused arguments, and can spot certain kinds of obvious mismatches in number of arguments vs. parameters.

When we look at JB's point on committing to specific semantics, we see something even more profound. It goes way beyond what even Java is capable of checking or automating.

The Formal Specification

JB is asking for a level beyond syntax, beyond type matching and off into "intent". JB appears to be looking for formal assertions of preconditions and postconditions that he can use to determine how to redesign methods to make them refactorable, and then how to refactor the changed design.

JB's formality would be a nice thing to capture. If every statement had a proper precondition and postcondition, then we could prove almost anything about our software except whether or not the loops actually terminated. (That can't be formally proven in a system with the same expressive power as software, it requires more sophisticated logical tools.)

Since Java and Python have added additional markers (annotations and decorators) JB's assertions could be captured, to an extent. You'd have to implement a simple "for all" and "there exists" predicate, but Python has a nice reduce that can be paired with a lambda that allows you to write a "for all"; from this you can built a "there exists".

I'm not sure how helpful formal assertions would be.

Pragmatic Refactoring

When working in the center of the refactoring use cases, IDE aids are helpful. When working at the fringe, they're just visual noise. Indeed, when redesigning something, I have to be sure not to look at any of the "helpful" messages from Eclipse because it's checking for errors using obsolete type information. When I've broken the whole thing down into a workbench full of parts, the semantic checks aren't even meaningful. Once I get it put back together again, automated checking can be handy to assure a complete job.

In Python, breaking the whole thing down as part of a redesign is so much simpler. We don't have the artifice of "interface" to keep to a single inheritance model with static type checking across multiple aspects of a class. We just move the methods around. We have multiple inheritance, and we don't need formal interface declarations.

Indeed, it's far, far easier to produce a working design in Python, and use that as a formal specification for a Java program. I can tweak and tinker, optimizing performance and simplifying without the rigid formality of Java. Adding proper class hierarchies and turning multiple inheritance into single+interface inheritance is typically a pretty easy transformation. Since I knew I was aiming at Java in the first place, I avoided Pythonisms that don't translate.

While it's true that we don't need Java's formality in Python, much of that formality is helpful. I find it easier to work with a proper inheritance hierarchy, one that has explicit Not Implemented exceptions to mark the place-holders. I like to have a tidy interface definition so that I can document the interface. This additional material makes refactoring slightly more complex, but could help an automated tool do some useful method matching among classes.

The Final Test

Without appropriate unit tests, refactoring is impossible. Even in Java, with a swanky IDE that checks everything, you still have potential problems which are uncheckable. In particular, a mis-named subclass method cannot be detected except by "near-miss" fuzzy-matching rules that will almost always work and will have false-positives. Only unit testing can locate this situation.

Unit testing absolutely is a stand-in for things the compiler can't check. You can portray the heavy use of unit testing as a negative ("the compiler can't be trusted") or as a pragmatic approach to verifying the things you can't formally state. All of the assertions in the world won't find a spelling mistake.

Worse, your formal declarations (post-condition assertions or type definitions) could just as easily be wrong. A tidy formal proof with a wrong piece of logic will derive an incorrect program. A misspelled class name may compile, but still fail a suite of tests.

Since the IDE can't register intent very well, it isn't a complete solution. In the case of redesign, it isn't even very helpful.

Python OODB (Revised)

2006-06-20T10:35:00-04:00

Simple object persistence (i.e., serialization to a file system) is what pickle, marshal and shelve do.

However, here's the next thing of some interest OODB's.

Zope's ZODB . The original OODB for Python, the backbone of Zope.

buzhug : "a fast, pure-Python database engine, using a syntax that Python programmers should find very intuitive."

PyPerSyst "fast, reliable, and flexible object persistence with a small footprint, suitable for embedding in other Python applications."

CouchDB "A stand-alone document store, [which] most closely resembles the Lotus Notes/Domino storage engine."

ObjectivityDB/Python "a high performance and robust Object-Oriented Database Management System [ODBMS]."

Axiom "Axiom is an object database, or alternatively, an object-relational mapper." The pleasant thing is that you don't really care which.

Durus "a persistent object system for applications written in the Python programming language."

http://www.itamarst.org/software/cog/ (not CoG , the Commodity Grid) is the Checkpointed Object Graph, which provides "semi-transparent persistence for large sets of interrelated Python objects."

Doctest beyond Python

2006-04-17T14:53:00-04:00

This is something that elevates Doctest into the realm of Pattern. Perhaps even above that.

The idea is so elegant: the document is the test, and the test procedure is the document.

There's a DRY clarity to the whole thing that is rather exciting. It is an elegant application of basic Literate Programming principles.

The best part is that it is difficult to do in other languages. Ruby and Perl have the necessary interactive execution modes. But in Java, it would be a nightmare to define all the required overheads to have a standardized exercise framework that paralleled the Python interactive execution mode.

This makes the "Doctest" pattern a key value proposition for any new language or environment. If you can implement the Doctest pattern, you have something that creates value by binding testing and documentation into one tidy package. If you can't implement the Doctest pattern, perhaps you should rethink your implementation because you can't easily compete against Python.

Python Object-Relational Mapping (Revised)

2006-04-13T02:37:00-04:00

Ian Bicking: A Blog http://blog.ianbicking.org/, provided some info on Py3K and Python Introspection http://blog.ianbicking.org/introspecting-expressions-in-py3k.html.

For me, the interesting part was his summary of Object-Relational Mapping. Mr. Bicking identifies two broad approaches: lambda introspection and operator overloading.

Lambda Introspection

Dejavu It primarily uses a generic Data Mapper architecture. It is more of an OODB backed by a relational store.

SQLComp The make_query method examines a lambda containing a list comprehension to create SQL. This is only queries, and isn't a complete ORM.

Operator Overloading

SQLObject This is a very complete ORM, cast in the some mold as Django.

SQLAlchemy This provides a Pythonic definition of SQL metadata and a mapping from the SQL metadata to Python class definitions. This is a very, very rich approach, allowing you to straddle the SQL and Object worlds explicitly.

PyORQ This is an older ORM with a few data types but a very "naked" use of overloaded operators to perform queries. Unlike the lambda overloading, the class provides operators that are set operations for queries.

Non-Introspective Approaches

Django , for example, encodes attributes and operators as keyword parameters to methods. It doesn't look inside the Python code, but parses the keywords.

PyDO2 encodes the query explicitly, using functions that mirror SQL operators or tuples that contain string names for the functions.

QLime is an ORM with functional notation, similar to PyDO2.

ORM (the Object-Relational Membrane) mostly captures SQL metadata in Python.

DBClass is focused on an easy way to hack around with SQL queries (to get data from procedures and so on).

Axiom is an object database, or alternatively, an object-relational mapper. It depends on Epsilon .

The Python wiki page on Higher Level Database Programming has additional notes and products that are high level.

Garden-Variety Relational Access

All of these modules provide the standard DB-API (PEP 249) interface to a SQL database.

The most visible access layer product is mx.ODBC for bare ODBC connectivity. This has the advantage of wide portability, and the disadvantage of the narrow ODBC interface.

PDO wraps a variety of other access methods into a single, combined package. I'm not precisely sure why it adds another interface layer, but it appears to simply do away with Cursor objects. However, it does provide a nice list of DB-API 2.0 modules for direct SQL access.

MySQLdb for MySQL

PySQLite and APSW are for the ultra-lightweight SQLite RDBMS.

The PostgresPy project will address many PostgreSQL topics. PyGreSql (aka pgdb), psycopg , PoPy , bpgsql .

kinterbasdb Firebird and Borland's Interbase

pyDB2 DB/2

cx_Oracle Oracle

adodbapi Python access to the MS Windows ADO interface

The Python wiki page on Database Interfaces also has a list of these product-specific access modules.

Recommendations

Rule 1. Do development with SQLite. Why? Eschew Features. Focus on RDBMS for relational store, focus on Python for processing. Stored Procedures and Triggers are a product-specific mine-field. Once the model passes unit tests, move to another RDBMS that supports concurrent users.

Rule 2. For OLTP, use an OR-Mapping and stay away from naked SQL. However (and this is a big however) you will likely be supporting ad-hoc reporting through SQL-based report writers. There are two extemes. At one end is Deja-Vu, which may be too far from the underlying SQL. The other end begins with SQLAlchemy, which may expose too much SQL; ORM and DBClass may be too light on object features.

Rule 3. For OLAP, you have two kinds of applications. Some parts (like dimension conformance) can use an OR-Mapping because they are OLAP-like. For some loading, aggregation and extraction, use direct SQL drivers for the chosen product. For the large-volume fact-oriented loads, use the vendor-supplied bulk loader. Portability is not your concern.

Agile Testing Goodies from PyCon 2006

2006-02-27T11:26:00-05:00

A number of testing frameworks were used. The Agile Testing tutorial provides a path through the toolsets, showing what you can do, and how you should do it.

Unit Testing: [Nose ], <{filename}/blog/2005/11/2005_11_09-compare_and_contrast_round_3_revised.rst>

Acceptance Testing: [FitNesse ]

Regression Testing: [TextTest ]

Functional Testing: [twill ]. A thorough analysis is at http://www.advogato.org/article/874.html.

Ajax Interaction Testing: [Selenium ] and PAMIE http://pamie.sourceforge.net/.

Literate Testing: [doctest ] and epydoc http://epydoc.sourceforge.net/.

testresources

2005-12-26T13:57:00-05:00

testresources http://www.robertcollins.net/unittest/testresources/ purpose appears to be to manage the resources used by a test suite.

Adding this resource management context extends the Test Suite to optimize tests around the resources. This can reshuffle the TestCases to minimize SetUp's. This can be useful in contexts where the Fixture includes Singletons or expensive resources.

While interesting, this package bends one of the common definitions of Unit Testing. If there is a complex resource dependency, the Fixture being tested isn't really isolated. This pushes beyond isolated unit testing with Mock objects into integration testing with real objects and real interfaces.

One can make the case that "unit" is intentionally vague; the Beck definitions refer to a Fixture as the design pattern. This could be a class, module or package, depending on your willingness to abstract. I agree that "unit" does not necessarily mean class. However, I do think that "unit" means isolated from other components.

testresources seems specifically designed for integration test, not unit test. I think it is miscategorized, and belongs to an unidentified species of products: integration testing frameworks.

SubUnit

2005-12-20T18:36:00-05:00

SubUnit's http://www.robertcollins.net/unittest/subunit/ purpose appears to be to manage testing via subprocesses.

Consequently, it can run external tests not in Python, it can fork a subprocess to manage the Fixture in an isolated process.

Adding this subprocess execution context extends the Test Runner implementation of the built-in unittest module. This can be useful in contexts where the Fixture includes Singletons or connection pools or other per-process design features.

Twisted Trial

2005-12-15T17:10:00-05:00

Trial is not really a stand-alone unit test framework. It is an extension to unittest focused on the testing needs for the Twisted framework.

The Trial how-to http://twistedmatrix.com/projects/core/documentation/howto/testing.html has some information. More valuable, perhaps are the API documents http://twistedmatrix.com/documents/current/api/twisted.trial.html.

Because of the asynchronous nature of Twisted, the Fixture pattern has a rather complex relationship with the Twisted Reactor. Also, since Twisted is a framework, not a single application, components are optional, and the TestSuite pattern is implemented with considerably more flexibility.

More Frameworks! (rev. 3)

2005-12-13T18:40:00-05:00

A wiki page on Python testing tools http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy identifies a number of additional unit testing tools. The wiki page provides a handy summary. I'll examine these in light of the Beck Unit Test design patterns to provide a little more detail on what they really do.

Twisted Trial
SubUnit
Testresources
PyUnitPerf
PeckCheck
PythonMock
pMock

Compare and Contrast (round 3, revised)

2005-11-09T19:38:00-05:00

The object-oriented unit testing framework began as Smalltalk's Beck Test framework http://www.xprogramming.com/testfram.htm. It evolved to the JUnit http://www.junit.org/index.htm>`_ ` <http://www.junit.org/index.htm%22%20target=%22NewWindow framework for Java. Beck defined four repeated patterns of unit testing software, covered in a previous posting <{filename}/blog/2005/11/2005_11_05-compare_and_contrast_round_1.rst>.

Nose, TestOOB, test.py (and TestGears) are significant revisions to the Test Suite and Test Runner parts of the unit test patterns. Additionally, these tools make efforts to implement the Diagnostics pattern, also.

The Fixture is implied by the context in which the tests are discovered. Nose can locate package, module and function tests; it uses TestCase class definitions, also. TestOOB and test.py sit more squarely on unittest, with the resulting focus on module-level testing.

In TestOOB and test.py, the TestCase class plus a flexible regular expressions or glob expression defines the test cases. test.py looks for packages of tests, using the path name of the package, as well as the module name. In Nose, the TestCase class can be used for compatibility, but this is not required; Nose will match a regular expression to locate tests.

The Results Check in nose can be done via the existing assert statement. Nose, pleasantly handles the "test which throws an exception" case: a test function that exits normally is a "pass". An AssertionError exception is a test failure; any other exception is an error. Since TestOOB and test.py sit on unittest, they depend on the complex set of assert methods, and the fail() method.

The Test Suite is implied by the collection of TestCase instances with the expected name forms in TestOOB. In Nose and test.py, it is the collection of modules, functions and methods with names that have the expected forms. Both cases make powerful use of Python introspection to track down the tests.

The Test Runner in nose can be a stand-alone nosetests program, or you can import nose; nose.main(). In the case of TestOOB, we have a testoob program, or we can import testoob; testoob.main()`. Nose has an interesting integration with Python distutils/setuptools. It adds a new "test" verb to setup.py. The test.py main program has a large number of options to fine-tune which tests are run

Nose supports the Diagnostics with output capture and a simple flag for producing additional details. TestOOB, in certain environments, will produce color output; it produces an XML test report as well as HTML test reports. TestOOB can launch the Python debugger as well as log failing assertions in detail. test.py can run pychecker, do tracing and refcount checking as part of the diagnostics.

TestOOB has some additional features for repeating and controlling the timing of the tests. While this is not sufficient to prove that an application lacks the kind of race condition that makes it behave poorly; it can help to provide some confidence for load testing. Similarly, test.py includes features for looping tests to look for memory leaks and race conditions.

Compare and Contrast (round 2)

2005-11-07T17:50:00-05:00

The object-oriented unit testing framework began as Smalltalk's Beck Test framework http://www.xprogramming.com/testfram.htm%22%20target=%22NewWindow. It evolved to the JUnit http://www.junit.org/index.htm%22%20target=%22NewWindow framework for Java. Beck defined four repeated patterns of unit testing software, covered in a previous posting <{filename}/blog/2005/11/2005_11_05-compare_and_contrast_round_1.rst>.

An additional pattern that py.test introduces is the Diagnostics pattern. This is a useful traceback or cached output. To make it useful, it is presented only for failing tests, and elides repetition in the event of recursions that lead to stack overflows.

py.test seems to deliver most of the Beck-defined features.

The Fixture is created by offering a number of setup/teardown functions, either at the module level (for a module or class) or within a class.

The Test Case is a module, class or function with an appropriate name. Either test_ or Test_ as a prefix is sufficient to define a test case.

The Results Check uses ordinary asserts and a special py.test.raises() function to cover all the bases. Personally, I prefer the JUnit approach to catching the expected exception and calling the fail() method for everything else.

The Suite is developed by implication through Python's powerful introspection: everything that looks like a test -- at the package, module and class level -- is a candidate. A regular expression can pick names, plus other global conditions can be examined to further refine the test protocols.

The Runner is a stand-alone py.test program that locates the tests, executes them and produces a log. Further, it produces Diagnostics focused on the failing tests.

Compare and Contrast (round 1)

2005-11-05T20:21:00-05:00

Some Basis for Comparison

The Fixture. The thing we are testing; a class or possibly a set of instances of a given class, or possibly something even larger. If we are testing more than one class at a time, we aren't really "unit" testing. So the fixture often includes stubs for missing classes.

The Test Case. A predictable reaction of the fixture. This should either work or fail. It can, of course also raise one of those egregious, unchecked-for errors that indicate fairly serious problems in a preliminary piece of software. Or, it may indicate something that was badly damaged during maintenance and is now raising errors instead of simply failing the regression test suite.

The Results Check. A specific assertion about the fixture's results.

The Test Suite. A collection of TestCases.

JUnit and unittest add a Test Runner pattern, also. This the top-level component that uses a Test Suite to create test results by executing each Test Case, assuring that each Results Check worked. The Test Runner can also assure that any Fixture Setup and Teardown is done correctly.

Legacy Frameworks

unittest delivers all the Beck-defined features. It should, it is the indirect descendant of the original framework. Having JUnit as an ancestor, however, leads to some clunky non-Pythonic features. In particular, Python features that Java lacks are ignored, including modules and free-standing functions.

doctest has an odd fit with the Beck framework. The fixture isn't well defined; since doctest has a module-centric view, a shallow copy of the module globals are given to each test, making the module globals the fixture. Each Case and Results Check is encoded in a docstring, usually by a cut and paste from an interactive testing session. The test suite is implied by the module structure.

unittest isn't terribly Pythonic. Doctest is module-focused, not class focused, and doesn't treat the notion of fixture very well.

IMO, module-based testing is a more useful level of unit testing. Individual classes, while important, rarely make sense in a vacuum. All of the test harness and stub classes required to test just one class seems like too much unproductive work. When the architecture changes, I may have to change a class definition as well as the test harness classes that stand in for this class in the unit testing framework.

Next Up, py.test, nose and testgears. Later, TestOOB and Sancho.

Python Unit Testing Frameworks (v3)

2005-11-02T00:12:00-05:00

Ned Batchelder : Blog [http://www.nedbatchelder.com/blog/index.html ] identifies no less than 6 unit testing frameworks for Python [http://www.nedbatchelder.com/blog/200510.html#e20051025T070731 ] and [http://www.nedbatchelder.com/blog/200411.html#e20041120T185622 ].

TestGears [http://www.turbogears.com/testgears/ ] is part of the TurboGears web uber-framework. It provides automatic discovery of test functions, simplifies suite development, and makes it easy to run tests zero configuration. Kevin Dangoor [http://www.blueskyonmars.com/ ] says he will deprecate this in favor of Nose. David Warnock [http://42.blogs.warnock.me.uk/2005/10/turbogears_cont.html ] says something similar.

TestOOB [http://testoob.sourceforge.net/ ] (Testing Out Of [the] Box) provides for new styles of output (HTML and color terminal), debugger launching, verbose asserts, parallel execution, and command-line utility testing

nose [http://somethingaboutorange.com/mrl/projects/nose/ ] provides an alternate test discovery and execution engine for unittest

unittest [http://docs.python.org/lib/module-unittest.html ], formerly known as PyUnit [http://pyunit.sourceforge.net/ ] (Thanks for the heads up, Tony [http://www.haloscan.com/comments/slott/E20051105152154/#29209 ])

doctest [http://docs.python.org/lib/module-doctest.html ]

py.test [http://codespeak.net/py/current/doc/test.html ]

Michal Watkins [http://mikewatkins.net/ ] adds Sancho, a unit testing framework [http://www.mems-exchange.org/software/sancho/ ].

Also, ZOPE has test.py [http://zopewiki.org/HowToRunZopeUnitTests ]. There is a derivative product, also, the SchoolTool Test Runner [http://svn.nuxeo.org/trac/pub/file/CalCore/trunk/test.py ].

Jeremy Hylton's blog has some notes [http://www.python.org/~jeremy/weblog/031014.html ] on unit testing, describing test.py.

Ian Bicking also has a list of complaints about the basic unittest interface [http://blog.colorstudy.com/ianb/weblog/2003/10/10.html#P11 ], many of which are answered by the add-ons.

One More Cool Thing About Python Is...

2005-10-22T15:25:00-04:00

So, confronted with 1.4M records of questionable data, what do we do?

We'll need a production program that can run daily from now until the end of time to cleanse the data and reject the truly evil records.
We don't have a good fix on what the data is.

We have some "specifications" but they're little more than a wish list of cleansing suggestions.

What to do?

Well, step 1 is obviously analysis to fully define the problem. What data do we have? What is the platonic ideal? What forms of badness are represented where data does not meet the ideal?

So, how do we analyze? We can't throw it into Excel. We can barely open the file in Textpad.

Python to the rescue

While the final application must be delivered in Java, Java's a terrible way to analyze unknown data. We would have a cycle of write, debug, build, test, run and scratch head over a fairly complex piece of Java code just to do simple frequency histograms.

The Python program we used was something on the order of

fq= {}
for line in file( 'sample.csv', 'r'):
    fields= line.split(',')
    aSuspect= fields[4].split()
    fq.setdefault( aSuspect[3], 0 )
    fq[ aSuspect[3] ] += 1
print fq

Given this template we can move around, analyzing fields individually or in groups, looking for the domain of badness and formulating a cleansing strategy.

Python reduces the development cycle to type-run-think analysis cycle which goes very quickly. And we can easily produce things we don't mind throwing away. We haven't invested much, we don't have the Urge to Preserve.

We can then dry-run our cleansing approach in a type-run-think design cycle. Again, the Urge to Preserve is negligible when there's such a tiny investment.

Compare the 8 lines of code above with the equivalent in Java. It would be, perhaps twice the size. The strong type checking slows development, and causes you to marry a class hierarchy that isn't really correct because it's too expensive to change.

And when we're done, we can simply translate to Java. I like this Python thing, it works.

S.Lott -- Software Architect - Python

A get_object_size() Function -- Again

BLUF

Background

What I Did

A get_object_size() Function [Updated]

BLUF

Background

Use Case

The Object Size Problem

The Documentation Says

What I Did

TODO

OO Design Principles: GRASP patterns

BLUF

GRASP Patterns

Controller

Creator

Indirection

Information Expert

Low Coupling

High Cohesion

Polymorphism

Protected Variations

Pure Fabrication

Conclusion

OO Design Principles: CUPID

BLUF

CUPID

Composable

Unix-Philosophy

Predictable

Idiomatic

Domain-based

Conclusion

OO Design Principles: SOLID

BLUF

SOLID Principles

ISP

LSP

OCP

DIP

SRP

Conclusion

OO Design Principles: ABC's

BLUF

Purely OO Design?

To ABC or not to ABC?

Conclusion

Is an Abstract Base Class still helpful?

When?

When is it necessary?

When is it superfluous?

Comma Comma

Where do we stand?

What to do?

Big Lesson

Domain-Specific Language

The Context

The Complication

Use Unicode

Joomla Conversion

The Goal

The Processes

Things That Aren't Easy

Embedded Images and Links

What Else? Oh, Right, Section Index

Finally, Broken Links

Conclusion

Database Migration, Part IV

How Joomla! Works

The Path

Conclusion

Database Migration, Part III

Things SQL Does Badly

Not Using SQL

How Do We Get There?

Step, the first

Step, the second

The AppTable class

The `AppTable` class