What’s more: this commutative diagram helps simplify and unify previous proofs of results in the study of invariant neural networks. Plus, it gives a blueprint for developing novel invariant neural network architectures.

In this post, I will explain what this all means. This should be of interest to people who like invariant neural networks, or are curious about applications of some basic topology, algebra, and geometry to the study of neural networks.

Definitions

First, an invariant neural network is a neural network \(f\) such that certain transformations of its inputs do not affect its outputs. We denote the set of transformations as \(G\), and assume that \(G\) is what is known as a group in algebra. Then a neural network \(f\) that is \(G\)-invariant satisfies \(f(gx) = f(x)\) for any \(g \in G\).

For instance, if \(f\) is an image classifier, then certain translations of the input images should not change the output of \(f\); a picture \(x\) with a duck in the bottom right corner and a picture \(gx\) with the same duck in the top left corner should both be classified as having a duck. Thus, \(f\) should approximately be invariant to the group \(G\) of translations. Convolutional neural networks for image classification are approximately invariant to translations.

We say that a network is universal if it can compute all continuous functions that we care about. In particular, we say that a \(G\)-invariant network architecture is universal if it can express any continuous \(G\)-invariant function.

In the commutative diagram above, \(\mathcal X\) is the domain that our input data lies on. We will assume \(\mc X \subseteq \RR^D\) is embedded in some Euclidean space of dimension \(D\). The quotient space \(\mathcal X / G\) is the set of all equivalence classes of inputs under the transformations of \(G\). In other words, it is the space \(\mathcal X\), but where \(gx\) is viewed as equal to \(x\) for any \(g \in G\). The quotient map \(\pi: \mathcal{X} \to \mathcal{X}/G\) is the map that sends an \(x \in \mathcal{X}\) to the equivalence class of all inputs that it can be transformed to, \(\pi(x) = \{ gx : g \in G\}\). Thus, \(\pi\) is \(G\)-invariant.

Commutative Diagram

A commutative diagram is a diagram where you can take any path through arrows and arrive at the same result. Let \(f: \mathcal{X} \to \mathbb{R}^s\) be any \(G\)-invariant continuous function. As an example, the above commutative diagram says that there is an \(\tilde f\) such that \(f = \tilde f \circ \pi\), and also an invertible \(\psi\) such that \(\tilde f \circ \psi^{-1} \circ \psi \circ \pi = f\).

First, consider the subdiagram shown here. This captures the so-called universal property of quotient spaces; for any continuous \(G\)-invariant function \(f: \mc X \to \RR^s\), there exists a unique continuous function \(\tilde f: \mc X / G \to \RR^s\) on the quotient space such that \(f = \tilde f \circ \pi\). However, this does directly not help us to parameterize \(G\)-invariant functions \(f\), as the quotient space \(\mc X / G\) that the \(\tilde f\) are defined on may look rather non-Euclidean (recall that the quotient space consists of equivalence classes of data points), so we may not know how to parameterize functions on such a domain.

Thus, we will want a topological embedding \(\psi : \mc X / G \to \mc Z \subseteq \RR^a\) of the quotient space into some Euclidean space \(\RR^a\), which means \(\psi\) is a continuous bijection with continuous inverse \(\psi^{-1}\). This brings us closer to parameterizing \(G\)-invariant neural networks. The two functions denoted with dashed red arrows in the commutative diagram are parameterized by neural networks: \[ \begin{align*} \phi & = \psi \circ \pi : \mc X \subseteq \RR^D \to \mc Z \subseteq \RR^a \\ \rho & = \tilde f \circ \psi^{-1}: \mc Z \subseteq \RR^a \to \RR^s. \end{align*} \] Of course, \(f = \tilde f \circ \pi = \tilde f \circ \psi^{-1} \circ \psi \circ \pi = \rho \circ \phi\). Note that \(\rho\) and \(\phi\) are compositions of continuous functions, so they are both continuous. Also, their domains are Euclidean spaces, so we can approximate them with standard neural network operations. \(\rho\) can be parameterized by an unconstrained neural network. However, \(\phi\) is required to be \(G\)-invariant, so it seems that we have not made progress, as our original goal was to parameterize \(G\)-invariant functions by neural networks. The next section shows how we can utilize structure in \(\phi = \psi \circ \pi\) to parameterize \(G\)-invariant neural nets.

Finding a quotient space embedding with structure

To parameterize the \(G\)-invariant \(\phi = \psi \circ \pi\), we need to find some structure in \(\psi \circ \pi\) that handles the \(G\)-invariance. For instance, for processing sets we want permutation invariance, and we can show that in this case for input \(x \in \RR^n\) (which represents a set of \(n\) real numbers) we may write \(\phi(x) = \sum_{i=1}^n \varphi(x_{i})\) for some suitable \(\varphi\) that does not have the \(G\)-invariance constraint. Thus, we may then parameterize \(\varphi\) with a standard neural network. There are different ways to find such structure in \(\phi\) or the quotient space embedding.

One way to find structure in \(\phi\) is to find structure in a generating set of \(G\)-invariant polynomials. A generating set \(p_{1}, \ldots, p_{l}\) of the \(G\)-invariant polynomials is a set of \(G\)-invariant polynomials such that every other \(G\)-invariant polynomial \(p\) can be written as a polynomial in \(p_{1}, \ldots, p_{l}\). Lemma 11.13 of [González and de Salas 03] shows that the map \(h: \mc X \to \RR^l\) given by \(h(x) = (p_{1}(x), \ldots, p_{l}(x))\) induces a topological embedding \(\tilde h: \mc X / G \to h(\mc X)\) such that \(h = \tilde h \circ \pi\). Thus, choosing \(\psi = \tilde h\) as our topological embedding above, we have that \(\phi(x) = h(x)\), so \(\phi\) contains all the structure that \(h\) does. This is why for the permutation invariant case, we can choose \(\phi(x) = \sum_{i=1}^n \varphi(x_{i})\); as the polynomials of the form \(p_{k}(x) = \sum_{i=1}^n x_{i}^k\) for \(k=1, \ldots, n\) are generating polynomials of the permutation invariant polynomials, we can write \[\phi(x) = h(x) = (p_{1}(x), \ldots, p_{n}(x)) = \sum_{i=1}^n (x_{i}, \ldots, x_{i}^n),\] so letting \(\varphi(x_{i}) = (x_{i}, \ldots, x_{i}^n)\), we have a nice sum decomposition structure on \(\phi\).

While this use of the \(G\)-invariant polynomials is theoretically general, it can be difficult to use in practice besides for particular choices of \(G\). This is because generators of \(G\)-invariant polynomials can be difficult to find for different \(G\), and they can take complicated forms with high degree polynomials (of up to degree \(|G|\) by Noether’s bound ¹). Thankfully, efficiently-computable generators are well known for two of the cases we consider below: when the groups are for permutation symmetries or for rotation/reflection symmetries.

Parameterizing with neural networks

Suppose that the data domain \(\mc X\) is compact, and that we have some structure such that \(\phi\) can be universally approximated by neural networks. Then \(\rho\) and \(\phi\) can be universally approximated by a neural network.

This is due to an application of standard universal approximation results, which say that neural networks can universally approximate continuous functions with domains that are compact subsets of some Euclidean space. If \(\mc X\) is compact, then one can show that \(\mc X / G\) is compact and thus \(\mc Z\) is compact. Hence, \(\phi: \mc X \to \mc Z\) and \(\rho: \mc Z \to \RR^s\) are continuous functions on compact subsets of Euclidean spaces, so they are amenable to approximation by neural networks.

Application 1: Deep Sets

We now give more details on the application of this commutative diagram to neural networks on sets. If we represent a set of \(n\) elements as a vector \(x \in \mathbb{R}^n\), then we want any function on the set to not depend on the order of the elements. Thus, letting \(G = \mathbb{S}_n\) be the group of permutations on \(n\) elements, we want a function \(f\) on sets to be invariant to permutations \(f(g x) = f(x)\), where \(gx = [x_{g(1)}, \ldots, x_{g(n)}]\) permutes the entries of \(x\) by the permutation \(g\).

The invariant DeepSets [Zaheer et al. 17] architecture parameterizes permutation invariant functions as \[ f(x) \approx \rho_\theta\left(\sum_{i=1}^n \phi_\theta(x_{i}) \right),\] where \(\rho_\theta\) and \(\phi_\theta\) are neural networks. Their proof that the architecture is universal uses a few computations and facts about symmetric polynomials (see their Appendix A.2).

We can instead prove universality using the above commutative diagram as follows. We take our generating set of \(G\)-invariant polynomials as the power sum symmetric polynomials: \(p_{k}(x) = \sum_{i=1}^n x_{i}^k\) for \(k=1, \ldots, n\). Thus, we know that in the above commutative diagram, we can write \[\phi = \psi \circ \pi(x) = \begin{bmatrix}\sum_{i=1}^n x_{i} & \ldots & \sum_{i=1}^n x_{i}^n\end{bmatrix} = \sum_{i=1}^n \begin{bmatrix}x_{i} & \ldots & x_{i}^n\end{bmatrix}.\]

Hence, in the DeepSets neural architectures, we can approximate \(\rho_{\theta} \approx \rho\), and \(\phi_\theta(y) \approx \begin{bmatrix} y & \ldots & y^n \end{bmatrix}\), which standard feedforward networks can do, so DeepSets is universal.

We note that this connection to the power sum symmetric polynomials as an embedding of the quotient space is also used in [Finkelshtein et al. 22], see their Appendix C.

Application 2: Rotation and Reflection Invariance

When dealing with certain types of point clouds of points in \(\mathbb{R}^d\), we may want to be invariant to rotations and reflections, which form the group \(G = O(d)\) of orthogonal \(d \times d\) matrices. [Villar et al. 21] proposed to use some classical results in invariant theory to parameterize \(O(d)\)-invariant neural networks. For learning an \(O(d)\) invariant function \(f: \RR^{n \times d} \to \RR^s\) on \(n\) points that satisfies \(f(Q v_{1}, \ldots, Qv_{n}) = f(v_{1}, \ldots, v_{n})\) for any \(Q \in O(d)\), they parameterize \[ f(v_{1}, \ldots, v_{n}) \approx g_{\theta}(V^\top V) = g_{\theta}((v_{i}^\top v_{j})_{i,j=1, \ldots, n}). \] In other words, \(g_\theta\) is a neural network acting on all inner products of two points \(v_{i}\) and \(v_{j}\). [Villar et al. 21] uses an argument based on the Cholesky decomposition to say that allowing \(g_\theta\) to be any arbitrary function gives universality.

To see that we only need continuous \(g_\theta\) and that we can thus use a neural network as \(g_\theta\), we note that the First Fundamental Theorem of \(O(d)\) from invariant theory says that the inner product polynomials \(p_{ij}(v_{1}, \ldots, v_{n}) = v_{i}^\top v_{j}\) are generators of the \(O(d)\)-invariant polynomials. Thus, in the above commutative diagram, we can write \[\phi(x) = \psi \circ \pi(x) = \begin{bmatrix} v_{i}^\top v_{j} \end{bmatrix}_{i,j=1, \ldots, n}.\] So the functions of the form \(\rho(\begin{bmatrix} v_{i}^\top v_{j} \end{bmatrix}_{i,j=1, \ldots, n})\) are universal. As \(\rho\) is unconstrained and continuous, we can approximate it by a neural network \(\rho_\theta \approx \rho\), so the networks of the form \(\rho_\theta( \begin{bmatrix} v_{i}^\top v_{j} \end{bmatrix}_{i,j=1, \ldots, n})\) are universal.

Application 3: Direct Products and Eigenvectors

Our recent paper [Lim et al. 22] inspired this blog post, as I discovered this common pattern underlying the commutative diagram through working on this project. In the paper, we develop universal architectures for a large class of invariances coming from direct products. If \(G_i\) is a group acting on \(\mc X_i\) for \(i=1, \ldots, k\), then the direct product \(G_1 \times \ldots \times G_k\) is the group of tuples \((g_1, \ldots, g_k)\) which acts on \(\mc X_1 \times \ldots \times \mc X_k\) as \((g_1, \ldots, g_k) (x_1, \ldots, x_k) = (g_1 x_1, \ldots, g_k x_k)\). In our Theorem 1 (the decomposition theorem), we show that a continuous function \(f(x_1, \ldots, x_k)\) that is invariant to the large group \(G = G_1 \times \ldots \times G_k\) can be parameterized as \[f(x_1, \ldots, x_k) \approx \rho_\theta\left(\phi_{1,\theta}(x_{1}), \ldots, \phi_{k,\theta}(x_{k}) \right),\] where \(\phi_{i,\theta}\) only needs to be invariant to the constituent smaller group \(G_{i}\), and \(\rho\) is unconstrained. Further, we can take \(\phi_{i,\theta} = \phi_{j,\theta}\) if \(\mc X_{i} = \mc X_{j}\) and \(G_{i} = G_{j}\). We prove this theorem precisely through the same-old commutative diagram, where we show that the \(\phi\) map has a product structure \[\phi(x_{1}, \ldots, x_{k}) = (\phi_{1}(x_{1}), \ldots, \phi_{k}(x_{k})),\] for some \(G_{i}\)-invariant functions \(\phi_{i}\) on the \(\mc X_{i}\).

We proved this result to apply it to a specific invariance arising from learning functions on eigenvectors. For a matrix \(A \in \RR^{n \times n}\) with an eigenvector \(v \in \RR^n\) of eigenvalue \(\lambda\), note that \(-v\) is also an eigenvector of \(A\) with the same eigenvalue. A numerical eigenvector algorithm could have returned either \(v\) or \(-v\), so we want a function \(f\) on eigenvectors \(v_{1}, \ldots, v_{k}\) to be invariant to sign flips: \[f(\pm v_{1}, \ldots, \pm v_{k}) = f(v_{1}, \ldots, v_{k}).\] There are in fact more invariances desired if there is a higher dimensional eigenspace, but we will not cover those here (see [Lim et al. 22] for more on this). This sign invariance is precisely invariance to the action of the group \(G = \{1, -1\}^k = \{1, -1\} \times \ldots \times \{1, -1\}\). Thus, our theorem says that as long as we can parameterize a universal network that is \(\{1, -1\}\) invariant, then we can use that to make a \(\{1, -1\}^k\) invariant function. Invariance to \(\{1, -1\}\) just means that \(h(-x) = h(x)\), so such a function is an even function, which can be simply parameterized as \(h(x) = \varphi(x) + \varphi(-x)\) for some unconstrained \(\varphi\). Thus, we can parameterize sign invariant functions on \(k\) vectors as \[f(v_{1}, \ldots, v_{k}) \approx \rho_\theta\left( \phi_\theta(v_{1}) + \phi_\theta(-v_{1}), \ \ldots, \ \phi_{\theta}(v_{k}) + \phi_{\theta}(-v_{k})\right).\] Note that we can use the same \(\phi_\theta\) for each vector \(v_{i}\) by our decomposition theorem, as \(\mc X_{i} = \mc X_{j}\) and \(G_{i} = G_{j}\) for each \(i\) and \(j\).

Limitations

While I think the strategy based on this commutative diagram may be useful for developing new invariant architectures (especially those that come from direct products of groups), this is far from solving every problem in invariant machine learning. For one, many invariant networks are built from equivariant layers followed by a final invariant pooling. Using equivariance has been shown to significantly improve performance in certain applications (e.g. note the success of networks with convolutional layers that are approximately translation equivariant, or see Figure 7 of [Batzner et al. 22] for an example in molecular dynamics). Also, this is mostly concerned with developing universal architectures, but universality is not sufficient for good performance. Inductive biases can greatly improve the performance of invariant architectures, as can be seen by the success of adding attention modules to Set architectures [Lee et al. 19] [Kim et al. 21] or as we saw by adding message passing to our SignNet architecture for eigenvectors of graph operators [Lim et al. 22]. Even if we do desire universality, this commutative diagram approach may not be computationally useful, as there may be no nice quotient space embedding with structure that we can leverage.

References

[Zaheer et al. 17] Deep Sets.
[Villar et al. 21] Scalars are universal: Equivariant machine learning, structured like classical physics.
[Lim et al. 22] Sign and Basis Invariant Networks for Spectral Graph Representation Learning.
[Finkelshtein et al. 22] A Simple and Universal Rotation Equivariant Point-cloud Network.
[Batzner et al. 21] E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials.
[Lee et al. 19] Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks.
[González and de Salas 03] C∞-differentiable spaces. Springer.
[Kim et al. 21] Transformers Generalize DeepSets and Can be Extended to Graphs and Hypergraphs.
[Kraft and Procesi 96]. Classical Invariant Theory.

Flag Images

Luckily for me, there was some nice data available that worked perfectly for this task. Wikimedia hosts svg files with the flags of all sorts of territories and states under a consistent format. What I did at first was directly scrape the flags of sovereign states from some Wikipedia lists. After running some computations on these, I came across this repository of region flags by Google, which has the Wikipedia flags including some US state and Canadian province flags that I did not have. Thus, I started using these svg files instead. In total, we have 326 flags (with some duplicates due to, for example, Bouvet Island using the flag of Norway). I converted each svg to a png file to allow for the analysis of each flag as a matrix of pixels. Technically, the flag pngs are not quite matrices of pixels, as each pixel has a red, green, and blue value. We simply map these images to grayscale to analyze them—as far as I can tell, there are no overlaps in which two colors map to the same grayscale value.

Ranks and Complexity of Flags

There is no doubt that the flags of low rank are simple and easy to unambiguously define. Any \(m \times n\) real matrix of rank \(k\) can be written as a sum of outer products \(\sum_{j=1}^k x_j y_j^T\), where \(x_j \in \mathbb{R}^m\) and \(y_j \in \mathbb{R}^n\), so \(k(m+n)\) real numbers specify the whole matrix. In fact, all that one has to do to describe the flags of Bolivia and Sweden is to specify a few colors and lines. However, the inverse is certainly not true—not all flags of high rank are complex. This is to be expected by basic linear algebra: any \(n \times n\) diagonal matrix with nonzero diagonal is full rank yet can be fully specified by an \(n\)-tuple of elements. In the same vein, the flag of the Republic of Congo is full rank due to its diagonal structure, yet it is composed of just three lines.

Another measure of complexity of a flag is given by the size in bytes of the flag’s svg file. The Republic of Congo’s svg is only 416 bytes, placing it as only the 183rd largest file in our set. After doing my analysis, I found this blog post which also ranks country flags by their svg file’s size in bytes, and funnily enough does so by using the size of Palau’s svg file as the smallest unit of measurement.

To compute ranks of the grayscale images, I simply use the rank function with default parameters in Julia’s default LinearAlgebra library. This function works by defining the rank as the number of singular values of \(A\) that have magnitude greater than \(k \cdot \epsilon \cdot \sigma_1\), where \(k\) is the smaller of number of rows and number of columns of \(A\), \(\epsilon\) is machine epsilon, and \(\sigma_1\) is the largest singular value of \(A\). Due to numerical error, the results are not perfect (e.g. we find that it computes the rank of the \(2 \times 200\) matrix with all elements equal to \(\pi\) as rank two), but we find that they are accurate in all low-rank examples besides Switzerland, for which the computed rank is 5 but the actual rank is 3.

Since flags have different aspect ratios and require different resolutions, the matrices are of different sizes, which makes a direct comparison of rank to be unfair. Thus, we instead consider the ratio between the rank and the maximum possible rank given the size of the matrix, \(\frac{\mathrm{rank}(A)}{\min(m,n)}\), which we call the percent rank of the \(m \times n\) matrix \(A\).

Altogether, \(15\) flags have full rank and \(43\) flags have rank one. Every one of the flags with full rank has some diagonal structure extending along an entire diagonal. While some of these full rank flags are simple like the Republic of Congo’s, some of them are—in my eyes—deserving of full rank, like Bhutan’s. The distribution of percent ranks covers much of \([0,1]\), with some bias towards rank one and full rank values.

Looking at the flags of lowest rank, we see some interesting patterns in choices by flag designers. Each rank one flag either consists of only horizontal or only vertical lines. None of these rank one flags have more than four different colors, and none of them have more than five lines. A lot of territories fly the flag of France. There are two different types of rank two flags: a bicolored flag with a cross and a multicolored flag with one vertical line with a few horizontal lines. The rank three flags either have an outlined cross or Switzerland’s cross that does not extend to the edges of the flag.

We find that rank and percent rank have a nice, somewhat linear positive relationship. Also, filesize and percent rank have a somewhat positive relationship, but there is clearly a lot of deviation in which flags with low filesize have high percent rank. However, the flags with the largest filesizes do tend to have large ranks; the complexity of an svg file describing a low rank flag is just never that high. It is easy to imagine a pathological case in which a flag is solely composed of a massive amount of thin horizontal lines, in which case we have a rank one flag with a large filesize. For some reason, no state has used such a flag.

Looking at the flags with the largest filesize, we see an assortment of flags with complex symbols and drawings.

Some of these large filesize flags still do not have very high rank. The flag of Pennsylvania has filesize 775,498 bytes and 62% rank, so it has the second largest filesize but merely 107th largest percent rank. While it has a detailed seal, the seal does not take up the whole flag, so it has many linearly dependent rows that are just entirely blue.

The case of Pennsylvania gives some nice intuition on these two flag complexity metrics, and suggests another way to measure complexity. We can measure each flag with a combination of these two metrics by taking the sum of the position (a better term would be rank or positional rank, but that would be confusing in the context of the linear rank) as measured by filesize with the position as measured by percent rank. Another way to combine the ranks would be through something like a rank product, but the simple sum maintains integer values. Here are the most complex flags as measured by this metric.

These flags do not just have detailed symbols and drawings, but in fact they have large drawings that span significant portions of the flag, or at least diagonal lines that do. Pennsylvania falls to position 27 in this ranking, but other flags like North Dakota’s (position 4) have a seal in the middle with a lot of empty columns; the difference is that North Dakota’s seal takes up a larger portion of the vertical space of the flag, so there are many linearly independent rows. There are still some flags that somewhat hack this ranking. The flag of Florida (position 1) has a small seal, but it has diagonal lines that give it full rank.

Without US States and Canadian Provinces

For anyone who does not care for US State or Canadian provinces, we also compute the filesize and position sum rankings after omitting flags for these subregions. Note that no US State or Canadian province flags have low-rank, while many have large filesizes. Some regions change relative order in position sum after omitting the subregions, in part due to omission of subregions making the filesize rankings more compact. For instance, Saint Pierre and Miquelon is now position 1, even though it was behind Tristan da Cunha when subregions were included.

Thoughts

The ranking of flag complexity by the sum of the rankings by percent rank and filesize seems pretty fair to me. I especially feel that the flag of Saint Pierre and Miquelon is deserving of the top spot, as it has neat designs that take up the whole flag. I am somewhat disappointed by the lack of variety in the designs of the flags of rank at most 3, but this is understandable given that flag designers probably did not care much for the rank of their flags. This type of computation would probably be pretty interesting for other types of flags and graphics. For example, the checkered flags used in auto racing have rank two and have a design distinct from the two types of rank two region flags.

Science is what we understand well enough to explain to a computer. Art is everything else we do. — Donald Knuth

If we import numpy and matplotlib,

import numpy as np
import matplotlib.pyplot as plt

then we can generate really nice art in just 3 lines of code.

As = np.random.randn(6,6,3)
vals = [val for s in np.linspace(0,1,500) for t in np.linspace(0,1-s,500) for val in np.linalg.eigvals(s*As[:,:,0] + t*As[:,:,1] + (1-s-t)*As[:,:,2])]
plt.plot(np.real(vals), np.imag(vals), 'mo', alpha=.5, markeredgewidth=.1, markeredgecolor='k', markersize=1.5)

Of course, this code was condensed for the novelty of having as few lines of code as possible. It is quite beautiful that essentially infinitely many (if not for the finiteness of the floating point system among other physical limitations) pretty images can be completely specified in so few lines of code. Compare this to the complexity of a Post-Impressionist oil painting, which would require data at the atomic level of every brushstroke to unambiguously store. To be fair, a better comparison would probably take into account the size of the library functions called along with the python interpreter, and it can be argued that a painting does not require atomic level information to understand or experience. Even so, the basic point still stands.

Coming in at a more modest 9 lines we have code that is more flexible and easy to read:

def eigenart(n, prec=500):
    A1 = np.random.randn(n,n)
    A2 = np.random.randn(n,n)
    A3 = np.random.randn(n,n)
    vals = []
    for s in np.linspace(0,1, prec):
        for t in np.linspace(0,1-s, prec):
            vals.extend(np.linalg.eigvals(s*A1 + t*A2 + (1-s-t)*A3))
    plt.plot(np.real(vals), np.imag(vals), 'mo', alpha=.5, markeredgewidth=.1, markeredgecolor='k', markersize=1.5)

The generated art is simply the set of all complex numbers that occur as an eigenvalue of some matrix in a convex set of matrices. Let \(M_n(\mathrm{F})\) denote the set of \(n \times n\) matrices with entries in \(\mathbb{F}\), \(\sigma(A)\) denote the set of eigenvalues of the matrix \(A\), and \(\mathrm{Co}(S)\) denote the convex hull of \(S\). For a set of matrices \(S \subseteq M_n(\mathbb{C})\), we define the hull spectra of \(S\) to be \[\mathrm{HS}(S) = \{\lambda \in \sigma(A) : A \in \mathrm{Co}(S) \}\] Thus, our eigenart function plots \(\mathrm{HS}(\{A_1, A_2, A_3\})\) for three random matrices \(A_1, A_2,\) and \(A_3\) in \(M_n(\mathbb{R})\) with entries drawn from a standard normal distribution. We can now understand certain properties of the generated art.

In the eigenart function, \(n\) is the size of the matrices. Larger \(n\) gives more complex hull spectra as each matrix in \(\mathrm{Co}(S)\) has more eigenvalues, and they interact in more complex ways. prec controls the number of convex combinations of matrices whose eigenvalues are computed and plotted. It can be increased to get rid of some of the areas of low density, or decreased to allow for less expensive computation. The plotting parameter alpha allows transparent markers so that overlap can be seen. Also, the small black marker edges allow for a shading effect, as areas that are denser with eigenvalues appear darker.

The hull spectra shown above are symmetric about the real axis—the generator matrices were chosen to be real, and the non-real eigenvalues of real matrices in \(M_n(\mathbb{R})\) come in conjugate pairs. If we instead take our random generators to have non-real entries as well, then the hull spectra does not have this symmetry. Also, real matrices tend to have some real eigenvalues with high probability (for instance, any \(n \times n\) real matrix with \(n\) an odd number has at least one real eigenvalue), so most hull spectra of real matrices contain a line segment or multiple line segments on the real line.

Let \(A_{s, t} = s A_1 + t A_2 + (1-s-t) A_3\) be a matrix in \(\mathrm{Co}(\{A_1, A_2, A_3\})\). Continuity and differentiability of the eigenvalues of a family of matrices is a technical subject, so what follows is rather hand-wavy. The entire set of eigenvalues of a matrix depends continuously on its entries. Some of the holes in the hull spectra are actual holes where there are no eigenvalues. However, some of the areas where there are holes or that are sparse with eigenvalues are actually mathematically completely solid; the choice of a finite number of matrices in the convex hull to evaluate at causes them to appear empty. These spots tend to occur near an eigenvalue of higher multiplicity, which is where non-differentiability of eigenvalues can occur. This explains sparse areas of eigenvalues near the real line, as when a non-real eigenvalue approaches the real line, its conjugate is another eigenvalue that approaches the real line. Elsewhere, the shading shows “folds” and other properties that seem to give the hull spectra depth. This can be explained by the fact that the hull spectra is a projection onto the first two coordinates of the higher dimensional surface consisting of points of the form \(\big(\Re\; \lambda(A_{s,t}), \Im\; \lambda(A_{s,t}), s, t\big)\) in \(\mathbb{R}^4\) (recall that the eigenvalues depend continuously on \(s\) and \(t\)).

The eigenvalues of a matrix are the roots of its characteristic polynomial, and any polynomial \(p\) has a corresponding matrix \(C\) (the companion matrix) such that the eigenvalues of \(C\) are precisely the roots of \(p\). From another viewpoint, the hull spectra \(\mathrm{HS}(S)\) can also be viewed as the roots of the characteristic polynomials of matrices in \(\mathrm{Co}(S)\) Others have created or discovered art from the roots of polynomials before: this post by John Baez and the work of Bahman Kalantari in polynomiography provide striking examples.

While hull spectra do provide amazing images, my interest in them arose from important problems in mathematics. One open problem that has been unsolved for decades is that of determining the set of complex numbers which occur as the eigenvalue of some \(n \times n\) doubly stochastic matrix. Equivalently, this set is the hull spectra of the \(n \times n\) permutation matrices. This article [1] provides a good survey of this problem and similar problems; I have also published two works in this field [2] [3].

[1] Charles R. Johnson, Carlos Marijuán, Pietro Paparella, and Miriam Pisonero. “The NIEP.” Operator Theory, Operator Algebras, and Matrix Theory, pp. 199-220 (2018).

[2] Eric Jankowski, Charles R. Johnson, and Derek Lim. “Spectra of convex hulls of matrix groups.” Linear Algebra and its Applications (2020).

[3] Amit Harlev, Charles R. Johnson, and Derek Lim. “The Doubly Stochastic Single Eigenvalue Problem: A Computational Approach.” Experimental Mathematics (2020).