Timetables are one of those components that look simple but contain a surprising amount of layout logic. For a project in 2026 I needed a version that supports multiple stages, adapts to the tallest session, and stays aligned across the entire timeline — all built in CSS.

Since the post is fairly long, here’s a brief overview of what’s coming:

1. Building the grid and placing the sessions — using subgrid, mod(), and round() to map times to grid lines.

2. Adding hour indicators — adding visual markers and handling horizontal overflow.

3. Sticky session headlines with scroll-driven animations — making the stage headers follow the scroll position.

Before diving in, here is the final result. This is what we’re building.

See the Pen Timetable | 03b by Nils Binder (@enbee81) on CodePen.

Part I: Building the grid and placing the sessions

Setting up the basic HTML

One of my main goals was to write clear, readable markup that defines each stage and its sessions. I also wanted a simple and intuitive way to assign a start and end time to every session. Here’s the base structure I ended up with:

html<div class="timetable">
  <div class="timetable--body">
    <div class="stage" style="--column: 2;">
      <h2 class="stage-headline">Main Stage</h2>
      <ol class="session-list" role="list">
        <li class="session">
          <!-- Any content for your session card goes here -->
        </li>
        
        <!-- Add more sessions below -->
      </ol>
    </div>
    
    <div class="stage" style="--column: 3;">
      <!-- Second stage -->
    </div>
  </div>
</div>

Let's break it down. We have a wrapper for the entire timetable, and a second wrapper for its body. The .timetable--body element isn’t strictly necessary yet, but will become useful later. Each stage gets its own container, marked with a headline and an ordered list holding all sessions for that stage.

Note: You may have noticed the inline styles that define which column each stage belongs to. This will be used later when placing the stages on the grid. I could have done this with pure CSS, but since the layout is already fairly complex, I decided it’s clearer and easier to understand when kept directly in the markup.

To assign time slots to each session, we can use custom properties directly on the <li> elements as inline styles:

html<li class="session" style="--start: 900; --end: 945;"> 
  <!-- Represents 9:00 - 9:45 -->
</li>
<li class="session" style="--start: 1015; --end: 1120;">
  <!-- Represents 10:15 - 11:20 -->
</li>
<li class="session" style="--start: 1400; --end: 1500;">
  <!-- Represents 14:00 - 15:00 -->
  <!-- am/pm 2pm - 3pm -->
</li>

Note: I’m European, so I’ll be using the 24-hour clock. Even if you prefer AM/PM, the --start and --end values need to stay in 24-hour format behind the scenes—otherwise the math gets messy.

Setting up base CSS

Basic variables

Now let's look at the CSS. For a component like this, I like to define a few useful variables up front:

css.timetable {
  --zebra-color-one: #224;
  --zebra-color-two: #003;
  
  --time-unit-height: 0.5rem;
  --stages: 2;
  --stage-gap: 1.5rem;
  --stage-width: min(100vw - 3rem, 22rem);
  
  --start-time: 9;
  --end-time: 18;
  --event-duration: calc(var(--end-time) - var(--start-time) + 1);
}

There’s already quite a bit going on here, so let’s break it down step by step.

We start simple: two color variables that we’ll reuse later.

Next is --time-unit-height, which is where things get more interesting. My timetable supports sessions that can start at 5-minute increments. That means one hour contains 12 possible slots. --time-unit-height defines the minimal visual height of one of those 5-minute units.

Increasing it makes every hour taller and gives more room for precise alignment, but it also means you’ll see more blank space during breaks or empty hours (like a long lunch slot).

Then we have --stages. I’m assuming you already know how many stages your event uses, and since this affects layout calculations, we store that as a variable.

Along with the number of stages, we define a gap between them (--stage-gap) and a consistent --stage-width. In this example, each stage should ideally be 22rem wide, but if that doesn’t fit, the layout will shrink to 100vw - 3rem.

We also store the --start-time and --end-time. Using both values, we calculate --event-duration, which gives us the total number of hours in our timeline. I’m adding one extra hour here purely for visual spacing — not for math accuracy, but for layout aesthetics.

Note: Most likely, you’ll be pulling your timetable data from an external source. When generating the HTML, you might not have access to the CSS files at build time. In that case, you can inject the number of stages, along with the start and end time, as inline styles on the .timetable element instead.

Grid definitions

Now we can use the variables we defined earlier to build the grid for our timetable:

css.timetable {
  /* ... */
  --rows: auto repeat(calc(var(--event-duration) * 12), minmax(var(--time-unit-height), auto));
  --columns: 1rem repeat(var(--stages), minmax(var(--stage-width), 1fr)) 1rem;
}

Rows: We start with an auto-width row at the top — this is where the stage headlines will appear. Below that, we add 12 rows for every hour (var(--event-duration) * 12), because we’re using 5-minute increments.

Each of those rows uses minmax(var(--time-unit-height), auto), which means:

• It has a minimum height defined by --time-unit-height

• but can grow if the session content needs more space

Columns: For the columns, we create one column per stage using repeat(var(--stages), ...). Each stage column has a minimum width defined by --stage-width, but can expand up to 1fr if there is extra room. For purely visual reasons, I also added a 1rem column at the beginning and the end. Later, we’ll place the hour indicators inside these outer columns.

Making use of grid and subgrid

To make sure every hour automatically grows to match the tallest session on any stage, all stages need to rely on the same underlying grid structure. We apply this to the .timetable--body element using the variables we defined earlier for columns, rows, and gaps.

Each stage then reuses that grid by setting grid-template-rows: subgrid.

A common mistake here is forgetting that a stage must span all rows before it can inherit them. To fix that, we set grid-row: 1 / -1, which makes each stage stretch from the top to the bottom of the timetable.

It doesn’t stop there: each session list also needs to use the same grid defined on .timetable--body. Because the session list sits right after the stage headline, it starts on the second row and continues all the way to the bottom (grid-row: 2 / -1).

css.timetable--body {
  display: grid;
  gap: 0 var(--stage-gap);
  grid-template-rows: var(--rows);
  grid-template-columns: var(--columns);
}

.stage {
  display: grid;
  grid-column: var(--column);
  grid-row: 1 / -1;
  grid-template-rows: subgrid;
}

.session-list {
  list-style-type: none;
  grid-row: 2 / -1;
  grid-column: 1;
  display: grid;
  grid-template-rows: subgrid;
}  

A little adjustment is missing here for smaller screen sizes. We want to be able to scroll though the timetable horizontally, so we add two lines of CSS to our .timeline--body:

css.timeline--body {
  max-width: 100%;
  overflow-x: auto;
  
  /* ... */
}

Placing the sessions onto the grid using round() and mod()

Now comes the trickiest part of the whole setup. To place each session in the correct position on the grid, we need to convert the time (stored as a number like 1045) into minutes. Remember, every session stores its start and end time as inline custom properties:

html<li class="session" style="--start: 1045; --end: 1120;">

Step 1: Convert hours to minutes

• First, we extract the hour portion of the number. We divide by 100: 1045 / 100 = 10.45

• Now we only want the full hours, so we round the number down: round(down, 10.45, 1) = 10

• Then we multiply that by 60 to get the number of minutes:

css.session {
  --start-hours-to-minutes: calc(round(down, var(--start) / 100, 1) * 60);
}

Step 2: Extract the remaining minutes

We still need the minutes portion (45 in 1045).

To do that, we use mod(), which returns the remainder of a division.

By taking mod(var(--start), 100), we get the last two digits of the number:

css.session {
  --start-minutes: mod(var(--start), 100);
}

Step 3: Remove minutes before the timetable starts

Next, we need to subtract all the minutes that happened before the timetable begins. For example, if the event starts at 09:00, we don’t want to count the minutes from midnight to 09:00.

We calculate those minutes like this:

css.session {
  --timetable-offset: calc(var(--start-time) * 60);
}

Step 4: Converting minutes to grid lines

Now we have everything we need to place a session on the grid.

We combine the minutes from the hour portion and the minute remainder (--start-hours-to-minutes + --start-minutes), then subtract the offset (--timetable-offset) so we only count the minutes within the visible timetable.

Since each grid row represents a 5-minute interval, we divide the result by 5.

CSS grid lines start at 1, so we add + 1 to shift the result by one line — otherwise, the first session would end up at line 0.

css.session {
  grid-row-start: calc(
    (var(--start-hours-to-minutes) + var(--start-minutes) - var(--timetable-offset)) / 5 + 1
  );
}

We need the same calculation for the end time, so the session knows where to stop. Putting it all together:

css.session {
  --timetable-offset: calc(var(--start-time) * 60);
  
  --start-hours-to-minutes: calc(round(down, var(--start) / 100, 1) * 60);
  --start-minutes: mod(var(--start), 100);
  --end-hours-to-minutes: calc(round(down, var(--end) / 100, 1) * 60);
  --end-minutes: mod(var(--end), 100);
  
  grid-row-start: calc(
    (var(--start-hours-to-minutes) + var(--start-minutes) - var(--timetable-offset)) / 5 + 1
  );
  grid-row-end: calc(
    (var(--end-hours-to-minutes) + var(--end-minutes) - var(--timetable-offset)) / 5 + 1
  );
}

With this in place, we already get a timetable that adapts its height to the content. Notice how the slot between 10:00 and 11:00 becomes much taller than the empty hour around noon during the lunch break.

See the Pen Timetable | 01 by Nils Binder (@enbee81) on CodePen.

Note: There’s also some extra styling for the session cards and the stage headlines. That’s not really part of this tutorial, so you can style those however you like.

Part II: Adding sticky hour indicators

Step 1: Adding hour indicators to the HTML and placing them on the grid

Although all sessions are already aligned correctly, it’s still hard to see when they start. To make this easier, we can add hour indicators that mark the beginning of each hour.

CSS grid does not provide a way to style a single grid row directly, the way we can with table rows in HTML. So we need to add our own elements to the markup. In this case, the hour markers are placed as the first child of .timetable--body, before the first stage:

html<div class="hours" aria-hidden="true">
  <div>
    <time datetime="09:00">09:00</time>
  </div>
  <div>
    <time datetime="10:00">10:00</time>
  </div>
  <div>
    <time datetime="11:00">11:00</time>
  </div>
  <!-- ... -->
</div>

We also need a small update to the CSS. Remember the first “empty” column we added in the column definition? Now we turn it into a defined width column that will contain the hour indicators. We add a new variable and use it in the grid definition:

css.timetable {
  /* ... */
  --th-width: 3.5rem;
  --columns: var(--th-width) repeat(var(--stages), minmax(var(--stage-width), 1fr)) 1rem;
}

Next, we need to place the hour indicators on the grid and make sure they use the same row and column setup as the rest of the timetable. Each direct child of .hours should span 12 rows, which represents exactly one hour in our layout.

This is also where the two zebra colors from the beginning come into play. By alternating the background on every second child, we get a subtle zebra pattern that makes it much easier to see where each hour begins and ends.

css.hours {
  --background: var(--zebra-color-one);
  display: grid;
  grid-row: 2 / -1;
  grid-column: 1 / -1;
  grid-template-rows: subgrid;
  
  > div {
    grid-row-end: span 12;
    background: var(--background);
    
    &:nth-child(odd) {
      --background: var(--zebra-color-two);
    }
  }
}

With this in place, the timeline becomes much easier to read.

See the Pen Timetable | 02a by Nils Binder (@enbee81) on CodePen.

Step 2: Making the hour indicators sticky

You may have noticed that the last example includes a third stage. On smaller screens this creates horizontal overflow, and because we set overflow-x: auto on .timetable--body, users can scroll sideways to reveal all stages.

However, when scrolling horizontally, the hour indicators disappear off the screen. To fix this, we keep each <div> as it is (because it controls the zebra stripes), but make the <time> elements themselves sticky:

css.timetable {
  /* ... */
  --th-width: 3.5rem;
  --stage-width: min(100vw - var(--th-width) - 3rem, 22rem);
}

.hours {
  time {
    display: block;
    position: sticky;
    inset-inline-start: 0;
    width: var(--th-width);
    height: 100%;
    background: rgb(from var(--background) r g b / 0.95);
    z-index: 2;
  }
 } 

Here's what is happening here:

• Since the hour indicators stay visible on the left, we subtract their width from the maximum width of each stage. That’s handled in the --stage-width variable on .timetable.

• <time> elements are inline by default, so we set display: block

• position: sticky with inset-inline-start: 0 keeps them visible on the left when scrolling

• the width matches the first column, and the height covers the full hour

• we reuse the zebra background, but make it slightly transparent

• and finally, because the hour indicators are the first elements inside .timetable--body, a z-index ensures they stay in front of the sessions

See the Pen Timetable | 02b by Nils Binder (@enbee81) on CodePen.

Step 3: Let each stage snap while scrolling

We already have horizontal overflow in place. Now it would be nice if each stage snapped into position so it rests neatly next to the hour indicators when scrolling. This only takes a few lines of CSS:

css.timetable--body {
  /* ... */
  scroll-snap-type: x mandatory;
}

.stage {
  /* ... */
  scroll-snap-align: start;
  scroll-margin-inline-start: calc(var(--th-width) + 0.5rem);
}

See the Pen Timetable | 02b by Nils Binder (@enbee81) on CodePen.

Part III: Sticky session headlines with scroll driven animations

This part is a bit hacky. I want the section headlines to be sticky, similar to the hour labels. But the timetable scrolls on the inline axis, and position: sticky only works within the element’s own scrolling container. When you scroll vertically (the block axis), the page (html) is the scroller, so you can’t pin something to the top if it lives inside a different scrolling context.

One workaround is to constrain the height of the timetable body so it becomes the vertical scroller as well. Then you’d scroll the timetable in both directions, and the headlines could stick to the top. In practice, though, a large two-axis scroll region can feel awkward — it’s not always obvious which container is scrolling. But see for yourself:

See the Pen Timetable | 02c by Nils Binder (@enbee81) on CodePen.

Scroll driven animations to the rescue

One way to solve this is to add a set of “fake” headlines before the actual timetable body. Since they’re duplicates of the real headlines, we mark them with aria-hidden="true" so they’re ignored by screen readers.

html<div class="timetable">
  <div class="timetable--head" aria-hidden="true">
    <div class="timetable--inner-head">
      <div class="stage-headline">Main Stage</div>
      <div class="stage-headline">Masterclasses</div>
      <div class="stage-headline">Startup Stage</div>
    </div>
  </div>
  
  <div class="timetable--body">
    /* ... */
  </div>
</div>

The entire .timetable--head sits before .timetable--body, which is the scrolling element. Because its parent scroller is still the root (html) element, these headlines can stick to the top of the page.

Let's add a bit of CSS to style them:

css.timetable--head {
  display: none;
  position: sticky;
  pointer-events: none;
  top: 0;
  z-index: 5;
  
  @supports (animation-timeline: --scroll-x) {
    display: block;
  }
}

.timetable--inner-head {
  display: grid;
  gap: 0 var(--stage-gap);
  grid-template-columns: var(--columns);
  
  background: var(--zebra-color-one);
  color: #fff;
  position: absolute;
  inset-inline-start: 0;
  inset-block-start: 0;
  min-width: 100%;
  box-shadow: 0 1px 0 rgb(255 255 255 / 0.2);
  
  > :first-child {
    grid-column: 2;
  }
}

For the .timetable--head, you can see that it’s hidden by default. The whole setup relies on animation-timeline, which isn’t supported in all browsers, so we only show the element when support is detected. We also set pointer-events: none because this is just a visual placeholder that mimics the real headline section and shouldn’t block any interaction.

The .timetable--head itself uses position: sticky so it stays fixed at the top of the page while scrolling.

In the .timetable--inner-head, we repeat the grid definitions from .timetable--body. Because the values are stored in variables, we can just reuse them here. While we used subgrid in the body to share the grid structure, the head exists in its own context, so it needs its own grid definition here.

See the Pen Timetable | 03a by Nils Binder (@enbee81) on CodePen.

Now the headlines are sticky, but on smaller screens they still don’t move along the inline axis, because they’re not part of the scrolling container. This is where scroll-driven animations come in.

The idea is:

• we create a named scroll timeline on .timetable--body, so other elements can react to its scroll position

• because .timetable--head is not a child of .timetable--body, we extend the timeline’s scope on the surrounding .timetable container using timeline-scope

• with that in place, we animate .timetable--inner-head so its horizontal position follows the scroll progress of .timetable--body

Here’s the CSS for that:

css.timetable {
  /* ... */
  
  /* Allow descendants to use the named scroll timeline */
  timeline-scope: --scroll-x;
}

.timetable--inner-head {
  /* ... */
  
  /* Apply the animation */
  animation: slide linear both;
  /* Use the named animation timeline */
  animation-timeline: --scroll-x;
}

/* Scroll-driven animation:
   - starts with the left edge aligned to the container’s left
   - ends with the right edge aligned to the container’s right */
@keyframes slide {
  from {
    inset-inline-start: 0;
    transform: translateX(0);
  }
  to {
    inset-inline-start: 100%;
    transform: translateX(-100%);
  }
}

.timeline--body {
  /* ... */
  
  /* Create a named scroll timeline on the inline axis */
  scroll-timeline-name: --scroll-x;
  scroll-timeline-axis: inline;
}

And that’s it 🎉 — all pieces in place. Here’s the final result:

See the Pen Timetable | 03b by Nils Binder (@enbee81) on CodePen.

Final notes

The functions round() and mod() became part of the CSS Baseline in 2024, and at the time of writing they’re supported by roughly 86% of browsers according to caniuse. For older browsers I’ve added a small JavaScript fallback that calculates the session positions and applies the grid placement inline, so the timetable still works even without full CSS support.

That’s all for this component. I hope the breakdown of the structure, the positioning logic, and the scroll-driven parts is useful if you ever need to build something similar. If you use this for your own timetable, I’d be curious to see the result.

Introduction

Historically, the term “amateur” referred to a non-professional lover of the arts, someone who pursued artistic activities out of passion rather than for economic gain. In an era when the boundaries between professional artists and cultivated laypersons were still fluid, amateurs could participate meaningfully in the cultural sphere and earn recognition for their work (Goethe & Schiller 1799).

This fragile balance began to shift in the nineteenth century, when the invention of photography and the emergence of modernism redefined the role of painting and displaced traditional craftsmanship in favor of conceptual innovation (Benjamin 1936).

Today, a parallel transformation is unfolding in software development: the automation of programming by large language models (LLMs) is once again moving value away from technical mastery toward creativity, abstraction, and design.

The Amateur in Classical Art

In the eighteenth and early nineteenth centuries, the amateur was regarded as a respected connoisseur and lover of the arts. Derived from the Latin amator (lover), the term “amateur” referred to individuals who engaged in artistic pursuits out of passion rather than professional ambition or economic necessity. At a time when the division between professional artists and cultivated amateurs was less rigid than today, amateurs could gain serious recognition for their contributions to cultural life (Goethe & Schiller 1799).

Technical skill was the primary criterion for evaluating artistic work during this period. Craftsmanship was seen as tangible proof of both the artist’s seriousness and the quality of their work. Particularly in fields such as portrait painting, landscape painting, and music, the precision of execution, composition, and mastery of material were essential to artistic value. This emphasis on technical proficiency enabled amateurs to be acknowledged, provided they demonstrated a high level of skill (Rebentisch 2017, p. 26ff.).

Moreover, even within the sphere of professional art, the production process was often collaborative. In many prominent workshops, especially during the Dutch Golden Age, masters such as Rembrandt or Rubens supervised large studios where assistants and pupils contributed to the production of artworks. Often, the master would create the compositional sketch or focal figures, while background elements and secondary details were executed by apprentices. As a result, the boundaries between individual authorship and collective creation were frequently blurred, and attributions remain uncertain to this day. This early form of division of labor in the arts reveals a tension between the singular “artistic genius” and the collaborative production process.

From a social perspective, amateur artistry fulfilled a dual function. On the one hand, it served to demonstrate education and cultivated leisure, particularly within the rising bourgeoisie. Artistic practice became an expression of comprehensive personal development, as formulated by thinkers such as Wilhelm von Humboldt. On the other hand, amateurs played an important role in disseminating artistic practices beyond professional circles: private musical performances, drawing salons, and the publication of poetry collections by non-professionals contributed significantly to the formation of a broader cultural public (Bourdieu 1992).

Only with the increasing professionalization of the art world in the course of the nineteenth century—and the institutionalization of specialized training programs—did the separation between the professional artist and the amateur become more pronounced. In the classical art world of the eighteenth century, however, the amateur remained a recognized figure, judged primarily by the standard of technical mastery rather than by innovation or theoretical discourse.

The Invention of Photography and Its Consequences

The invention of photography in the early nineteenth century marked a profound turning point in the history of the visual arts. With the ability to create precise representations of reality through technical means, painting lost its central role as the primary medium for realistic depiction. Portrait painting in particular, which had served important social functions in both aristocratic and bourgeois contexts, found itself facing technological competition that was faster, more affordable, and more objective (Benjamin 1936).

This new competition forced painting to redefine its role. Rather than continuing to serve primarily as a medium for faithful reproduction, artists began to explore subjective perceptions, emotional atmospheres, and experimental forms of representation. Movements such as Impressionism, Expressionism, and later Abstract Art emerged as direct responses to these challenges. Consequently, technical mastery—control of perspective, anatomy, and light—gradually gave way to conceptual considerations and personal expression.

Painting survived this crisis by emancipating itself from its traditional mimetic functions and establishing new values: innovation, authenticity, and subjectivity. Thus, artistic evaluation shifted from the mastery of established techniques to the originality of ideas and the distinctiveness of individual style.

The transformation of painting in response to technological innovation illustrates a broader dynamic: when a technical function can be automated or reproduced more efficiently by new media, the focus of human creative work tends to shift toward areas where intuition, conceptual thinking, and innovation cannot easily be mechanized. This historical development provides an important framework for understanding contemporary shifts in other fields, particularly the impact of automation on programming and software development.

Parallels to Programming

The historical trajectory observed in the visual arts—where technological innovation displaced traditional craftsmanship and redirected human creativity toward conceptual domains—finds a striking parallel in the contemporary field of software development. For much of its history, programming has been regarded primarily as a craft: a discipline requiring meticulous technical skill, mastery of complex systems, and a deep understanding of abstract logic. The programmer was seen as a kind of artisan, comparable to the classical painter or composer, whose expertise was rooted in extensive, often arduous, training and practice.

However, as in the case of the arts, programming has always contained an element of division of labor. In large-scale software projects, it became increasingly common to distinguish between developers responsible for the detailed implementation of code and software architects who conceived the overarching system design. While the architects formulated high-level solutions and conceptual frameworks, developers executed the technical realization, similar to how apprentices in a painter’s workshop contributed to the realization of a master’s vision.

From Craft to Concept: The Changing Role of the Programmer

Recent advances in artificial intelligence, particularly the rise of large language models (LLMs), are now disrupting this traditional model. Tasks that once demanded considerable technical effort—such as writing boilerplate code, debugging, or even generating complex algorithmic solutions—can now be performed, or at least significantly accelerated, by AI systems. The routine, repetitive aspects of programming are thus increasingly subject to automation, much like realistic portraiture was once superseded by the camera.

This development is transforming the role of the human programmer. Mere technical execution is no longer sufficient to secure a professional identity. Instead, greater emphasis is placed on skills that machines cannot easily replicate: the ability to formulate meaningful problems, design robust and flexible architectures, and devise creative, context-sensitive solutions. In this emerging landscape, the programmer must increasingly act as a conceptual thinker, strategist, and innovator—rather than merely as a craftsman.

Mere technical execution is no longer sufficient to secure a professional identity.

The social consequences of this shift mirror those observed in the historical transformation of the art world. Just as technical proficiency alone ceased to guarantee artistic recognition in the modern era, technical coding skills alone may soon no longer guarantee professional success. Instead, recognition and value will increasingly depend on an individual’s capacity for abstraction, critical thinking, and creative synthesis.

For decades, programmers have enjoyed an elevated social status. They have been celebrated as the architects of the digital future, commanding high salaries and cultivating a professional identity marked by exclusivity and self-confidence. Yet with the rise of artificial intelligence capable of automating significant portions of coding work, this self-image is undergoing a dramatic destabilization.

Whereas previous waves of digitalization primarily affected external fields—such as retail, logistics, or administrative work—today it is the very agents of digitalization who find their own roles threatened. Programmers, once the initiators of disruption, are now themselves subject to it. This inversion marks a profound psychological shift: professions once believed to be immune to automation must now confront their own obsolescence.

Conclusion

The trajectory from the disappearance of the amateur in classical art to the current transformations in programming reveals a striking historical continuity: technological innovation repeatedly displaces traditional craftsmanship, forcing human creativity to migrate into conceptual and strategic domains.

In the arts, the invention of photography precipitated a crisis of representation that ultimately redefined artistic value, shifting emphasis from technical execution to originality, expression, and theoretical engagement. Similarly, in software development today, the rise of artificial intelligence is automating many routine programming tasks and challenging the established identity of programmers as technical artisans. The human contribution is increasingly concentrated on areas that demand abstraction, critical thinking, and creative problem-solving.

This shift is not merely technical or economic but profoundly social and psychological. The destabilization of traditional professional identities—particularly among programmers who once stood at the forefront of digital disruption—illustrates the broader societal implications of automation. As expertise is redefined and new hierarchies emerge, individuals and institutions alike must rethink the nature of work, creativity, and human agency in an increasingly automated environment.

Resource List

Benjamin, Walter. “The Work of Art in the Age of Mechanical Reproduction.” 1936. In Illuminations, edited by Hannah Arendt and translated by Harry Zohn, 217–251. New York: Schocken Books, 1968.

Bourdieu, Pierre. The Rules of Art: Genesis and Structure of the Literary Field. Translated by Susan Emanuel. Stanford, CA: Stanford University Press, 1996. (Original French edition 1992).

Goethe, Johann Wolfgang von, and Friedrich Schiller. Correspondence, 1794–1805. Edited by Karl Heinemann. Berlin: Aufbau Verlag, 1980. (Letters dated 1799 cited in text).

Rebentisch, Juliane. Theorien der Gegenwartskunst zur Einführung. 2nd revised ed. Hamburg: Junius, 2017.

Photos

Jean-Baptiste-Simeon Chardin, "The Young Draughtsman"

Peter Paul Rubens, “The Garden of Love”

Louis-Jacques-Mandé Daguerre (1844), photographed by Jean-Baptiste Sabatier-Blot

Marcel Duchamp, “Fountain”, photograph by Alfred Stieglitz

We have all seen amazing artistic or realistic images generated by AI. However, one main challenge for AI image generation is to find use cases that solve problems or boost process performance. One such use case is changing a Model's clothes to your desire while keeping the overall image intact. This can be a massive benefit for the fashion industry and e-commerce domain. This post will describe the general idea behind the setup used for clothes replacement and some challenges and possible fixes.

Use Cases

This blog post focused on replacing or swapping clothes from Person A to Person B. Using a ComfyUI workflow like this, it is also possible to replace a Person's clothes with a “Product Shot” of a clothing piece.

• Virtual Try-On: One of the main problems of online shopping is the lack of ability to try on clothes to see if the color or fitting would suit you. AI clothes replacement can mitigate this, leading to better customer experience and reduced returns. Additionally, it could be extended with personal AI fashion assistants to provide feedback or search for similar clothing pieces.

• Fashion Campaigns: Booking models, locations, and photographs for a campaign can be expensive and time-consuming, especially for smaller brands. For viral or trending campaigns, it is vital to be fast; with clothes replacements, it is possible to recreate viral Memes or trends with your fashion. In the early stages of a campaign concept, it can help to test different ideas in a more realistic setting.

• Storytelling/Education: Clothe replacements can be used as digital installations to show historical wardrobes for educational purposes. It can also be used in storytelling scenarios, especially in addition to consistent character techniques to create fitting images for your story.

Setup/Workflow

This workflow aims to replace the clothes from an Image of Person A and Place the selected clothes on Person B. We want to specify which clothes to replace and if we will replace the whole outfit or just certain parts, e.g., the pants or the top. To achieve this, we have to work with image segmentation tools to find the parts of the image that represent clothes. The workflow uses the fashion segmentation custom node from https://github.com/1038lab/ComfyUI-RMBG/tree/main. Before we start the segmentation process, we remove the background from the original image of Person A to optimize the quality of the result. For this segmentation, it is better to have a higher segmentation offset so that we don’t have holes or missing parts in the image since it is used for the style reference in the FLUX.1 Redux model. Here are some examples of different segmentations for different selections:

For Person B, we must create a mask for the parts/clothes where the replacement should happen. This is done by using image segmentation to generate a mask of the segmented parts of the image. This time, we use the clothes segmentation from https://github.com/1038lab/ComfyUI-RMBG/tree/main since it allows us to specify body parts and regions more freely. In this case, we also want to use a more narrow mask and a bit of blur for the mask because this mask will be used for the Flux.1 Fill model to be inpainted. The blur helps smooth edges, leading to a better output quality. Following are some examples of masks created by fashion segmentation, which allow us to also mask body parts like arms/legs that are not covered by clothes.

The segmented clothes from Person A are then used as the style reference for the flux redux model. The Flux.1 Fill model then inpaints the masked Part of Person B with a style referenced from the Flux.1 Redux model. This approach allows us to replace only the clothes and keep the rest of the Image of Person B intact. We enhance the output quality using the inpaint crop and inpaint stitch approach using this custom node: https://github.com/lquesada/ComfyUI-Inpaint-CropAndStitch. This means we crop the image to fit the generated mask, which allows for more details in the inpaint process, and then stitch the altered part back into the original image so that the rest stays the same.

Examples and Challenges

In the following section, I will provide some examples of what is possible with this workflow and some inside information on the challenges and adjustments that can be made to enhance the quality of the outputs.

Pattern:

This can be a challenging task for AI image generation because, with patterns, we need to follow the style reference very closely; small changes in the pattern can lead to a different look. Humans are good at detecting if a pattern is matching. Even if we can see directly where the changes are, something is off. This would be bad for our use cases. So, the pattern needs to be followed very strictly. To boost performance, use the FluxFill model with weight_dtype: fp8_e4m3fn. It also helps if the style-referenced image is high resolution so many details can be detected and preserved.

Fonts/Text:

It was a massive problem for AI image generation to generate text and apply certain font styles to images for a long time. Fashion pieces use text, so for our use cases, it is possible to also inpaint with text and the correct font. This is possible with the Flux.1 Fill model. However, this can still be challenging, and some text is not fully applied at the first attempt. This can be fixed with more capable models in the future.

Body Types:

A Big challenge is different body types, which can be very tricky for the replacement. If you need to apply the correct body type, it can be helpful to adjust the mask sizes of your Person A image, but this can lead to a drop in the quality of the replacement itself. Here, it is best to adjust this to your needs. Or have base images with similar body type features.

Pose/Position:

We aim to preserve as much of the original image as possible and keep the overall body position intact. This is very important for many use cases, e.g., replacing clothes for a campaign shooting. The crop-and-stitch approach used in this workflow further enhances the fact that we only change the relevant aspects of the image and try to avoid or minimize other changes to the original image.

Conclusion

FluxRedux + FluxFill are powerful models/tools that allow us to get impressive results using this ComfyUI workflow. These results can be used for the described use cases, boosting productivity and creativity. There are a lot of fine-tuning capabilities with the mask sizes and providing decent input images. Even more capable models will fix some problems/challenges; human adjustments to the workflow can correct others.

If you have any questions or even an interesting project, feel free to ping us https://9elements.com/contact/

For the past three years, 9elements has worked collaboratively with UK agency Kore, helping the World Health Organization (WHO) to visualize global public health data on data.who.int, WHO’s central data hub.

Open health data plays a key role for guiding public health decisions and informing the general public. The data needs to be presented and visualized well so health professionals, scientists, policymakers and citizens can access, understand and use it.

The WHO Data portal, launched in May 2023, is a joint effort of public health professionals, data scientists, visualization experts, UX designers, accessibility specialists and software developers.

Under the project leadership of Kore, 9elements developed a central piece of data.who.int, the tailor-made data visualization framework. This work needs to meet the highest quality standards. The WHO principles require the content to be accessible to everyone, everywhere in the world.

Data Design Language

Our work is based on the WHO Data Design Language, produced by Kore and conceived by a diverse team led by the world-renowned data visualization designer Moritz Stefaner. The core design values are:

• Clear: Data presentations are tailored to information needs, understandable and approachable.

• Open: Data presentations are accessible by following the Web Content Accessibility Guidelines (WCAG). They are international, available in the six official WHO languages.

• Robust: Using solid and lean technologies, the data is presented in a variety of channels and sizes. The presentations have alternative access modes, adapt to user preferences and reading situations.

• Transparent: The data presentations reveal uncertainty, precision, provenance and coverage of the data.

Comparing countries on powerful dashboards

The data visualization framework features several pages:

Indicator pages like Healthy life expectancy at birth (HALE) allow to compare countries and regions and break down the data by sex and age group, for example.

Country pages like the Democratic Republic of the Congo give an overview of the public health in one country by showing key health statistics like demographics, life expectancy and causes of death.

The COVID-19 dashboard tracks cases, deaths, vaccinations and current variant circulation.

The Triple Billion progress dashboard tracks the world’s progress towards improving the lives of people through access to universal health coverage, protection from health emergencies and through better health and well-being.

Tailor-made data visualization framework

While there are several mature data visualization kits, there was no solution that met WHO’s strict requirements and integrated well with their data infrastructure.

We developed a tailor-made, light-weight data visualization framework for public health data on a world scale. To deliver data experiences with the highest quality, it implements the clear vision and guidelines of the Data Design Language.

The visualization framework is both extensible and flexible. It produces interactive data experiences in the browser, but also text, static HTML and SVG and pixel-based images. The goal is to serve the most robust data presentation suitable for the medium, the reading context and the client capabilities.

The framework connects the WHO databases to combine data with metadata, translations, and formatting rules to present the data in a meaningful way and to put it into perspective.

Individual facts, tables and charts are combined into interactive widgets to make complex datasets accessible. These widgets compare the data across time, geographical and political entities, like countries and regions, as well as categories, like sex and age group.

We built a chart creator where WHO content editors can explore the data and be guided to help them build a suitable visualization. For data curators, we have built a powerful data explorer for more than 2,000 indicators.

Under the hood, the visualization framework is implemented in portable JavaScript code that runs in the browser, in Node.js, and in cloud functions. The individual charts and widgets are implemented in D3 and Svelte, a well-established duo that produces lean HTML, CSS, SVG and fast JavaScript.

Building your next data visualizations

At 9elements, we have been visualizing data for our clients for more than 10 years. In 2013, we developed GED VIZ for the Bertelsmann Foundation, visualizing global economic relations. From 2014 on, we developed the front-end and the chart rendering of the OECD Data Portal. In 2015, we contributed to the World Economic Forum Inclusive Growth Report.

Let us discuss how we can help you to explore, present and visualize the data of your organization or business! Contact us.

Credits

This project was developed for Kore, an agency for positive public impact through strategy, design and information architecture, based in the UK. With thanks to their wider team, below, on this work for WHO.

Data visualization, product design, UX/UI, cartography and accessibility: Alice Thudt, Christian Laesser, Fred Wheeler, Maarten Lambrechts, Matt Hollidge, Moritz Stefaner, Philippe Rivière, Sarah Fossheim and Yaseed Chaumoo.

Thanks to all WHO staff involved with the project.

Learn more

• WHO Data – data.who.int

• WHO Data Design Language documentation

• Overview on the WHO Data Design Language

Fine Tuning Flux.1-dev Model

Let’s take a look at how to fine-tune a Flux.1-dev model. Flux.1 is an impressive image generation model from https://blackforestlabs.ai/ that has instantly become one of the most prominent image generation models. One significant benefit is that it can be fine-tuned with just a few images.

This post will provide a basic guide to the steps needed to fine-tune a model. But first, let me introduce you to Oscar.

Dataset preparation

For fine-tuning, you need a set of images to use for training. In this case, we aim to leverage Flux's powerful generalization capabilities and train it to generate images of Oscar in various styles and scenarios. For our training, we used 11 images of Oscar. You can view the dataset here.

We are training a LoRa for character creation, so we select images for our dataset based on the following guidelines:

1. Consistent Iconic Features:
   Ensure all images include consistent, iconic features. In our case, this could be Oscar's "color" or "raccoon look." In other instances, iconic features might include eye color, clothing, hairstyle, or hair color.

2. Variants in Poses and Expressions:
   Include images with diverse poses and expressions, camera angles, and body positions. You may incorporate various art styles, but do not alter the iconic features defined in point one.

3. Maximize Output with Few Images:
   Before adding an image to your dataset, consider its unique benefits. Does it add variety or enhance the dataset while preserving the iconic features?

4. Avoid Group Shots:
   Exclude images where the character is part of a group with other characters, as these can dilute the focus on the main subject.

5. General Image Quality:
   Use high-quality images. Prefer lossless formats like PNGs and avoid pixelated or blurry images. Higher-resolution images are preferred as they allow for more accurate feature extraction.

The next step in preparing the dataset is to create labels or captions for the images. For the training process, we require the following structure:

plaintext-- images
---- image_1.png
---- image_1.txt
---- image_2.png
---- image_2.txt
....

The quality of labels can significantly influence the results of fine-tuning. However, labeling each image manually can be a tedious task. In our case, we used florence2 to generate the initial labels for our dataset. In our case, we just used the generated captions since the results were decent enough, but here are some key points to consider when reviewing and refining your labels:

• Describe Iconic Features: Ensure that the iconic features relevant to your fine-tuning are clearly described in the label.

• Remove Unnecessary Details: Omit irrelevant details about the background or style that do not contribute to the fine-tuning objective. This helps focus the process on the relevant task.

• General Style Description: Include a description of the general style of the image in one place, such as "photorealistic", "anime", or "cartoon style".

• Camera Angle and Position: Describe the camera angle and character position to provide more context for the model.

• Trigger Word Usage: If you use a trigger word for your LoRa, ensure it is present in the label. For example, in our case, the iconic features are always linked to the trigger word OSCAR_OSFC in the labels.

Model training

We used one L40S GPU on RunPod.io to generate our labels and execute the training process. You can find more information about running a RunPod instance here.

The training process was conducted using the ai-toolkit. Before running a training session with the AI Toolkit, install all necessary dependencies.

bashgit clone https://github.com/ostris/ai-toolkit.git
cd ai-toolkit
git submodule update --init --recursive
python3 -m venv venv
source venv/bin/activate
pip3 install torch
pip3 install -r requirements.txt

To access the Flux.1-dev model, you need to log in to Hugging Face. This can be done using the huggingface-cli.

Follow these steps:

1. Create a token in your Hugging Face account settings.

2. Use the token to log in to Hugging Face by running the following command in your terminal:

huggingface-cli login

Navigate to the /config/example directory and open the train_lora_flux_24gb.yaml file.

Adjust this configuration file to suit your specific needs. Below is the configuration used to train the OSCAR_OSFC LoRa:

yaml---
job: extension
config:
  # this name will be the folder and filename name
  name: "oscar_new_v1"
  process:
    - type: 'sd_trainer'
      # root folder to save training sessions/samples/weights
      training_folder: "output"
      # uncomment to see performance stats in the terminal every N steps
      performance_log_every: 1000
      device: cuda:0
      # if a trigger word is specified, it will be added to captions of training data if it does not already exist
      # alternatively, in your captions you can add [trigger] and it will be replaced with the trigger word
      trigger_word: "OSC4R_0SFC"
      network:
        type: "lora"
        linear: 32
        linear_alpha: 32
      save:
        dtype: float16 # precision to save
        save_every: 250 # save every this many steps
        max_step_saves_to_keep: 12 # how many intermittent saves to keep
        push_to_hub: false #change this to True to push your trained model to Hugging Face.
        # You can either set up a HF_TOKEN env variable or you'll be prompted to log-in         
#       hf_repo_id: your-username/your-model-slug
#       hf_private: true #whether the repo is private or public
      datasets:
        # datasets are a folder of images. captions need to be txt files with the same name as the image
        # for instance image2.jpg and image2.txt. Only jpg, jpeg, and png are supported currently
        # images will automatically be resized and bucketed into the resolution specified
        # on windows, escape back slashes with another backslash so
        # "C:\\path\\to\\images\\folder"
        - folder_path: "./images"
          caption_ext: "txt"
          caption_dropout_rate: 0.02  # will drop out the caption 5% of time
          shuffle_tokens: false  # shuffle caption order, split by commas
          cache_latents_to_disk: true  # leave this true unless you know what you're doing
          resolution: [ 512, 768, 1024 ]  # flux enjoys multiple resolutions
      train:
        batch_size: 1
        steps: 3000  # total number of steps to train 500 - 4000 is a good range
        gradient_accumulation_steps: 1
        train_unet: true
        train_text_encoder: false  # probably won't work with flux
        gradient_checkpointing: true  # need the on unless you have a ton of vram
        noise_scheduler: "flowmatch" # for training only
        optimizer: "adamw8bit"
        lr: 1e-4
        # uncomment this to skip the pre training sample
#        skip_first_sample: true
        # uncomment to completely disable sampling
#        disable_sampling: true
        # uncomment to use new vell curved weighting. Experimental but may produce better results
#        linear_timesteps: true

        # ema will smooth out learning, but could slow it down. Recommended to leave on.
        ema_config:
          use_ema: true
          ema_decay: 0.99

        # will probably need this if gpu supports it for flux, other dtypes may not work correctly
        dtype: bf16
      model:
        # huggingface model name or path
        name_or_path: "black-forest-labs/FLUX.1-dev"
        is_flux: true
        quantize: true  # run 8bit mixed precision
#        low_vram: true  # uncomment this if the GPU is connected to your monitors. It will use less vram to quantize, but is slower.
      sample:
        sampler: "flowmatch" # must match train.noise_scheduler
        sample_every: 250 # sample every this many steps
        width: 1024
        height: 1024
        prompts:
          # you can add [trigger] to the prompts here and it will be replaced with the trigger word
#          - "[trigger] holding a sign that says 'I LOVE PROMPTS!'"\
          - "[trigger] holding a sign that says 'I LOVE PROMPTS!'"
          - "[trigger] playing chess at the park, bomb going off in the background"
          - "[trigger] holding a coffee cup, in a beanie, sitting at a cafe"
          - "[trigger] building a log cabin in the snow covered mountains"
          - "[trigger] playing the guitar, on stage, singing a song, laser lights, punk rocker"
          - "hipster [trigger] with a beard, building a chair, in a wood shop"
          - "[trigger], in a post apocalyptic world, with a shotgun, in a leather jacket, in a desert, with a motorcycle"
          - "[trigger] scateboarding across a buisy city"
          - "[trigger] as a wizzard in an dungeons and dragons setting"
        neg: ""  # not used on flux
        seed: 42
        walk_seed: true
        guidance_scale: 4
        sample_steps: 20
# you can add any additional meta info here. [name] is replaced with config name at top
meta:
  name: "[name]"
  version: '1.0'

Here are the options we adjusted compared to the default example file:

1. trigger_word: OSCAR_OSFC
   This helps the Flux.1 model retains its generalization capabilities while allowing us to trigger the LoRa specifically for the desired aspects when creating new OSCAR images.

2. folder_path
   Adjust this to the path where your images and labels are stored.

3. steps: 3000
   We increased the training steps to 3000 to create a slightly overfitted model, which works well for our specific case.

4. prompts
   Customize these to fit your needs. It's a good practice to include a variety of scenarios with different camera angles and styles in the prompts.

Running the training process

Once the configuration is finalized, run the training process with the following command:

python3 run.py config/examples/train_lora_flux_24gb.yaml

The full training process on 1 L40S GPU took around 1:45min. We specified that the model should save progress every 250 steps. These saved models can be accessed and used in a ComfyUI workflow to test and apply your trained model.

Examples of the training process

The following examples were generated using the same configuration:

• Left Image: Generated using the LoRa after 250 steps.

• Middle Image: Generated after 1500 steps.

• Right Image: Generated with the fully trained model.

As you can see, our training resulted in a fine-tuned model capable of generating Oscar images across different scenarios and artistic styles. Additionally, the slightly overfitted model ensures a consistent generation of Oscar images, as demonstrated with the wizard example.

Conclusion and Use Cases

It’s truly remarkable that with just a few images, we can train a model tailored to our specific task while retaining the powerful abilities of the base model to generate stunning images. Even with a limited or suboptimal dataset, achieving good results is possible. Moreover, there are plenty of opportunities to refine the configurations or labels to improve the fine-tuning process further.

Fine-tuning a LoRa is an excellent way to bring consistency to AI-generated images. Here are some additional use cases where fine-tuning shines:

1. Fashion Product Photos
   LoRas trained on your products, such as a new T-shirt collection, can generate images of AI-generated models wearing your products. This eliminates the need to hire models, book a studio, or secure outdoor locations for photoshoots, saving both time and resources.

2. Guided Creativity
   Fine-tuning enables the use of AI’s powerful image-generation capabilities to visualize ideas while maintaining consistent styles or characteristics. This guided creativity ensures that all images adhere to a cohesive artistic vision or thematic identity.

3. Safety
   Fine-tuned models offer stricter guidelines and are less likely to generate hallucinations or outputs that stray from the desired purpose. This ensures that the generated content is accurate and aligned with approved use cases, reducing the risk of harmful or inappropriate outputs. This makes fine-tuned models safer and more reliable for sensitive or professional applications.

If you have any questions or even an interesting project, feel free to ping us https://9elements.com/contact/

For an internal project called ‘ControlManiac’, which helps us with all the boring numbers juggling in our software studio, I had to add a global filter to the navigation bar, which lets me select the division I want to filter the data for on whatever view I am currently on.

Since we often switch views while using this app, it should keep its selection. It should also keep it upon reloading or revisiting the page and upon deployment of a new version.

This is what it looks like; it is the toggle element on the right:

The Home, Projects, and Costs navigation options are all separate live views, within the same live_session.

I thought there would be a quite easy solution, but it turned out that all common approaches had their downsides:

• Keeping it in the assigns only is not an option, since this is cleared upon changing the current live view.

• Storing it in a session cookie would ensure it survives a full page reload but brings two problems: First, the session information is not reloaded upon changing the live views within the same live_session, because this is done entirely via web sockets. Thus, we would get the initial value the session head at the time of the first HTTP request. This brings me to the second problem: I neither can update the session since there is no HTTP request upon live navigation, and the session cookie cannot be updated via web socket.

• Storing it in an ETS table solves the problem of switching live views within the same live_session, but of course, it would not survive an app restart, as is the case upon deployment of a new version.

I could, of course, pursue the session approach and just make sure to add a parameter every time I switch between the main live views, but this does feel very odd.

I eventually decided to go with a classic cookie-based session combined with an additional layer stored in an ETS table.

Preparing the pipeline

To be able to load data via plug and on_mount, we first implement a module ControlManiacWeb.DivisionSelector and added it to the pipline:

elixirdefmodule ControlManiacWeb.Router do
  use ControlManiacWeb, :router

  pipeline :browser do
    ...
    plug :fetch_current_division
  end

  ...

  scope "/", ControlManiacWeb do
    pipe_through [...]

    live_session :require_authenticated_user,
      on_mount: [
        ...,
        {ControlManiacWeb.DivisionSelector, :mount_current_division}
      ] do
      ...
    end
  end

  ...

The basic idea for these two functions is as follows:

• fetch_current_division pug is added to the pipeline. Its task is to fetch the current division from cache or session and add it to conn.assigns.

• mount_current_division is used for the live views. Its task is to load the current division from cache and add it to socket.assigns.

Storing settings in the ETS table

Before we come to the implementation of these functions, lets have a look at how we implement the cache using an ETS table.

elixirdefmodule ControlManiac.SettingsCache do
  use GenServer

  @name __MODULE__
  @tab :settings_cache
  @ttl 60 * 60 * 24 * 31

  # Client

  def start_link(_), do: GenServer.start_link(__MODULE__, [], name: @name)

  def insert(key, value) do
    expiration = :os.system_time(:seconds) + @ttl
    :ets.insert(@tab, {key, value, expiration})
  end

  def get(key, default \\ nil) do
    lookup =
      case :ets.lookup(@tab, key) do
        [{_, value, _} | _] -> value
        [] -> default
      end

    lookup
  end

  # Server

  def init(_) do
    :ets.new(
      @tab,
      [:set, :named_table, :public, read_concurrency: true, write_concurrency: true]
    )

    {:ok, []}
  end

end

What this module does is quite simple. It gets up the table within a GenServer, to ensure it exists when the application starts. It then offers functions to insert and get entries, which directly write into the ETS table to avoid the GenServer becoming a bottleneck.

Retrieving data

We now can use our SettingsCache together with normal sessions to implement the desired behavior for the pipeline. Please note that this implementation assumes we have a user present for whom we can store the data, and that we want to store the data on a per-user basis.

elixirdefmodule ControlManiacWeb.DivisionSelector do
  import Plug.Conn

  alias ControlManiac.Accounts.User
  alias ControlManiac.SettingsCache

  @default_division -1
  @session_division_key "current_division_id"
  @cache_division_key :division_cache

  def fetch_current_division(conn, _opts) do
    current_user = conn.assigns[:current_user]

    assign(
      conn,
      :current_division,
      get_division_from_cache(current_user) ||
        get_division_from_session(conn, current_user) ||
        @default_division
    )
  end

  defp get_division_from_cache(nil), do: nil

  defp get_division_from_cache(%User{id: user_id}) do
    SettingsCache.get({@cache_division_key, user_id}, nil)
  end

  defp get_division_from_session(conn) do
    get_session(conn, @session_division_key)
  end

  ...

end

This code first attempts to get the current division from the cache. If this is not successful, it tries to get it from the session, reverting to a default value (which is hard-coded to -1 / ‘All’) if this fails, too.

Handling data for live views

So here comes the implementation of the on_mount/4 function used for live views:

elixirdef on_mount(:mount_current_division, _params, session, socket) do
  division_id =
    get_division_from_cache(socket.assigns[:current_user]) ||
      get_division_from_session_map(session) ||
      @default_division

  {
    :cont,
    Phoenix.Component.assign(
      socket,
      :current_division,
      division_id
    )
  }
end

As you might notice, there again is a fallback to the session map with get_division_from_session_map/1, although I mentioned at the beginning that this will include old values from the first http request, and not the value the user selected last. So why do we do that?

We do this to handle a specific edge case: In case the server restarts and the socket connection is initiated again, we lost our cache. Upon reconnecting the socket there is no run through the plug pipeline (so, no fetch_current_division/2 invoked), but since reconnection is done via XHR request, we get updated session info. This is especially useful after deployments: When the system reconnects, we get the correct value from the session.

We need the fallback to session info anyway because of this edge case, so we can also use it as a fallback for the first requests, where the ETS table cache might not be warm although a division is set in the session. This way, we don’t have to make sure the cache is up to date in fetch_current_division/2: We use the value from the initial session, and as soon as the user changes the division, the ETS table cache entry is created and the fallback will never be reached.

Since in on_mount/4 we get the information stored in the session as a map, the function to retrieve it is slightly different:

elixirdefp get_division_from_session_map(session) do
  if division_id = Map.get(session, @session_division_key) do
    String.to_integer(division_id)
  else
    nil
  end
end

Storing data in session despite using sockets

So how do we make sure to store the most recent value in the session, given that by navigating only within a single live_session, we won’t have HTTP requests?

We do so by triggering a JS hook, which then performs an XHR request to store the actual data, an idea I copied from FullstackPhoenix.

We first implement a controller with the sole purpose of adding data to the session, given it is within a list of allowed keys:

elixirdefmodule ControlManiacWeb.StoreSessionController do
  use ControlManiacWeb, :controller

  @allowed_keys ~w(current_division_id)

  def create(conn, params) do
    updated_conn =
      Enum.reduce(params, conn, fn {key, value}, acc_conn ->
        if key in @allowed_keys do
          put_session(acc_conn, String.to_atom(key), value)
        else
          acc_conn
        end
      end)

    send_resp(updated_conn, 200, "")
  end
end

In order to have something we can hook the JS to we add a div to the root layout:

html<div id="store-session" phx-hook="StoreSession"></div>

We can now attach a JS handler to the phx-hook in order to trigger the actual request.

javascript// store_session.js

export const StoreSession = {
  mounted() {
    const token = document.querySelector('meta[name="csrf-token"]').getAttribute('content')

    this.handleEvent('store_session', data => {
      fetch('/store_session', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'X-CSRF-Token': token
        },
        body: JSON.stringify(data)
      })
    })
  }
}

// And don't forger to register the hook in app.js

import { StoreSession } from "./store_session"

let liveSocket = new LiveSocket("/live", Socket, {
  ...
  hooks: {
    StoreSession: StoreSession
  }
})

At last, we have to make sure this hook is triggered upon selecting another division. For this, we have to do two things: Tell the DivisionSelector module we implemented in the beginning to store the updated version to the cache, and triggering the hook to store it in the session.

elixirdef handle_event("switch-division", %{"id" => id}, socket) do
	DivisionSelector.update_division(
    socket.assigns.current_user,
    String.to_integer(id)
  )

  socket =
    socket
    |> push_event("store_session", DivisionSelector.map_for_session(id))
    |> assign(:current_division, id)

  {:noreply, socket}
end

To make it work, we add the following two functions to the DivisionSelector:

elixirdef update_division(%User{id: user_id} = user, division_id) do
  SettingsCache.insert({@cache_division_key, user_id}, division_id)
end

def map_for_session(id) do
  %{@session_division_key => id}
end

Note we added map_for_session/1 for consistency, to ensure the keys used to store data in cache and session are defined in one single place.

In the real application, we also used Phoenix.PubSub to notify current views about the division change, which might be a topic for another blog post some time.

Conclusion

Given that this is what I thought to be a very simple problem, I was quite surprised that I was not able to find an easier solution. But, on the other hand, handling global state is hard, no matter the framework, and at least within Elixir Phoenix, we are able to implement a solution in a very explicit and controllable way.

We've all been there: you're working on a website and need to display a row of logos—clients, partners, sponsors—you name it. However, logos come in all shapes and sizes, and making them look good together can be quite challenging. How do you get them to play nice and look visually appealing without spending hours tweaking each one?

This challenge becomes even trickier when you don't know in advance which logos will be in your logo row.

The Common Approach: Same Height for All

The most straightforward solution is to set all logos to the same height. Let's see how that looks.

See the Pen Logo Strip | 01 by Nils Binder (@enbee81) on CodePen.

Note: All the logos are wrapped in an additional div (.logo). This is because we want a plain block element without intrinsic sizing, which images with width and height attributes typically have. By setting the height on the outer div and giving the image a height of 100% and width of auto, we ensure consistent sizing across all logos.

While this approach is easy to implement, it doesn't always yield a visually balanced logo strip. Wide logos might appear too big.

Adjust Heights Based on Aspect Ratio

To create a more harmonious logo strip, we can adjust the height of each logo based on its aspect ratio. If a logo is significantly wider than it is tall (aspect ratio greater than one), we'll reduce its height accordingly. The wider the logo, the smaller its allowed height. This way, all logos maintain a visual balance.

To achieve this, we need to know the original width and height of each logo. This shouldn't be a problem when working with a CMS, as you can usually access the image dimensions easily. And if you're creating the images yourself, it's even easier.

Setting Up the HTML:

On each of the .logo divs, we'll set two custom properties: --width and --height. Be sure to use unitless values here because in CSS calculations, you cannot remove a unit once it's there. 🙂

html<div class="logo-row">
  <div class="logo" style="--width:49; --height: 48;">
    <img src="first-logo.svg" alt="" />
  </div>
  <div class="logo" style="--width:228; --height: 48;">
    <img src="second-logo.svg" alt="" />
  </div>
  ...more logos...
</div>

The CSS Magic:

css.logo-row {
  --base-height: 3rem;
  --scale-factor-horizontal: 0.1;

  display: flex;
  justify-content: center;
  align-items: center;
  flex-wrap: wrap;
  gap: 3rem 2rem;
}

.logo {
  --base-ratio: calc(var(--width) / var(--height));
  aspect-ratio: var(--base-ratio);

  --factor-horizontal: min(
    var(--scale-factor-horizontal) * -1 * var(--base-ratio) + var(--scale-factor-horizontal) + 1,
    1
  );
  height: max(
    var(--base-height) / 2,
    var(--base-height) * var(--factor-horizontal)
  );

  & img {
    width: 100%;
    height: auto;
  }
}

There is a lot going on here. Let's break it down step by step.

1. Setting Base Variables

We set --base-height and --scale-factor-horizontal on the .logo-row. These variables will be used in our calculations for each logo.

css.logo-row {
  --base-height: 3rem;
  --scale-factor-horizontal: 0.1;
  /* ... */
}

2. Calculating Aspect Ratio

For each .logo, we calculate its aspect ratio and store it in --base-ratio.

css.logo {
  --base-ratio: calc(var(--width) / var(--height));
  aspect-ratio: var(--base-ratio);
  /* ... */
}

3. Calculating the Scaling Factor

Now comes the juicy part! We need to calculate a scaling factor that adjusts the height of each logo based on its aspect ratio. This ensures that wider logos are scaled down proportionally, maintaining visual harmony across the logo strip.

The Formula
We use linear interpolation to compute the scaling factor, which returns 1 when the aspect ratio is 1. Here's the formula:

css--factor-horizontal: min(
  (-1 * var(--scale-factor-horizontal) * var(--base-ratio)) + 
  var(--scale-factor-horizontal) + 1,
  1
);

Since we know the value of the --scale-factor-horizontal variable, we can insert it here to simplify:

css--factor-horizontal: min(
  (-0.1 * var(--base-ratio)) + 1.1,
  1
);

Let's Plug in Some Numbers

• Aspect Ratio of 1: (-0.1 * 1) + 1.1 = 1

• Aspect Ratio of 2: (-0.1 * 2) + 1.1 = 0.9

• Aspect Ratio of 3: (-0.1 * 3) + 1.1 = 0.8

This means a logo with an aspect ratio of 2:1 will have 90% the height of a logo with a 1:1 aspect ratio.

4. Adjusting the Height

We set the height of each logo by multiplying the base height by the calculated scaling factor. To ensure that even the widest logos don't become too small, we also use a max function to prevent the height from dropping below half of the base height.

cssheight: max(
  var(--base-height) / 2,
  var(--base-height) * var(--factor-horizontal)
);

See the Pen Logo Strip | 02 by Nils Binder (@enbee81) on CodePen.

Adding Adjustments for Portrait Logos

So far, we've been dealing with logos that are square or wider—that is, logos with an aspect ratio of 1 or greater. But occasionally, you might need to include a portrait-oriented logo that's taller than it is wide, with an aspect ratio less than 1.

The challenge with portrait logos is that their aspect ratios are between 0 and 1 while landscape logos have a ratio that is between 1 and infinity. Therefore the interpolation method we used for wider logos doesn't make a noticeable difference here, and scaling them using the same factor doesn't adjust their size enough.

To solve this, we'll introduce a new resize factor specifically for portrait logos, called --factor-vertical.

For the final height-calculation, instead of using a max function to prevent logos from becoming too small, we'll switch to a clamp function. This lets us set both minimum and maximum height limits, ensuring that portrait logos don't become too big and overshadow the rest.

Here's the final version of our logo adjustment technique. You can experiment with the two scaling values and adjust the base height of the logos to see how it affects the overall appearance of the logo strip. If you have ideas on how to further refine this solution or insights you'd like to share, I'd love to hear from you! Feel free to reach out and share your suggestions.

See the Pen Logo Strip | final by Nils Binder (@enbee81) on CodePen.

UPDATE Sept. 25: Scaling Logos to Equal Area

I want to give a big shoutout to Roman for suggesting an alternative solution to our logo sizing challenge. His method aims to scale all logos so that they occupy the same area as a square logo would. In other words, the product of the width and height (the area) of each logo becomes consistent across all logos.

How it Works

• Calculating the Square's Area: First, we take the base height and multiply it by itself to get the area of a square logo with that height.

• Finding the Scale Factor: Next, we determine the scale factor by dividing the square's area by the area of each logo (its width multiplied by its height). Since we're scaling both dimensions uniformly, we take the square root of that ratio.

• Scaling the Logo: We then multiply the height of each logo by this scale factor. This adjusts the logo's size so that it occupies the same area as the square logo.

The beauty of this approach is that it works seamlessly for both portrait and landscape logos, ensuring a consistent visual presence across the logo strip.

Adjusting the Visual Impact

While Roman's solution is mathematically sound, I found that making all logos occupy exactly the same area can sometimes lead to visual inconsistencies, especially when logos have extreme aspect ratios. To address this, I introduced a strength variable that allows us to control how strongly the scaling is applied.

• Calculating the Scaled Height: First, we compute the scaled height using the scale factor from Roman's method.

• Applying the Strength Factor: Then, we interpolate between the base height and the scaled height using the strength variable. When the strength is set to 1, we get the full effect of the scaling. When it's set to 0, all logos use the base height without any scaling. Values in between allow for fine-tuning the visual balance to suit your specific needs.

A Caveat: Dealing with Units in CSS Calculations

One challenge with this method is that to calculate the scale factor accurately, we need unitless values. However, CSS variables often carry units, which can complicate calculations that require pure numbers.

To overcome this, we need to convert the base height into a unitless pixel value. This can be achieved using a clever CSS trick involving trigonometric functions, specifically atan2 and tan. By using these functions, we can isolate the numerical value from the unit, allowing us to perform the necessary calculations without units interfering.

If you want to learn more about this, you can have a look at my blog post "Speed vs. Duration – A Use Case for Mixed Unit Division" where I delve into this topic in more detail.

See the Pen Logo Strip | refork by Nils Binder (@enbee81) on CodePen.

Logos Used in This Post

In case you're curious about the logos featured in our examples, here's where they come from:

• 9elements Logo – 9elements.com

• img.ly Logo – img.ly

• CSS Café Meetup Logo – css.cafe

• Das Ruhrgebiet Magazin Logo – dasruhrgebiet.de

• 9elements Suggestion for CSS 4/5 Logo - new CSS logo discussion

• Beloved CSS Owl Selector Logo

• 10xD Logo – 10xd.de

By the way, if you’re on the hunt for a shiny new logo, all of these were crafted by the talented team at 9elements. Just saying! 😉

Building flexible data visualizations for international sites

For our international clients, we have created dynamic charts and data visualizations for the web. Charts typically render shapes like lines and paths, rectangles and circles. They contain text for titles, axis labels, numerical values and legends.

SVG is the good fit for this purpose. It embeds directly into HTML and pairs well with CSS. However, for dynamic data visualizations on the web, SVG poses a challenge.

Responsive charts and the problems of SVG

The websites we build feature responsive layouts and fluid typography. We employ CSS Flexbox and Grid together with media and container queries to fit in the content. There is not one single fixed presentation, but many possible presentations depending on the content and the reading environment.

In contrast, SVG does not have layout techniques like Flexbox, Grid or even Normal Flow. In SVG, all shapes are absolutely positioned. Text does not wrap automatically. The shapes and text need to be laid out manually by the code that generates the SVG.

SVG does scale continuously, as the name says – but for charts on the web, we usually do not want that. A small chart should not look like a downscaled big chart. Text would become unreadable, shapes would become tiny pixel mush – even with techniques that prevent the scaling of some graphical features.

For charts on the web, we want quantitative and qualitative responsive scaling. A small and a large chart should be designed and laid out differently. A small chart should focus on clear, distinguishable marks that represent the data. A large chart should take advantage of the screen estate to show more items and details as well as provide context.

For example, a line chart with multiple lines may switch to small multiples on smaller viewports or containers.

We have typically implemented this responsiveness with client-side JavaScript logic. JavaScript is able to read the container size and measure text in order to compute all shape coordinates and sizes. This often involves decollision with force simulations.

This approach has severe disadvantages. The cycle of forcing the browser to compute the style, reading sizes and setting positions leads to layout thrashing and slows down the chart rendering.

When the container size changes, for example due to a browser resize or orientation change, the JavaScript needs to compute all SVG positions and sizes from scratch. Assuming this takes 50-100ms per chart, a page with 20 charts freezes the browser for 1-2 seconds.

HTML, CSS and SVG hybrid

Horizontal bar charts are simple yet effective, intuitive and accessible visualizations. They are versatile regarding the bar design, labeling, value placement and axes. And they can be highly flexible regarding the container size.

We have a pretty solid implementation of a responsive bar chart. In narrow containers, the row label is shown on top of the bar. In wide containers, it is shown next to the bar.

This chart is a hybrid of HTML, CSS and SVG. We wanted to use essential CSS layout methods like Flexbox instead of re-implementing layout algorithms in JavaScript. However, the synchronization with the SVG parts is still slow, complex client-side JavaScript code.

Bar chart in plain HTML & CSS

We were wondering: Can we achieve this with HTML and CSS alone, preferably without SVG and with less JavaScript logic? We fiddled around, but never finished this idea.

Then we saw the beautiful responsive bar charts of State of JS, made with HTML & CSS only. On narrow viewports, they use a two-column grid:

On wide viewports, this is a three-column grid with subgrids that inherit the column setup:

These well-made charts encouraged us to try to migrate our bar charts to HTML & CSS.

Grid setup

For a start, we rebuild the basic structure:

See the Pen Responsive bar chart with ticks by molily (@molily) on CodePen.

In the narrow version, the each row (li element) is a two-column grid:

cssdisplay: grid;
grid-template-columns: minmax(0, 1fr) min-content;
grid-template-areas:
  "dimension value"
  "bar bar";
position: relative;

In the wide version, the wrapper (ol element) is a three-column grid:

csscssdisplay: grid;
grid-template-areas: "dimension bar value";
grid-template-columns: fit-content(10rem) 1fr min-content;

The row (li) is a subgrid that spans all columns:

csscssdisplay: grid;
grid-template-columns: subgrid;
grid-template-areas: none;
grid-column: 1 / -1;

Real-world requirements

Our real bar chart, however, is much more complex and has the following requirements:

• Internationalization with bidirectional text: We're building charts for sites in six languages and two text directions: Left-to-right (LTR, like English and Russian) and right-to-left (RTL, like Arabic and Hebrew).

• Positive and negative values. Bars grow to both sides.

• Row labels may have an arbitrary length and should wrap and align nicely.

• Value labels should be positioned at the end of the bars, not inside them or in a separate column.

• Do not repeat the axis tick lines for each row if it's avoidable.

This is the solution we came up with:

Responsive bar chart in HTML & CSS

This is version 2 which implements essential feedback from Vesa Piitinen.

See the Pen Bar chart V2 by molily (@molily) on CodePen.

Let's dive into the implementation.

Responsive grid setup

The HTML structure looks like this:

html<div class="bar-chart">
  <div class="ticks" aria-hidden="true">
    <div class="tick" style="inset-inline-start: {percent}%">{tick value}</div>
    <!-- … more ticks … -->
  </div>
  <ol>
    <li>
      <div class="dimension">
        <span class="dimension-label">{dimension label}</span>
      </div>
      <div class="bar" style="margin-inline-start: {bar start}%; width: {bar width}%">
        <span class="bar-label">{ bar label }</span>
      </div>
      <div class="value" aria-hidden="true">{ bar label again }</div>
    </li>
    <!-- Bars with negative values require a class is-negative: -->
    <li class="is-negative">
      <div class="dimension">
        <span class="dimension-label">{dimension label}</span>
      </div>
      <!-- And the value need to placed before the bar: -->
      <div class="value" aria-hidden="true">{bar label again}</div>
      <div class="bar" style="margin-inline-start: {bar start}%; width: {bar width}%">
        <span class="bar-label">{bar label}</span>
      </div>
    </li>
    <!-- … more li elements … -->
  </ol>
</div>

In the narrow version, the .bar-chart wrapper is a three-column grid:

cssdisplay: grid;
grid-template-columns: min-content 1fr min-content;
grid-template-areas:
  "dimension dimension dimension"
  "valuePaddingStart bar valuePaddingEnd";

The ol element and li elements are subgrids:

cssdisplay: grid;
grid-column: 1 / -1;
grid-template-columns: subgrid;

In the wide version, the wrapper becomes a four-column grid:

cssgrid-template-areas: "dimension valuePaddingStart bar valuePaddingEnd";
grid-template-columns: fit-content(10rem) min-content 1fr min-content;

Each row remains a subgrid.

Bidirectional text

Internationalization is where HTML and CSS shine compared to SVG.

Our JavaScript code that generates SVG charts is full of if (isLTR) {…} else {…} conditionals. In SVG, the origin of the coordinate system is always top left. X coordinates need to be calculated using those LTR/RTL switches.

In HTML and CSS, we can simply use logical properties like inset-inline-start/-end, margin-inline-start/-end as well as padding-inline-start/-end to solve most left-to-right vs. right-to-left differences. When laying out the boxes in CSS, we can work with the text direction.

For example, each bar is a Flexbox container with the value label nested inside. Then the label is positioned next to the bar: For positive values, we add a box with ::before plus content: '' with a padding-inline-start of 100%. For negative values, we add a box with ::after plus content: '' with a padding-inline-end: 100%. These boxes push the label out of the bar so it sits right next to it.

We still need to handle positive and negative values differently, but by using Flexbox, logical properties and the current text direction, we don't need to handle left-to-right and right-to-left differently.

The bar labels are also rendered into the columns named valuePaddingStart and valuePaddingEnd. These invisible placeholders ensure the columns have the correct width to accommodate the value labels. So the labels appear twice in the DOM. The placeholders have aria-hidden="true" and visibility: hidden though.

Tick lines spanning the full height

Our goal to put the axis tick lines in the DOM only once instead of repeating them for each row complicates the grid. The challenge is to constrain the tick lines in the bar column horizontally, but let them span the whole grid vertically.

This is possible with grid-row: 1 / -1 given the grid has explicit rows. It does not work with an arbitrary number of implicitly-created rows.

So we defined an outer grid that has two fixed rows. The ticks are then positioned in the first row, spanning two rows.

css.ticks {
  grid-column: bar;
  grid-row: 1 / span 2;
}

The list of bars is then positioned in the second row and spans all columns of the parent grid. It creates a subgrid that inherits the grid configuration from the parent grid.

cssol {
  display: grid;
  grid-row: 2;
  grid-column: 1 / -1;
  grid-template-columns: subgrid;
}

The subgrid may then create an arbitrary number of implicit rows. It remains nested in the second row of the outer grid.

Minimal example on CodePen:

See the Pen Grid: Span whole grid by molily (@molily) on CodePen.

Accessibility considerations

Accessibility of data visualizations is a top priority for us and our clients. In our SVG charts and HTML / SVG hybrids, we have assigned ARIA roles and accessible labels so graphical shapes have proper semantics and textual representation. In the accessibility tree, these charts appear either as lists (like ul or ol elements) or tables (like the table element) so users can read and navigate the chart in a familiar way.

While we have made SVG charts accessible, it is simpler and more robust to use semantic HTML directly. The shown HTML and CSS bar chart uses plain ol and li elements with built-in ARIA roles. Screen readers and other assistive tools read out the labels and values.

Edge with JAWS on Windows:

Chrome with VoiceOver on MacOS:

Recap

Today's websites feature responsive layout and fluid typography. Data visualizations should adapt these design techniques.

While responsive and accessible SVGs are possible, they require manual client-side JavaScript logic. HTML and CSS allow us to create charts using declarative layouts and bidirectional positioning without computing positions and preventing overlap manually.

We've demonstrated this for a bar chart. We've also created HTML, CSS and SVG hybrids where each technology does what it is good at.

Building your next data visualizations

At 9elements, we have been visualizing data for our clients for more than 10 years. In 2013, we developed GED VIZ for the Bertelsmann Foundation, visualizing global economic relations. From 2014 on, we developed the front-end and the chart rendering of the OECD Data Portal. In 2015, we contributed to the World Economic Forum Inclusive Growth Report. The bar charts described in this article are part of a long-term work for an international organization in the public health sector.

Let us discuss how we can help you to explore, present and visualize the data of your organization or business! Contact us.

Acknowledgements

Thanks to my colleagues Nils Binder, Julian Laubstein and Matthias von Schmettow for this collaboration.

Thanks to Vesa Piittinen for substantial feedback and many valuable ideas on how to improve and simplify the HTML and CSS. Please have a look at Vesa's version of the bar chart which demonstrates more clever optimizations.

Thanks to the data visualization designers Alice Thudt, Christian Laesser and Moritz Stefaner for their stellar work on the Data Design Language. Thanks to Sarah Fossheim for their accessibility reviews and insights.

Thanks to the Devographics team behind the “State of HTML/CSS/JS“ surveys for the inspiration.

Thanks to our client for the opportunity to work on ambitious data visualizations.

Animating elements in CSS can be challenging, especially when you want to ensure smooth transitions across different screen sizes. One issue that I stumble upon quite often is the inability to divide two length values in CSS. For example, calc(100vw / 1px) isn't valid. While we can adjust length values based on the viewport width, such as using clamp(1rem, 0.818rem + 0.91vw, 1.5rem) for font sizes, this approach doesn't work for other value types like <angle> or <time> because they do not accept length values. If you want to use other unit types that depend on the viewport width, you need to strip away the unit and have it as a <number> type.

Until recently, I thought we needed JavaScript to get the viewport width as a unitless number. But then I watched Matthias Ott's fantastic talk at this year’s CSS Day conference. He mentioned a post by Jane Ori, where she explains a clever technique. She combines atan2() with tan() to get a <number> value for the viewport width. Although atan2 can be buggy, as she explains, it works well enough for a quick demo to show where these values could be useful.

Note: I highly recommend reading Jane's article for a deeper understanding. But for now, don't worry too much about the math involved. The key takeaway is that tan(atan2(y, x)) will give us the value of x/y without any units. We'll use this to obtain the viewport width in pixels but as a type instead of a type. I will get back to this in a minute, and we will go through it step by step.

The duration dilemma

When you animate elements with CSS, you usually set a fixed duration for the animation. This works well if you know the exact distance the element will move. However, if you don't know the distance the speed of the element changes. For example, if you want to animate an element from the left side of the screen to the right, the duration remains constant, but the element's speed varies depending on the viewport width. On small screens, the element moves much slower than on larger screens.

See the Pen duration vs. speed | 01 by Nils Binder (@enbee81) on CodePen.

Duration based on screen-size

Sure, you could adapt the duration using media queries to make it look nicer. But this requires creating many steps, causing the animation to jump from one duration to another. It feels like adaptive design limited to specific screen sizes. It would be much easier if we could tie the duration directly to the viewport width.

Let’s do this with the technique described by Jane Ori:

First, we need to register a property with a <length> syntax. We'll call it --100vw since it will be set to 100vw. This step might seem unnecessary because we're essentially storing the value 100vw in a variable named --100vw. However, it is necessary due to the current implementation quirks when atan2 is used with two different length units.

css@property --100vw {
  syntax: "<length>";
  initial-value: 0px;
  inherits: true;
}

With this in place, we can define three custom properties in the root:

1. set the --100vw to 100vw,

2. use the tangent and arctangent combination to calculate the unitless pixel width

3. calculate the duration based on the px-width

css:root {
  --100vw: 100vw;
  --px-width: tan(atan2(var(--100vw), 1px));
  --duration: calc(var(--px-width) / 1000 * 2s);
}

In this example, --px-width is divided by 1000 and then multiplied by 2s. For a screen width of 1000px, this results in a duration of 2s. On a screen width of 500px, it results in a duration of 1s.

See the Pen duration vs. speed | 02 by Nils Binder (@enbee81) on CodePen.

You can see that instead of defining the duration, we now define the speed of the object. This ensures the animation speed is consistent across screen sizes. On larger screens, it takes a bit longer for the circle to move from one side to the other compared to smaller screens.

Advanced example

Here's another example where the unitless width is used in another animation, and the result is used in a sin() function to create a bird flying in a wavy line across the screen. You can also play with the duration and the wave frequency:

See the Pen Bird with controlled speed by Nils Binder (@enbee81) on CodePen.

Is this a hack? Should I use this in production?

When complex CSS is involved, it's easy to label it as a hack. However, I believe that a true CSS hack implies exploiting quirks or inconsistencies in how different browsers render CSS. The techniques used in this post utilize CSS that functions as intended, so I wouldn't classify them as hacks.

As for using these techniques in production, be aware that registered custom properties are not yet supported in Firefox at the time of writing. Support is expected in the next version (128), so consider using this approach as a progressive enhancement.

I'm also very interested in seeing other use cases, so please tag me or 9elements if you experiment with similar techniques.

In this article, I'm sharing a customizable CSS auto-grid layout. It's a grid component that allows for adjustments in minimum column widths, horizontal and vertical gaps, and even the maximum number of columns to fit various design needs.

Since CSS Grid Layout became supported by all major browsers back in late 2017, it has opened up a lot of possibilities for designing web layouts. However, I've noticed that many new developers still find it challenging to grasp. I totally get it; it took me a while to get the hang of using new concepts like repeat(), minmax(), and fr myself. So, I thought it would be helpful to share a simple auto-grid component that we frequently use in our web development projects at 9elements.

The HTML setup for this is pretty straightforward. We start with a div that has the class auto-grid and attach a list of items to it. Our demo items don't contain any content, so we'll add a background color and a fixed height to them. This way, you'll be able to see them.

See the Pen auto-grid 01 by Nils Binder (@enbee81) on CodePen.

Now let’s start setting up our grid:

css.auto-grid {
	display: grid;
	grid-template-columns: 1fr 1fr 1fr;
	gap: 1rem;
}

By setting display: grid;, we're telling the container to act as a grid. Then, we're defining our columns. With grid-template-columns: 1fr 1fr 1fr;, we're creating three columns of equal width. The fr unit stands for "fraction" and represents a share of the available space. And we add a gap of 1rem, that makes it a lot easier to see the columns.

No one enjoys writing the same code over and over again. Luckily, CSS Grid offers a handy shortcut. Instead of listing 1fr three times, you can use repeat(3, 1fr). This way, if you ever need to change the number of columns, you can simply adjust the number in the repeat function.

Get started with auto-fit

Up until now, our focus was on the number of columns, without considering their width. To specify a fixed width, we can replace the fr unit with a fixed measurement. Furthermore, instead of defining a set number of columns, we can use the auto-fit keyword. This allows us to create as many columns as can fit within the parent container, adjusting dynamically to the available space.

css.auto-grid {
	display: grid;
	grid-template-columns: repeat(auto-fit, 10rem);
	gap: 1rem;
}

See the Pen auto-grid 02 by Nils Binder (@enbee81) on CodePen.

This approach works well, but often, you might not want your grid items fixed at exactly 10rem. Instead, you'd prefer a minimum size of 10rem, with the ability for items to expand if additional space is available. This can be achieved by combining a fixed size with the flexible fr unit inside a minmax() function, which is specifically designed for grid template declarations. We set the minimum width to 10rem and allow the columns to grow with the available space, up to a maximum of 1fr.

css.auto-grid {
	display: grid;
	grid-template-columns: repeat(auto-fit, minmax(10rem, 1fr));
	gap: 1rem;
}

See the Pen auto-grid 03 by Nils Binder (@enbee81) on CodePen.

There's an important aspect we haven't yet covered: what happens if the grid container is smaller than the columns' minimum width. Typically, this would cause an overflow, which is seldom desirable in a layout. To avoid such overflow, we need to ensure that the minimum size specified in the minmax function does not exceed the container's width. This is where the min() function comes in handy. Unlike minmax, min() can be used outside grid template declarations. It compares a set of values and selects the smallest. For instance, min(10rem, 100%) will choose 10rem as long as the container's width exceeds 10rem. When the container's width is less than 10rem, 100% becomes the chosen value, preventing overflow. In our grid setup, this adjustment would look like:

css.auto-grid {
	display: grid;
	grid-template-columns: repeat(auto-fit, minmax(min(10rem, 100%), 1fr));
	gap: 1rem;
}

See the Pen auto-grid 04 by Nils Binder (@enbee81) on CodePen.

Making things customizable with custom properties

To make our auto-grid component more versatile, we'll introduce CSS custom properties (variables). This will allow us to accommodate different use cases more effectively. Since we're about to make the column width declaration more complex (no worries, it'll be clear), our first step is to move the calculation into a variable. This way, we can easily reuse it in our grid template declaration. We'll also create a variable for the minimum column width, with a default value of 10rem. Here's how it looks:

css.auto-grid {
	/* Variable for column width calculation */
	--column-width: min(var(--auto-grid-min-size, 10rem), 100%)

	display: grid;
	/* Apply the calculated column width */
	grid-template-columns: repeat(auto-fit, minmax(var(--column-width), 1fr));
	gap: 1rem;
}

To make our grid's spacing even more customizable, we'll introduce variables for the gap property too. This is where we get to play with some variable fallback magic. The gap property can take two values: one for the vertical gap between rows and another for the horizontal gap between columns. When only one value is given, it applies equally to both vertical and horizontal gaps. We'll define two variables: --auto-grid-gap for the general gap and --auto-grid-gap-vertical for the optional vertical gap. The neat trick here is to use chained fallbacks for the vertical gap, allowing it to default to the general gap value, and then to 1rem if neither is defined. Here’s how we implement this:

css.auto-grid {
	/* Variable for column width calculation */
	--column-width: min(var(--auto-grid-min-size, 10rem), 100%);

	display: grid;
	/* Apply the calculated column width */
	grid-template-columns: repeat(auto-fit, minmax(var(--column-width), 1fr));

	/* gap with fallback */
	gap: 
		var(--auto-grid-gap-vertical, var(--auto-grid-gap, 1rem))
    var(--auto-grid-gap, 1rem);
}

See the Pen auto-grid 05 by Nils Binder (@enbee81) on CodePen.

Our auto-grid component is now versatile enough to accommodate about 90% of layout scenarios. It's designed to maximize the use of available space by dynamically adjusting the number of columns within a single row, ensuring a clean, overflow-free display even on smaller screens or within constrained containers. What makes this particularly satisfying is the ability to easily customize and reuse the component across your project, simply by tweaking the custom properties.

Adding a max-column-count

Sometimes I'm asked, "Can we set a limit to the number of columns? I don't want an endless sea of them." The answer is a resounding yes—we can definitely put a cap on column counts. However, a word of caution: as we venture into this territory, be prepared. The CSS gets a bit trickier from here on out.

Let's say we aim for a max of four columns. With four columns, we're kind of aiming for each to take about 25% of the width. But, there's a catch—the gaps. They take up some space too, so it's more like 25ish %, not exactly to the dot, unless you set your gaps to zero.

Now, what happens if we tweak the numbers? If we go down to three columns, each one gets a bit more room, about 33% of the space. But if we add an extra, making it five, then each column has to slim down to fit, getting around 20% each.

Next, we need a trick to cap our column widths at about 25%. We're okay with column widths stretching beyond that, say to 33%, but anything less, like 20%, just doesn't cut it. That's where the CSS max() function comes into play. We've seen min() in action, picking the smallest value. Well, max() is its counterpart, always grabbing the largest value from what we give it.

Let's tweak our --columns-width variable in our CSS. We're going to envelop it with a max() function.

css--column-width: max(
	25%, /* This is the 25ish %. We'll look at this in a second */
	min(var(--auto-grid-min-size, 10rem), 100%)
);

This way, if there's space for a fifth column, and each column's width might drop to around 20%, our setup ensures that the columns don't shrink beyond 25% because our max() function prioritizes the larger value, keeping our columns nicely sized.

Mind the gap

To address the gap issue, we can't use a flat 25% because it doesn't account for gaps. Instead, we adjust the desired column width by subtracting the gap width from 25%. This approximation isn't perfect—it subtracts the total gap from the width as if there were gaps for all four columns, not just the three in-between. However, this discrepancy doesn't impact the layout due to the minmax function with 1fr, allowing columns to expand into available space.

For clarity, we'll separate the calculation of the maximum column width into its own variable:

css--max-column-width: 100% / 4 - var(--auto-grid-gap, 1rem);

--column-width: max(
	var(--max-column-width),
	min(var(--auto-grid-min-size, 10rem), 100%)
);

And voila, now the auto-grid will never show more than 4 columns, and will nicely reduce the column-count when the surrounding container get’s smaller.

To infinity and beyond

In our quest to make the max-column-count flexible, we introduce a variable called --auto-grid-max-columns. For those moments when no specific count is provided, our aim is an endless expanse of columns, limited only by the available space. Rather than resorting to an arbitrarily high number like 999 to simulate this effect, CSS allows us to elegantly use the infinity constant within calc() operations. This makes infinity a much cleaner and semantically accurate choice, as dividing by infinity effectively sets the value to 0, perfectly suiting our scenario for an unlimited number of columns. In practice, it looks like this in CSS:

css--max-column-width: 100% / var(--auto-grid-max-columns, infinity) - var(--auto-grid-gap, 1rem);

--column-width: max(
	var(--max-column-width),
	min(var(--auto-grid-min-size, 10rem), 100%)
);

Finally – here you can see everything combined in action.

See the Pen final auto-grid by Nils Binder (@enbee81) on CodePen.

Wrapping up

I hope this guide has made CSS Grid and creating reusable components a bit clearer. What's really cool about this is you don’t have to know everything about it to use it. This means in a big team, anyone can take this component, tweak the settings, and it'll work for their needs. It's all about making things easier and more flexible for everyone involved.

Motivation

This article focuses on leveraging ComfyUI beyond its basic workflow capabilities. You have created a fantastic Workflow and want to share it with the world or build an application around it. By hosting your projects and utilizing this WebSocket API concept, you can dynamically process user input to create an incredible style transfer or stunning photo effect.

Introduction

This blog post describes the basic structure of a WebSocket API that communicates with ComfyUI. Generating images through ComfyUI typically takes several seconds, and depending on the complexity of the workflow, this time can increase. We utilize a WebSocket connection to track progress and allow us to give real-time feedback to the user. Using these endpoints without a WebSocket connection is possible, but this will cost you the benefits of real-time updates.

Code for a basic WebSocket API structure can be found here: Basic WebSocket API.

Utilized ComfyUI endpoints

ComfyUI already has predefined endpoints ComfyUI endpoints, which we can target. Furthermore, ComfyUI also offers a WebSocket interface. For the API described later in this blog post, we do not need to modify this file, as it already provides everything we need.

• @routes.get('/ws') ⇒ Returns the WebSocket object, sends status and executing messages

• @routes.post("/prompt") ⇒ Queues prompt to workflow, returns prompt_id or error

• @routes.get("/history/{prompt_id}") ⇒ Returns the queue or output for the given prompt_id

• @routes.get("/view") ⇒ Returns an Image given a filename, subfolder, and type ("input", "output", "temp")

• @routes.post("/upload/image")⇒ Uploads an image to ComfyUI, given image_data and type ("input", "output", "temp")

API Setup

To achieve this, we first create a connection to the WebSocket path of ComfyUI within the API. This is done through the "/ws"route. We can auto-generate the client_id with an uuid for our basic API use case. And as server_address, we use our locally running ComfyUI.

pythondef open_websocket_connection():
  server_address='127.0.0.1:8188'
  client_id=str(uuid.uuid4())
  ws = websocket.WebSocket()
  ws.connect("ws://{}/ws?clientId={}".format(server_address, client_id))
  return ws, server_address, client_id

API endpoints

Queue Prompt

Sends a prompt to a ComfyUI to place it into the workflow queue via the "/prompt" endpoint given by ComfyUI. The parameters are the prompt, which is the whole workflow JSON; client_id, which we generated; and the server_address of the running ComfyUI instance

pythondef queue_prompt(prompt, client_id, server_address):
  p = {"prompt": prompt, "client_id": client_id}
  headers = {'Content-Type': 'application/json'}
  data = json.dumps(p).encode('utf-8')
  req =  urllib.request.Request("http://{}/prompt".format(server_address), data=data, headers=headers)
  return json.loads(urllib.request.urlopen(req).read())

Get History

Fetches the history to a given prompt ID from ComfyUI via the "/history/{prompt_id}" endpoint. As parameters, it receives the ID of a prompt and the server_address of the running ComfyUI Server. ComfyUI returns a JSON with relevant Output data, e.g., the Images with filename and directory, which we can then use to fetch those images.

pythondef get_history(prompt_id, server_address):
  with urllib.request.urlopen("http://{}/history/{}".format(server_address, prompt_id)) as response:
      return json.loads(response.read())

Get Image

Retrieves an image from ComfyUI based on path, filename, and type from ComfyUI via the "/view" endpoint. ComfyUI returns the raw image data.

pythondef get_image(filename, subfolder, folder_type, server_address):
  data = {"filename": filename, "subfolder": subfolder, "type": folder_type}
  url_values = urllib.parse.urlencode(data)
  with urllib.request.urlopen("http://{}/view?{}".format(server_address, url_values)) as response:
      return response.read()

Upload Image

Uploads an image to ComfyUI using multipart/form-data encoding via the "/upload/image" endpoint. This function opens an image file from the specified path and uploads it to ComfyUI. As params, we specify the input_path, which is the path to the image file we want to upload, the name, which will be the filename of the uploaded image and server_address for running ComfyUI. As optional params, we can define if we want to Upload the image into a different directory ("output", "temp") and if we want to overwrite an existing image for that path and filename. The image is uploaded as 'image/png'.

pythondef upload_image(input_path, name, server_address, image_type="input", overwrite=False):
  with open(input_path, 'rb') as file:
    multipart_data = MultipartEncoder(
      fields= {
        'image': (name, file, 'image/png'),
        'type': image_type,
        'overwrite': str(overwrite).lower()
      }
    )

    data = multipart_data
    headers = { 'Content-Type': multipart_data.content_type }
    request = urllib.request.Request("http://{}/upload/image".format(server_address), data=data, headers=headers)
    with urllib.request.urlopen(request) as response:
      return response.read()

API Workflow

To use ComfyUI workflow via the API, save the Workflow with the Save (API Format). If you don't have this button, you must enable the "Dev mode Options" by clicking the Settings button on the top right (gear icon). Check the setting option "Enable Dev Mode options". After that, the Button Save (API Format) should appear.

Basic prompt to image generation

As a first step, we have to load our workflow JSON. In my case I have an folder at the root level of my API where i keep my Workflows.

pythondef load_workflow(workflow_path):
  try:
      with open(workflow_path, 'r') as file:
          workflow = json.load(file)
          return json.dumps(workflow)
  except FileNotFoundError:
      print(f"The file {workflow_path} was not found.")
      return None
  except json.JSONDecodeError:
      print(f"The file {workflow_path} contains invalid JSON.")
      return None

Manipulating workflow

The goal is to reuse a defined workflow over and over but also be able to change the positive and negative prompts to generate different images. The technique can also be used to change other aspects of the workflow according to your needs, like the "cfg" or "steps". You need to find the relevant nodes in your workflow and adjust the values. One important aspect is that the seed has to be regenerated every time since ComfyUI will not generate a new Image if the seed hasn't changed.

pythondef prompt_to_image(workflow, positve_prompt, negative_prompt='', save_previews=False):
  prompt = json.loads(workflow)
  id_to_class_type = {id: details['class_type'] for id, details in prompt.items()}
  k_sampler = [key for key, value in id_to_class_type.items() if value == 'KSampler'][0]
  prompt.get(k_sampler)['inputs']['seed'] = random.randint(10**14, 10**15 - 1)
  postive_input_id = prompt.get(k_sampler)['inputs']['positive'][0]
  prompt.get(postive_input_id)['inputs']['text'] = positve_prompt

  if negative_prompt != '':
    negative_input_id = prompt.get(k_sampler)['inputs']['negative'][0]
    prompt.get(negative_input_id)['inputs']['text'] = negative_prompt

  generate_image_by_prompt(prompt, './output/', save_previews)

Calling the API

pythondef generate_image_by_prompt(prompt, output_path, save_previews=False):
  try:
    ws, server_address, client_id = open_websocket_connection()
    prompt_id = queue_prompt(prompt, client_id, server_address)['prompt_id']
    track_progress(prompt, ws, prompt_id)
    images = get_images(prompt_id, server_address, save_previews)
    save_image(images, output_path, save_previews)
  finally:
    ws.close()

1. Establish a WebSocket connection to ComfyUI

2. Queue the prompt via API call

3. Tracking the progress of our prompt by using the WebSocket connection

4. Fetch the generated images for our prompt

5. Save the Images locally

Tracking progress

The WebSocket connection enables us to track the progress of the workflow. We want to know which node we are currently on and how many nodes there are in a workflow to have an update and understanding of the progress. We also want to track the progress of the K-Sampler in detail since this is the node, which takes most of the time to finish, and we want to know at what step the K-Sampler currently is and how many steps in general there are.

pythondef track_progress(prompt, ws, prompt_id):
  node_ids = list(prompt.keys())
  finished_nodes = []

  while True:
      out = ws.recv()
      if isinstance(out, str):
          message = json.loads(out)
          if message['type'] == 'progress':
              data = message['data']
              current_step = data['value']
              print('In K-Sampler -> Step: ', current_step, ' of: ', data['max'])
          if message['type'] == 'execution_cached':
              data = message['data']
              for itm in data['nodes']:
                  if itm not in finished_nodes:
                      finished_nodes.append(itm)
                      print('Progess: ', len(finished_nodes), '/', len(node_ids), ' Tasks done')
          if message['type'] == 'executing':
              data = message['data']
              if data['node'] not in finished_nodes:
                  finished_nodes.append(data['node'])
                  print('Progess: ', len(finished_nodes), '/', len(node_ids), ' Tasks done')

              if data['node'] is None and data['prompt_id'] == prompt_id:
                  break #Execution is done
      else:
          continue
  return

Handling images

We fetch and save the images by first fetching the history for a given prompt ID. After that, we search for relevant Images in the output that the history endpoint provides us. Images from "Save Image" nodes are listed in "output". Preview Images are listed as "temp". We can specify if we want to save the previews as well. In the last step, we save the images locally in our specified output directory.

pythondef get_images(prompt_id, server_address, allow_preview = False):
  output_images = []

  history = get_history(prompt_id, server_address)[prompt_id]
  for node_id in history['outputs']:
      node_output = history['outputs'][node_id]
      output_data = {}
      if 'images' in node_output:
          for image in node_output['images']:
              if allow_preview and image['type'] == 'temp':
                  preview_data = get_image(image['filename'], image['subfolder'], image['type'], server_address)
                  output_data['image_data'] = preview_data
              if image['type'] == 'output':
                  image_data = get_image(image['filename'], image['subfolder'], image['type'], server_address)
                  output_data['image_data'] = image_data
      output_data['file_name'] = image['filename']
      output_data['type'] = image['type']
      output_images.append(output_data)

  return output_images

def save_image(images, output_path, save_previews):
  for itm in images:
      directory = os.path.join(output_path, 'temp/') if itm['type'] == 'temp' and save_previews else output_path
      os.makedirs(directory, exist_ok=True)
      try:
          image = Image.open(io.BytesIO(itm['image_data']))
          image.save(os.path.join(directory, itm['file_name']))
      except Exception as e:
          print(f"Failed to save image {itm['file_name']}: {e}")  

Example

Workflow used for this example: Basic prompt-to-image workflow

• model: sdXL_v10VAEFix.safetensors

• sampler_name: dpmpp_3m_sde

• scheduler: karras

• steps: 22

• cfg: 6.2

• positive prompt changed by API to: Woman in a red dress standing in middle of a crowded place, skyscrapers in the background, cinematic, neon colors, realistic look

Basic Image to Image generation

First you have to build a basic image to image workflow in ComfyUI, with an Load Image and VEA Encode like this:

Manipulating workflow

For the most part, we manipulate the workflow in the same way as we did in the prompt-to-image workflow, but we also want to be able to change the input image we use. Therefore, we need to modify the name of the image in the image loader and upload the image before queueing the prompt.

pythondef prompt_image_to_image(workflow, input_path, positve_prompt, negative_prompt='', save_previews=False):
  prompt = json.loads(workflow)
  id_to_class_type = {id: details['class_type'] for id, details in prompt.items()}
  k_sampler = [key for key, value in id_to_class_type.items() if value == 'KSampler'][0]
  prompt.get(k_sampler)['inputs']['seed'] = random.randint(10**14, 10**15 - 1)
  postive_input_id = prompt.get(k_sampler)['inputs']['positive'][0]
  prompt.get(postive_input_id)['inputs']['text'] = positve_prompt

  if negative_prompt != '':
    negative_input_id = prompt.get(k_sampler)['inputs']['negative'][0]
    prompt.get(negative_input_id)['inputs']['text'] = negative_prompt

  image_loader = [key for key, value in id_to_class_type.items() if value == 'LoadImage'][0]
  filename = input_path.split('/')[-1]
  prompt.get(image_loader)['inputs']['image'] = filename

  generate_image_by_prompt_and_image(prompt, './output/', input_path, filename, save_previews)

Calling the API

pythondef generate_image_by_prompt_and_image(prompt, output_path, input_path, filename, save_previews=False):
  try:
    ws, server_address, client_id = open_websocket_connection()
    upload_image(input_path, filename, server_address)
    prompt_id = queue_prompt(prompt, client_id, server_address)['prompt_id']
    track_progress(prompt, ws, prompt_id)
    images = get_images(prompt_id, server_address, save_previews)
    save_image(images, output_path, save_previews)
  finally:
    ws.close()

1. Establish a WebSocket connection to ComfyUI

2. Upload input image to ComfyUI

3. Queue the prompt via API call

4. Tracking the progress of our prompt by using the WebSocket connection

5. Fetch the generated images for our prompt

6. Save the Images locally

Example

Lets make some subtle changes to the image of the first Example. Change the color of the dress but keep the overall composition of the image.

Workflow used for this example: Basic image-to-image workflow

• model: sdXL_v10VAEFix.safetensors

• sampler_name: dpmpp_3m_sde

• scheduler: karras

• steps: 22

• cfg: 8

• diffusion: 0.6

• positive prompt changed by API to: Woman in a white dress standing in middle of a crowded place, skyscrapers in the background, cinematic, monotone colors, dark, dystopian

Outcome

As you can see hosting your ComfyUI workflow and putting it behind an API is a straightforward task. Devil is in details e.g. when you want to work with custom nodes and models but everything is doable.

If you have any questions or even an interesting project feel free to ping us https://9elements.com/contact/

Svelte is one of the best bets on technology we have ever made. It’s
fast, easy to learn and very ergonomic to use. It gives our teams a
quick way to build responsive web apps that can handle a ton of data and
complex interactions. Together with d3, it has become one of our go-to frameworks for data visualization. The new beta version of Svelte 5 already looks super interesting. Among
the more recent and drastic changes is the introduction of runes, which claims to make your Svelte apps faster and more maintainable, among other things. Since a lot of our projects could benefit from those promises, we wanted to see if the hype was warranted.

In all of our examples, we'll basically be working with the following type. It's a very simple object with a time and a value field:

javascripttype DataPoint = { time: Date, value: number }

The Svelte 4 component

As an example, we’re building a line chart with the time on the x-axis and the values on the y-axis. The initial component to render a basic chart looks like this:

html<script>
	import { extent, max } from 'd3-array';
	import { scaleLinear, scaleUtc } from 'd3-scale';
	import { line } from 'd3-shape';

	/** @type {{time: Date, value: number}[]}} */
	export let dataPoints;
	export let now = new Date();
	export let mainChartWidth = 800;
	export let mainChartHeight = 100;

	// Calculate the limits of our input data
	$: maxValue = max(dataPoints, (point) => point.value);
	$: [minTime, maxTime] = extent(dataPoints, (point) => point.time);

	// Use the limits to create x and y scales
	$: x = scaleUtc()
		.domain([minTime ?? now, maxTime ?? now])
		.range([0, mainChartWidth]);

	$: y = scaleLinear()
		.domain([0, maxValue ?? 0])
		.range([mainChartHeight, 0]);

	// Use the scales to create a generator
	/** @type {import("d3-shape").Line<{time: Date, value: number}>} */
	$: lineGenerator = line(
		(point) => x(point.time),
		(point) => y(point.value)
	);

	// Use the generator to create the path commands for our chart
	$: d = lineGenerator(dataPoints);
</script>

<svg
	width={mainChartWidth}
	height={mainChartHeight}
	viewBox="0 0 {mainChartWidth} {mainChartHeight}"
>
	<path {d} />
</svg>

<style>
	svg {
		stroke: black;
		stroke-width: 0.1em;
		fill: none;
		stroke-linejoin: round;
		stroke-linecap: round;
	}
</style>

This component looks like a very familiar, Svelte component for rendering a simple chart. Nothing much to see here.

Sidenote: Quick introduction to runes

Runes, and the $state rune specifically, are the new way to declare reactivity within and outside Svelte components. In Svelte 4 a simple counter state variable would look like this:

html<script>
	let count = 0;

	function increment() {
		count += 1;
	}
</script>

Rewriting the example using runes it looks like this:

html<script>
	let count = $state(0);

	function increment() {
		count += 1;
	}
</script>

It might look like a small change, but behind it lies a fundamental shift in the way Svelte updates its reactive variables. The $state rune turns our state into a signal which allows Svelte to track changes on a granular level and update the DOM in a performant way.

The Svelte 5 (beta) component

Let’s try this again with Svelte 5. The fifth version of the framework allows us to use reactive state outside the Svelte component context. All we have to do is create a \*.svelte.js file, and we can use all the required runes. This gives us the opportunity to model our reactive data using classes. So we start by modeling our DataPoint in a new file called ./datapoint.svelte.js:

javascriptexport class DataPoint {
  /** @type {Date} */
  time = $state(new Date());

  /** @type {number} */
  value = $state(0);

  /**
   * @param {Date} time
   * @param {number} value
   */
  constructor(time, value) {
    this.time = time;
    this.value = value;
  }
}

$state

This newly defined type has the same structure as our original one, but with the difference that it’s using the newly introduced $state rune. These so-called state fields will desugar to simple getters and setters that access the signals created behind the scenes:

javascriptexport class DataPoint {
  #time = $.source($.proxy(new Date()));

  get time() {
    return $.get(this.#time);
  }

  set time(value) {
    $.set(this.#time, $.proxy(value));
  }

  #value = $.source(0);

  get value() {
    return $.get(this.#value);
  }

  set value(value) {
    $.set(this.#value, $.proxy(value));
  }

  /**
   * @param {Date} time
   * @param {number} value
   */
  constructor(time, value) {
    this.#time.v = $.proxy(time);
    this.#value.v = $.proxy(value);
  }
}

This makes our DataPoint class behave like a regular class, but Svelte adds fine-grained reactivity behind the scenes that is fully within our control. We control which fields are reactive and which aren’t. Our new Svelte 5 component looks very similar but has small differences:

html<script>
	import { extent, max } from 'd3-array';
	import { DataPoint } from './datapoint.svelte.js';
	import { scaleLinear, scaleUtc } from 'd3-scale';
	import { line } from 'd3-shape';

	/**
	 * @type {{
	 * 	dataPoints: DataPoint[]
	 * 	mainChartWidth?: number
	 *	mainChartHeight?: number
	 *	now?: Date
	 * }}
	 */
	const { dataPoints, mainChartWidth = 800, mainChartHeight = 100, now = new Date() } = $props();

	// Calculate the limits of our input data
	const maxValue = $derived(max(dataPoints, (point) => point.value));
	const [minTime, maxTime] = $derived(extent(dataPoints, (point) => point.time));

	// Use the limits to create x and y scales
	const x = $derived(
		scaleUtc()
			.domain([minTime ?? now, maxTime ?? now])
			.range([0, mainChartWidth])
	);

	const y = $derived(
		scaleLinear()
			.domain([0, maxValue ?? 0])
			.range([mainChartHeight, 0])
	);

	// Use the scales to create a generator
	/** @type {import("d3-shape").Line<DataPoint>} */
	const lineGenerator = $derived(
		line(
			(point) => x(point.time),
			(point) => y(point.value)
		)
	);

	// Use the generator to create the path commands for our chart
	const d = $derived(lineGenerator(dataPoints));
</script>

<svg
	width={mainChartWidth}
	height={mainChartHeight}
	viewBox="0 0 {mainChartWidth} {mainChartHeight}"
>
	<path {d} />
</svg>

<style>
	svg {
		stroke: black;
		stroke-width: 2px;
		fill: none;
		stroke-linejoin: round;
		stroke-linecap: round;
	}
</style>

$props

Reading the component code from top to bottom, the first thing that pops into view is the new $props rune. Our export let statements are gone and have been replaced by a simple destructuring:

javascriptconst {
  dataPoints,
  mainChartWidth = 800,
  mainChartHeight = 100,
  now = new Date(),
} = $props();

In contrast to the old style of using export let this allows us to fully type our component props in more ways than was previously possible.

$derived

Another thing we see, almost immediately, is the absence of $: reactive statements. They have been replaced by the $derived rune. The new rune works similar to the old reactive statements, with the main difference being, that the dependencies of the derived value don’t have to be visible to the compiler. That means that these two examples function the same way:

javascriptconst counter = $state(0);

const doubled = $derived(counter * 2);

javascriptconst counter = $state(0);

function double() {
  return counter * 2;
}

const doubled = $derived(double());

This is because Svelte 5 tracks dependencies at runtime instead of tracking them at compile time. Combined with the existence of \*.svelte.js files, this means that we can refactor out reactive code into different files. We are no longer restricted to keeping reactive code in .svelte files.

The stress test

Go to https://stress-testing-svelte.vercel.app/ to try the examples for yourself. You can find the sources on GitHub. To stress test our examples and to get a feel for the improvements, we created a site that renders a variable number of charts with 1000 data points each. The number of charts is controllable by the user and can be adjusted in 10 point steps. We also added a button that “simulates” the mutation of the data points as fast as possible using requestAnimationFrame.

The result

The problem with the Svelte 4 component arise as soon, as too many instances of this chart are active on the current page (e.g. 100). At that point, the example dies spectacularly with the infamous ERR_SVELTE_TOO_MANY_UPDATES error. This is not an error per se, by the way. Svelte tries to shield you from having recursions in your reactive statements, which is absolutely desirable. But we don’t have any recursion in our line chart, and 100 charts with 1000 data points for each chart isn’t that contrived.

Meanwhile, the Svelte 5 example, while getting slower, continues to work.

Conclusion

As requirements and expectations for data visualization evolve and get more complex, Svelte continues to keep up with the times while keeping its promise of speed and simplicity. The latest iteration of the framework has shown us this in particular. It will most likely accompany us for a long time.

There are many AI tools for image generation, and it seems like a new one is added every day. The results are often very impressive, yet they can also feel quite random. I wanted to try what I could achieve by setting a clear goal and striving to reach it.

My idea: a movie poster for a fictional sequel to Disney's live-action adaptation of The Little Mermaid. The main character should be a blend of King Triton and the sea witch Ursula.

I initially tried with Midjourney. The images are impressively "real," but I couldn't manage to create a hybrid human-octopus character with a human upper body and an octopus lower body.

So, I turned to DALL-E by OpenAI. Here, it was much easier to create images that were closer to what I wanted. However, the results are much more artificial, resembling the graphics of video games.

After seeing these images, I thought we might not be there yet, or at least my skills weren't sufficient to create a photo-realistic image with the desired content. A few days later, my social media timeline was flooded with posts about Magnific AI; an upscaling tool that enlarges AI-based images while adding details. So, I revisited my merman creations to see if I could enhance the already created images. And I must say, the result was mind-blowing for me. Not only were the original images doubled in size from 1024px in width to 2048px, but amazing details were also added, making the image immediately appear much more realistic (as realistic as one can get with a merman with tentacles. 😅).

My enthusiasm was reignited. Although I found the result very impressive, a closer look revealed some flaws:

• He appears to be both under water and in the water.

• The tentacles seem to emerge from nowhere and don't appear to be connected to a body.

• An eye is growing on his left upper arm.

• ...

From there, I began working with Photoshop. I first cropped the lower part of the image, and then had the entire image expanded. While Photoshop's in-painting options have not been convincing to me, the outpainting features sometimes yield very impressive results. This allowed me to extend the tentacles downward and give the entire scene more space on all sides.

After enlarging the image twice, I was working with a resolution of over 4000px in both width and height. Unfortunately, the background generated with Photoshop was always somewhat blurry. So, I used DALL-E and Magnific AI again to create a sunken underwater city. I was quite impressed with the result.

I then rather amateurishly incorporated part of this image into my existing one. With more time and talent, one could certainly have created a good montage using classic Photoshop skills. However, I used my basic montage, halved its size, and then enlarged it again with Magnific AI.

From this point, it was a mix of classic Photoshop retouching processes and AI-supported modifications. I smoothed out irregularities, adjusted skin tones, and added more details to the background. Finally, the movie title was missing, which I created in Figma and then integrated into the existing file.

In conclusion, I can say that I am very satisfied with the result. I couldn't have created such a poster without AI support. However, it's important to note that a lot of work went into this as well. The necessary effort will surely decrease significantly in the future, but it's very important to understand the strengths and weaknesses of the tools we currently have at our disposal.

Innovating a company from within is a complicated task. You need to identify painpoints in your current processes, you need to create benchmarks so that you can validate changes and you need to have hypotheses to get a gut feeling which are the most important things to work on and to prioritize.

This is why we helped to build the DISROOPTIVEs web application. We digitalized the process in all of its facets and help you to document and keep track of your goals.

DISROOPTIVEs web application is focussing on finding innovative solutions for their customers in a multi-step process which involves gathering data from various regions of the world (wide web) to do exactly that: identify painpoints, benchmark companies and how they improved, create hypotheses for the customer solutions and many more. The application itself featured a sophisticated realtime multi-user environment and every step of the way also needed user-input.

The next Frontier

With the rise of ChatGPT (and LLMs in general) our client had the plan to raise the bar even further. The way LLMs are able to reason about tasks has increased fantastically and we can just recommend every company to investigate every digitalized process to figure out if it can be fundamentally improved.

For DISROOPTIVE we had the following goals:

1. Let the LLM figure out the painpoints about your processes just from providing the project briefing / mission statement.

2. Make suggestions about possible benchmarks.

3. Derive customized hypotheses automatically.

Though a LLM won't replace your chief digitalization officer you'll be surprised about the answers. With some little prompt engineering we managed to create unconventional thoughts but more important the LLM helped us not to forget the basics.

The client had dabbled with ChatGPT before, so he knew already that the model is in principal capable of providing desired output for each of the steps mentioned above. It was only a question of how the solution would look and feel like.

The first Painpoints

I'm making this sound bad, but like stated above the first goal actually was to have painpoints generated only being given the initial project briefing. So we grabbed an OpenAI API key, placed a button in the frontend and hooked it up to have the backend feeding the prompt to OpenAI's GPT model. Since our data model was already existent we knew what properties the answer should contain, and since we wanted multiple painpoints in a single go, we asked exactly that.

Ask and you shall be given

Following the initial problem description the prompt contained "Answer in valid JSON format like this: [{ key: 'string', values: [string, string, string]},{ key: 'string', value: [string, ...], ..." } ".  Checking the API responses showed that the answer was structured in the requested way so we could easily built on top of that and create some database records that get attached to the project. The first step of the way was already complete. It was only a rather raw implementation at first but it worked like a charm. Well most of the time...

I said JSON, damnit!

Turns out that, from time to time and rather rarely, the answer wasn't JSON. Sometimes just a string, sometimes just an array of strings. Explicitly reminding the model of the RFC8259 JSON format somewhat fixed the issue but ever since the release of GPT-4 Turbo and its introduction to response_format AKA "JSON-Mode", this problem hasn't surfaced anymore. So everything looked promising, we had a functioning prompt for generating painpoints grouped thematically which lead to database records that enriched an in app project based only on the information provided by the project briefing. Great! We went on to build a little dashboard section where the prompts and number of requested objects could be edited by the admin, leaving only the system prompts regarding the desired response structure hardcoded as the system relied on it.

Uh, that leaves a benchmark

The next step involved the generation of benchmarks. Benchmarks against other companies were originally rather research intensive. 'What companies are there, what they do and where they excel at and how this could be useful to the current problem at hand'. And this time, instead of just transforming a given text into rather short and precise painpoints, GPT was prompted to provide knowledge on said question.

Ask for more and you shall be given more, way more

This time the prompt got more sophisticated as was the underlying data model but the basic principle of "Answer in valid JSON, put everything in an array" remained. Checking the first response revealed all the information necessary to create bechmarks in the database. Great again! But this time, we noticed an significant increase in response time. Triggering the process in the frontend, to having the benchmarks show up started to take minutes. The response time for the painpoints was laughably low. but it involved only a couple of words or sentences at best per item. Benchmarks can feature multiple paragraphs, this is on a completely other level in terms of output. Once the response was processed and stored in the database the result looked great, but from the user perspective it kind of felt odd, to wait so long for a response, while in contrast ChatGPT starts to answer immediately.

Gently down the stream

Being reminded by ChatGPTs streamed responses in the browser, we started to look into streaming the API response aswell. Upon sending the request, we saw an immediate response starting with "[", then "{" and it only took a couple of seconds until the first "}" had been streamed back to us. Ofcourse the stream continued, presenting the next objects, each starting and ending with the curly brackets. The next time we ran this, the code was modified to check for a pair of opening and closing curly brackets, cut out everything between them, parsing it as JSON and creating the record. All while the streaming continued in the background and the next pair of curly brackets is identified and processed. The result? Instead of waiting a couple of minutes for the complete response with a number of benchmarks, we manged to deliver benchmarks in a couple of seconds to the user and while he's busy checking out the first update the next one is nearly done being processed aswell, leading to a much smoother experience.

Visual communication

The last and final step involved the combination of the previous outputs into a new one, so we built another endpoint that took the recipie and ingredients, prompted GPT and processed the response on the fly, this time assisted by Dall-E to create some visual representation of the result. And while Dall-E 2 failed to amaze, Dall-E 3 is a huge leap forward in quality, rivaling Midjourney and the likes but already providing an API.

The Impact

Seeing GPT-4 and now GPT-4 Turbo at work is just incredible. And DISROOPTIVE's use case seems exceptionally suited for AI assistance, removing the grunt work and allowing the users to focus on making strategic decisions, revolutionizing the workflow.

Are you interested in what DISROOPTIVE is doing? Do you have a project that could benefit from AI integration? Or do you want to build something from the ground up? We'd love to hear from you! Get in touch with us! 

Intro

This Blog post aims to provide a basic overview of the concepts and buzzwords in the AI realm. This is not a technical deep dive or a blog post that explores nuances of different concepts. This blog post introduces each concept or term with a definition or description, followed by examples or additional context. Lastly, links to further resources are provided for those interested in delving deeper.

Firstly, I would like to provide additional information on the three overarching concepts of the blog post. These concepts are AI (Artificial Intelligence), ML (Machine Learning), and generative AI. The following Venn diagram shows the relationship between these concepts. AI and ML are often used interchangeably, but ML is a subset of AI. So, not everything we call AI is based on ML, but everything based on ML can be called AI. The topic of generative AI mainly uses machine learning concepts, but there are examples of generative AI that are not based on ML.

AI (Artificial Intelligence)

AI describes solving tasks or problems by computers/machines, which require a form of human-like intelligence.

Examples:

• Natural language processing

• Natural language generation

• Sentiment Analysis

• Image recognition

• Chess Engines

Further definition:

• https://www.merriam-webster.com/dictionary/artificial intelligence

• https://www.ibm.com/topics/artificial-intelligence

ML (Machine Learning)

ML is a subfield of AI that describes the development of solutions for tasks or problems by computers/machines using data and algorithms without a programmer explicitly specifying the solution path through program code.

Use cases include:

• Natural language generation

• Image recognition

• Autonomous driving

Further definition:

• https://www.merriam-webster.com/dictionary/machine learning

• https://www.ibm.com/topics/machine-learning

NN (Neural Network)

Neural Networks are a computer architecture that is inspired by the human brain. Neural Networks are a core principle of machine learning. They consist of interconnected nodes (neurons). Neural networks typically consist of an input layer, one or more hidden layers, and an output layer. Neural networks process inputs using a variety of mathematical operations and subsequently produce an output.

Further definition:

• https://www.britannica.com/technology/neural-network

• https://www.merriam-webster.com/dictionary/neural%20network

DL (Deep Learning)

DL is a machine learning subfield based on neural networks with multiple input, hidden and output layers. Various transformations and feature extractions can be performed in the different layers to recognize more complex patterns.

Use cases include:

• Natural language processing

• Image recognition

• Recommender systems

Further definition:

• https://www.ibm.com/topics/deep-learning

• https://www.ibm.com/blog/ai-vs-machine-learning-vs-deep-learning-vs-neural-networks/

Model

A model in the context of neural networks refers to the structure of the layers, activation functions, and nodes with associated weights and biases. The model is created and optimized through machine learning by learning from input data and recognizing patterns. The goal of a model is to provide predictions based on input data.

Well-known models include:

• GPT (Large Language Model)

• AlexNet (Computer Vision Model)

• AlphaZero (Chess Engine based on Machine Learning)

Example: I want to make statements based on pictures as input, determining whether the image contains an astronaut or an alien. An abstract model could look like this. We divide the properties of astronauts and aliens into "all" and "some". When the model receives the first image, it guesses because it has no prior data. Let's say it guessed “astronaut”. The Model now thinks all astronauts in the world will look like this. So, every property of the image is put in the “all” category. The model gets a different picture and will most likely say alien, even if it is an astronaut, because the new image will not check everything in the “all” properties. The model then updates itself and puts properties shared by both astronauts in “all” and the rest in “some”. This process will repeat for the whole training dataset so that the Model can adjust what are “all” and what are “some” properties for astronauts and aliens.

Additional information:

• https://www.databricks.com/glossary/machine-learning-models

Weights, Activation function, and Biases

Weights are numeric values representing the strength of connections between nodes in a neural network. They determine the extent to which the output of one node influences the subsequent node.

Activation functions are Non-Linear Functions that calculate the output value of a neuron. Based on the output value, the neuron will be activated or not. For the calculation, an activation function sums all inputs and weights connected to the neuron and adds a bias.

Biases are numeric values independent of the previous layer. Each node has its bias value and can control the activation of that node by shifting the activation function right or left. This enables more flexibility for the neural network.

Additional information:

• https://machine-learning.paperspace.com/wiki/weights-and-biases

• https://www.ijeast.com/papers/310-316,Tesma412,IJEAST.pdf

Model Parameter

Model Parameters are a model's internal values learned from the input data during training. In neural networks, parameters include weights and biases. These parameters are optimized to enable the neural network to make accurate predictions about the input data.

Additional information:

• https://machinelearningmastery.com/difference-between-a-parameter-and-a-hyperparameter/

Training/Learning

Training/Learning describes the iterative process in which data is passed to a neural network. After each iteration, the learnable parameters of the neural network are adjusted to achieve an optimal solution for the given problem.

For training models, a dataset is required. There are different strategies for using this data for training. However, using different data for training and validating the model is essential.

Additional information:

• https://www.telusinternational.com/insights/ai-data/article/how-to-train-ai

Hyperparameter

Hyperparameters are values provided to the model from the outside to make adjustments that can influence the learning process and its performance, among other things. The optimal settings for a model are usually determined through systematic tuning of the parameters or experimentation.

The learning rate is one Hyperparameter that can be set; it can specify how much the weights between the neural network nodes are adjusted after each iteration.

Additional information:

• https://machinelearningmastery.com/difference-between-a-parameter-and-a-hyperparameter/

AGI (Artificial General Intelligence)

AGI is also referred to as strong AI. An AGI can independently learn new problem-solving strategies that are not explicitly included in its original model or training data, and it can handle various tasks in different contexts.

So far, all the AI systems we have developed are not AGI. There are different assessments regarding when and whether it will be possible to create an AGI.

Further definition:

• https://www.techopedia.com/definition/31618/artificial-general-intelligence-agi

Narrow/Weak AI

Narrow/Weak AI refers to systems designed to perform specific tasks. These systems rely on their training data and underlying model and cannot independently learn new skills or solve problems beyond their original scope.

For example, an AI like AlphaZero excels at chess better than any human player and does not know how to play Go or Tic-Tac-Toe. There are systems, such as in a self-driving car, where multiple weak AI's work together. However, each AI system is only responsible for its specific domain.

Further definition:

• https://www.techopedia.com/definition/32874/narrow-artificial-intelligence-narrow-ai

Big Data

Big Data refers to large amounts of data, typically internet, mobile, health, or traffic. These data sets often exceed the storage capacity of standard databases and require specialized solutions. This data is used as the basis for data preprocessing to train models.

Further definition:

• https://dictionary.cambridge.org/de/worterbuch/englisch/big-data

Bias

Note: This concept of Bias has nothing to do with the biases in the scope of the nodes in a neural network!

Bias describes the process by which the outputs of AI systems can be based on certain assumptions or prejudices. Various factors can contribute to the emergence of bias in an AI system, such as the training data, the model used, and the evaluation of the outputs. Bias can distort the results of an AI system and lead to unequal treatment of specific individuals or groups.

For example, if you feed an AI system with pilot data and then ask it to define the characteristics of a good pilot, it will likely describe a white male person because these characteristics apply to many pilots. However, humans know these are different from the criteria we seek when describing a pilot.

Additional information:

• https://brie.berkeley.edu/sites/default/files/brie_wp_2018-3.pdf

• https://hbr.org/2019/10/what-do-we-do-about-the-biases-in-ai

Alignment

Alignment describes the process of evaluating the outputs of an AI according to predefined values, usually ethical or social values. This feedback can be used to ensure that the model operates within this value framework.

OpenAI has employed many individuals in precarious working conditions in the so-called Global South for the alignment of ChatGPT, who evaluate outputs and prompts based on the following values defined by OpenAI:

• Truthfulness

• Harmlessness

• Helpfulness

An example of the negative consequences that can occur with AI systems when they lack alignment is demonstrated by Microsoft's chatbot Tay. Within a few hours, the chatbot became radicalized and had to be shut down after 16 hours.

Additional Information:

• https://www.nytimes.com/2016/03/25/technology/microsoft-created-a-twitter-bot-to-learn-from-users-it-quickly-became-a-racist-jerk.html

• https://cdn.openai.com/papers/Training_language_models...

Reinforcement Learning

Reinforcement Learning describes a form of model training where the model can decide how to fulfill a task. The model receives rewards or punishments at specific time points or after actions. The goal of the model is to obtain the highest possible rewards.

Use cases include:

• Self-driving cars

• Gaming

• Resource management

Additional Information:

• https://developer.ibm.com/learningpaths/get-started-automated-ai-for-decision-making-api/what-is-automated-ai-for-decision-making/

Supervised Learning

Supervised Learning describes a form of model training where datasets consisting of pairs of input and corresponding output data are used. Based on the input, the model makes a prediction and then compares it to the output data.

Use cases include:

• Classification problems

• Regression problems

• Anomaly detection

Additional information:

• https://www.ibm.com/topics/supervised-learning

• https://www.researchgate.net/publication/229031588_Supervised_Learning

Unsupervised Learning

Unsupervised Learning describes a form of model training where no predefined goals for the results are set. The aim is to find patterns, structures, or relationships based on the given input data.

Use cases include:

• Clustering problems

• Anomaly detection

• Recommender systems

Additional information:

• https://www.ibm.com/topics/unsupervised-learning

Overfitting

Overfitting is a problem that can occur during training when the model becomes too closely adapted to the training data, losing the ability to make accurate predictions for new data.

For example, let's say we want a model that can determine if there is an astronaut in the input image. We train the model with 10,000 images and find that after training, the model can almost accurately predict whether there is an astronaut with nearly 100% accuracy. However, when we use our validation dataset, the model only achieves an accuracy rate of 55%. Essentially, the model is guessing whether it is an astronaut or not. The model has likely failed to learn to recognize what an astronaut is but instead has memorized the training data.

Additional information:

• https://elitedatascience.com/overfitting-in-machine-learning

• https://www.ibm.com/topics/overfitting

Fine Tuning

Fine Tuning, often referred to as transfer learning, describes adapting a pre-trained model to improve its performance in a specialized subset of the problem or to solve a new task closely related to the original one. This eliminates the need to completely retrain a model.

OpenAI's GPT model can be used as a text recognition and generation base model. Based on this model, additional datasets, such as internal product descriptions, can be added, and the GPT model can be fine-tuned to deliver better results for the new datasets.

Additional information:

• https://platform.openai.com/docs/guides/fine-tuning

Prompt / Prompt Engineering

Prompt is the user input, usually a few words or a sentence, that describes what a generative AI should produce.

Prompt Engineering involves optimizing prompts, i.e., determining which prompts effectively achieve the desired results. Important aspects include providing context to the prompt and experimenting with different prompts.

Additional information:

• https://dev.to/github/prompt-engineering-for-ai-what-is-prompt-engineering-and-how-to-get-good-results-from-ai-engines-5ch6

Generative AI

Generative AI describes AI systems that generate various outputs using user inputs (prompts). Use cases include text, code, image, video, or music generators.

Well-known generative AI systems include:

• ChatGPT (Text)

• Midjourney, Stable Diffusion, DALL-E (Image)

• Murf, Lovo, Whisper (Voice)

Additional information:

• https://arxiv.org/pdf/2301.04655.pdf

Clustering

Clustering is a problem in the AI context where one searches for relationships or patterns in large datasets that overarching clusters can describe. Unsupervised learning is commonly used for this type of problem.

In this process, algorithms calculate the similarity between data points and create a network of data points where the similarity is represented by distance. This allows for the identification of groupings (clusters). A popular algorithm to detect clusters is the K-Means Algorithm.

Additional Information:

• https://www.geeksforgeeks.org/clustering-in-machine-learning/

Image reference:

https://commons.wikimedia.org/wiki/File:K-means_convergence.gif

Classification

Classification is a problem in AI where the input is mapped to an output value from a set of defined output values. The outputs can be discrete classes such as "Yes/No," "Categories," or "Labels." Supervised learning is commonly used for this type of problem.

The example given in the model section is a typical example of a classification problem. Pictures that can be mapped into distinct classes.

Additional information:

• https://www.geeksforgeeks.org/regression-classification-supervised-machine-learning/?ref=gcse

Regression

Regression is a type of problem in the AI context where the output values are continuous and numerical. The goal is to make predictions in the form of numerical output values based on the input. Supervised learning is commonly used for this type of problem.

An example of regression is predicting house prices based on square meters, number of rooms, and the city. A model trained on this dataset should be able to predict the price for a new data point given the input of square meters, number of rooms, and the city.

Additional information:

• https://www.geeksforgeeks.org/regression-classification-supervised-machine-learning/?ref=gcse

Sentiment Analysis

Sentiment Analysis describes identifying a text's sentiment, attitude, or evaluation.

Use cases include:

• Market research

• Monitoring social media

• Political analysis

Additional information:

• https://www.sciencedirect.com/science/article/pii/S187705091630463X

Hallucination

Hallucination is a process in which a generative AI produces an output unsupported by the training data or contradicts known facts.

Other terms for the concept of hallucination are:

• confabulation

• delusion

• Some even call it: bullshitting.

There are different reasons why an output can be described as a hallucination:

• Data quality of the training data: There are attacks on LLMs where you poison the training data with wrong information

• Lack of context, e.g., there are two Persons with the same name relevant to the given prompt, and the model mixes up data from the two persons.

Additional information:

• https://www.nytimes.com/2023/05/01/business/ai-chatbots-hallucination.html

Language Models / Large Language Models

Language models calculate the distribution and probabilities of a word or word sequences based on the previous context or the previous sequence. They enable computers/machines to process and generate natural language.

Large language models are a particular class of language models with a huge number of parameters and are trained on a large amount of data. Large language models can, therefore, recognize more complex language patterns and capture larger contexts.

Well-known models include:

• GPT

• N-Gram Model

• LTST Model (Long Term Short Term)

• LaMDA

• Llama

Additional information:

• https://www.techopedia.com/definition/34948/large-language-model-llm

Transformer

Transformers are a computer architecture used in neural networks to transfer one sequence of characters to another. Transformers are used in computer systems for natural language processing.

Transferring one sequence of characters to another is the classical approach in language translation. A simple look-up table cannot be used in translation because words in different languages often have multiple meanings, and sentence structures vary between languages.

Additional Information:

• https://deepai.org/machine-learning-glossary-and-terms/transformer-neural-network

• https://arxiv.org/abs/1706.03762

Image reference:

https://arxiv.org/abs/1706.03762 Figure 1: The Transformer - model architecture.

GPT (Generative Pre-trained Transformer)

GPT is the most well-known language model used by the ChatGPT application developed by OpenAI. It can process prompts and generate text as output. The generated text includes both natural language and computer code.

Additional information:

• https://openai.com/chatgpt

Token

Tokens are representations of strings of characters (words or subwords) used to convert natural language into a machine/computer-understandable system.

The tokenization used by OpenAI for words and punctuation in the English language yields the following:

• 1 token ~= 4 chars in English

• 1 token ~= ¾ words

• 100 tokens ~= 75 words

Additional information:

• https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them

NLP (Natural Language Processing)

NLP (Natural Language Processing) is a branch of AI development that focuses on enabling machines/computers to process natural/human language.

Use cases of NLP include:

• Speech recognition

• Translation

• Sentiment analysis

Additional information:

• https://www.ibm.com/de-de/topics/natural-language-processing

NLG (Natural Language Generation)

NLP is a branch of AI development that focuses on enabling machines/computers to generate natural/human language and thus communicate.

Areas of NLG (Natural Language Generation) include:

• Chatbots

• Voice assistants

• Content generation

Additional information:

• https://www.jair.org/index.php/jair/article/view/11173/26378

OpenAI

OpenAI is a company based in the USA that is engaged in developing and researching AI systems. The stated goal of OpenAI is to create AGI (Artificial General Intelligence). Well-known products from OpenAI include ChatGPT and DALL-E.

Additional information:

• https://openai.com/

• https://www.washingtonpost.com/technology/2023/02/06/what-is-openai-chatgpt/

Building a reliable search for a web application is hard. Building it for a video-based platform is even harder. For Medmile, a German platform offering bite-sized video courses for doctors, we relied on descriptions created by the lecturers who recorded the courses to power search and discovery.

The problem: The lectures were focused on top-notch video content and not on writing extensive descriptions. Thus, the real content, the information we needed for the search engine to offer meaningful results, was ‘hidden’ in the course itself, in the audio track, to be precise.

Whisper and GPT to the rescue.

The general idea is simple: We transcribe the videos using Whisper, use GPT to generate summaries of its content, and calculate embeddings we can use for search and discovery.

Let’s have a look at the process:

Transcribing the video

To transcribe the video, we first have to extract the audio track. Since we cannot transcribe the whole track at once, we split it into parts, creating chunks reliably small enough to be processed by OpenAI’s Whisper API.

Generating these chunks can easily be done with FFmpeg, in this case, used in a custom video processor hooked into ActiveStorage:

rubydef process
  filename = blob.filename
  new_filename = ActiveStorage::Filename.new("#{filename.base}.flac")

  tempfile_pattern = "#{Dir.tmpdir}/#{new_filename.base}-%02d#{new_filename.extension_with_delimiter}"

  blob.open tmpdir: Dir.tmpdir do |file|
    system(
      self.class.ffmpeg_path, "-y",
      "-i", file.path,
      "-vn",
      "-f", "segment",
      "-segment_time", "200",
      "-acodec", "flac",
      tempfile_pattern,
      exception: true
    )
  end

  # ...
end

These chunks can then be sent to Whisper using the Ruby OpenAI Gem.

rubydef client
  @client ||= OpenAI::Client.new
end

def transcribe(tempfile)
  response = client.audio.transcribe(
    parameters: {
      model: "whisper-1",
      file: tempfile
    })

  response['text'] or raise "No text in transcription response..."
end

Instead of just combining these chunks as raw text, we use GPT to generate summaries, both to make sure to stay within OpenAI’s limits but also to generate descriptions that could eventually be shown to the user as a more extensive description of the video. To allow implementation of our search functionality, we calculate the embeddings on the video summary and store it with the video, using Neighbor, a gem providing nearest neighbor search for Rails and Postgres.

rubydef generate_summary(text)
  response = client.chat(
    parameters: {
        model: "gpt-3.5-turbo",
        messages: [{ role: "user", content: "Please summarize in maximum 850 characters:\\n#{text}\\n\\nSummary:"}],
        temperature: 0.3,
    })

   response.dig("choices", 0, "message", "content") or raise "No text in API response: #{response}"
end

def calculate_embeddings(text)
  response = client.embeddings(
    parameters: {
      model: "text-embedding-ada-002",
      input: text
    }
  )

  response.dig("data", 0, "embedding") or raise "No embedding in API response: #{response}"
end

def summarize(video)
  summaries = []

  summaries = video.raw_transcriptions.map do |transcription|
    generate_summary(transcription)
  end

  video.gpt_summary = generate_summary(summaries.join(" "))
  video.gpt_summary_embedding = calculate_embeddings(video.gpt_summary)
end

Storing vectors

The Neighbor readme on Github offers a good explanation of how to use the gem, but for the sake of completeness, here are the few lines you need to store vectors:

rubyclass AddEmbeddingsToCoursesAndVideos < ActiveRecord::Migration[7.0]
  def change
    enable_extension "vector"   
    add_column :videos, :gpt_summary_embedding, :vector, limit: 1536
  end
end

class Video < ApplicationRecord
  has_neighbors :gpt_summary_embedding, normalize: true
  self.filter_attributes += [ :gpt_summary_embedding ]

  # ...
end

With Euclidean distance, we would not need normalized vectors, but since we are still experimenting, we decided it wouldn’t hurt to add that flag.

Using calculated embeddings for search

To perform a search based on a user query, we calculate the embeddings on the query using the same methods used for calculating embeddings on the video summaries. Then, we use these embeddings to look for the nearest neighbors within a set distance.

Thanks to Neighbor, this is done quite easily (at least in this simplified example):

rubyMAX_DISTANCE = 0.55

embeddings = generate_summary(query)

@ai_results = 
  Video.nearest_neighbors(:gpt_summary_embedding, embeddings, distance: "euclidean")
    .limit(10)
    .to_a
    .select { |result| result.neighbor_distance < MAX_DISTANCE }

The results are already ordered by distance to the query, so in theory, the best match should also be the first in the list. As usual, when it comes to AI, you have to experiment with real data and real queries to fine-tune things like maximum distance, as well as play with the queries used to generate the summaries.

Conclusion

We just released this function for registered users. Being in beta does not replace the ‘classic’ search function but rather provides an enhancement, displayed in an additional section when results seem to be good (a.k.a. close) enough, clearly marked as ‘AI results’.

First real user feedback is overwhelmingly positive, and we’ll continue working on enhancing the search function itself and using the generated video summaries to provide helpful support for lecturers in the form of AI-generated suggestions when it comes to writing video descriptions.

Do you have a project that would benefit from utilizing AI to support users and lead them to the right content? Don’t hesitate to get in touch with us.

Automated testing controversies

In software development, there are few topics as controversial as test automation.

The first controversy revolves around the fundamental value of automated tests. What can automated testing achieve and contribute to correct, bug-free software that works for the user? What does a valuable test look like?

The second debate is about whether the effort of testing is worth it – for the end user, for the business, the development team. Testing takes time, costs money and requires infrastructure.

Testing is a skill that developers need to learn in addition to countless other skills. Many developers, even seasoned ones, are beginners when it comes to automated testing. They often feel unproductive, blocked and frustrated when they have to write tests.

The third argument deals with the practical implementation of tests. How should we test a particular feature? Which tools and frameworks should we use? What makes a reliable test? And last but not least, what color should the bike shed in the garden be painted?

While I follow these discussions with some ironic distance, I do not want to dismiss them. We need to discuss these important questions again and again, especially with beginners. I sympathize with everyone who struggles writing tests. And I admire those who teach testing with respect and an open mind.

What do I know about testing anyway?

In my professional work, I have delved deeply into automated testing of websites and web applications. I have published the online books Robust JavaScript and Testing Angular in which I tried to share my experience.

You would think I have a solid opinion on the subject. I sure know something about testing. But I have reached a Socratic point where I know that I know nothing certain about testing.

In a recent client project, we wrote and maintained a large code base with an extensive test suite. That changed my mind in several ways I would like to describe here.

Giving automated testing a central role

For our client, we developed a data visualization framework written in HTML, SVG, CSS and JavaScript. The data experiences are rendered both on the server and the client. The visualized data is mostly pulled from HTTP APIs.

Automated testing played a central role in the project. We pursued the following goals:

• Tests should give us strong confidence that all bits and pieces work as designed and continue to work after a change.

• Tests should cover the complex logic that transforms and visualizes different kinds of data. It’s almost impossible for manual testing to cover these cases.

• Tests should cover accessibility features that are not immediately obvious for users without assistive technologies.

• Tests should be easy and straight-forward to write. They should run fast and reliably.

The initial testing setup was:

• Unit tests of plain JavaScript functions and UI components

• Integration tests of public HTML/JavaScript APIs and Node.js HTTP services

• End-to-end tests running against full web pages

• Static code analysis with linters, static types and accessibility checkers

We wanted build and test all parts on every change on a continuous integration server to give our developers quick feedback. We have successfully used GitHub Actions for this purpose. After some optimization, the builds, checks and tests take 15-20 minutes to run.

Unit and integration tests

We started using the Jest test runner for tests of plain JavaScript code, Node.js services as well as Svelte and Preact components. Jest is the de facto standard for testing JavaScript code. It was originally developed at Facebook alongside React and later React Native and released in 2014.

Testing UI components with jsdom

While Jest runs under Node.js, it emulates browser JavaScript APIs with jsdom. This enables web developers to test their client-side JavaScript code – for example, Svelte components – in a pure Node environment. Such tests run faster than spinning up a fully-fledged web browser.

Rendering UI components with jsdom makes sense for testing on the abstraction level of the DOM framework, Svelte or Preact in our case. We trust them to perform the proper DOM updates. We trust jsdom to implement the DOM correctly. So we test against the resulting HTML tree. But we need to keep in mind that other browser APIs might or might not be properly emulated by jsdom.

Since jsdom merely emulates JavaScript running in the browser, there a numerous large and small differences. Most importantly, the generated HTML and CSS is never rendered. Elements do not have a size, a visibility or other computed styles. Therefore, features that revolve around visual rendering need to be tested in real browsers.

Testing UI components by simulating user interaction

Based on my experience with automated testing of user interfaces on the web, I support this statement:

The more your tests resemble the way your software is used, the more confidence they can give you. – Kent C. Dodds

This is the motto of the Testing Library, which is a whole family of libraries for testing HTML DOM and JavaScript frameworks that render HTML.

The Testing Libraries provide handy utilities for testing components in React, Angular, Svelte, Preact etc. But what makes them special is the underlying testing methodology.

The Testing Libraries promote high-level tests that do not test implementation details. The tests should interact with the HTML document as it is presented to the user: Text, buttons, links, form fields, elements with certain ARIA roles. Tests should deal with these HTML elements rather than component instances. Quite like end-to-end tests which by nature know nothing about JavaScript frameworks and component organization.

Consequently, the Testing Library recommends finding elements by text content, accessible name, form field label or ARIA role. This way, testing focuses on what matters to the user while promoting semantic markup and good accessibility practices.

In the client project, accessibility was a core requirement. So the Testing Library proved to be particularly beneficial: If the test finds and clicks a button with a certain accessible name, it also guarantees that the button is indeed a button element with this very label. And not a meaningless div that happens to have a click handler.

Switching from Jest to Vitest

In the last years, Facebook (Meta) pulled out of the maintenance of Jest, transferred the legal ownership to the OpenJS foundation and started a community funding. Despite being one of the pillars of the JavaScript ecosystem, Jest has become a project a few volunteers are thanklessly maintaining in their free time.

Our biggest issue with Jest was the lack of proper ECMAScript modules (ESM) support. More and more packages are published as ESM only and code transformers output ESM only. It became harder and harder to test our growing code base with Jest. We did not succeed to get the tests running with Jest’s experimental ESM support.

Another issue was code parity between production and tests. For the development and production build, we used Vite, which has excellent ESM support. For Jest, we had to use a different build pipeline with different code transformers. So the tested code slightly differed from the production code.

We started evaluating Vitest as an alternative to Jest. Since we were using Vite, the Vitest test runner would tightly integrate with the existing build pipeline. Vitest offers almost the same APIs as Jest and supports jsdom as well.

While the Jest ecosystem did not make the progress we wished for, Vitest grew into a viable alternative to Jest. We evaluated Vitest three times over the course of one year. The first and second attempt, we ran into bugs and incompatibilities. Luckily, the Vitest team addressed them quickly while improving the stability and performance. The third attempt, we were finally able to migrate our tests from Jest to Vitest.

In the larger ecosystem, Jest is still nine times more popular than Vitest, but Vitest is gaining users quickly.

End-to-end tests

For end-to-end (E2E) tests, we used Cypress and Playwright right from the start. Both are “second generation” end-to-end testing frameworks that orchestrate Chromium, Firefox and WebKit browsers.

Our in-depth tests are written for Cypress and run on Chromium. Additional high-level tests are written for Playwright and run in Firefox. Why that?

Using the Vite build tool and @vitejs/plugin-legacy, we produce two builds:

• A modern build that requires ECMAScript Module (ESM) support. In particular, <script type="module"> plus dynamic imports plus import.meta.

• A legacy build for browsers without ESM support. This mostly targets old Chrome and Safari versions. A significant number of people have mobile devices that cannot be updated.

Because of these two builds, we used using two end-to-end testing frameworks: Cypress tested the modern build with Chrome. Playwright tested the legacy build with Firefox.

Firefox supports ESM since version 60 (May 2018). When the feature was still in beta, the config option dom.moduleScripts.enabled switched it on. When the feature became stable, the option was enabled per default and allowed to switch it off. This is what we did to mimic an old browser without ESM support.

You might wonder, why not use Cypress for testing the legacy build as well? After all, Cypress does allow running tests in Firefox and does allow passing launch options. But the Cypress in-browser test runner itself is written in JavaScript using ESM syntax! Disabling ESM support would paralyse the test runner.

When your testing trick suddenly stops working

With version 117, Firefox removed the feature flag for ECMAScript Modules. That makes sense since ESM is an essential web feature today.

Unfortunately, this change made our dual setup with Cypress and Playwright obsolete. The tests still pass on Playwright with Firefox 117, but they test the modern build, not the legacy build.

We are still figuring out how to test the legacy build without much cost and effort. For now, we have pinned the Playwright version at 1.37, so it launches Firefox 115. This is possible because Playwright versions and browser versions are hard-wired.

Despite these hiccups, Cypress and Playwright are both excellent tools that served us well. Tests for Cypress and Playwright are easy to write and execute reliably. End-to-end testing as a whole improved significantly thanks to these projects.

Recently, Cypress and Playwright gained the experimental ability to test JavaScript UI components. They became a “one-stop shop” for UI testing in real browsers. We have not compared this approach to the existing Vitest-based tests in this project, but we will definitely do so in the future.

Static checks with TypeScript and linters

In his influential articles on testing JavaScript, Kent C. Dodds extends the familiar Testing Pyramid so it becomes a Testing Trophy: Static code checks form the foundation. Unit, integration and end-to-end tests rest on this foundation.

These checks are especially important for JavaScript, a dynamically-typed language with many pitfalls. Static analysis preserves API contracts, checks for code slips that may lead to runtime errors and enforces coding guidelines for robustness, performance and accessibility.

For us, TypeScript has proven to be an indispensable tool for software design, for implementation and for maintaining code correctness. Our project’s modular and flexible architecture is held together by type contracts: Configuration objects, function signatures, component props and data objects. These rules are codified in TypeScript types and enforced by the TypeScript type checker.

Many developers understand TypeScript as a new language with new syntax on top of JavaScript. This is true if you write .ts files instead of .js files. In our recent project, we wrote only a few .ts files with type declarations. The implementation itself used plain JavaScript with type annotations in JSDoc. This way, we got strict type checks without the need to compile TypeScript to JavaScript code.

For static code analysis, TypeScript, ESLint, svelte-check as well as axe-core are powerful, reliable and mature. They form the basis of automated testing. While we went all-in with static types in the latest project, you can decide which level of strictness you want with these tools.

Pragmatic testing insights

This project challenged my beliefs about software testing in a good way. Here are my main insights:

• Testing fundamentally means struggling with tools – for better and for worse. Improving the software, improving the tests and improving the tools go hand in hand.

• Tools determine your testing practice and also shape how you reason about testing conceptually. It is also well-known that testing shapes and improves your design decisions as well as the code implementation. In the best case, this leads to an upwards spiral of doing, learning and understanding.

• The tooling landscape progresses quickly, especially for testing UI components implemented JavaScript. You cannot stick with one setup over the course of years. We had to find a way to keep up with these changes and benefit from them.

• Testing is trying out, going back and forth. Throwing out your plans, making pragmatic decisions.

Credits

Thanks to my 9elements teammates Daniel Hölzgen, Julian Laubstein, Leif Rothbrust, Matthias von Schmettow and Philipp Doll who contributed to this project.

Work with us!

Thanks for reading this article! If you are planning an ambitious web project and need a strong design and development partner, 9elements is the right agency for you.

We are open for business so please get in touch with us.

If you want an automated system to follow a given set of rules reliably, no matter how complex they are, symbolic logic is the perfect fit for the job. In particular, default logic is well suited when translating experts’ knowledge into machine-readable definitions since it closely relates to how humans describe real-world rules. Unfortunately, these systems are not well suited to be used by humans, especially when not trained to use logical programs.

This is where LLMs come in: They (kind of) understand what the user means, no matter if they could use the exact correct terms or descriptions. On the other hand, LLMs have problems reliably following rules, especially if these become more complex, and are prone to hallucinations, especially during more extended conversations.

To mitigate the downside of both approaches, we started experimenting with combining them, letting each approach shine where it is best.

A chatbot for professional caretakers

The scenario is the professional caretaker situation in a hospital, where the system is located in an (audio) chatbot, sitting at each patient’s bed. Proper documentation in this scenario represents a considerable challenge due to usually being done after the actual work on the patient, with little to no time. In addition, a lot of background knowledge in the medical field and about the concrete case is needed to decide on the next steps sensibly and to notice possible critical derived problems.

The bot should listen to what is said during the work, update its current beliefs of the patient’s status, and use its medical background knowledge to derive additional information to generate warnings for situations a professional caretaker might overlook.

In this scenario, we have a patient who had surgery three days ago. The patient is well, but then her temperature is rising. While this might be okay in other situations, after surgery, it might indicate complications due to the surgery or the anesthetics used.

This warning should be reliably derived from the given input, so this part is what is done in a logical program, not the LLM itself.

Generic approach

The overall system has three tasks when reacting to a message from a user:

1. Generate a written response representing what the system understood, as direct feedback to the user, as an additional and immediate validation if everything went right.

2. Generate a written warning in addition to the text response when its derived knowledge contains a new problem that might require the user's attention.

3. Provide a representation of the updated, derived knowledge state, i.e., used to update tables and charts.

To achieve this, we combine LLMs, in this example via the OpenAI API using a gpt-3.5-turbo model, with logical programs, in this example, by using the answer set solver DLV.

Predicate extraction

To extract logic predicates from a written or spoken user message, we instruct GPT to extract logic predicates, allowing only a very limited set of predicates.

prologI want you to extract logical predicates from user input.
Only extract predicates, do not write anything else, no explanation, no excuse.

Use only the limited alphabet, nothing else, do not extend the alphabet:

START
data(temperature, TEMP). with TEMP integer between 30 and 45
data(painLevel, LEVEL). with LEVEL integer between 0 (no pain) and 5 (severe pain)
condition(stability, STABILITY). with STABILITY in [stable, unstable]
condition(backgroundInfo, INFO). with INFO [surgery, operation, procedure, allergic]

[...]

END

This is the text to extract predicates from:

START
#{message}
END

For the first message in the above example, the answer from GPT might look as follows:

prologdata(temperature, 37)
condition(stability, stable)
condition(backgroundInfo, surgery)

In order to be able to keep track of the patient and react to changes over time, we need to add an explicit representation of time to the predicates. We do this by adding a timestamp to all predicates received from the LLM, and adding the timestamp itself to the agent’s beliefs.

prologtimeStamp(1).
data(temperature, 37, 1).
condition(stability, stable, 1).
condition(backgroundInfo, surgery, 1).

These transformed predicates are then added to the beliefs of the agent.

Deriving knowledge

To derive new knowledge based on the agent’s current beliefs and background knowledge, we use an answer set solver, in this case DLV. First, we give the agent an understanding of time and how to derive current values.

prolognewerTimestamp(T) :- timeStamp(T), timeStamp(T2), T < T2.
currentTimeStamp(T) :- timeStamp(T), not newerTimestamp(T).

newerData(TYPE, T) :- data(TYPE, _, T), data(TYPE, _, T2) , T < T2.
dataBetween(TYPE, T1, T2) :- data(TYPE, _, T1), data(TYPE, _, T2), data(TYPE, _, T3), T1 > T3, T3 > T2.
currentData(TYPE, VALUE) :- data(TYPE, VALUE, T), not newerData(TYPE, T).

[...]

We now can use this knowledge to derive information about possible problems, in this example related to rising temperature after a surgery, due to the possible use of anesthetics.

prologanesthetics :- backgroundInfo(surgery).
anesthetics :- backgroundInfo(operation).

[...]

currentProblem(T1, risingTemperatureAfterOperation) :- data(temperature, X1, T1), data(temperature, X2, T2), anesthetics, X1 > X2, T1 > T2, not dataBetween(temperature, T1, T2), not newerData(temperature, T1).

Note that in this case, we derive the information about possible problems from the use of anesthetics. In a real live, more complex scenario, there might be a lot of other reasons that may have caused the administration of anesthetics, which might trigger the same warning. This derived knowledge is called the epistemic state of the agent.

Fighting hallucinations

In addition to deterministically applying rules, we aimed to fight hallucinations in possibly long running conversations with the agent. To achieve this, we do not provide the conversation history in the API call but extract all current beliefs regarding data from the agent’s epistemic state and add it to the prompt, leaving out background information and derived problems. In the above example, we would fill in the current part:

prologYou can use the following background information:

START
data(temperature,37)
condition(stability,stable)
END

Generating output

In the last step, we generate the agent’s output.

For generating the written response in the conversation, we ask the LLM to generate a brief confirmation on the predicates it detected, feeding it it’s previous response.

prologConfirm briefly and short the updates to a patient's file based on the following logic predicates:

data(temperature, 37)
condition(stability, stable)
condition(backgroundInfo, surgery)

We extract all data points, background information, and derived problems from the agent’s epistemic state to provide the actual data. It can be used to fill tables, charts, or journals of the professional caretaker.

In case a problem is detected, we use the LLM to generate a written warning in addition to the ‘raw’ predicate in the data output:

prologAct as a warning system for professional caretakers. Generate a concise warning explanation for the following problem:

currentProblem(risingTemperatureAfterOperation)

Conclusion

While it is possible to mitigate the downsides of both approaches to some extent, this work only features as a proof of concept and still leaves a lot of work to be done.

Although the answer set solver works reliably, it depends on the correct transformation of user input to predicates. While this worked surprisingly well with GPT and Bard in our tests, both systems did not reliably stick to the limited alphabet. We tried to mitigate this problem by explaining to the LLM how to deal with uncertain transformations to keep the derived data clean and allow the system to display what part was not understood, allowing the user to react and refine the input.

In addition, for generating reliable problem descriptions, the LLM should be fine-tuned or provided with expert knowledge with a lang-chain approach to ensure the generated warnings make sense.

However, this represents an exciting approach that we will continue to pursue in our internal research and projects that rely on reliable rule adherence.

Do you have a project that would benefit from a better interface that still reliably fits within the processes? Don't hesitate to get in touch with us.

Many thanks to Nils Binder for styling the example application and to Tina Hoelzgen for her professional input on the topic of caretaking.

As developers, we often find ourselves faced with the task of translating design elements like colors, font sizes, and spacing into our code. Alternatively, as designers, we need to summarize and provide this information to the developers. At 9e, this has frequently proven to be quite labor-intensive. Due to this, we sought a more efficient approach for the long term.

The introduction of Figma variables has provided us with precisely this opportunity, as these variables are essentially identical to the ones subsequently created in our CSS file. If we could extract them from Figma, it would significantly reduce our workload. I was enthusiastic about this initial idea, and it's exactly what I wanted to explore further.

Let's Begin: The Figma Plugin Adventure

At the beginning I was not sure what exactly the tool should look like. So first I searched for an existing tool that provides the function of exporting variables for CSS. I found only one plugin that exports variables as JSON only. This gave me my first idea to create a website where users can upload a JSON file and generate variables based on that data.

However, the concept of a website where you upload a JSON file generated by a Figma plugin just to obtain its CSS code seemed like a significant detour to me. Additionally, this approach would create a dependency on that specific plugin, necessitating constant checks for potential changes or lack of support.

So I decided to explore the possibility of creating a Figma plugin. The challenge now was to figure out where to start.

After a quick search, I stumbled upon an introductory tutorial from Figma that provided a solid initial insight into the world of Figma plugins. Figma also offers excellent documentation for its plugin API.

Figma API

After setting up my codebase using the tutorial, I was ready to start. But first, I needed a clear MVP (Minimum Viable Product). The MVP was essential for identifying the functions to prioritize.

For my MVP, users would select their variable collections in Figma, convert these to CSS variables, and then copy the generated code.

This approach helped me estimate the necessary features and understand my requirements from the Figma API. My main goal was to access the collections and individual variables via the API. Thankfully, the API provided several methods to retrieve this information:

Collections & Variables

I am able to obtain an enumeration of all the collections present in the document or

javascript// getLocalVariableCollections
const localCollections = figma.variables.getLocalVariableCollections();

search specifically for a collection by its ID. However, I only receive the ID from the getLocalVariableCollections() function. Therefore, I required both of these functions.

javascript// getVariableCollectionById
const collection = figma.variables.getVariableCollectionById('VariableCollectionId');

What you receive from the API in both cases is a Collection-Object, which appears like this.

javascriptconst variableCollection = {
    id: "VariableCollectionId:1:4",
    defaultModeId: "1:0",
    hiddenFromPublishing: false,
    key: "245b6e0ea91efcd1b77173cd1fbdeb6b6a04936f",
    modes: [
        { name: 'Mode 1', modeId: '1:0' }
    ],
    name: "Primitives",
    remote: false,
    variableIds: [
        /* ... List of variable IDs ... */
    ]
};

This provides information not only about the modes present in the individual collections but also about the variables associated with each collection, as specified in the variableIds.

In the second step, I searched for the variables associated with the respective collection using the variableIds obtained from that collection. For this purpose, the Figma API provides a dedicated function. Additionally, I have included below an example of the object returned by the API.

javascript//getVariableById
const variable = figma.variables.getVariableById(variableId);

javascriptconst variable = {
    id: "VariableID:10:337",
    description: /* ... */,
    hiddenFromPublishing: false,
    key: "b638043cd1accac7941f530d286d1c1954b12289",
    name: "color/Gray/750-60%",
    remote: false,
    resolvedType: "COLOR",
    scopes: [
        /* ... List of scopes ... */
    ],
    valuesByMode: {
        "1:0": { r: 0.7450980544090271, g: 0.7450980544090271, b: 0.7450980544090271, a: 0.6000000238418579 }
    },
    variableCollectionId: "VariableCollectionId:1:4"
};

Exactly as I was writing these lines of the blog article, I noticed a minor error in the assignment of variables to their respective modes. The correct approach is to assign the variables based on the ModeId from the collection with the Id found in valuesByMode. However, a different solution had been applied previously. This goes to show that a solid understanding of the API is invaluable.

Once I had established this structure for my plugin, my foundational framework was ready, and I could finally begin.

By the way, with this function, you can also directly retrieve all variables. Personally, I didn't require this functionality, but I included it for the sake of completeness.

javascript//getLocalVariables
const localVariables = figma.variables.getLocalVariables('STRING'); // filters local variables by the 'STRING' type

My new friend in development

After successfully outputting the variables, my next step was to convert the RGBA data from the object into hex and RGB formats, as well as to specify all pixel (px) values in rems (rem), given its contemporary relevance. Additionally, I aimed to alphabetically sort the output variables. However, for instances where certain variables were very similar and had a numeric value like '0rem–3.125rem', I intended to sort them by size as well, to get something like this:

css:root {
  --spacing-0: 0rem;
  --spacing-2xs: 0.125rem;
  --spacing-xs: 0.3125rem;
  --spacing-s: 0.625rem;
  --spacing-m: 0.9375rem;
  --spacing-l: 1.25rem;
  --spacing-xl: 1.5625rem;
  --spacing-2xl: 1.875rem;
  --spacing-3xl: 2.5rem;
  --spacing-4xl: 3.125rem;
}

Developing this sorting mechanism manually would have been a hefty task for me. While this type of sorting algorithm might be a breeze for some, it posed quite the challenge for me. I'm grateful to ChatGPT for generating the subsequent code, saving me from diving too deep into those complexities.

javascript// This function sorts an array of objects based on a complex name attribute.
function sortObjectsByAttributeNew(objects) {
  // Sort the objects by splitting their names into parts and comparing them.
  const sortedByName = objects.sort((a, b) => {
    const aParts = a.name.split("/");
    const bParts = b.name.split("/");

    // Get the main names by excluding the last part (value) of the name.
    const aMainName = aParts.slice(0, -1).join("/");
    const bMainName = bParts.slice(0, -1).join("/");

    // Compare the main names, using locale-based comparison.
    if (aMainName !== bMainName) {
      return aMainName.localeCompare(bMainName);
    } else {
      // If main names are the same, compare the last part (value) of the names.
      const aValue = aParts[aParts.length - 1];
      const bValue = bParts[bParts.length - 1];
      const nameComparison = aValue.localeCompare(bValue);
      if (nameComparison !== 0) {
        return nameComparison;
      } else {
        // If last parts of names are also the same, compare values directly.
        return a.value.localeCompare(b.value);
      }
    }
  });

  // Group the sorted objects by their main names.
  const groupedByName = {};
  sortedByName.forEach((obj) => {
    const mainName = obj.name.split("/").slice(0, -1).join("/");
    if (!groupedByName[mainName]) {
      groupedByName[mainName] = [];
    }
    groupedByName[mainName].push(obj);
  });

  // Sort and concatenate the grouped objects, producing the final sorted array.
  const sortedNumberObjects = [];
  for (const mainName in groupedByName) {
    const groupedObjs = groupedByName[mainName];
    const sortedObjs = sortNumberObjectsByValue(groupedObjs);
    sortedNumberObjects.push(...sortedObjs);
  }

  // Return the fully sorted array of objects.
  return sortedNumberObjects;
}